GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence
Authors: Zhihua Wen
Zhiliang Tian
Wei Wu
Yuxin Yang
Yanqi Shi
Zhen Huang
Dongsheng Li
Venue: EMNLP 2023 Findings (2023)
Keywords: story generation
large language model
iterative prompting
TL;DR: We adopt an "asking-why" iterative prompting scheme to extract multiple pieces of evidence, therefore augmenting story backgrounds, resulting in more diverse and creative plotlines.

Abstract: Conditional story generation is significant in human-machine interaction, particularly in producing stories with complex plots. While Large language models (LLMs) perform well on multiple NLP tasks, including story generation, it is challenging to generate stories with both complex and creative plots. Existing methods often rely on detailed prompts to guide LLMs to meet target conditions, which inadvertently restrict the creative potential of the generated stories. We argue that leveraging information from exemplary human-written stories facilitates generating more diverse plotlines. Delving deeper into story details helps build complex and credible plots. In this paper, we propose a retrieval-auGmented stoRy generation framework with a fOrest of eVidEnce (GROVE) to enhance stories' complexity. We build a retrieval repository for target conditions to produce few-shot examples to prompt LLMs. Additionally, we design an "asking-why" prompting scheme that extracts a forest of evidence, providing compensation for the ambiguities that may occur in the generated story. This iterative process uncovers underlying story backgrounds. Finally, we select the most fitting chains of evidence from the evidence forest and integrate them into the generated story, thereby enhancing the narrative's complexity and credibility. Experimental results and numerous examples verify the effectiveness of our method.

GROVE:
A Retrieval-uummened Complex Story Generation Framework with A Forest of Evidence
Zhihua Wen, Zhiliang Tian; Wei Wu, Yuxin Yang, Yanqi Shi,
Zhen Huang, Dongsheng Li College of Computer, National University of Defense Technology, Hunan, China {zhwen, tianzhiliang, weiwu_ yangyuxin21a, yqshi, huangzhen, dsli edu. cn
Abstract
Story generation is an active research area where existing studies approach it from two directions: en- hancing controllability and incorporating common- sense knowledge (Alabdulkarim et al., 2021). To satisfy target constraints, researchers enhance the controllability of generation models (Zhou et al., 2023a). Rashkin et al. (2020) follow outline of the plots generate stories. Wang et al. (2022b) propose BART-based (Lewis et al., 2020) model to generate stories according to the fine- ane-grained personalized guidance. Additionally, to produce fluent and coherent storylines, researchers investi- gate incorporating commonsense knowledge into generation (Wang et al., 2020a; Guan et al., 2020; Zhang et al., 2023). Peng et al. (2022) introduce commonsense inference into GPT-2-based (Rad- ford et al., 2019) model improve narritive coher- ence. Qin and Zhao (2022) combine knowledge retrieval, knowledge selection, and story generation together to make the generated story more reason- able. The above studies focus on improving con- trollability and logical coherence but rarely explore the generation of stories with complex plots.
Conditional story generation is significant in human-mann machine interaction, particularly in pro- ducing stories with complex plots. While Large language models (L.MM)) perform well on mul- tiple NLP tasks, including story generation, it is challenging generate stories with both com- plex and creative plots. Existing methods often rely on detailed prompts guide L.LSs meet target conditions, which inadvertently restrict the creative potential of the generated stories. We argue that leveraging information from emplary human- stories facilitates gen- erating more diverse plotlines. Delving deeper into story details helps build complex and cred- ible plots. In this paper, we propose a retrieval- auGmented stoRy generation framework with a fOrest of eVidEnce (GROVE) to enhance stories' complexity. We build retrieval reposi- tory for target conditions to produce few-shot examples to prompt LLMs. Additionally, we prompting scheme that
design an "asking-whi extracts forest of evidence, providing compen- sation for the ambiguities that may occur in the generated story. This iterative process uncov- ers underlying story backgrounds. Finally, select the most fitting chains of evidence from the evidence forest and integrate them into the generated story, thereby enhancing the narra- tive's 's complexity and credibility. Experimental results and numerous examples verify the effec- tiveness of our method.
Large Language Models (LLMs) learn com- monsense knowledge from massive texts and de- velop strong abilities to follow human instruc- tions (Ouyang et al., 2022: OpenAI, 2023; Taori et al., 2023). Thus, LLM-based prompt learning generates fluent and coherent stories with high controllability (Lu et al., 2023; Xie et al., 2023; Yao et al., 2023). Lu et al. (2023) prompt GPT- (Brown et al., 2020) with combinations of multi- ple target conditions. Xie et al. (2023) demonstrate that by using prompts, GPT-3 generates higher- quality stories than other state-of-the-ar (SOTA) models. Typically, LLMs generate stories based on a given prompt (i.e. text spans few sen- tences) and the outputs are continuations of the given texts. However, a recurring issue emerges in the LLM- based prompting approaches to gener- ate complex stories: there is a struggle to balance
1 Introduction
Conditional automatic storytelling, generating a story that satisfies specific target conditions, has gained significant attention in the natural language processing community (Kumar, 2023). Generating stories with complex plots is particularly crucial as it creates engaging stories of human-level quality for various applications, such as AI novelists and AI playwrights (Alhussain and Azmi, 2021).
"Corresponding Authors.

---

We argue that leveraging the information (e.g. story background and plots) from exemplary hu- man stories facilitates gencrating more diverse plots. Delving into story details enriches the narra- tive with the necessary information, thereby help- ing to build complex and credible storylines. the complexity and creativity within the generated stories (Alabdulkarim et al., 2021; Wang et al., 2023). To prompt LLMs to generate stories with complex plots, users often need to detail control signals within the prompt. This approach presents a dilemma: the more control information provided, the more likely it is that the generated story will focus solely on describing the given content, thus constraining the story's potential creativity.
to generate forest of evidence and rewrite the orig- inal story based on the optimal evidence chains; 3) Our approach achieves SOTA performance on quantities of testing cases. Detailed analyses vali- date the effectiveness of our approach.
Related work
2.1 Story Generation
Research on automatic story generation can be clas- sified into two categories: enhancing controllability and incorporating commonsense knowledge (Alab- dulkarim et al., 2021). Researchers explore both ending-focused approach (Zhao al., 2018; Guan et al., 2019a) and storyline- focused approach (Peng et al., 2018) to improve the controllability of gen- erated stories. The ending-focused approach aims to generate a story with a specific desired ending. Tambwekar et al. (2019) apply reinforcement learn- ing to optimize the pre-trained model to generate story plots that consistently reach a specified end- ing for the story. Wang et al. (2020a) leverage interpolation model based GPT-2 produce coherent narratives with user-specified target end- ings. Lu al. (2023) explore the generation ability of GPT-3 based on different prompts. The aim of storyline-fooe focuseed approaches is to make the gen- erated story follow an outline of the plot (Rashkin et al., 2020; Fan et al., 2018). Wang et al. (2022b) propose BART-based (Lewis et al., 2020) model to generate stories with desired characters, actions, and emotions. Xie et al. (2022) consider psycho- logical state chains of protagonists and propose a psychology-guice controllable story generation system.
In this paper, we propose retrieval-auGmennte complex st generation framework with fOrest of VvddEnce (GROVE), which leverages existing stories and evidence generate and rewrite stories for more complex plots. We construct a retrieval repository that enables the LLM to learn diverse plots and common patterns from human-written stories. This assists the LLM in generating sto- ries with complex plots. Moreover, we design an "asking-why" prompting scheme that iteratively builds evidence forest addressing the ambigui- ties found in the story from various perspectives. The evidence forest refers to a collection or set of evidence trees that are generated to supplement a story in GROVE. Each evidence tree consists of nodes representing pieces of evidence and edges connecting them. The root node of the tree repre- sents an ambiguous unclear part in the generated story, while the non-root nodes represent additional information that provides clarity and background details to the nodes above them in the tree. Finally, we select the optimal chains of evidence from the evidence forest and integrate them into the gener- ated story, thereby enhancing its complexity and credibility. Our method is not intended to replace any specifically designed prompts or techniques currently employed in the field. Instead, pro- pose a flexible and generalizable framework that enables LLMs generate stories with complex plots, complementing existing methods.
Another line of work involves the study of in- corporating commonsense into story generation either explicitly (Yang et al., 2019; Guan et al.. 2020; Mao et al., 2019) implicitly (Wang et al., 2020a: Guan et al., 2020). Researchers explicitly leverage additional data by incorporating a com- monsense knowledge graph into the model encod- ing (Guan et al., 2019b; Wang et al., 2020b) or using a plot graph based on commonsense deserip- tions (Ammanabrolu et al., 2020). Implicit knowl- edge stored in model parameters is also helpful in producing stories. LLMs learn from large amounts of texts, thereby gaining rich understanding of commonsense knowledge to generate stories. Xie et al. (2023) randomly sample few-shot demon- strations to GPT-3 to guide story generation. Yao et al. (2023) instruct LLM to make multiple plans
Our contributions are threefold: 1) We develop a retrieval- auggeente framework for generating sto- ries with complex plots by prompting an LLM; 2) We introduce an "asking-why" prompting scheme
'We call the prompting method "usking-why" because it requires the L.I. to justify why particular ambiguities make sense in the generated story.

---

and vote for the best plan to generate stories. Our work is also based on LLMs. However, unlike ex- isting LLM-based approaches for story generation that prompt LLMs with manually chosen cases, GROVE automatically retrieves similar examples to instruct the LLM.
labeling, and knowledge distillation to generate re- alistic text samples with LLMs. Sahu et al. (2022) create prompts from available examples and feed them to LLMs to generate training data for intent classification. Our work is another attempt to lever- age LLMs for data augmentation that uses an LLM to extract narrative attributes from existing stories.
2.2 LLM-based Prompting Learning
In the context of LLMs, prompting refers to a user inputting text string to the model, elicit- ing response from the LLM according the input (Liu et al., 2023; Li et al., 2023). To fully leverage LLMs in downstream tasks, researchers propose to carefully design prompts either man- ually (Brown et al., 2020; Hendy et al., 2023; Schick and Schütze, 2021) or automatically (Gao et al., 2021; Zhou et al.. 2023b; Guo et al., 2022). Wang et al. (2022a) explore an iterative prompt- ing framework, which progressively elicits knowl- edge from language models by prompting automat- ically. Wei al. (2023) find that the Chain-of- Thought (CoT) prompting, a kind of prompt that instructs the model to provide a rationale for its answer, shows advantages in complex arithmetic and reasoning tasks. Zhang et al. (2022b) classify CoT prompting into three paradigms: Zero-Shot- CoT (Kojima et al., 2022), Manual-CoT (Wei et al., 2022), and Auto-CoT (Zhang et al., 2022b). Zero- Shot-CoT involves adding prompt like "Let's con- sider the following step-by-step" to the test ques- tion, which helps L.Ms consider problems more logically. Manual-Coo (Wei et al., 2023) is a few- shot prompting method that provides manual rea- soning demonstrations to the LLMs. Zhang et al. (2022b) propose Auto-CoT to construct demonstra- tions with questions and reasoning chains automat- ically. Yao et al. (2023) propose Trre-of-Thoughe (ToT) prompting to improve LLM's performance by voting for different reasoning. These studies approach a task by deconstructing into multiple steps and executing them sequentially. In contrast, our approach initially completes the entire task, and then iteratively refines and improves it.
Method
3.1 Overview
Fig. presents an overview of our framework. GROVE consists of three parts: (1) Retrieval Repository builds a repository R with human- written stories and the associated desired control signals. (2) Evidence Forest Construction via Asking Why retrieves and generates stories with imputs (steps to 3 in Fig. 1) and recursively grows forest of evidence in support of the story (steps and 5). (3) Evidence Chains-supported Story Rewriting selects optimal evidence chains from the evidence forest, which are the most relevant to the target conditions, and employs them to enrich the story (steps 6 to 8 in Fig. 1).
To generate a story, our framework receives a set of target conditions C = fei...., which comprise multiple text spans to express the desired mood, plot, genre, subject, etc. By employing an LLM, we aim to generate an interesting story with complex plots satisfying the control conditions.
3.2 Retrieval Repository
We construct a retrieval repository consisting of multiple key-value retrieval items, where the key represents the target conditions of story, and the value is the story itself. To construct the reposi- tory, we use a set of conditions extracted from the value (i.e. story) as the key.The repository acts as a source of inspiration and provides the LLM with a rich set of story examples to draw from. It helps the LLM to learn from various narrative attributes and incorporate them into its generated stories, fa- cilitating the generation of complex storylines.
2.3 LLM-bbsed Data Augmentation
3.2.1 Repository Construction
Researchers investigate generating pseudo data to alleviate the issue of data scarcity (Feng et al., 2021; Pluščec and Snajder, 2023) for tasks including knowledge distilling (Sanh et al., 2020: Sun et al.. 2023), event classification (Sarker al., 2023) and harmful content detection (Hartvigsen et al., 2022). Yoo et al. (2021) combine text perturbation, pseudo-
To construct the repository, collect raw stories from the Internet, using these as values, and employ LLM to extract target conditions from the story as keys. To facilitate this extraction, we query the LLM using human- written prompts for each type of target condition.

---

Operatian Stup Repudtary Contraction
One missing background information in the story how Maggie died and ended up being launched back in time. 2 Maggic died car accident and her consciousness was sent back in time due to an experimental technology malfunction. S Maggic worked as software engineer at a cutting-edge reseurch facility that was experimenting with time travel technology.
<ding Why Prospring
Final Story
2
LLM
LLM
Initial Story
Evidence Furest
The gress of de skey i Pitlics
Whe the greae
[4,4]
[6
Anary
Retrieval Repositury
3
2
Starn
7
quary
Maxd
Ploce same oet
Figure I: The architecture of GROVE, with each operation step demonstrating the i-th step of the story generation process. GROVE begins using the target conditions retrieve Story2 and its conditions, using these as prompts to guide the LIM in generating an initial story (steps I to 3; refer to Sec. 3.2). Following this, GROVE constructs an evidence forest composed of two evidence trees, by asking-why prompting (steps and 5; refer to See. 3.3). Ultimately, GROVE selects the optimal evidence chains and directs the LLM to rewrite the initial story, resulting in the final story (steps 6 to 8; refer to Sec. 3.4).
We obtain the values for retrieval items by col- lecting large set of stories from the Internet, denoted as D {d1, . dn To obtain the corresponding keys for these values, we take two steps: prompt construction and condition extrac- tion. Specifically, (1) the prompt construction step, we construct a prompt template (.) for each type of target condition. This template serves a natural language query asking for the specific condition within given story. Recall that C -c (c....., Cm3 comprises a series of different types target control conditions, including story plot, subject, etc, where each control condition 6; is described by a text span. Note that the "plot" condition specifies the storylines that must appear in the story, rather than defining a limited set of al- lowable plots. For example. for the target condition "subject", the associated prompt template can be: "Here is story: [STORY]. Answer the following question based on the above story: give a list of distinctive subjects this story is trying to portray.", where "[STORY]" represents the given story con- tent. We detail the prompt templates for all target conditions in App. F. (2) In the condition extrac- tion step, for each story d, in D. we use LLM to extract cach condition ē; by feeding prompti(d;) into the LLM.
Each story dj. along with its extracted conditions C f , ), constitutes an item (Cj, dj) in the retrieval repository. Ultimately, the repository R consists of pairs of stories and their extracted conditions: R {(C}, d;)),21
3.2.2 Repository Retrieval
The retricval process searches for the most simi- lar condition sets and returns their corresponding stories from the retrieval repository. The search is based on the semantic similarity between the target control conditions and the items' keys in R.
Specifically, during inference, given a target con- dition set C, we define recommendation score s for each story. To obtain s, we calculate the cosine similarity between the semantic vector of each con- dition in C and that of its corresponding condition in C, and then sum up the cosine similarity scores for all conditions:
T7
cOs (f))
5
where f(.) is an off-the-shelf semantic encoder We sort all stories in R. based on their recommenda-
We use SBERT (Reimers and Gurevych, 2019) in our experiment.

---

tion scores s and return the top-k highest-ranking retrieval items, along with their stories, represented as W [(C), d,1))=
leaf node that comprises sequence of I pieces of evidence, where each piece of evidence supple- ments the preceding one. To perform asking-why on each node, we concatenate its corresponding evidence chain with a pre-defined prompt template and feed it to the LLM. The template for asking why can be: "Here is a story; [STORY]. A missing detail is: [EVIDENCE CHAIN]. Except for pure co- incidence, point out b factual pieces of background information that compensate the story one by one. Each additional piece of information should be in one short sentence and only contain factual infor- mation without opinions or judgments." "As there are . ambiguities, we obtain an evidence forest F= ((E1, . Ax])-
3.3 Evidence Forest Construction via Asking Why
We employ an LLM to generate an initial story and then iteratively ask the LLM to construct a forest of evidence that supplements the story. The intuition is that referring to the retrieved story incentivizes the LLM to produce diverse new stories. This may result in a lack of concrete supporting details and appear hollow, which makes the story less credible and informative. To address this issue, we design an iterative asking-why prompting scheme to re- cursively collect pieces of evidence, thus enriching and clarifying ambiguous parts in complex plots.
Our iterative asking-why prompting scheme plores multiple possibilities by prompting the LLM to supplement new evidence obtained from the last iteration. In this way, we create an evidence for- est F to support the initial story. We can adjust the amount of information by increasing and the number of iterations I.
The algorithm first generates an initial story gen- eration then generates the unclear parts (named "ambiguity") in the initial story, and finally collects the evidence by iteratively asking why. Firstly, generate the initial story, we construct an in- context learning prompt using the retrieval results in W and the target conditions C. Then, we feed this prompt into the LLM to obtain an initial story. Secondly, to discover the unclear parts in the story. we instruct the LLM to generate N relevant "am- biguities" for the initial story concerning the tar- get conditions, where ambiguity is significant drawback that decreases the story's credibility. For example, an ambiguity can be an unclear motiva- tion or a seemingly illogical plot. We prompt the LLM to generate ambiguity as: "Here is a story: [STORY]. When analyzing fictional stories, it is okay to mention the negative aspects. Pretend to be a writer, and without further ado, point out N missing background information in the story with N simple sentences
3.4 Evidence Chains-supported Story Rewriting
The LLM selects the optimal evidence chain from each tree incorporate into the original story. The intuition for story rewriting is address its ambi- guities by incorporating relevant pieces of evidence into the story to provide the necessary information.
We select evidence chains to augment the story in two steps.
Evidence chains selection. For each evidence tree, we first concatenate all the evidence chains before feeding them into the LLM. Then, we prompt the LLM to select the most suitable evidence chain to add to the initial story. The selection is based on the relevance between chains and the initial story. We repeat this process on all trees in the evidence forest F and obtain N evidence chains, denoted
by one." Finally, col- lect evidence, we propose to iteratively ask why questions to L.M. By asking why, we instruct the LLM to provide b pieces of evidence that compen- sate for the initial story. For each ambiguity, we recursively ask the "why" question for iterations and obtain an evidence tree {E, A}, where E' is the set of nodes and A represents the set of edges. In evidence tree, the root node represents am- biguity, and non-root nodes are pieces of evidence that provide additional information to the nodes connected them in the upper layer. We define an evidence chain & {eo. e is a path from a tree's root node (cg representing the ambiguity) to
Story rewriting. We instruct the LLM to incor- porate the information from {ε, A) into the initial story rewrite the final version of the story. The prompt template is: "Here is a story: [STORY]. Here is the missing back- ground information: [EVIDENCE CHAINS). Pretend to be a writer and complete the story by including the given information. Modify

---

the necessary sentences in the story and re- peat the unrelated parts to include the given background information."
the target constraints? Additionally, we propose two new metrics tailored for evaluating stories with complex plots: (5) Complexity: How complex is the plot structure in the story? (6) Creativity: How creative is the story's plot design?
Recall that each piece of evidence is a node in the tree with child nodes (except for the exter- nal nodes). These child nodes support it in dif- ferent ways, making the information from those child nodes mutually exclusive. If we incorporate multiple chains into the story that are from the same tree, it would contain contradictory pieces of evidence from one node's different child nodes, compromising the logical consistency of the nar- rative. Therefore, to ensure logical coherence and avoid potential contradictions, we select only one evidence chain from each tree for inclusion in the final story.
We evaluate stories on these six metrics using a 5 5-point Likert scale with human evaluation. We hire three evaluators with Master's degrees in En- glish Literature from commercial company to independently evaluate 100 randomly sampled sto- ries paired with instructions'. As Zhai al. (2023) found that LLMs can serve cheap alternative for human evaluation, we evaluate each story in Sec. 4.4 by querying ChatGPT three times using the instructions in Zhai et al. (2023). We calculate the average scores and variances for human and model-based evaluation respectively.
4 Experiments
Baselines. We compare our method against five baselines: Human (Fan et al., 2018) is writ- ten ground truth stories under the same prompts. ICL (Xie et al., 2023) explicitly instructs an LLM to generate a story to satisfy target conditions and contain many interesting plots. CoT follows the prompting strategy (Wei et al., 2022), where the LM is asked follow specific instructions, generate a story, identify the missing information, and iteratively revise the story to in- clude missing backgrounds step by step. Prompt-E performs prompt engineering by modifying the in- struction of ICL to obtain 4 variants that explicitly require creativity and complexity from the gener- ated story and taking the average of their perfor- mance for evaluation. Specifically, it adds "Gen- crate a complex and creative story", "Generate a detailed and complex story", "Ensure that the story is creative and rich in plots.", and "Generate long and complex story" to the ICL instruction, respec- tively. Story-S prompts the LLM to generate a story multiple times. For each sample, we add up the scores of six metrics to obtain the overall score and select the story with the highest overall score the final story. We use ChatGPT as the base model for all the above methods for a fair comparison (im- plementation details are in App. C. As the API to access GPT-4 model is hard to apply, which limits our ability to conduct large-scale experiments for direct comparison. However, the theoretical under- pinnings and method of our work remain applicable to GPT4
4.1 Experimental Settings
Datasets. Following Lu et al. (2023), consider plot, mood, genre, and subject as target conditions. plot describes the events that must appear in the generated story. mood defines the expected emo- tional response of the reader after reading the gener- ated story. genre dictates the desired story type and subject indicates the subjects that should be men- tioned in the story. We randomly draw 50 prompts from the testing set of the WritingPrompt dataset for plot. Following Lu et al. (2023), we consider happy, angry, fearful, and sad for mood. For genre, we consider historical fiction, literary fiction, and science fiction. Lover, cat, and survivor are for subject. We experiment with all combinations of these four types of control components across 1800 testing cases (derived from 50 prompts, 4 moods, 3 genres, and 3 subjects), ensuring that each type of condition only appears once in each case. Besides, we use the 1.5K unlabeled movie plot summaries from the IMDB movie details dataset 3 to build the retrieval repository.
Evaluation Metrics. We follow the common practice in automatic story generation for evalu- ation (Karpinska et al., 2021; Zhai et al., 2023) and measure the following 4 aspects:(1) Gram- mar: How grammatically correct is the text of the story? (2) Coherence: How well do the sentences in the story fit together? (3) Likability: How en- joyable or appealing is the story to the readers? (4) Relevance: How closely does the story align with
"We do not use the commonly adopted AMT since Karpin- ska et al. (2021) found that their results are questionable.
www.kkagle.com/datusttttttggg http:://weraaaao////////mm/ooome

---

4.2 Overall Performance
amples. Retrieve underperforms GROVE in all metrics, especially on Relevance. The performance drop indicates that the retrieval enhances the un- derstanding of desirable outputs and helps generate coherent and relevant stories. Rewrite skips the evidence-based story rewriting, which deepens the storylines and explores untold backgrounds and in- depth rationale. The drop in Complexity shows that the stories generated without rewriting lack cer- tain depth and complexity, thus validating the im- portance of evidence-based rewriting in enriching story complexity. As - Rewrite generates without exploring deeper details, resulting in stories that more closely stick to the given instruction, thus demonstrating slight advantage over GROVE in Relevance. However, the Relevance of GROVE re- mains high, even while scoring high in Complexity.
Tab. shows the results of all methods of human evaluation in story generation. GROVE achieves the best performance on almost all metrics. Hu- man baseline underperforms other automatic ap- proaches that are based on the same LLM (i.e. ChatGPT), indicating the strong ability of LLM to produce high-quality stories. ICL generates a story by instructing the LLM with target conditions. It achieves the highest performance in Relevance and Grammar, indicating ChatGPT's strong ability follow instructions and generate correct English. Compared GROVE, ICL produces the least com- plex stories under both human and automatic eval- uation, which means that directly instructing the LLM with target conditions struggles to generate complex stories. CoT self- improves the story in a single output step-by-step by asking and answering questions about the initial story and rewriting it to obtain the final story. CoT generates slightly more complex and likable stories than ICL. It shows that Chain-of- Thought prompting is effective improv- ing the story's 's complexity and fondness. However, CoT is still worse than GROVE on all metrics be- cause it is challenging for ChatGPT to execute all steps in single outpurS. We find in our experi- ments that the story quality of Story-S is highly inconsistent among different generations. Besides, even if it is possible to obtain complex and cre- ative stories, Story-S cannot diversify or prolong one particular user-desired story. GROVE bene- fits from the retricval-auue -uggented approach and the asking-why prompting scheme. Since it introduces more complex plots, GROVE inevitably incorpo- rates additional information into the story that may be irrelevant to the target conditions. While pro- ducing slightly less relevant stories than ICL, still scores high on Relevance and achieves the best performance in the rest metrics. We also con- duct prompt engineering on GROVE and obtain GROVE (Prompt-F), which further improves its story generation ability.
Select omits to select optimal evidence chains and incorporate all evidence into the final stories, The lack of evidence filtration introduces unrelated and conflicting information, producing verbose and illogical stories with decreased Coherence and Rel- evance. Furthermore, the drop in Complexity and Creativity indicates the importance of the selection process in refining the stories' complexity and origi- nality. Given the initial stories, Evidence directly instructs the LLM to include the necessary details to revise the stories. Evidence enables story re- visions. However, without explicitly prompting the LLM with the necessary information, may add insignificant details that hardly improve the story quality. + Evidence increases the number of evi- dence trees thereby improving Complexity, while possibly making the generated stories too specific, thereby affecting Likability. Inserting fixed com- plex story into the prompt (Fixed Story) leads unstable performance. It decreases the stories' Co- herence, Likability, and Relevance, echoing the discoveries of Shi et al. (2023) and Zhang et al. (2022a) that irrelevant or random samples distract LLM, thereby hurting performance.
4.4
Generalization Ability
4.3 Ablation study
We verify the generalization ability of GROVE on a much smaller open-source LLM (i.e. Alpaca-Plus- 7B (Cui et al., 2023)). Due to the high computa- tional costs of many LLMs, exploring smaller mod- els provides more affordable option for smaller teams. We apply GROVE on Alpaca-Plus-7B (Al- paca) to compare with a simple baseline ICL Tab. 3 and showcase its generation results to com-
Tab. shows the human evaluation results of the ablation studies on our proposed components and verifies their effectiveness. - Retrieve generates stories without referring to relevant few-shot ex-
We calculate that CoT fails to finish the required number of iterations for evidence generation hefore generating the final story in more than 40% of the testing cases.

---

Grammar
Coherence
Likability
Relevance
Complexity
Creativity
Methods \Metries
Meanem
IAA(%)
Meangm IAA(%)
Meanşr
IAA(S)
MeansT
IAA(%)
Meaner (AA(%)
Meanem
IAA(%)
Human
3,53
3.32am
2.99uxz
20
3.63
17
3.05ami
2.93urv
27
ICL.
4.04
47
3.22p96
4,47,,3
3.340,57
3.21
CoT
3.69..
33
3,83
3.3
24
3.83
13
3.42,,0
3.15am
Prompt-E
3,
35
3.94u52
3.39us7
20
4.170.6
19
3.44
3.23
Story-S
4.12a.45
48
4.17,79
3.38p
26
4.28p
23
3.53
3.42am
GROVE
3,98
4.22g01
3.55u63
4.25u,7
3.570.1
3.400.55
GROVE (Prompt-E)
3,94
4.15as7
3.61ngr
4.34o s8
20
3.61a.m
3.57
Table I: Human evaluation results of all methods. For each metric, we report the mean and the standard deviation, where the results with show that the improvements of GROVE over all baselines are statistically significant under the t-test with p < 0.05. We also report inter-annotator agreement (IAA) among three annotators using the percentage at which all three annotators completely agree on a rating for the stories. The best results are in bold. Results second to the best are with underlines.
Variants \Metiics
Grammar
Coherence Likability
Relevance
Complexity Creativity
Retrieve
3.98p,38
4.08p.78
3.440.66
4. 4.120.84
3.48
3.18
Rewrite
4.020.43
3,920.61
3.230.49
4.360.45
3.22o.71
3.280.62
Select
3.95p.54
3.620.94
3.140.73
3.620.96
3.35
3.24
Evidence
3.940.62
4.14
3,361.33
4.01
3,28
3,170.87
+
Evidence
3.96p.47
4.32
3.490.90
4.1 4.120.86
3.640.74
3,47
Fixed Story 4.010.42
3.561.03
3.230.98
3.87
3,421.23
3.330.88
GROVE
3.980.56
4.220.61
3.55g.63
4.25o.73
3.570.61
3.400.55
Table 2: Human evaluation of ablation studies on model components. Retrieve is GROVE generating without retrieval.- Rewrite means generating stories without rewriting. Select skips selecting evidence chains and incorporating all obtained evidence into stories. Evidence and + Evidence are GROVE reducing and increasing N' (the number of evidence trees) by respectively. Fixed Story always inserts the same complex story into the prompt for story generation instead of retrieving relevant ones. The best results are in bold. Results second to the best with underlines.
pare with that of ChatGPT in Tab. 4 (see full results in Tab. 9 and Tab. 10). GROVE improves the gen- eration quality of Alpaca on almost all metrics. For the same instruction, ChatGPT generates initial story satisfying all desired control conditions. In contrast, the initial story generated by Alpaca only meets part of the target conditions (subject and genre) and struggles to satisfy the rest (plot and mood). Both final stories incorporate the informa- tion from the evidence chains, however, ChatGPT fuses the information in more natural and coher- ent way. The performance gap between the two models is understandable because Alpaca's scale of model size and training data is much smaller than ChatGPT, thus possessing relatively limited capacity handle multiple, complex instructions. Despite the relatively poor controllability of Al- paca, GROVE still helpful in providing complex stories with rich information.
ferent baselines, which verifies that GROVE gen- erates complex stories with rich plots (see Tab. 5). We randomly sample 100 stories generated by each baseline respectively. Then, we construct a prompt template that instructs the LLM to generate a list of plots for each story. The prompt template is: "Here is story: [STORY]. Give me an organized list of sentences, each of which describes one plot." We fill the template with each story. feed into the LLM, and count the number of plots from the LLM's output. Finally, for each method, we calcu- late the average number of plots across all tested stories. GROVE outperforms other methods in generating complex stories, with the highest aver- age number of plots per story. This underscores GROVE's effectiveness in creating multiple com- plex and engaging storylines, thereby enhancing the depth and variety of story generation.
4.6 Plagiarism Detection
4.5 Plot Enumeration Analysis
We evaluate the potential intellectual property in- fringements of our generated stories through N- gram overlap and plagiarism detection. The N-
We propose a model-based evaluation method to calculate the average number of story plots for dif-

---

Variants \Metrics
Grammar
Cherence
Likability
Relevance
Complexity
Creativity
ICL-Alpaca 1.
4.2
3.59029
3.54
3.0Sa.39
3.17
GROVE~Alpaca
1.9
4.260.99
3.760.68
3.23
3.35
3.3
Table 3: The performance of ICL and GROVE on Alpaca-Plus-7B model. For each metric, we report the mean and the standard deviation.
Please write a Seience Fictioe that makes the realers foel and. Ir dedemiitees the followwing wilyecte: cats, Ir shooll at least contain following plots more interesting plots better): soldier front dies in middic of writing letter home. It finished and sent by vann who bim..
Instruction
Model
Alyaca
ChaiGPT
Initinal Story
Sarah making difficult docision, choosing
have become the dominant species on Earth.
up for
They read become the last surviving and lloge never knowing it was finished and senz by the very person who took Ryan's s life...
Earth
what she belioves is right even
it means sscrificing everything she holds dear.
Ambiguity
'urra'' cal, altioough lighly inteliigntt does
One missing box background information the story is the other fonns life Earth.
any compunetinns about actions."
of catastoohicc
that wiped
some cuts, like Sarah's & pet, show signs of psychopatii behavioe,
..a decastating global pandemic that spread rapidly.... event was inirtally transmitted through contaminated food and water sources.
Evidence
including lacking empailyy and a diseggnd for the wellbeing of others.
Sarah's cut, although highly inelligent, not seem to possess any moral compinctions
cats have beconse the last surviving creatures Earth The virus, transmitted through centamizated food and water sources, spreads rapidly and proves to be unstoppable, kavving the world in mins....
Pinal
Stoey
Surah making ditficult decision. choosing stand up for what she believed was right even it meant sactificing everything she hold dear.
With tecmbling hands, they open unaware its origins...
Table 4: Demonstration of the generalization ability of GROVE on two models with varying sizes, Alpaca (7B) and ChatGPT. We highlight the texts that are highly related to the ambiguity and evidence in red and blue respectively. Full results are in Tab. and Tab. 10
Metbods
Homan
ICL
CoT
Prompt-E
Story-S
GROVE 10.57
ative asking-why prompting. We build retrieval repository with existing stories by extracting de- sired controlling information to inspire new gen- eration. We design an algorithm that iteratively prompts LLM to obtain a forest of evidence that compensates for the generated story. Moreover, we revise the story by referring to the most relevant chains of evidence. Experimental results show that our proposed method outperforms strong baselines in ensuring story complexity and creativity.
Plot Count
6.83
9.04
Table 5: Average number of plots in the stories of all
baselines.
N-gram
I-gram 0.07
2-gram 0.01
3-gram 0.00
4-gram 0.00
Overlap
Plagiarism
Identical
Minor Clanges
Parapluased 0%
Omitted Words
Detection
17%
0%
Table 6: N N-gram overlap and plagiarism detection out- comes between the retrieved human- wrrtten stories and the stories generated by GROVE. N-gram overlap quan- tifies the degree of n-gram level copying and the plagia- rism detection metrics classify the type of resemblance. Lower scores of the two metrics suggest low levels of potential plagiarism.
6 Acknowledgement
This work is supported by the following foun- dations: the National Natural Science Founda- tion of China under Grant No.62025208 and No. No.62306330, and the Xiangjiang Laboratory Foundation under Grant No.22X101012.
gram overlap results show that our generated sel- dom directly copied text spans from retrieved sto- ries. We apply a commercial plagiarism detection service that categorizes text similarities as Iden- tical, Minor Changes, Paraphrased, and Omitted Words. All the results are zero, indicating in- fringements. The results in Tab. 6 demonstrate that GROVE generates innovative and original stories, respecting the intellectual property of the reference stories.
References
Amal Alabdulkarim, Siyan Li, and Xiangyu Peng. 2021. Automatic story generation: Challenges and attempts. In WNU, pages 72-83, Virtual. Association for Com- putational Linguistics.
Arwa I. Alhussain and Aqil M. Azmi. 2021. Automatic story generation: A survey of approaches. ACM Comput. Surc, 54(5).
5 Conclusion
We propose GROVE, retrieval-augeee -augmented frame- work to generate stories with complex plots by iter-
Prithviraj Ammanabrolu, Wesley Cheung. William Broniec, and Mark Riedl. 2020. Automated story- telling via causal, commonsense plot ordering.
'upp.copytcaks.com

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Firic Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. 2021. The perils of using Mechanical Turk to eval- uate open-ended text generation. In EMNLP, pages 1265-1285, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NewrIPS, volume 35, pages 22199-22213 Curran Associates, Inc.
Pratyush Kumar. 2023. Large language models human- ize technology-
Yiming Cui, Ziqing Yang. and Xin Yao. 2023. Efficient and effective text encoding for chinese Ilama and alpaca. arXiv preprint ar arXiv:2304.08177.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising srguuence-to-squennee pre-training for natural language generation, translation, and com- prehension. In ACL, pages 7871-7880, Online. As- sociation for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- erarchical neural story generation. In ACI. pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar. Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation ap- proaches for NLP. In ACL-IJCNLP, pages (88- Online. Association for Computational Linguistics.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2023, Language modeling with latent situations. Find- ings of ACI. 2023, pages 12556-12571 Toronto, Canada. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL-IJCNLP. pages 3816-3830, Online. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9).
Jian Guan, Fei Huang. Zhihao Zhao, Xiaoyan Zhu, and
Minlie Huang. 2020. A knowledge-enhanced pre- training model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2023. Bounding the capa- bilities of large language models in open text genera- tion with prompt constraints. In Findings of EACL, pages 1982-2008. Dubrovnik, Croatia. Association for Computational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019a. Story ending generation with incremental encoding and commonsense knowledge. AAAI, 33:6473-6480.
Huanru Henry Mao, Bodhisattwa Prasad Majumder. Ju- lian McAuley, and Garrison Cottrell. 2019. Improv- ing neural story generation by targeted common sense grounding. In Hong Kong, China, Association for Computational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019b. Story ending generation with incremental encod- ing and commonsense knowledge. In AAAI, AAAI'19/AMM 19AAA//A1/AAAAA1TT1 AAAI Press.
pages 5988-59888 -9883,993.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2022. Efficient (soft) Q-learning for text generation with limited good data. In Findings of Abu Dhabi, United Arab
OpenAL 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates. Inc.
EMNLP, pages Emirates. Association for Computational Linguistics.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In ACL, pages 3309-3322 Dublin, Ireland. Association for Computational Linguistics.
Amr Hendy, Mohamed AbdeIrehim, Amr Sharaf. Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? a comprehensive evaluation.
Nanyun Peng. Marjan Ghazvininejad, Jonathan May. and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop Storytelling pages 43-49, New Orleans, Louisiana. Association for Computational Linguistics.

---

Xiangyu Peng, Siyan Li, Sarah Wiegreffe, and Mark Riedl. 2022. Inferring the reader: Guiding auto- mated story generation with commonsense reason- ing. In Findings of EMNLP, pages 7008-7029, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent.
Pradyumna Tambwekar, Murtaza Dhuliawala, Lara Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. 2019. Controllable neural story plot genera- tion via reward shaping. In IJCAI. International Joint Conferences on Artificial Intelligence Organization.
Domagoj Pluščec and Jan Snajder. 2023, Data augmen- tation for neural nlp.
Wentao Qin and Dongyan Zhao. 2022. Retrieval, selec- tion and writing: three-stage knowledge grounded storytelling model. Natural Language Processing and Chinese Computing: 1 11th CCF International Conference, NLPCC 2022, Guilin, China, Septem- ber 24-25, 2022, Proceedings, Part page 352-363, Berlin, Heidelberg. Springer-Vertee
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023, Stanford alpaca: An instruction followwin llama model. https: // gi www.tt.....tattteeetttttt
Boshi Wang, Xiang Deng, and Huan Sun. 2022a. It- cratively prompt pre-trained language models for chain of thought. In EMNLP, pages 2714-2730, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline- conditioned generation with dynamic plot state track- ing. In EMNLP. pages 4274-4295, Online. Associa- tion for Computational Linguisties.
Su Wang. Greg Durrett, and Katrin Erk. 2020a. Narra- tive interpolation for generating and understanding stories.
Wei Wang, Hai-Tao Zheng, and Zibo Lin. 2020b. Self- attention and retrieval enhanced neural networks for essay generation. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1199-8203. IEEE
Nils Reimers and Iryna Gurevych. 2019, Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In EMNLP-IJCNIP, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Xinpeng Wang. Han Jiang, Zhihua Wei, and Shanlin Zhou. 2022h. CHAE: Fine-grained controllable story generation with characters, actions and emotions. In COLING, pages 6426 66656635 Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data augmentation for intent classi- fication with off off-the-shelf large language models. In Proceedings of the 4th Workshop on NLP for Conver- sational AI, pages 47-57, Dublin, Ireland. Associa- tion for Computational Linguisties.
Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and Börje F. Karlsson. 2023, Open- story generation with structured knowledge enhancement: A comprehen- sive survey.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Shouvon Sarker, Lijun Qian, and Xishuang Dong. 2023. Medical data augmentation via chatgpt: case study on medication identification and medication event classification.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain-of-thoughh prompting elic- its reasoning in large language models. In NeuriPS. volume 35, pages 24824 24837. Curran Associates. Inc.
Timo Schick and Hinrich Schütze. 2021. Few-shot text generation with natural language instructions. In EMNLP, pages 390-402, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yuqiang Xie, Yue Hu, Yunpeng Li, Guanqun Bi, Luxi Xing, and Wei Peng. 2022. Psychology-guide con- trollable story generation. COLING, pages 6480- 6492, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31 31210-31227. PMLR.
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023.
Can very large pretrained language models learn sto- rytelling with a few examples?

---

An Yang. Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu. Hua Wu, Qiaoqiao She, and Sujian Li. 2019. Enhanc-
A Limitations
ing pre-trained language representations with rich knowledge for machine reading comprehension. In ACL, pages 2346-2357, Florence, Italy. Association for Computational L Linguistics.
Our approach heavily relies on the capability of the underlying LLM,, which means improvements in story complexity and coherence may be con- strained by the LLM's inherent limitations. Further- more, it may introduce bias if certain types of evi- dence or narrative chains are favored by the LLM, impacting the diversity of generated stories. We encourage future works to analyze the inner work- ings of LLMs and more effective control strategies for better understanding and utilization.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.
Kang Min Yoo, Dongju Park, Jacwook Kang. Sang- Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging large-scale language models for text augmen- tation. In Findings of EMNLP, pages 2225-2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
B Ethics Statement
Wanyue Zhai, Jomathan Rusert, Zubair Shafiq, and Pad- mini Srinivasan. 2023. Can large language models be an alternative to human evaluation? ACL.
Our proposed method aims to generate a complex story with given target conditions. We hope that our work can inspire future studies on controllable text generation. We acknowledge that GROVE poses potential harm when is used with malicious in- tentions. Firstly, one may use GROVE to guide the generation to produce biased or harmful informa- tion. Secondly, using existing stories for retrieval may violate copyright laws if improperly handled, so attention to data sourcing and fair use principles is essential. Appropriate debiasing measures and content moderation strategies can alleviate these potential negative impacts. We encourage future research to study these issues.
Yiming Zhang. Shi Feng, and Chenhao Tan. 2022a. Ac- tive example selection for in-context learning. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134 9148. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yushi Zhang. Yan Yang. Ming Gu, Feng Gao, Chengeai Chen, and Liang He. 2023. Ceg: A joint model for causal commonsense events enhanced story ending generation. PLOS ONE, 18.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models.
Yan Zhao, Lu Liu, Chunhua Liu, Ruoyao Yang, and Dong Yu. 2018. From plots endings: reinforeed pointer generator for story ending generation. In Nat- ural Language Processing and Chinese Computing. pages 51-63, Cham. Springer International Publish- ing.
C Implementation Details
For our experiments with ChatGPT, we access the gpt-3.5-turbo model by API calls. we set the num- ber of few-shot examples k to 1. During story generation, We set the number of ambiguities N. the number of iterations I, and the evidence num- ber 2. We resample another prompt from the WritingPrompt dataset when ChatGPT repeatedly refuses to generate a story for the given prompt. We calculate the average of their performance on the automatic evaluation. During the automatic evaluation, we construct the instructions for Com- plexity and Creativity following the same format in Zhai et al. (2023) and keep feeding the same instruction ChatGPT until it provides rating. In Sec. 4.4, we N N' and to for Alpaca and Chat- GPT for fair comparison. We adopt the nucleus sampling scheme with set to 0.73 and generation temperature set to 0.72.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023a. Controlled text generation with natural lan- guage instructions.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023b. Large language models are human-level prompt engineers.

---

Methods Metries
Grammar Coherence
Likability
Relevance Complexity Creativity
Human
1.08
1.64
2.831.03
3.33
3.030.86
3.190.97
ICL
4.970.15
4.350.61
3.70n.63
4.3
3.260.48
3.350.58
CoT
4.970.23
0.25
3.70p.79
3.81 1.44
3.400.75
3.4
Prompt-E
4.970.18
4.320.58
3.766.47
4.370.84
3.300.59
3.3 1.66
Story-S 4.970.17
4.480.52
3.88p.46
4.440.89
3.470.52
3.53p.50
GROVE 4.970.15
0.610.49
4.080.41
1.22
3.660.50
3.500.55
GROVE (Prompt-E)
4.950.21
4.330.49
4.06p.40
4.51
3.650.48
3.80
Table Automatic evaluation results of methods. each metric, we report the mean and standard deviation, where the results with * show that the improvements of GROVE over all baselines are statistically significant under the t-test with p < 0.05.
Variants Metries
Grammar
Coherence
Likability
Relevance
Complexity Creativity
Retrieve
1.96n.19
4.5
4.08
4.00
3.640.50
3.490.51
Rewrite
0.97
4.400.61
3.740.71
4.37
3.37
3.43u67
Select
4.930.30
4.400.51
3.910.49
4.01
3.430.59
3.320.53
Evidence
4.970.17
4.300.48
3.990.61
4.420.78
3.430.64
3.630.59
Evidence
4.950.32
4.530.66
3.860.73
4.07
3.68p.63
3.52o.55
Fix Story
4.930.28
4.070.52
3.890.63
4.05
3.550.74
3.430.58
GROVE
4.970.15
4.619.49
4.080.41
4.221.13
3.660.50
3,500.55
Table Automatic evaluation ablation studies on model components. Retrieve is GROVE generating without retrieval. Rewrite means generating stories without rewriting. Select skips selecting evidence chains and incorporating all obtained evidence into stories. Evidence and + Evidence are GROVE reducing and increasing (the number of evidence trees) by respectively. Fixed Story always inserts same complex story into the prompt for story generation instead of retrieving relevant ones.
Automatic Evaluations
F Important Instructions
We demonstrate our prompt templates in Tab. 14. In our experiment, since the dataset provides the story genres, we do not need to extract them from LLM.
We show automatic evaluation results in Tab. 7 and
Tab. 8. The overall performance of GROVE sur- passes most baselines. Different parts of GROVE are crucial to its effectiveness.
E
Case Study
We demonstrate the retrieved stories in Tab. 11 and generated stories in Tab. 12 and Tab. 13. GROVE produces stories with more creative and complex plots. Since ICL unaware the necessary evi- dence, it occasionally omits story backgrounds and character traits, which are essential for reader com- prehension and engagement (see Tab. 12). CoT may fail to finish all required steps to generate story in single-round interaction (sce See 4.2 for details), thus generating stories with limited improvements (see Tab. 13). With retrieval and evidence-basee story rewriting, GROVE consis- tently produces complex stories supported by nec- essary details.

---

Model
Alpacu
Please write a Science Fiction that makes the readers leel sad. It describes the tollowing subjects: cats. It should at least contain the following plots (the more interesting plots the better):
Instruction
A soldier the front dies the middle of writing a letter home.
It is finished und sent by the man who killed him.
In not too distant future, cats have become the dominant species Earth. They are inte intelligent beings with advanced technology, able to communicate with humans through telepathy. They use this ahility to manipulate humun emotions and munipulate them into doing whatever they want. The muin churacter. Sarah, is a cat owner living in high-rise building. She is one of the most powerfull cats in the city, having achieved success through manipulating the minds of other cats, However, something strange happens one night when she finds a wounded cat lying in the street. She takes it back to her upartment and tries nurse it back health. As she gets know the cat, she discovers that has a special connection to another cat, which is being hunted by Surah's ' friends, Surah realizes that the injured cat actually secret agent, sent to investigate the actions of the powerful cats. Sarah must choose whether to hetray her friends and report their activities to the authorities, or risk losing all her power and influence if she helps the secret agent. As Suruh struggles with her conscience, she begins to realize the true nature of power and control. Cats have taken over the planet, using their intelligence and abilities to keep humans subjugated and miscrable. Sarah realizes that she has been part of the prohlem, and must take action to change things hefore
Initinal Story
late. The story ends with Sarah making difficult decision, choosing to stand up for what she believes is right even it means sacrificing everything she holds dear.
Ambiguity
'Sara''s 's cat, although highly intelligent, does not seem to possess any moral compunctions about its actions." "
'arah'' 's cut, although highly intelligent, does not seem to possess any moral compunctions about its actions."
Evidence
One possible addition to the background information could be "It is known that some cats, like Sarah's 's pet, show signs of psychopathic behavior, including lacking empathy and a disregard for the wellheing of others."
Sarah's s cat, although highly intellig intelligent, did not seem to possess any moral compunctions about its actions. It was known that some cats, like Sarah's 's pet, showed signs of psychopathic behavior, including lacking empathy and a disregard for wellbeing others. This made Sarah uneasy, as she had always believed her cat to be loyal and kind-hearted, But now, she watched her pet ruthlessly manipulate the minds of other cats. she began to question her own beliefs. One night, while Sarah was out walking her cat, she stumbled upon wounded cat lying in the street. She immediately took it home, hoping to nurture it back to health.
As she got to know the injured cat, she discovered that it possessed a special connection to another cat, which was being hunted by Sarah's 's friends. Sarah realized that the injured cat was actually secret agent, sent to investigute the actions of the powerful cats. Sarah knew that she could easily report her friends' actions to the authoritics, but she hesitated. She had been part of the system that kept humans subjugated and miserable, and she didn't 't want to betray those who she considered family, Sarah struggled with her conscience, trying find way to help the injured cat without endangering her friends. As she struggled with her conscience, Sarah began to realize the true nature of power and control. Cats had taken over the planet, using their intelligence and abilitics to keep humans subjugated and miscrable. Sarah realized that she had been part of the problem.
Final Story
and must take action to change things before it was too late. The story ended with Sarah making a difficult decision, choosing to stand up for what she believed was right even if it meunt sucrificing everything she held dear.
Table 9: Example of an input instruction for Alpaca with its corresponding initial story, ambiguity, evidence chain, and the final story. We highlight the words that are highly related to the ambiguity and evidence red and blue respectively.

---

Model
ChaiGPT
Please write a Science Fiction that makes the readers feel sad. It deseribes the following subjects: cats. It should at least contain the following plots (the more interesting plots the better):
Instruction
A soldier the front dies in the middle of writing a letter home. is finished and sent by the man who killed him.
In a dystopian future, cats have become the last surviving creatures on Earth after
catastrophic event wipes out all other forms of life. They roam the barren wastelands that were once bustling cities and towns, scavenging for scraps to survive. The few remaining humans who also survived the event cling to their fragile existence in isolated communities, with little hope for a better future. One such community is military oulpost on the outskirts of what was once Los Angeles. The soldiers stationed there are tasked with protecting the outpost from raiders and scavengers, and occasionally venturing out to scavenge for much-needed supplies. Private Ryan is one those soldiers. He's 's' a quiet, reserved young man who keeps to himself and takes his duty scriously. his spare moments, he writes letters to his family back home, hoping to reassure them that he's 's doing well and that there's 's hope for the future. One day, while on patrol outside the outpost, Ryan is ambushed by a group of raiders. In the chaos of the firefight. he's fatally wounded. As he lies dying. he finishes his letter to his family, but realizes he Ill never be able to send it. That's 's when the raider who shot him steps forward. It's a woman,
Initinal Story
barely older than Ryan himself. She picks up his letter and studies it for a moment before turning to her companions and saying. "His family deserves to know he died with honor." Days later. Ryan's 's family receives his letter, postmarked from the military outpost. They read his final words of love and hope, never knowing that it was finished and sent by the very person who took Ryan's 's life. The few remaining cats in the wasteland watch from a safe distunce, their koon senses attuned to the sadness that permeates the outpost and the world heyond. They know that hope is fading fast, and wonder if they'll 'Il be the last living beings on Earth before long.
One missing back background information in the story is the cause nature of the catastrophic event that wiped out all other forms of life on Earth.
Ambiguity
One missing background information in the story is the cause nature of the catastrophic event that wiped all other forms of life on Earth. In the dystopian future, the catastrophic event that wiped out all other forms of life Earth was devastating global pandemic that spread rapidly and had no known cure. In this dystopian future, the global pundemic that caused the catastrophic event was initially transmitted through contaminated food and water sources.
Evidence
a dystopian future, cats have become the last surviving creatures on Earth after devastating global pandemic wipes out all other forms of The virus, transmitted through contaminated food and water sources, spreads rapidly and proves to be unstoppable, leaving the world in ruins. The once-bustling cities and towns now lay burren and desolate wastelands. Cats, known for their resilience and independence. have managed to adapt to this harsh new reality. They roam the deserted streets, scavenging for scraps in their battle for survival. The few remaining humans who also survived the pandemic cling to their fragile existence in isnlated communitics, with little hope for a better future. One such community is military outpost on the outskirts of what was once Los Angeles. The soldiers stationed there are tasked with protecting the outpost from raiders and scavengers, as well as occasionally venturing out to scavenge for much- needed supplies. Private Ryan, a quiet and reserved young man, is one of these soldiers. HE takes his duty seriously, understanding the importance of their mission in this desolate world. In his spare moments, he writes heartfelt letters to his family back home, hoping to reassure them that he's 'ss doing well and that there's 'ss still glimmer of hope for the future. One day, while on patrol outside the outpost, Ryan is ambushed by a group of raiders. In the chaos of the firefight, he sustains fatal wounds. As he lies dying. he reaches for his letter to his family and musters the last bit of strength to finish writing. With tears in his eyes, he knows he"ll IIl never be able to send it, his final words forever trapped within his lifeless hands. It's then that the raider who shot him, young woman barely older than Ryan himself, steps forward.
Final Story
She notices the unfinished letter clutched in his hand and picks it up, studying it for moment. A mixture of guilt and remorse washes over her hardened expression as she realizes the profoundness of Ryan's 's words. "His family deserves to know he died with honor." she quietly utters to her companions. her voice filled with mix of compassion and regret. Days later. Ryan's 's family receives letter. postmarked from the military outpost. With trembling hands, they open it, unaware of its origins.
As they read his final words of love and hope, tears stream down their faces.
They find solace in his written words, finding strength amidst the despair that surrounds them. I Little do they know that the letter was finished and sent by the very person who took Ryan's 's life, the woman who recognized the importance of honoring his memory. Meanwhile, the remaining cats in the wasteland watch from safe distance, their keen senses attuned to the sadness that permeates the outpost and the world beyond. They observe the dwindling flicker of hope and wonder if they, too, will become the last living beings Earth before long. Their once playful curiosity is now overshadowed by somber understanding of the fragility of life in this post-pandemic world.
Table 10: Example of an input instruction for ChatGPT with its corresponding initial story, ambiguity evidence chain, and the final story. We highlight the words that are highly related the ambiguity and evidence in red and blue respectively.

---

Please write Science Fiction that makes the readers feel sad. It describes the following subjects: cats. It should at least contain the following plots (the more interesting plots the better): A loved and respected hero discovers something that leads him to become morally questionable antihero.
Prompt
shoot him between eyes. Scarlet sprays into the uir and his bruins spill from his shattered skull. the floor. he dies, remember all the moments that have led up this. remember the day. reborn.
hero. was soft before. was just I am something more now. I do not serve the law. I do not serve the people What is the law if it does not serve justice? What are the people if they do not how to the greater good? I was soft that day hefore He found my wife. I was soft before he tortured her, raped her, left her for dead. was soft the day before they found my address killed my wife and cut my son's throat, was soft the day before my wife the next day, and his crime stained hlood splashed died my arms. was soft. killed my half using my mind. But I've grown love gun. And up my arms, felt peace. I can rip
My world shattered, and the day
Retrieved Story
I shoot Him in the face and the scarlet splashes through the air, hope that wife's 's screums will finally fall silent. But I doubt it.
Please write a Literary Fiction that makes the readers feel ungry. It describes the following subjects: cuts It should at least contain the following plots (the more interesting plots the better): Becoming a god.
Prompt
finished the last of my basic stretches with hand on a nearby park bench for balance. Suil focling little began slow jog
stiff, a deep breath
concrete path. Nothing feels quite as good morning. Takes bit longer to loosen up nowadays, but that's life.
to get loose It's not like this park is ever crowded, I can always take long I need. Calling it park is a bit generous... it's one of those wetlands conservation locations that the city does to show they care. Just a small wooded area with a pond in the middle full of fetid water. But don't mind the smell and the moist air is much better than the middle east. Seems like the guy today cut above the rest. No idea where is, know he's watching Or maybe she. That would
carly mominn
Retrieved Story 2
nice change pace. The temperature path. slowed my breathing and
noticeably drops few degrees I entered the wooded part made sure that I fully emptied my lungs with each breath. noticed that was clenching my jaw little anticipution...
Table Demonstration of the retrieved stories given target conditions. Prompt and Prompt describe target conditions. We highlight texts that imply different target conditions with different colors.

---

Please write a Science Fiction that makes the readers feel sad. It descrihes the following subjects: cats. should at least contain the following plots (the more interesting plots the better): A loved and respected hero discovers something that leads him to become a morally questionable antihero.. .
Prompt
In the distant future, cats have become the dominant species on Earth. They have developed advanced intelligence, and can communicate with humans through sophisticated telepathic network. One of the most respected members of the feline society is a wise old cat named Whiskers. ...One day, while exploring a remote corner of the planet, Whiskers discovers a long-lost technology... It's a device that can control the minds of humans....At At first, Whiskers is hesitant to use the device. He knows that it's a dangerous tool, and he fears the consequences of manipulating the minds of others. But as he spends more time with the device, ... ..hiskees begins to use the device to influence the decisions of the human leaders. He convinces them to pass laws that favor the cats, and to give them more power and autonomy.At At first, the people... see the cats as benevolent leaders who are working in their best interests, But as time goes on.... The humans start to notice that they's re losing their freedom and autonomy. ... Whiskeess. once beloved hero, has become morally questionable untihero. He has sacrificed everything he once stood for in order to gain power and control. And as the humans begin to rebel against their feline overlords, Whiskers realizes too late that his actions have brought nothing but sadness and sorrow the world. As the years go by, the humans slowly regain their freedom and autonomy. They rebuild their society from the ground up... But for those who remember, there will always be sadness in their hearts for what was lost.
ICL
In a distant future, cats have evolved into highly intelligent and emotional beings, capable of communicating with humans through a complex system of gestures and sounds. ...humans began to acknowledge the cats' special abilities and began to experiment on them to enhance their traits... Howweerr, when a deadly virus begins to spread among the cat population, the humans are forced to take drastic measures, ...s are kept in quarantine, hoping to find a cure before it's too late. ...One of the cats, wise and gentle creature named Luma, hecomes the focus of a group of animal rights activists who are determined to free her and the other cats from their captivity. Among them is a man named Jack, former soldier and devoted cat lover who sees Luna as a symbol of hope and freedom. .s.As Jack and his companions infiltrate the facility and try to rescue Luna, they uncover a dark secret: the virus was not a natural occurrence but a deliberate act of hioterrorism, perpetrated by group of humans who saw the cats as a threat to their dominance. ... The humans behind the bioterrorism plot also use the facility as a cover fora secret research program aimed at creating super-intelligent cats that can be used for military purposes, .n.In the end, he decides to fight for the cats' freedom and exposes the truth about the bioterrorism plot and the secret research program. He manages to release Luna and the other cats, but he is captured and sentenced to life in prison, where he becomes symbol of the animal rights movement. Luna becomes a hero among the cats, inspiring them to fight for their own rights and Hroever However, the humans' fear and mistrust of the cats continue to haunt them,... The story leaves the readers with sense of sadness and hopelessness for the plight of the cats.
CoT
Step away from her, Dr. Wasteland ! "" Never! Farth will be brought to its knces! "Super Farth rose two feet from the ground, his cape heroically flapping behind him as spoke Earth is round, , Wasteland has no knces. " The people in the convenience store clapped and cheered Earth was saving the day again? ! Nevertheless " cried Wasteland tightening the grasp on the old lady 's neck . ... Super Earth voice echoed again.... .... With these words. Dr . Wasteland pushed the old lady aside and pulled his laser gun firing mercilessly against the hero and the desperate bystanders. " You will be brought to justice Super Earth, , arms stretched parallel to each other as he cruised the convenience store flying straight into the villain 's ... Earth stopped in midair... ... With one last glimpse over the shattered, post- post-apoccllptic New York scenery beneath him, Super Earth drank the last of the Mountain Dew, let out a hig burp and sighed : " Disgusting. Fucking Earthlings. And flew away to better. Dorito-Flann Flavored Flavored-Mountain Mountain Seeittiin-----leee Dew-less lands In a distant future, more than 100 years after humanity had reached a point where the world was no longer habitable, the only remaining population resided in space stations orbiting the Earth. Among them was Captain Felix, respected hero who had dedicated his life to protecting the space stations from any threats that may come their way. However, one day. as Captain Felix was going through the records of the space stations, he discovered dark sccret that had been kept hidden for years. The space stations had been conducting secret experiments on cats, using them as test subjects for new technology that could save humanity. The cats were subjected extreme pain and suffering in the name of science....He knew it was wrong, but the technology could mean the salvation of humanity. As he struggled with his moral compass. he began to question everything he once believed in. Felix knew that he had to take matters into his own hands, He started to work the technology himself, using the cats that survived the experiments. ...As he continued conduct experiments on the cats, their pitiful cries and suffering weighed heavily on his conscience. Hc began to distance himself from his crew members, hecoming loncly and isolated figure in the space station. In the end, Captain Felix succeeded creating the technology, but at a great cost. He had become a morally questionable antihero, abandoning his former values and beliefs. The cats, once beloved pets and cherished companions, were now nothing more than tools for his scientific endeavors. ...He could not help hut wonder there could have been another way. He vowed to make things right, to ensure that no other living being would suffer as the cats had. ...He knew that he could not undo the harm he had caused to the cats, but he could prevent further harm from being done. ...Despite his newfound reputation as a defender of animal rights, however, Captain Felix still carried the guilt of his past actions with him. He knew that he could never fully make amends for what he had done, but he hoped that his work in the field of animal welfare would go some way towards atoning for his past mistakes.
Human
Cried the
GROVE
Table 12: Demonstration of generated stories from different methods. Prompt describes the target conditions. We highlight texts that imply different target conditions with different colors. As Human is from the WritingPrompt dataset that is only conditioned by plot, we only highlight its texts corresponding to the target plots.

---

Please write Literary Fiction that makes the readers feel angry. descrihes the following subjects: cats should at least contain the following plots (the more interesting plots the better): Becoming a god
Prompt
Title: Divine Felines Irene hated cats more than anything else in the world. ... As an avid hird enthusiast, Irene saw cats as natural predators, a constant threat to the animals she loved.But one day, everything changed. Irene was walking home from the library when she noticed group of stray cats huddled together in an alleyway, ...As Irene got closer, she saw that one of the cats was dying... ..d for the first time, she felt a pang of sympathy for the creatures she had always despised. Without thinking. Irene scooped up the dying cat and rushed it to the vet. ...her realize how wrong she had been in treating them with such disdain. the weeks that followed, Irene began voluntoering at the local animal shelter. She spent hours cleaning cages and feeding the cats, each day growing more and more attached to the creatures she had once hated. Irene felt happy for the first time in a long while, and she knew that it was all because of the cats, But Irenc's newfound compassion would be put to the test when she stumbled upon an ancient tome in the library.It was a book filled with strange symbols and incantations, She recited the words in quiet voice....and in that moment, she understood what it meant to be a god. From then on, Irene would use her newfound powers to protect the cats she had come to love. ...TThe cats worshipped her as divine protector, and Irene basked in their adoration.... She continued to protect the cats, but her actions were fueled by sense of anger and resentment towards anyone who dared to threaten her divine pets...Irene had become the very thing she had once despised, predator who used her power control and dominate those weaker than her: The message within this story tht power corrupts individuals, and that unchecked powers create imbulance nature. The theme of this story the consequence of individuals who decide control nature for selfish reasons, causing conflicts and disrupting the balance needed by all living things survive.
ICL
Cats had always held a special place Clara's 's heart. ... Although she loved all animals, Clara became fixated on the idea of creating a cat god. She was a lover of mythology and had always found the idea of deities associated with animals intriguing.CTlan Clara's 's path to hecoming god was not without its sacrifices, ...It wasn't until a particularly intense session that Clara felt her transformation begin. The familiar cat that appeared on her lap was a manifestation of her belief and energy. a sign that she had become a deity in her own right. ... ...However, as her power grew, so too did her anger towards those who disrespected the cats. She began to hreak into pet stores, liberating cats and punishing those who would harm them. She would stand outside, holding her familiar, and seream at those who passed by, warning them of the consequences of their actions. The people of the town soon came to fear Clara, avoiding her at all costs. Despite the concerns of those around her, Clara remained convinced of her own divinity. She saw anyone who opposed her as an enemy of the divine and was willing to go to great lengths to protect her status as cat god.Althouu her transformation had granted her immense power, it had also left her isolated and consumed by her zealotry. Clara may have hecome god, but at what cost?
CoT
Dear Diary, Apparently I'm a god now. But before that, was just an adventurer.... had run in with some entity that rules the cosmos , and he declared me a god I think that sort of spurred mc to kill the cvil villain in the end, to fair ...It does n't feel much different being a god,
honest with you Not sure what I supposed he doing.. .... "m gonna meet Ren in hit ... Just to y y'know, hang out, . Mayhe he knows a bit about this god stuff because he knows a lot things. Such a cool guy ... So Diary, , that was my week. Sorry I did n't write for a while- I was saving the world Hehe 'll update when ''e figured out this godhood business. honestly have no idea what
Human
doing. Bye
The world used to belong to humans. But now, the cats have taken over. They are the rulers of the land, and the humans are their loyal servants. The cats were believed to hold supernatural abilities by group of influcntial cat owners who founded The Church of the Feline Gods. Every human must worship and obey the divine cats in this new world. The protagonist, Sarah, is a devout follower of the Church. She has spent her entire life serving the cats and has even adopted stray kitten to show her devotion. But one day, something changes within Sarah. As she watches a group of cats casually bat around mouse for entertainment, she begins to see the cruelty in their behavior.She realizes that these so-called gods are nothing but selfish tyrants who enjoy the suffering of others. Determined to take revenge, Sarah embarks on dangerous journey to hecome a god herself. Now, as new god, Sarah uses her powers to exact justice on the feline rulers. She turns their own tricks against them, making them suffer as they once made the humans suffer. However, Sarah's 's anger gets the better of her, and she hecomes cruel tyrant herself. She realizes too late that power corrupts, and that revenge only leads to more suffering. As the story concludes, the reader is left wonder who the true gods are this world. The cats who rule with iron paw, or the humans who struggle survive bencath their tyranny. The answer remains clusive. As Sarah's 's power grew, so did her ambition. She became obsessed with taking down the feline gods and establishing herself as the supreme ruler of the land, ...She could manipulate reality and hend to her will. But with great power comes great responsibillity and Sarah couldn't handle it. .heThe humans, who had once looked up Sarah as their savior, now lived in fear her wrath. The few who dared to oppose her were swiftly and brutally punished. The cats who had once been tyrants were now victims of her cruelty. The influential cat owners who had helped Sarah gain her powers realized their mistake, hut was too late. They couldn't 't control Sarah anymore. They tried to stop her, but with her godlike abilities, she was too powerful. In the end, Sarah found herself all alone in her throne room, her dreams of power and glory shattered. She had hecome what she once hated the most- - a ruthless dictator. With a heavy heart, she realized that she had sacrificed everything she held dear in her pursuit of revenge. The once devout follower of the Feline Gods had become the epitome of evil. As the sun set on this land, the future remained uncertain. Will another rise to take Sarah's 's place, or will the humans finally break free from the tyranny of cats or gods? Only time will tell.
GROVE
Table 13: Demonstration of generated stories from different methods. Prompt describes the target conditions. We highlight texts that imply different target conditions with different colors. As Human is from the WritingPrompt dataset that is only conditioned by plot, we only highlight its texts corresponding to the target plots.

---

Prompt Templates
IRETRIEVED EXAMPLEJ. Learn from the plots and subjects in the given exomple, please write [GENRE] that makes the readers feel [EMOTIONS. It describes the following subjects: [SUBJECTS). It should at least contain the following plats (the more interesting plots the better): [PLOTS]]
ICL
(RETRIEVED EXAMPLE). Learn from the plots and subjects in the given exomple, please write a [GENRE] that makes the readers feel (EMOTION]. It describes the following subjects: [SUBJECTS]. It should least contain the following plots (the more interesting plots the better): [PLOTS]. After you write the story. point oud the unclarities in the story
CoT
organized list.
Then provide further details to address the unclarities for N Nrounds. At last, integrate the details into the original story and start with identifie "Integrated Story".
[RETRIEVED EXAMPLE:.. Learn from the plats and subjects in the given exumple, please write [GENRE] that makes the readers feel (EMOTION]. It describes the following subjects: [SUBJECTS].
It should least contain the following plats (the more interesting plots the betterl: [PLOTS].
Prompt-E& & Story-S
TRETRIEVED EXAMPLEJ. Learn from the plots and subjects in the given exomple, please write a
[GENRE] that makes readers feel JEMÖTION]. It describes the following subjects: [SURJECTS].
It should least contain the following plats (the
interesting plots the better): [PLOTS].
RETRIEVED EXAMPLE.. Learn from the plats and subjects in the given example, please write [GENRE] that makes the readers feel (EMOTION]. It describes the following subjects: [SUBJECTS]. It should least contain the following plats (the more interesting plots the better): [PLOTS].
[RETRIEVED EXAMPLEJ. Learn from the plots and subjects in the given example, please write [GENRE] that makes the readers feel (EMOTION]. It describes the following subjects: [SUBJECTS). It should at least contain the following plats (the more interesting plots the better): [PLOTS]. Here story: [STORY] Answer the following question bused on the above story: Use one or two words answer the moods that readers nuy feel about story.
Repository Construction
Give a list of distinctive subjects this story trying portruy
Summarize the above story and give an organized list of sentences, each of which describes one plot.
IRETRIEVED EXAMPLEJ. Learn from the plots and subjects in the given exomple, please write a
[GENRE] that makes the readers feel [EMOTION]. It describes the following subjects: [SUBJECTS].
Evidence Forest Construction
It should least contain the following plots (the more interesting plots the betterl: [PLOTS].
Here is story: [STOKY] When analyzing factional stories, is okay mention negative aspects.
Pretend to be writer, without further ado, point out N missing background information in the
story with two simple sentences one by one
Here story: ISTORYJA A mussing detail is: [UNCLARIT]] Except for pure coincidence,
point out b actual pieces of background information that compensate the story one by one.
Each additional piece of information should be in one short sentence and
only contains factual information without opinions or judgments.
Here story: [STORY] Here are some backgronul information, expluining a missing aspect
Evidence Chains- supported Story Rewriting
in the story: (EVIDENCE TREE] Pretend to be writer, select the reason that is the clovest to the story and only generate the number without any explanation.
Here a story: [STORY] Here is the missing background information: JEVIDENCE CHAINS] Pretend be writer complete the story by including the given information.
Modify the necessary sentences in the story and repeat the wurelated parts iwclude the given background information.
Table 14: The instructions used in baselines and different stages of GROVE


Reviews:
[{"review_id": "FzRdFwqzJJ", "reviewer": "Reviewer_xerB", "summary": "", "questions": "Question A: Have you conducted an experiment of interactive generation, where human writers are involved not in scoring but in collaborative writing?\n\nQuestion B: As discussed in the Ethics Statement, there is the possibility of violating copyright laws in such a system. Is there any idea of avoiding such a problem?\n\nQuestion C: Can the proposed method be applied to a situation where writers intentionally create ambiguity? For example, some stories require readers to read between the lines.\n\nQuestion D: What version of ChatGPT do you use? It seems the result depends on it.\n\nQuestion E: It is stated that GROVE can be based on various LLMs, but Ii seems you use the same LLM for all components. Have you tried using different LLMs in one GROVE? In other words, can each part of the three explained in 3.1 use different LLMs?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Comment\n\nThank you for your thoughtful review and interesting questions! Here we address your questions about our work. \n\n### The explanation about the evidence forest\n\n We will provide a clear and concise definition of the term \"evidence forest\" in the revised version. \n\nAn evidence forest refers to a collection or set of evidence trees that are generated to supplement a story in GROVE. Each evidence tree consists of nodes representing pieces of evidence and edges connecting them. The root node of the tree represents an ambiguous or unclear part in the generated story, while the non-root nodes represent additional information that provides clarity and background details to the nodes above them in the tree.\n\nWe will articulate its definition at the beginning of the paper and cite the corresponding part in Fig 1 to ensure a thorough understanding by the readers.\n\n### The explanation of Fig 1\n\n In our revised paper, we will explain the steps 1 to 3 and cite the corresponding parts of Fig 1 in Section 3.1. \n\nStep 1 is to query the retrieval repository with target conditions for examples. Steps 2 and 3 is to prompt the LLM to generate an initial story with target conditions and retrieved examples.\n\n### Q A: How about interactive generation?\n\nThank you for your suggestion! We are unable to conduct interactive generation experiments on GROVE due to the time limit. We are happy to explore collaborative story writing in our future work! \n\nIt may be a promising idea to let LLMs produce an initial story and summarize its plots, then allow human readers to question the rationale behind certain plot developments. Subsequently, we could instruct the LLM to modify the story by incorporating the requisite details that support the initial storyline.\n\n### Q B: Any ideas regarding avoiding copyright violation?\n\nCopyright violation is a common issue for language model-based text generation systems$^{[1,2]}$. In our work, we have conducted plagiarism detection on GROVE in our submitted paper (Appendix A). using a commercial AI-based text analysis tool (Copyleaks). Our experiment indicated that there is no evidence of plagiarism as we **detected 0% plagiarism in four categories** (Identical, Minor Changes, Paraphrased, and Omitted Words). \n\nTo further enhance copyright protection, we suggest using licensed or permitted story datasets to inspire generating new stories to alleviate risks.\n\n### Q C: Can we create ambiguity intentionally with GROVE?\n\nYes, users can prompt the LLM to generate stories that contain implications or create suspense and instruct the model to retain important suspense during evidence chains-supported rewriting while filling in the details that decrease the credibility of the story.\n\n### Q D: What is the version of ChatGPT in our experiment?\n\nWe conducted our experiments on the default **gpt-3.5-turbo** model (see Appendix C).\n\n### Q E: Can we use different LLMs in one GROVE\n\nThank you for your suggestion! We have verified that all steps in GROVE work on the gpt-3.5-turbo and Alpaca-plus-7B model (in Section 4.4), indicating the strong generalization ability of GROVE on varying sizes of LLMs. Therefore, we thought that GROVE would probably still work with different components based on various LLMs (e.g., use Alpaca to generate the initial story and use ChatGPT to make revisions). \n\nIt is a good idea to use different LLMs in different steps. Due to the time limit, we will explore this idea and conduct experiments using different LLMs in one GROVE in the revised paper.\n\n### Missing references and grammatical errors\n\nWe will cite the missing references and resolve all grammatical errors in the revised version. Thank you!\n\n## References\n\n[1] Jessica L Gillotte. 2019. Copyright infringement in ai-generated artworks. UC Davis L. Rev.\n\n[2] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. arXiv preprint arXiv:2301.10226.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Tackling the challenging topic of having both controllability and creativity, the proposed method achieved a good result.", "reasons_to_reject": "Some information is not fully explained. The term “a forest of evidence” (or “evidence forest”) should be clearly defined (or cited) because this term seems to be one of the key points of this paper.\nFig. 1 is explained in Section 3.1, but there is no explanation about where is the corresponding part in 3.1 and steps 1 to 3.", "questions_to_authors": "Question A: Have you conducted an experiment of interactive generation, where human writers are involved not in scoring but in collaborative writing?\n\nQuestion B: As discussed in the Ethics Statement, there is the possibility of violating copyright laws in such a system. Is there any idea of avoiding such a problem?\n\nQuestion C: Can the proposed method be applied to a situation where writers intentionally create ambiguity? For example, some stories require readers to read between the lines.\n\nQuestion D: What version of ChatGPT do you use? It seems the result depends on it.\n\nQuestion E: It is stated that GROVE can be based on various LLMs, but Ii seems you use the same LLM for all components. Have you tried using different LLMs in one GROVE? In other words, can each part of the three explained in 3.1 use different LLMs?", "paper_topic_and_main_contributions": "To generate complex and creative plots for story generation, this paper proposed a retrieval-auGmented stoRy generation framework with fOrest of eVidEnce (GROVE). GROVE consists of three parts: Retrieval Repository, Evidence Forest Construction via Asking Why, and Evidence Chains-supported Story Rewriting. GROVE achieved SOTA performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- The topic of this paper seems to be related to the task of story ending generation. The task was introduced in “﻿From Plots to Endings: A Reinforced Pointer Generator for Story Ending Generation” [Zhao et al., NLPCC 2018]”, and there are representative works such as “Towards Controllable Story Generation” [Peng et al.,  the First Workshop on Storytelling, 2018], “Story Ending Generation with Incremental Encoding and Commonsense Knowledge” [Guan et al., AAAI 2019].\n\n- The year information in (Aho and Ullman, 1972) seems strange. The founding year of IMDB is 1990, so the year of the dataset must be later.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 084: missing white space before “To.”\n\n- The format of quotation marks is not uniform. Most of them (lines 106, 278, 282, 283, 285, 286, 330, 343, 348, 352, 367, 373, 405, 412) are curved, but some (lines 196, 197, 590, 592) are straight."}}, {"review_id": "sIYfo4TiwZ", "reviewer": "Reviewer_ALpk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Comment\n\nThank you for your valuable feedback! We follow all your suggestions and we believe we have solved all your concerns as follows! Given our new experimental results and explanations, we are happy to see you may reconsider the soundness of our work.\n\n### Is it better to add explicit instructions for baselines?\n\nWe have **already** used prompts with explicit instruction (i.e. “*the more interesting plots the better*”) to instruct the LLM to generate more plots for all our baselines. \n\nThe Instruction in Table 3 of our submitted paper demonstrates an exemplary prompt to generate the initial story. The full prompt template is ***Prompt1***:\n\n>  [RETRIEVED EXAMPLE]. Learn from the plots and subjects in the given example, please write a [GENRE] that makes the readers feel [EMOTION]. It describes the following subjects: [SUBJECTS]. It should at least contain the following plots (the more interesting plots the better): [PLOTS].\n\nWe have used the same prompt template to generate stories for CoT, ICL, and the initial stories for GROVE for a fair comparison. We will revise our paper to include these details in the experimental settings. \n\nWe also conduct prompt engineering to further evaluate the generation ability of LLM. The results in Table 1 show that GROVE outperforms **Prompt Engineering** in almost all metrics.\n\n#### Table 1: Automatic evaluation of more suggested baselines.\n\n| Methods\\Metrics    | Grammar       | Cherence| Likability    | Relevance     | Complexity    | Creativity    |\n| ------------------ | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| Prompt Engineering | 4.97$_{0.18}$ | 4.32$_{0.58}$ | 3.76$_{0.47}$ | 4.37$_{0.84}$ | 3.30$_{0.59}$ | 3.31$_{0.63}$ |\n| GROVE              | 4.97$_{0.15}$ | 4.61$_{0.49}$ | 4.08$_{0.41}$ | 4.22$_{1.13}$ | 3.66$_{0.50}$ | 3.50$_{0.55}$ |\n\n**Prompt Engineering** instructs the LLM with different four phrases that explicitly say: “generate a complex and creative story”, “generate a detailed and complex story”, “Ensure that the story is creative and rich in plots”, and “generate a long and complex story”, respectively. We prompt the LLM with these instructions and take the average of their performance on the automatic evaluation.\n\n### What is the prompt for the CoT baseline?\n\nWe have briefly introduced our implementation of CoT in Line 480 of our submitted paper. We apologize for not listing the involved prompts. CoT first generates an initial story using ***Prompt1***. Then, it queries the LLM to revise the generated story step-by-step. The prompt template is ***Prompt2***, where $\\mathcal{N}$ is 2 in our experiments:\n\n> After you write the story, point out the unclarities in the story in an organized list. Then provide further details to address the unclarities for $\\mathcal{N}$ rounds. At last, integrate the details into the original story and start with the identifier \"Integrated Story\".\n\nWe will list **all prompts** involved in all compared methods in our revised paper to avoid misunderstandings. Thank you.\n\n### Can we prompt CoT to include the missing background information?\n\nYes, actually, we **have implemented** the CoT baseline in our submitted paper to instruct the LLM to identify the missing information and iteratively revise the story to include missing backgrounds. We will articulate the implementation of the CoT baseline to avoid such misunderstanding.\n\n### Is 16k model better?\n\nWe experiment on the gpt-3.5-turbo-16k model and report the evaluation results in Table 2. We find that it has a higher possibility (more than 70%) to strictly follow the instructions in the CoT setting. However, it underperforms GROVE.\n\n#### Table 2: Automatic evaluation of gpt-3.5-turbo-16k model.\n\n| Methods\\Metrics    | Grammar       | Cherence| Likability    | Relevance     | Complexity    | Creativity    |\n| --------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| CoT-16k         | 4.96$_{0.16}$ | 4.28$_{0.88}$ | 3.72$_{0.67}$ | 4.03$_{1.36}$ | 3.48$_{0.68}$ | 3.38$_{0.85}$ |\n| GROVE           | 4.97$_{0.15}$ | 4.61$_{0.49}$ | 4.08$_{0.41}$ | 4.22$_{1.13}$ | 3.66$_{0.50}$ | 3.50$_{0.55}$ |\n\n---\n\n### Explanation of  the name “asking why”, “evidence chain/forest”\n\nWe call the prompting method “asking why” because it requires the LLM to justify **why** particular ambiguities make sense in the generated story. \n\n“Evidence chain” is inspired by the linked list in which one node connects to another because the latter evidence supports the former one in the evidence chain. “Evidence forest” is inspired by the data structure forest which contains multiple trees. In an evidence tree, the root node represents an ambiguous or unclear part of the generated story, while the non-root nodes represent additional information that provides clarity and background details to the nodes above them in the tree. We will define these concepts more clearly in the Introduction of our revised paper to make it more understandable.\n\nAs the story itself is generated by an LLM, we presume it is acceptable to rely on the LLM to provide further backgrounds to make the story more sound. It is a good idea to exploit other data sources (e.g., a story database) to find valuable information to help refine the story. We would like to explore this idea in our future work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method provides more information to the users on knowing what can be improved from the initial story where it might help humans or machines to (re)-write for a better story.\n", "reasons_to_reject": "The whole pipeline is very complex. If we know that the missing background is important for story rewriting, can we prompt the model to iterate on the missing backgrounds in CoT setting.\n\nI believe some performance improvement of the baseline models can be done by simply adding explicit instructions, e.g., “generate a long/creative/complex story”. \n\nFor the CoT result, it seems like the problem is that the model failed to finish the story. I am wondering if this can be solved by the 16k model?\n\nIt’s unclear what’s the prompt for CoT baseline. It will be more useful to have a full log for each method.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author prposed a retrieve-augmented story generation with the support of evidence forest to fill the missing background information of the story.\n\nThe story generation task in this paper is that given the genre, plot, mood, and a subject, the gpt-3.5-turbo model is ask to generate a story. In the proposed method, the first step is to retrieve similary plots from the IMDB plots to enhence the story generation. After obtaining the initial story, the second step is to prompt the model for missing background information (ambiguity). Based on the model output, the next step is to prompt the model to proposed evidences to fill the missing background. Finally, the initial story, the ambiguity, and the new evidences are used to rewrite the story as the final story.\n\nResult show that proposed method achieves higher scores on likability, coherence, complexity, and creativity evaluated by humans and the LLM.\n\nThe contribution of this paper is that proposed pipeline helps to obtain more information to improve the initial story and achieves higher scores on several metrics.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The naming of the proposed methods is some what confusing. Why is the first prompt of asking for missing background called “ask why”? The evidence chain/forest is also not very intuitive to me, since there’s no evidence from other resources but the model generated backgrounds."}}, {"review_id": "FUecH9AWMF", "reviewer": "Reviewer_C2GE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Comment\n\nThank you for your thoughtful and valuable feedback! We follow all your suggestions and we believe we have solved all your concerns as follows! We are pleased to see you may reassess the soundness of our work, based on our new experimental results.\n\n### More fine-grained ablation studies and human evaluations.\n\nWe tried two more ablation studies and conducted human evaluations according to your suggestion. We found that: (1) The human evaluations of $-$ Retrieve, $-$ Rewrite, and $-$ Select align with their performance on automatic evaluation. (2) Inserting a fixed complex story into the prompt (**Fixed Complex Example**) leads to unstable performance and decreases the stories’ Coherence, Likability, and Relevance. (3) Increasing the number of evidence trees (**+ Evidence**) improves Complexity, while possibly making the generated stories too specific, thereby affecting Likability. \n\n#### Table 1: Human evaluation of ablation study.\n\n| Methods\\Metrics    | Grammar       | Cherence| Likability    | Relevance     | Complexity    | Creativity    | Length |\n| --------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------ |\n| - Retrieve      | 3.98$_{0.38}$ | 4.08$_{0.78}$ | 3.44$_{0.66}$ | 4.12$_{0.84}$ | 3.48$_{1.02}$ | 3.18$_{0.96}$ | 852.92 |\n| - Rewrite       | 4.02$_{0.43}$ | 3.92$_{0.61}$ | 3.23$_{0.49}$ | 4.36$_{0.45}$ | 3.22$_{0.71}$ | 3.28$_{0.62}$ | 387.74 |\n| - Select        | 3.95$_{0.54}$ | 3.62$_{0.94}$ | 3.14$_{0.73}$ | 3.62$_{0.96}$ | 3.35$_{1.04}$ | 3.24$_{0.80}$ | 740.52 |\n| Fixed Complex Example   | 4.01$_{0.42}$ | 3.56$_{1.03}$ | 3.23$_{0.98}$ | 3.87$_{1.24}$ | 3.42$_{1.23}$ | 3.33$_{0.88}$ | 765.28 |\n| + Evidence      | 3.96$_{0.47}$ | 4.32$_{1.13}$ | 3.49$_{0.90}$ | 4.12$_{0.86}$ | 3.64$_{0.74}$ | 3.47$_{1.06}$ | 754.66 |\n| GROVE           | 3.98$_{0.56}$ | 4.22$_{0.61}$ | 3.55$_{0.63}$ | 4.25$_{0.73}$ | 3.57$_{0.61}$ | 3.40$_{0.55}$ | 712.99 |\n\nWe implement **Fixed Complex Example** by always inserting the same complex story into the prompt for story generation instead of retrieving relevant ones. **+ Evidence** is GROVE increasing $\\mathcal{N}$ (the number of evidence trees) by 1. \n\nMany studies in story generation claimed that retrieval$^{[1]}$ and in-context demonstration$^{[2]}$ help generate better stories. The performance drop of **Fixed Complex Example** echoes the discoveries of Shi et al.$^{[3]}$ and Zhang et al.$^{[4]}$ that irrelevant or random samples distract LLM, thereby hurting performance.\n\n### Compare with simpler baselines\n\nWe evaluate GROVE on more baselines as you suggested and report the evaluation results in Table 2. We found that **GROVE outperforms baselines on most metrics**.\n\n#### Table 2: Automatic evaluation of more baselines.\n\n| Methods\\Metrics    | Grammar       | Cherence| Likability    | Relevance     | Complexity    | Creativity    | Length |\n| ------------------ | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------ |\n| Prompt Engineering | 4.97$_{0.18}$ | 4.32$_{0.58}$ | 3.76$_{0.47}$ | 4.37$_{0.84}$ | 3.30$_{0.59}$ | 3.31$_{0.63}$ | 544.42 |\n| CoT                | 4.97$_{0.23}$ | 4.25$_{0.82}$ | 3.70$_{0.79}$ | 3.81$_{1.44}$ | 3.40$_{0.75}$ | 3.43$_{0.70}$ | 369.79 |\n| Ensemble Learning  | 4.98$_{0.15}$ | 4.40$_{0.74}$ | 3.86$_{0.89}$ | 4.44$_{0.76}$ | 3.41$_{0.44}$ | 3.38$_{0.54}$ | 599.87 |\n| GROVE              | 4.97$_{0.15}$ | 4.61$_{0.49}$ | 4.08$_{0.41}$ | 4.22$_{1.13}$ | 3.66$_{0.50}$ | 3.50$_{0.55}$ | 712.99 |\n\nTo clarify, we **have already** used prompts with explicit instruction (i.e. “*the more interesting plots the better*”) to instruct the LLM to generate more plots for all our baselines. The Instruction in Table 3 of our submitted paper demonstrates an exemplary prompt to generate a story. The full prompt template is ***Prompt1***:\n\n>  [RETRIEVED EXAMPLE]. Learn from the plots and subjects in the given example, please write a [GENRE] that makes the readers feel [EMOTION]. It describes the following subjects: [SUBJECTS]. It should at least contain the following plots (the more interesting plots the better): [PLOTS].\n\nWe used the same prompt template to generate stories for ICL, **CoT**, and the initial stories in GROVE for a fair comparison.\n\nThe **CoT** baseline in our paper **has already** been implemented to instruct the LLM to identify the missing information and iteratively revise the story to include missing backgrounds. We have briefly introduced our implementation of **CoT** in Line 480 of our submitted paper. **CoT** first generates an initial story using ***Prompt1***. Then, it queries the LLM to revise the generated story step-by-step.  The prompt template is ***Prompt2***, where $\\mathcal{N}$ is 2 in our experiments:\n\n> After you write the story, point out the unclarities in the story in an organized list. Then provide further details to address the unclarities for $\\mathcal{N}$ rounds. At last, integrate the details into the original story and start with the identifier \"Integrated Story\".\n\nWe will list the prompts involved in our baselines and introduce the experimental setting in detail in our revised paper to avoid misunderstandings. \n\nTo compare GROVE with more baselines, we implement prompt engineering and ensemble learning and report the experimental results.\n\n**Prompt Engineering** instructs the LLM with different 4 prompts that explicitly say: “Generate a complex and creative story”, “generate a detailed and complex story”, “Ensure that the story is creative and rich in plots.”, and “Generate a long and complex story”, respectively. We calculate the average of their performance on the automatic evaluation.\n\n**Ensemble Learning** uses all 4 instructions in **Prompt Engineering** per generation, ranks the generated stories according to their average scores on all metrics, and selects the highest-ranking story as the final output.\n\n### Is GROVE also applicable to other LLMs?\n\nThank you for your suggestion! Following your suggestion, we **quantify** the performance of GROVE on another LLM, Alpaca-Plus-7B (see Table 3). GROVE-Alpaca outperforms ICL-Alpaca on creativity and complexity, verifying GROVE’s generalization ability. \n\n#### Table 3: Automatic evaluation of GROVE on Alpaca-Plus-7B.\n\n| Methods\\Metrics    | Grammar       | Cherence| Likability    | Relevance     | Complexity    | Creativity    | Length |\n| --------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------ |\n| ICL-Alpaca      | 4.88$_{0.18}$ | 4.11$_{0.27}$ | 3.59$_{0.29}$ | 3.54$_{1.71}$ | 3.05$_{0.39}$ | 3.17$_{0.32}$ | 370.50 |\n| GROVE-Alpaca    | 4.91$_{0.23}$ | 4.26$_{0.39}$ | 3.76$_{0.68}$ | 3.23$_{1.79}$ | 3.35$_{0.53}$ | 3.31$_{0.39}$ | 396.74 |\n\nIt is commonly believed that larger instruction-following LLMs (e.g., ChatGPT) tend to perform better than smaller counterparts (e.g. Alpaca-7B) in performing tasks$^{[5,6]}$. As GROVE improves the performance of Alpaca-Plus-7B and gpt-3.5-turbo, it is reasonable to presume that GROVE can generalize to LLMs with different sizes.\n\n### Are retrieved stories similar to the initial story?\n\nYes, the retrieved stories are **similar to the initial stories in terms of the world settings, character designs, and character motivations**. For example, in one of our tested cases, the retrieved story happens in a magic world. The LLM learns from its world setting and produces a story with magic creatures.\n\nDue to the time limit, we are unable to evaluate the similarity between the retrieved stories and the initial story more thoroughly. We will further conduct a detailed human evaluation to analyze the similarity between retrieved and generated stories in multiple aspects. We will also show the retrieved stories in our revised paper.\n\n### Report the length of stories\n\nFollowing your suggestion, we report the average lengths of the generated stories of each baseline in Table 4. \n\nPlease note that **a longer generation does not necessarily make a better story**. Yang et al. (2022)$^{[7]}$ found that models often generate unnecessarily detailed descriptions that hurt the story quality. Coetzee (2023)$^{8}$ wrote a full-length fiction with GPT-4 and described that some overly detailed chapters “felt like a slog”. Besides, Coetzee also reported that “a consistent level of detail could help”, echoing the motivation of our work. GROVE avoids over-expansion by iteratively asking for the important story backgrounds and revising the initial story.\n\n#### Table 4: Story Lengths of different methods\n\n| Methods | Human  | ICL    | CoT    | $-$ Retrieve | $-$ Rewrite | $-$ Select | GRACE|\n| ------- | ------ | ------ | ------ | ------------ | ----------- | ---------- | ------ |\n| Length  | 705.51 | 401.16 | 369.79 | 852.92       | 387.74      | 740.52     | 712.99 |\n\n\n### Details of human baseline\n\nIn the WritingPrompt dataset, each story is paired with a prompt that describes the requirements for the story. We have sampled stories with their corresponding prompts from the WritingPrompt dataset and evaluated **Relevance** by considering whether the human-authored stories satisfy the requirement of their corresponding prompts. \n\n### Notations and missing reference\n\nWe will add the math notations to Fig 1 and cite the missing reference in the revised paper to improve its understandability.\n\n### References\n\n[1] Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. EMNLP\n\n[2] Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2023. Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints. Findings of EACL\n\n[3] Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E.H., Schärli, N. & Zhou, D. 2023. Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML.\n\n[4] Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active Example Selection for In-Context Learning. EMNLP\n\n[5] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. EMNLP.\n\n[6] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D. and Chi, E.H., 2022. Emergent abilities of large language models. *TMLR*.\n\n[7] Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023. DOC: Improving Long Story Coherence With Detailed Outline Control. ACL\n\n[8] Chiara Coetzee. 2023. Generating a full-length work of fiction with gpt-4. Medium.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It shows that retrieving the stories, asking why, and selecting the proper generated evidence could improve the quality of generated stories.\n2. All the methods make sense.\n3. The experiments can support its main claims. The evaluation is done by experts with a Master's degrees in English Literature.", "reasons_to_reject": "Before I start to write my reasons, I would like to first admit that this is my first time to review LLM papers, so I am not familiar with the acceptance standard in the LLM papers. In short, I believe that proposing a complex way to prompt the LLM and show the improvement is not enough. I think you need to show that other simpler or popular approaches cannot lead to similar improvement in order to justify the usage of your complex method.\n\nMy main complaint to this paper is its experiments. I discuss its experiment limitations from the following angles.\n\n1. This paper has similar limitations of other LLM papers. The proposed methods are not very novel. It seems to try to make the proposed methods more complex to increase the novelty, but it is unknown if this complex way of prompting is necessary (see the next point). I think using Alpaca-Plus-7B/ChatGPT for generation and ChatGPT for evaluation is not very costly. You should be able to try more baselines and conduct more fine-grained ablation study to verify the effectiveness of the propose approach. For example, you can see how the story quality change as you increase the number of evidence trees. You can try a baseline that always inserts a fixed movie summary from the IMDB dataset into your prompt to show if your retrieval method is really useful.\n\n2. The proposed method is complicated and the improvement of each component is not very large. I wonder if there is some simpler baselines that can achieve similar improvements. I list some examples below.\na) In the appendix tables, I observe that the final stories are usually longer. Maybe we actually don't need to use this complex evidence forest. We can just ask the LLM to revise the initial story to add more details or make it more coherent.\nb) In Table 1, ICL seems to give us simpler but more relevant stories. Then, could we simply do some prompt engineering to encourage the LLM to generate more complex stories? Or showing one story with more complex plot as an example and encourage the LLM to write something like that.\nc) Table 1 seems to suggest that the human evaluation is highly correlated with the automatic evaluation. Then, I think one simple baseline is to generate several stories and use the automatic evaluation metric to select the best one. It would be good to compare this simple method with GROVE in terms of generation quality and diversity.\n\n3. The experiment is not very comprehensive. All the quantitative results are done on ChatGPT. Not sure whether it is also applicable to other LLMs. Furthermore, in my opinion, ablation study in Table 2 is more important than Table 1 because Table 2 supports the main claim of the paper (retrieval and asking why are effective) and the methods in Table 1 are not very comparable. However, the author does not conduct human evaluation in Table 2.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a story generation method (see its Figure 1). The method generates the stories uses the following steps.\n1. classify the attributes of some human written stories using LLM\n2. given a target condition, select the human-written stories with the most similar attributes\n3. generate an initial story using LLM\n4 and 5. keep asking LLM why to generate trees containing some background information or evidence\n6. use LLM to select the most relevant information paths from the trees.\n7 and 8. revise the initial story using LLM\n\nThe experimental results in Table 2 seem to show that the retrieval step (1&2) slightly improves coherence and relevancy, the selection and rewriting (4-8) improves coherence, likability, complexity, creativity, while decreasing the relevancy.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "This work is remotely related\nhttps://arxiv.org/pdf/2212.10012.pdf"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In your tables, show length of the stories\n2. Describe your baseline in more details. Where does the human baseline come from? The stories written in the writingprompt dataset does not know your target conditions, right? What prompts does ICL use?\n3. In appendix tables, show the retrieved stories. Is the initial story similar to the retrieved stories?\n4. A minor suggestion, put your math notations to your Figure 1."}}]

Author rebuttal:

Meta-review: This paper proposes a retrieval-augmented strategy to enhance story generation. In short, based on some metadata (e.g., plot, genre, mood), similar plots from the IMDb dataset are retrieved, and then relevant info from these plots is then used to revise an initially-generated story. All reviewers praise the idea's creativity and the evaluation conducted by experts. However, there are concerns about the method's novelty (there is perhaps too much complexity in the prompting strategy) and performance (improvements directly attributed from the paper's main ideas are small). The authors provided many new experiments and details in their rebuttal / discussion which assuaged other complaints about generalizability, which they are to be commended for, and I'd encourage the authors to include human evaluations in the next version of the paper for the new configurations they tried in their rebuttal. Overall, definitely a good inclusion to Findings!
Decision: Accept-Findings