STORYWARS: A Dataset and Instruction Tuning Baselines for Collaborative Story Understanding and Generation
Year: 2023

Abstract:
Collaborative stories, which are texts created through the collaborative efforts of multiple authors with different writing styles and intentions, pose unique challenges for NLP models. Understanding and generating such stories remains an underexplored area due to the lack of open-domain corpora. To address this, we introduce STORYWARS, a new dataset of over 40,000 collaborative stories written by 9,400 different authors from an online platform. We design 12 task types, comprising 7 understanding and 5 generation task types, on STORYWARS, deriving 101 diverse story-related tasks in total as a multi-task benchmark covering all fully-supervised, few-shot, and zero-shot scenarios. Furthermore, we present our instructiontuned model, INSTRUCTSTORY, for the story tasks showing that instruction tuning, in addition to achieving superior results in zero-shot and few-shot scenarios, can also obtain the best performance on the fully-supervised tasks in STORYWARS, establishing strong multi-task benchmark performances on STORYWARS.1

heading: 1 Introduction
text: Storytelling is crucial due to its vital role in human experience, history, and culture dating back to the earliest days of humanity. Humans possess the unique storytelling ability to structure a sequence of events, whether factual, fictional or a mixture of both, and create a coherent narrative that conveys a big picture while also including intricate details. Current story generation systems usually mimic this ability by starting with a plot then crafting the story. This can be done by linearly expanding (Peng et al., 2018, Yao et al., 2019, Martin et al., 2017) or hierarchically developing (Xu et al., 2018, Fan et al., 2018, Fan et al., 2019, Rashkin et al. 2020, Goldfarb-Tarrant et al., 2020) the story based on the given plot. Collaborative storytelling
1We make our data, code, and models publicly available at https://github.com/ylndu/storywars
WARS dataset. In each turn, the author leaves a "floor" for the next author to continue collaboratively .
is distinctly challenging because there is no predetermined plot or story outline of events. Instead, collaborative stories are created through the collective efforts of multiple authors. Each author contributes a section sequentially, while also attempting to express their own personal intentions within the context of the jointly crafted and jointly owned story. It is a more challenging problem as it requires not only the ability to generate text, but also the capability to understand the previous context and contributions written by other authors.
3044
Large Language Models (LLMs) (Devlin et al. 2019, Liu et al., 2019, Yang et al. 2019, Raffel et al. 2019, Brown et al. 2020, Zhang et al. 2022, Chowdhery et al. 2022, Touvron et al. 2023) have demonstrated exceptional performance on various understanding and generation benchmarks, indicating their potential in addressing natural language processing (NLP) challenges related to collaborative storytelling. This prompts an intriguing question within the research community: How could LLMs synergize both their understanding and generation capabilities via multitask learning to address the challenges of collaborative storytelling?
We present STORYWARS, a dataset of over 40,000 stories gathered from an online collaborative storytelling platform2. Figure 1 shows an example story in the STORYWARS dataset. Each story contains rich information including its title, genres given by the initial author, chapters written by different authors, and human ratings including stars and likes. Each chapter was written by exactly one author and the previous author might leave a collaborative floor (Coates, 1997) for the next author to continue. Therefore, for a model to generate a continuing chapter, it needs to understand the preceding context, including the title, genres, and the writing styles and intentions of previous authors conveyed in the collaborative floor.
Due to the multitask nature of collaborative storytelling and the rich information of the STORYWARS, we design 12 task types, including both understanding and generation task types, as a multitask benchmark for an initial probe of collaborative storytelling. We follow the task definition from FLAN (Wei et al., 2021), where each task type contains multiple tasks. In the end, our benchmark contains 101 tasks in total, split such that it covers all fully-supervised, few-shot, and zeroshot learning application scenarios. It is important to note that prevailing multitask NLP benchmarks are either focusing on understanding (e.g. Wang et al., 2018, Wang et al., 2019) or generation (e.g. Gehrmann et al., 2021, Khashabi et al., 2021, Liu et al., 2021) alone, or only a subset of the learning scenarios. To our knowledge, we are the first to propose a story benchmark that contains both understanding and generation in all three scenarios.
Large language models have been shown to not only be fully-supervised, few-shot, and zero-shot
2www.storywars.net Unfortunately, the website has closed down by the time of writing this paper. Some stories could be recovered from https://archive.md/sAOOq
learners but also multitask ones. Instruction Tuning (Wei et al., 2021, Sanh et al., 2022, Chung et al., 2022) has been the state-of-the-art approach for zero-shot and few-shot scenarios. However, it has not yet been applied in the fully-supervised setting. We evaluated Instruction Tuning on the benchmark and we found that in addition to achieving state-ofthe-art results in zero-shot and few-shot scenarios, when combined with single-task fine-tuning, Instruction Tuning can surpass single-task fine-tuning alone, resulting in a consistent performance boost of 1.53 points on average for all tasks.
Our contributions are as follows: • We introduce a novel collaborative story
dataset STORYWARS that comprises 40k stories written by 9.4k different authors, with rich information such as genres and human ratings, to promote research in the field of collaborative storytelling.
• We propose a new benchmark based on STORYWARS that consists of 7 understanding and 5 generation task types, totaling in 101 tasks for testing the fundamental abilities of LLMs to model collaborative stories. The benchmark covers the fully-supervised, few-shot, and zero-shot scenarios.
• We present INSTRUCTSTORY, a instructiontuned model that demonstrates strong performance on the STORYWARS benchmark in all three learning scenarios. In addition, we show for the first time that we could extend Instruction Tuning with a single-task finetuning stage to achieve superior performance and obtain robust performance boost.

heading: 2 Related Work

heading: 2.1 Story Datasets
text: The most popular story datasets that have been widely used by many story generation systems in the past are ROCStories (Mostafazadeh et al., 2016) and WritingPrompts (Fan et al., 2018). ROCStories comprises five-sentence commonsense short stories, and WritingPrompts includes 300k opendomain prompt-story pairs, neither of which are collaboratively written. On the other hand, Storium (Akoury et al., 2020) and roleplayerguild (Louis and Sutton, 2018), are collaborative and written by multiple authors in turns, but in a game setting. The key distinction of our STORYWARS dataset is that the stories are both collaborative and open-domain. For a comparison of these datasets, refer to Table 1.

heading: 2.2 Multitask NLP Benchmarks
text: Existing multitask NLP benchmarks tends to focus on evaluating either understanding (Wang et al., 2018, Wang et al., 2019) or generation (Gehrmann et al., 2021, Khashabi et al., 2021, Liu et al., 2021) capabilities of NLP models. There are taskspecific benchmarks that address both, such as those for dialog (Mehri et al., 2020) and code (Lu et al., 2021). For the task of storytelling, the LOT benchmark (Guan et al., 2022) focuses on both aspects but is limited to Chinese and has fewer tasks than our proposed STORYWARS dataset. BIGbench (Srivastava et al., 2022), which includes 204 tasks for understanding and generation, only tests zero-shot and few-shot abilities without finetuning. STORYWARS provides a benchmark for story understanding and generation with 101 tasks spanning all zero-shot, few-shot, and full-supervised scenarios for various applications.

heading: 2.3 Multitask NLP and Instruction Tuning
text: Current multitask LLMs mainly follow two approaches. The first approach involves finetuning, such as with ExT5 (Aribandi et al., 2022) and Muppet (Aghajanyan et al., 2021), where the model is made more generalized through multitask finetuning and then fine-tuned again on downstream tasks. The second approach focuses solely on zero-shot and few-shot performance, with the goal of bridging the gap between finetuning and these performance levels, as seen in FLAN (Wei et al., 2021), T0(Sanh et al., 2022), FLAN-T5 (Chung et al., 2022), and ZeroPrompt (Xu et al., 2022). These models often utilize Instruction Tuning or similar frameworks. In this paper, we extend Instruction Tuning’s capabilities to achieve superior performance in the full-supervised scenario as well.

heading: 3 Methodology

heading: 3.1 The STORYWARS Dataset
text: We obtained the STORYWARS dataset from storywars.net, an online collaborative storytelling platform where users can pitch ideas and create stories. However, once an initial chapter is published, the story becomes part of the Story Wars community and can be contributed to by other users. For a continuing chapter to be officially recognized, it must be voted in by other users, resulting in a high quality of stories on the platform.
We scraped and parsed the stories on Story Wars, ending up in obtaining 76k stories. We then used FastText (Bojanowski et al., 2017) language identification to filter for English stories and further cleaned the dataset by removing noisy stories based on GPT-2 perplexity (Radford et al., 2019). We also removed stories that are shorter than 30 words or stories with chapters that are shorter than 10 words. To further ensure the quality of the dataset, we also remove stories that have very low human ratings, such as likes and stars.
In consideration of ethical issues, we employed OpenAI Content Moderation APIs3 and the Detoxify4 toxicity classifier to identify and remove potentially harmful content, such as toxicity, obscenity/sexual content, threats, insults, identity hate, and self-harm posts from the dataset. Furthermore, to safeguard user privacy, we replaced all URLs, email addresses, and phone numbers with special tokens <URL>, <EMAIL>, and <PHONE>.
After thorough data cleaning, we obtained a final dataset of 40,135 stories written by 9,494 authors. Due to the fact that the long tail of genres is very noisy, we made the simplifying assumption that each story contains a single dominant genre, if any. Each story in the dataset was structured with sev-
3https://beta.openai.com/docs/api-reference/moderations 4https://github.com/unitaryai/detoxify
eral key elements, including a title, a genre (which could be empty), the numbers of likes and stars received, the authors and the corresponding chapters.
We denote an arbitrary story in the dataset as s ∈ S, where S = {(p, (ci, ai)ti=0, g, rl, rs)}. That is, each story si is denoted by a 5-tuple of a title p, chapter-author pairs (ci, ai) of t turns, a genre g, a likes rating rl, and a stars rating rs.

heading: 3.2 The Multitask Benchmark

heading: 3.2.1 Story Understanding Tasks
text: Genre Classification Understanding the genre of a story is essential for collaborative storytelling models to comprehend the context. The genre classification task involves identifying the genre of a story. This task can be formulated as a binary text classification problem, where given a story, the task is to predict whether it belongs to a specific genre g. This can be represented as g = f(c1, c2, ..., ct). Authorship Attribution Identifying the author of a text is a crucial step in understanding the writing style of an individual. Authorship attribution, traditionally, is the task of determining the author of a given text. In this paper, we formulate the task of authorship attribution as identifying the author of a specific chapter, represented as a = f(c). Authorship Verification Authorship Verification, in contrast to author attribution, is the task of determining whether two texts have been written by the same author by comparing their writing styles. The task is represented as y = f(ci, cj), where y is a binary variable. Connectivity Inference Understanding the chapter shifts in long-range stories can be a beneficial ability for collaborative storytelling. Following Sun et al. (2022), we also include the connectivity inference task, where the goal is to determine whether two given chapters are consecutive in a story. The task is represented as y = f(cn, cm). Temporal Inference Inspired from the Connectivity Inference task, we also aim to evaluate a model’s ability to understand the temporal relationships between chapters in a story. The Temporal Inference task involves determining whether two chapters in the same story are in the correct chronological order. For example, (ci, ci+1) and (ci, ci+5) would be considered positive instances, while (ci+5, ci) would not. The task is represented as y = f(cn, cm), where y is a binary variable. Story Scoring Understanding human ratings of a story is crucial for generating texts that align with
human preferences. Many dialog-related applications rely on human labelers to rate texts based on different criteria, e.g. LAMDA (Thoppilan et al., 2022). Since STORYWARS contains human ratings in the form of likes and stars, we propose to include a regression task for story scoring as a task type. We follow Raffel et al. (2019) and normalize the story ratings to a range from 0-10, with rounded scores to the nearest increment of 0.1, and convert the float to string. Given a rating score, such as rl, the task is represented as rl = f(c1, c2, ..., ct). Story Segmentation Although stories are already divided into chapters, it is still possible to evaluate models’ ability to identify chapter boundaries within a story, where one chapter concludes and another begins, in order to encourage the model to capture discourse-level information. We design the task of story segmentation as c1, b1, c2, b2, ..., bt−1, ct = f(s), where bi is the boundary between two chapters.

heading: 3.2.2 Story Generation Tasks
text: Next Chapter Generation The next chapter generation problem is defined as an generation task that takes previous chapters and genre information as input, and then generates the subsequent chapter. This is represented as ck+1 = f(c1, c2, ..., ck, g). Conditional Story Generation The conditional story generation problem is defined as an generation task that also takes previous chapters and genre information as input, but then generates the entire continuation of the story until the conclusion instead. It further evaluates an NLP model’s capability to plan and organize the story. This is represented as ck+1, ck+2, ..., ct = f(c1, c2, ..., ck, g). Chapter Infilling In line with Ippolito et al. (2019), the chapter infilling task evaluates an NLP model’s ability to generate an intermediate chapter given the context of a preceding and subsequent chapter. This is represented as ck = f(ck−1, ck+1). Global Infilling Building on the chapter infilling task, the global infilling problem considers more extensive context information, including both preceding and subsequent chapters. This is represented as ck = f(c1, c2, ..., ck−1, ck+1, ..., ct). Temporal Ordering Following Lin et al. (2021), we also include a task that unscrambles chapter sequences based on temporal information, except that we simplify the problem by eliminating the requirement for the NLP model to infill masked chapters. This is represented as c1, c2, ..., ct = f(permute(c1, c2, ..., ct)).

heading: 3.2.3 The Benchmark
text: Benchmark task statistics The 12 task types translate into 101 tasks based on STORYWARS, with 96 understanding tasks and 5 generation tasks. It is worth noting that the majority of the understanding tasks are genre classification tasks (60) and author attribution tasks (30). Out of the 60 genre classification tasks, we split them into 27 fullysupervised, 10 few-shot, and 23 zero-shot datasets, according to the genre frequency so that the split closely aligns with realistic data distribution. For the fully-supervised and few-shot tasks, we divided the data into training, dev, and test sets. For the zero-shot tasks, we used all the data as a test set by sampling. The remaining task types were used for fully-supervised scenarios. It is important to mention that all of the data in the fully-supervised, few-shot, and zero-shot scenarios are disjoint to prevent data leakage. The overall task data statistics can be found in the Table 2. Evaluation metrics For the genre classification, author attribution, author verification, temporal inference, and connectivity inference tasks, we use F-1 score as the evaluation metric, due to the imbalance nature of the task data. For the story scoring tasks, in line with Raffel et al. (2019) for regression tasks, we use Spearman correlation coefficients as the evaluation metric, because it measures monotonic relationships. For the story segmentation task, we use Boundary Similarity (Fournier, 2013) as the evaluation metric. For the generation tasks, following the suggestions introduced in Chhun et al. (2022), Qin et al. (2019), and Gangal et al. (2021),
we use BERTScore (Zhang* et al., 2020) as the evaluation metric, as it has been shown by Chhun et al. (2022) to have better correlation with human evaluation at both the story-level and system-level for story generation systems than other automatic metrics including frequently-used BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). Also, Gangal et al. (2021) points out that in the narrative reordering problem, similar to our temporal ordering task, BERTScore also correlates quite well with human evaluations. We recognize that there is currently no widely accepted or reliable automatic evaluation metric in the field of story generation, and the use of automatic evaluation in this field is often criticized. However, for the purpose of fast and fair comparison, we chose to follow previous work and use the current best available metric, even though we acknowledge that it may not be perfect.
For evaluating the model performance, we calculate the macro-average of the performance on all tasks within each task type, this allows us to compare models across different task types. The metrics for understanding, generation, and overall performance are determined by the macro-average of the scores across the corresponding task types.

heading: 3.3 The INSTRUCTSTORY Framework
text: The main goal of instruction tuning is to evaluate the performance of unseen tasks in zero-shot and few-shot learning scenarios, and to show that it can improve the gap between zero-shot and fullysupervised learning performances. Additionally, we are interested in how instruction tuning can improve the performance of fully-supervised tasks.
To accomplish our goal, we propose a two-stage training approach called INSTRUCTSTORY. In the first stage, we use instruction tuning as a form of pre-finetuning Aghajanyan et al. (2021). During this stage, we use instructions instead of task prefixes proposed in Muppet Aghajanyan et al. (2021) to enhance the model’s ability to generalize to new instructions. In the second stage, after instruction tuning with the fully-supervised task mix, we use single-task finetuning to continually train the model for each fully-supervised task. We use T5-largelm-adapt (770m) as the base model for instruction tuning INSTRUCTSTORY and all of the training tasks are from the STORYWARS fully-supervised training split. Figure 2 illustrates the overall INSTRUCTSTORY framework. The instructions we used are included in Appendix A.1.

heading: 4 Experimental Results

heading: 4.1 Baselines
text: We include several strong baseline models with a comparable number of parameters. For understanding tasks, we include BERT-large (345m), RoBERTa-large (354m), and DeBERTav2-xlarge (900m) as baselines. For generation tasks, we include GPT2-medium (345m), GPT2large (774m), and OPT-350m as baselines. These models all have comparable or near-comparable numbers of parameters. To demonstrate the effectiveness of our method, we also include T5-largelm-adapt (770m) as a baseline model in the overall comparison. In addition, for the few-shot and zero-shot scenarios, we include the state-of-the-art instruction tuning model FLAN-T5-large (Chung et al., 2022) as a comparison baseline.

heading: 4.2 Experimental Setup
text: To train INSTRUCTSTORY, we use instruction tuning on T5-large-lm-adapt for 5 epochs using the fully-supervised task mix. We use the Adam optimizer with a learning rate of 5e-5 and a batch size of 64. At each gradient step, examples are randomly sampled from all tasks. The maximum input and target sequence lengths are set to 1024, and any longer inputs or targets will be truncated.
For the fully-supervised learning scenario, both INSTRUCTSTORY and all the baselines are finetuned on a single task for 10 epochs for each
task. The best performing checkpoint for each task is chosen based on the performance on its dev set. Note that BERT-large, RoBERTa-Large, and DeBERTa-v2-xlarge all have a maximum sequence length of 512, while GPT2-medium and GPT2Large have a maximum sequence length of 1024 and OPT-350m has a maximum sequence length of 2048. We truncate the data instances based on the respective max sequence lengths of the models.
For the few-shot learning scenario, we finetune all the models and use early stopping based on the dev set performance. Also, we are unable to use in-context learning demonstrations like in Chung et al. (2022), as the story lengths are often too long to fit within the max input sequence length.
For the zero-shot scenarios, we only compare INSTRUCTSTORY with T5 and FLAN-T5, as the other baseline models have poor zero-shot performance.
More information about training specifics and hyperparamters can be seen in Appendix A.2.

heading: 4.3 Main Results
text: Fully-supervised Results The fully-supervised results are presented in Table 3. We show that INSTRUCTSTORY can achieve a 1.53 point increase in the overall average score compared to the singletask finetuned T5 baseline. Additionally, for understanding tasks, INSTRUCTSTORY outperforms T5 by 2.06 points. When compared to other strong understanding baselines including BERT, RoBERTa, and DeBERTa, INSTRUCTSTORY also achieves
the best results. For generation tasks, INSTRUCTSTORY outperforms T5 by 0.77 points. It also achieves favorable performance when compared to other strong generation baselines such as GPT2medium and GPT2-large, although performing a little bit worse than OPT-350m. We hypothesize that the difference in performance between OPT-350m and INSTRUCTSTORY is due to the base model, specifically the size of the pretraining corpus (35B tokens vs 180B tokens).(Zhang et al., 2022)
Few-shot Results The few-shot results are shown in Table 4. For the few-shot scenario, INSTRUCTSTORY achieves the highest score of 61.44, followed by FLAN-T5 which achieved the second highest score of 59.45, outperforming all the T5, BERT, RoBERTa, and DeBERTa baselines. This demonstrates that even when instruction-tuned on a different dataset distribution, FLAN-T5 can still achieve competitive results when further fine-tuned for few-shot tasks.
Zero-shot Results We can see the zero-shot results in Table 5. In the zero-shot scenario, we compare INSTRUCTSTORY with T5 and FLAN-T5, and we can see that INSTRUCTSTORY has a significant improvement in zero-shot performance, a 28.08 increase from T5 and a 12.21 increase from FLANT5. This is expected because our instruction tuning training task mix has a similar, though unseen, data distribution to the zero-shot test sets.

heading: 4.4 Discussions
text: INSTRUCTSTORY brings a robust improvement in performance. By comparing T5 and INSTRUCTSTORY in Table 3, we see that INSTRUCTSTORY scores higher than T5 in every task type. The performance gain is consistent across all task types. Even on the task level, INSTRUCTSTORY achieves better results than T5 in 24 out of 27 genre classification tasks and 23 out of 30 authorship attribution tasks. This indicates that in fully-supervised scenario, one can confidently use the power of instruction tuning to improve performance.
Ablation: Instruction tuning with both understanding and generation tasks is more effective than instruction tuning with only understanding tasks or only generation tasks. Table 6 illustrates this by comparing the fully-supervised, fewshot, and zero-shot genre classification scores of INSTRUCTSTORY, its variants ISU, and ISG, where ISU and ISG are instruction tuned with understanding tasks mix and generation tasks mix, separately. From the table, we can see that IS > ISU > ISG > T5 across all zero-shot, few-shot, and fully-supervised learning scenarios, which indicates that instruction tuning with a mix of understanding and generation tasks is better than instruction tuning with only one of them.

heading: 5 Conclusion
text: We introduced a novel dataset STORYWARS and a multitask benchmark for collaborative story understanding and generation. Our proposed INSTRUCTSTORY model, which leverages instruction tuning as multitask pre-finetuning, outperformed both its single-task finetuning baseline and other strong models on the STORYWARS benchmark and established strong performance in all zero-shot, fewshot, and fully-supervised learning scenarios. We hope that our newly proposed STORYWARS dataset will serve as a catalyst for research in the field of collaborative storytelling and inspire further advancements in this area.

heading: 6 Limiations
text: Our proposed INSTRUCTSTORY method utilizes both single-task finetuning and instruction tuning to achieve good results. However, when finetuned on a new task, the model may suffer from the problem of catastrophic forgetting and lose its multitasking generalization abilities. Recent research by Scialom et al. (2022) has investigated this issue in instruction-tuned models and proposed a technique called Rehearsal to mitigate it. However, this work primarily focuses on zero-shot scenarios and does not address fully-supervised learning. It would be of interest to explore whether it is possible to finetune on a single task while preserving the model’s multitasking abilities and generalization capabilities. We leave this question as an area for future research.
Additionally, it is important to note that our approach of single-task finetuning for each downstream task results in multiple models being required to be served simultaneously, which can lead to increased computational costs. In practice, this is a trade-off that must be carefully considered, as it requires balancing performance requirements with the resources available. It can be an important factor to consider when implementing this approach in real-world settings.
In the end, a proper and thorough evaluation of collaborative story generation remains an on-going research. While automatic evaluation metrics such as BERTScore has the best human correlations at story-level and system-level per Chhun et al. (2022), it may not be comprehensive enough in evaluating the highly creative output of collaborative story generation. There is a need for more nuanced and sophisticated metrics that can capture the complexity and diversity of collaborative stories. Therefore, the development and validation of appropriate evaluation methods is crucial for progress in this field.

heading: 7 Ethical Considerations
text: In Section 3.1, we have discussed our procedures to identify and remove potential harmful content and user privacy information. However, it is important to also consider the broader ethical implications of using AI in collaborative storytelling. These include issues such as ensuring fair and unbiased representation, protecting data privacy, and preventing the use of AI-generated content for harmful purposes. For example, AI-generated stories or
characters may perpetuate stereotypes or reinforce societal biases if they are trained on biased data. Therefore, it is crucial to consider and address these ethical issues in order to create inclusive and responsible AI-generated stories that do not harm individuals or groups.

heading: A Appendix
text: A.1 Instruction Template examples Please refer to Table 7 for the instruction template examples.
A.2 Hypterparameters Please refer to Table 8 for the hyperparameters.
A.3 Full results tables Please refer to Table 9, Table 10, Table 11, and Table 12 for all full results.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation is section 6 after conclusion

heading: 3 A2. Did you discuss any potential risks of your work?
text: under ethical considerations in section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 dataset
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3.1
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 specifies the number of parameters of models.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3.2.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this question as an area for future research. Additionally, it is important to note that our approach of single-task finetuning for each downstream task results in multiple models being required to be served simultaneously, which can lead to increased computational costs. In practice, this is a trade-off that must be carefully considered, as it requires balancing performance requirements with the resources available. It can be an important factor to consider when implementing this approach in real-world settings. In the end, a proper and thorough evaluation of collaborative story generation remains an on-going research. While automatic evaluation metrics such as BERTScore has the best human correlations at story-level and system-level per Chhun et al. (2022), it may not be comprehensive enough in evaluating the highly creative output of collaborative story generation. There is a need for more nuanced and sophisticated metrics that can capture the complexity and diversity of collaborative stories. Therefore, the development and validation of appropriate evaluation methods is crucial for progress in this field