Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
Authors: Lixing Zhu
Runcong Zhao
Lin Gui
Yulan He
Venue: EMNLP 2023 Findings (2023)
Keywords: Narrative Understanding
Reading Comprehension
Summarization
Question Answering
TL;DR: This paper surveys narrative understanding and provides insights into possible new tasks.

Abstract: Narrative understanding involves capturing the author's cognitive processes, providing insights into their knowledge, intentions, beliefs, and desires. Although large language models (LLMs) excel in generating grammatically coherent text, their ability to comprehend the author's thoughts remains uncertain. This limitation hinders the practical applications of narrative understanding. In this paper, we conduct a comprehensive survey of narrative understanding tasks, thoroughly examining their key features, definitions, taxonomy, associated datasets, training objectives, evaluation metrics, and limitations. Furthermore, we explore the potential of expanding the capabilities of modularized LLMs to address novel narrative understanding tasks. By framing narrative understanding as the retrieval of the author's imaginative cues that outline the narrative structure, our study introduces a fresh perspective on enhancing narrative comprehension.

Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
Lixing Zhu
Runcong Zhao Lin Gui
Yulan He
*Department of Informatics, King's College London
'The Alan Turing Institute, UK
[Lixing. Zhu, Runcong Zhao, Lin.Gui, Yulan. Jekcl........ ac. uk
Abstract
narrative understanding and, thereby preventing readers from fully understanding the true intention of the authors.
Narrative understanding involves capturing the author's cognitive processes, providing insights into their knowledge, intentions, beliefs, and de- sires. Although large language models (LLMs) excel in generating grammatically coherent text, their ability to comprehend the author's thoughts remains uncertain. This limitation hinders the practical applications of narra- tive understanding. In this paper, we con- duct comprehensive survey of narrative un- derstanding tasks, thoroughly examining their key features, definitions, taxonomy, associated datasets, training objectives, evaluation met- rics, and limitations. Furthermore, we explore the potential of expanding the capabilities of modularized LLMs to address novel narrative understanding tasks. By framing narrative un- derstanding as the retrieval of the author's imag- inative cues that outline the narrative structure, our study introduces fresh perspective on en- hancing narrative comprehension.
Narrative understanding has been explored through various approaches that aim to recognize thoughts within narratives (Mostafazadeh et al., 2020; Kar al., 2020; Lee et al., 2021; Sang et al., 2022). Still, these approaches often frag- mented, focusing on diverse tasks scattered across multiple datasets, obfuscating the fundamental el- ements (e.g.. the characters, the events and their relationships) of the narrative structure (Ouyang and McKeown, 2014, 2015; Cutting, 2016). To ad- dress this gap. in this paper. we lay the foundation and provide comprehensive synthesis of the afore- mentioned narrative understanding tasks. We start with the key features of this genre, followed by formal definition of narrative understanding, We then present a laxonomy of narrative understanding tasks and their associated datasets, exploring how these datasets are constructed, the training objec- tives, and the evaluation metrics employed. We proceed to investigate the limitations of existing approaches and provide insights into new frontiers that can be explored by leveraging current modular- ized LLMs (c.g., GPT with RLHF (Ouyang et al., 2022)), with particular focus potential new tasks.
1 Introduction
When reading a narrative, it is common for read- ers to analyze the author's cognitive processes, in- cluding their knowledge, intentions, belicfs, and desires (Castricato et al., 2021; Kosinski, 2023). In general, narrative is a medium for personal experi- ences (Somasundaran et al., 2018). Although Large Language Models (LLMs) have the capability to generate grammatically coherent texts, their ability accurately capture the author's 's thoughts, such the underlying skeletons outline prompts de- vised by the authors themselves (Mahowald al., 2023), remains questionable. This is supported by cogmitive research that bilingual individuals tend to convey more precise thoughts compared to mono- lingual English speakers (Chee, 2006). The po- tential deficiency in tracing thoughts within nar- ratives would hinder the practical application of
To sum up, our work firstly aligns disparate tasks with the LLM paradigm, and categorizes them based on the choices of context and input- output format. Then it catalogues datasets based on the established taxonomy. Subsequently. it intro- duces Bayesian prompt selection as alternative approach define the task of narrative understand- ing. Finally, it outlines open research directions.
Definition of Narrative Understanding
Narrative texts possess distinct characteristics, which are different from other forms of discourse. Elements such as point view, salient characters,
"Equal contribution. Corresponding author.

---

and events, which are associated or arranged in a particular order (Chambers and Jurafsky, 2008;
Narrative Consistency Check involves deter- mining whether an assertion aligns with the nar- rative or contradicts it. This task encompasses various scopes, ranging from the entire narrative structure (Ouyang and McKeown, 2014) to dis- course structure (Mihaylov and Frank, 2019) and ultimately, the constituents of the narrative, such as agents and events (Piper et al., 2021; Wang et al., 2021).
Ouyang and McKeown, 2015; Piper et al., 2021), giving rise to a cohesive story synopsis known as the plots (Hühn et al., 2014). Its scope spans across various genres, including novels. fiction, films, theatre, and more, within the domain of lit- erary theory (Genette, 1988). Although it is un- necessary to endorse particular narrative theory, some elements are commonly encountered in com- prehension. For example, readers have to under- stand the causation or relationship that goes be- yond a timeline and delve into the relationships between the characters (Worth, 2004). From a model-theoretic perspective, narrative understand- ing can be described as process through which the audience perceives the narrator's constructed plot thoughts (Czarniawska, 2004; Castricato et al., 1021).
For example, Granroth- SrradrothWiildnng and Clark (2016) designed a Multiple Choice Narrative Cloze (MCNC) prediction task, where stories are struc- tured as a sequence of events. Each event is rep- resented by a 3-tuple, which comprises the verb lemma, the grammatical relation, and the associ- ated entity. They aimed to predict the subsequent event from given set of options, framed in the context of story cloze. Furthermore, Chaturvedi et al. (2017) extended this prediction task to en- compass the prediction of a story ending based on its existing content. Similarly, the ROC story cloze task (Mostafazadeh et al., 2016), addressed by Cai et al. (2017), involves choosing the most plausible ending. There are various approaches developed for story ending prediction, such as the incorporation of commonsense knowledge (Li et al., 2018b), utilization of skip-thought embed- dings (Srinivasan al., 2018), entity-driven recur- rent networks (Henaff' et al., 2017: Liu et al., 2018). scene structure (Tian et al., 2020), centrality or salience of events (Zhang et al., 2021), and con- textualized narrative event representation (Wilner et al., 2021), respectively. Simple and well- established, the Story Cloze Test does not cover the core aspects of narrative structure, though. Roem- mele and Gordon (2018a) introduced an advance- ments in this task by predicting causally related events in stories using the Choice of Plausible Al- ternatives (COPA) (Roemmele al., 201 dataset. Each instance in the COPA dataset contains three sentences: a Premise, Alternative and Alterna- tive 2, with the Premise describing an event and the Alternatives proposing the plausible cause or effect of the event. Building upon this, Qin et al. (2019) aligned the ROC story cloze and COPA dataset with HellaSwag (Zellers et al., 2019) and introduced the counterfactual narrative reasoning. This task involves re-writing the story to restore narrative consistency. Their proposed Traeel dataset features 29, 849 counterfactual revisions to initial story endings. Ippolito et al. (2020) further
To this end, we define narrative understanding as the process of reconstructing the writer's creative prompts that sketch the narrative structure (Ouyang and McKeown, 2014; Fan et al., 2019). In line with Brown et al. (2020), we adopt the practice of using the descriptions of NLP tasks as context to accommodate different paradigms. Additionally, we employ the LLaMA (Touvron et al., 2023) tax- onomy to dichotomize this data-oriented task as either multiple-choice or free-form text comple- tion. Let {In, Wn denote the dataset where 21:N are narratives, and y1:N the annotated sketches, narrative understanding aims to predict Y given X by optimizing (V(ii. context). Existing literature can be roughly categorized based on the format of Yn and how context is described. To align with the classical NLP tax- onomy, we specify context as a single prompt from either Reading Comprehension (Section 2.1), Summarisation (Section 2.2), or Question Answering (Section 2.3). In the rest of this sec- tion, the context will be further elaborated and specialized in more narrowly-defined tasks to re- fine the taxonomy, resulting in the reformatting of yn. accordingly.
2.1 Narrative Reading Comprehension
In machine reading comprehension, the context prompt is instantiated as a single prompt of "select- ing which option consistent with the story", and yo. is structured as categorical label(s) that corre- spond to the available options.

---

Story Close
ROC (Mostafazadeh al. 2016)
Evens-Centric
Plausible Alter-
COPA)
COIA (Roemmele et al., 2011)
Counterfactual
Qin al. (2019)
Consistency Checking
Coreference ldentinicttion
Patil et (201 Ja-
fain Finlayson (2019)
Chancler-Cotte
Porsonality
(20223)
Other blements
Winzare et al. (2019)
Multiple Elements
Kar (2020)); Mostafazadel al. (20200
Reading Com-
prehensioo
Component Sogmentation
William Croh (2017); Pegulampidi et (2019)
Analysas
Event Relation Extraction
Kolomiyess et al. (2012); Revi al. (2021)
Abstractive
Ouyang al. (2017)
CoepssL
Extractive
Lehr al. (2013): Papalampidi al. (2020)
Narrative
Narraiiv Sum- marisation
Event
Zhao al. (2022a)
Object-Speeific
Character
Zhang
Brahman (3021)
Multiple Elements
(2022)
NurariveQA (Kočisky al.. 2018); BookQA (An- pelidis et al., 20199; Mon et (20200) TallMe- Why (Lal et al., 2021); HowQA (Cai et al., 2022)
Geseration Basse
Answer Generation
Nanative Answoring
Retrieval Based
Lahutov al. (2011); BookQA (An- gelidis al., 2019); Mou et al. (2020)
Question Cieneration
Namative(A (Kolisky
FaarytaeeAA 2022)
Figure I: Typology of Narrative Understanding. Some literature sources repeated since they contain both types of datasets or onput-oulput schemes. schemee..
expanded the scope of the story cloze task to the
nition (NER) and dependency parsing. In contrast, recent approach introduced the TVShowGuess dataset, which simplified speaker guessing multiple-choice selection (Sang et al., 2022). How- ever, difficult for some narrative-specific NLP tasks, e.g., NER, to determine whether labelling 'a talking cup' as a 'person' is appropriate. To miti- gate this, some approaches take character-centric perspective. Brahman et al. (2021) introduced two new tasks: Character Identification, which assesses the alignment between character and unidenti- fied description, and Character Description Gener- ation, which emphasizes generating summaries for individual characters. Other work targeted person- ality prediction (Yu et al., 2023b), or used the off- the-shelf LLM to perform role extraction (Stamm- bach et al., 2022). To delve into the psychology of story characters and understand the causal connec- tions between story events and the mental states of characters, Rashkin et al. (2018) introduced dataset, StoryCommonsense, which contains the annotations of motivations and emotional reactions of story characters.
entire narrative and proposed sentence-level lan- guage model for consecutive multiple-choice pre- diction among candidate sentences on the Toronto Book Corpus (Zhu et al., 2015).
In addition to event prediction, Wanzare et al. (2019) introduced the concept of prototypical events, referred to as scenarios, to incorporate essential commonsense knowledge in narratives. They also introduced benchmark dataset for sce- nario detection. Building upon their work, Situ- ated Commonsense Reasoning (Ashida and Sug- awara, 2022) aimed deriving possible scenarios following a given story. Other attributes, such as a movie's success and event salience, have been explored in studies (Kim et al., 2019; Otake et al., 2020; Wilmot and Keller, 2021).
In line with the event consistency, characters, also known as participants, play crucial role in linking narratives (Patil al., 2018: 8; Jahan and Finlayson, 2019). Patil al. (2018) employed Markov Logic Network to encode linguistic knowl- edge for the identification of aliases of participants in narrative. They defined participants as enti- ties and framed the task as Named Entity Recog-
While much existing work on narrative under- standing focuses on specific aspects, Kar et al.

---

(2020) considered the multifaceted features of nar- ratives and created multi-label dataset from both plot synopses and movie reviews. Chaturvedi et al. (2018) identified narrative similarity terms of plot events, and resemblances between characters and their social relationships. Mostafazadeh et al. (2020) built the General Lized and COntextualized Story Explanations (GLUCOSE) dataset from chil- dren's stories and focused on explaining implicit causes and effects within narratives. It includes events, locations, possessions, and other attributes of the curated claims within the stories.
or absent. To address this issue, William Croft (2017) proposed to decompose the storyline by con- sidering individual temporal subevents for each par- ticipant in a clausal event, which interact causally. Bamman et al. (2020) focused on resolving coref- erence in English fiction and presented the LitBank to resolve the long-distance within-documes men- tions. Building upon this work, Yu et al. (2023a) released corpus of fiction and Wikipedia text to facilitate anaphoric reference discovery. Yan et al. (2019) introduced a more complex structure called Functional Schema, which utilizes language mod- els, to reflect how storytelling patterns make up the narrative. Mikhalkova et al. (2020) introduces the Text World Theory (Werth, 1999; Wang et al., 2016) to regulate the structured annotations of nar- ratives. This annotation scheme, profiling the world in narrative, is expanded in (Levi et al., 2022) by adding new narrative elements. Situated reason- ing datasets, such Moral Stories (Emelin et al., 2021), 1), target the branching developments narra- tive plots, specifically focusing on if-else scenarios. Tools have also been created for annotating the se- mantic relations among the text segments (Raring et al., 2022).
Structural Analysis of Events: Plot and Story- line Extraction The narrative structure encom- passes sequence of events that shape story and define the roles of its characters (Hearst, 1997; Cut- ting, 20. Unlike narrative consistency checking, which focuses on elementary consistency, this task involves two key objectives. First, it aims to extract a clear and coherent timeline that underlies the nar- rative's 's progression. Second, it aims to sequence the relationship of key factors to construct the plot of the narrative (Kolomiyets et al., 2012).
In the first line of approaches, Ouyang and McK- cown (2015) considered significant shifts, referred to as Turning Points, in a narrative. These turning points represent the reportable events in the story. Li et al. (2018a) divided typical story into five parts: Orientation, Complicating Actions, Most Re- portable Event, Resolution and Aftermath, which are annotated in chronological order, capturing the temporal progression of the story. The temporal re- lationships within the narrative can be extracted and structured into a database of temporal events (Yao and Huang, 2018). a similar vein, Papalampidi et al. (2019) strived to identify turning points by segmenting screenplays into thematic units, such as setup and complications. Anantharama et al. (2022) developed pipeline approach that involves event triplet extraction and clustering to reconstruct time series of narrative clusters based on identified topics.
In addition to the aforementioned approaches for plot storyline construction, several joint mod- els have been developed to simultaneously uncover key elements and predict their connections. A no- table work is PlotMachines (Rashkin et al., 2020) which involves an outline extraction method for automatic constructing the outline-story dataset. In another study, Lee et al. (2021) employed Graph Convolutional Networks (GCN) to predict entities and links on the StoryCommonsense and DesireDB datasets (Rahimtoroghi et al., 2017).
2.2 Narrative Summarisation
Narrative summarisation, often referred as Story Retelling (Lchr et al., 2013), can be specified by restricting On to a paraphrase that captures the essense of the original literature. Similar to other summarisation tasks, narrative summarisation can be extractive or abstractive, depending on whether the paraphrase is text snippets directly extracted from the story or is generated from input text.
Based on the types of event relations such as tem- poral, causal, or nested, storylines can be organised timelines (Ansah et al., 2019; yang Hsu et al., 2021), hierarchical trees (Zhu and Oates, 2012), or directed graphs (Norambuena and Mitra, 2021; Yan and Tang, 2023). Current approzches often con- struct storylines using stated timestamps (Ansah et al., 2019; Revi et al., 2021). However, challenges arise in narratives where time details may be vague
Early tasks, such Automated Narrative Retelling Assessment (Lehr al., 2013), primar- ily focused on recapitulation story elements in the form of a tagging task. Subsequently, Narrative Summarisation Corpora (Ouyang et al., 2017; Pa- palampidi et al., 2020) were developed to facilitate

---

more comprehensive understandings of narratives, The former is designed for abstractive summariza- tion, while the latter intended for informative sentence retrieval, taking into account the inherent narrative structure. IDN-Sum (Revi et al., 2020) provides a unique view of summarisation within the context of Interactive Digital Narrative (IDN)
original story.
To effectively handle long context, Tay et al. (2019) introduced a Pointer-Gennratorr framework to sample useful excerpts for training, and chose between extraction and generation for answering. Meanwhile, the BookQA (Angelidis et al., 2019) approach targeted Who questions in the Narra- tiveQA corpus by leveraging BERT locate rel- evant content. Likewise, Mou et al. (2020) pro- posed a two-step approach which consists of evi- dence retrieval to build a collection of paragraphs and a question-answering step to produce an an- swer given the collection. Mou et al. (2021) sur- veyed open-domain QA techniques and provided the Ranker- Reader solution, which improves upon the work of Mou et al. (2020) with newer ranker and reader model.
games. Recent work has proposed benchmarks that require machines to capture specifie narrative ele- ments, e.g.. synoptic character descriptions (Zhang al., 2019b) and story rewriting anchored in six dimensions (Lee et al., 2022). the same vein, Goyal et al. (2022) collected span-level annotations based on discourse structure to evaluate the coher- ence of summaries for long documents. Brahman et al. (2021) presented character- view by introducing two new tasks: Character Identifica- tion and Character Description Generation. More recently, benchmark called NarraSum (Zhao et al., 2022a) has been developed for large-scale narrative summarisation, encompassing salient events and characters, albeit without explicit framing.
Unlike pipeline approaches, Mihaylov and Frank (2019) converted the free-text answers from Narra- tiveQA into text-span answers and used the span answers as labels for training and prediction. Other attempts have been made adapt Narrative for extractive QA. For example, Frermann (2019) modified the dataset into an extractive QA format suitable for passage retrieval and answer span pre- diction. On the other hand, the TellMeWhy (Lal et al., 2021) dataset combined the commonsense knowledge and the characters' motivations in short narratives when designing the questions, present- ing new challenge for answering why-questions in narratives. Kalbaliyev and Sirts (2022) reviewed the WhyQA challenges, and Cai et al. (2022) col- lected how-to questions from Wikihow articles build the HowQA dataset, which serves as a testbed for a Retriever Retriever-Generatorr model.
2.3 Narrative Question Answering
Answering implicit, ambiguous, or causality ques- tions from long narratives with diverse writing styles across different genres requires deep level of understanding (Kalbaliyev and Sirts, 2022). From the task perspective, Narrative QA can be categorized based on either the format of yn. or the task prompt that referred to as context. The format of yn. could be categorical, where an answer is provided as a span specified by starting-ending positions. Alternatively, it can be free-form text that is generated. The context could be specified as answer selection//eenrration or question generation.
Generator
Numerous research works have focused on pro- viding accurate answers curated questions, with a specific focus on event frames (Tozzo et al., 2018), or questions related external common- sense knowledge (Labutov et al., 2018). They all fall into the category of retrieval based QA, where relevant information is selected from narratives. In contrast, the NarrativeQA (Kočiský et al., 2018) dataset took a different approach by instructing the annotators to ask questions and express an- swers in their own words after reading an entire long document, such book movie script. This resulted in high-quality questions designed by human annotators, and human-genente answers.
Generating meaningful questions is impor- tant aspect of human intelligence that holds great educational potential for enhancing children's 's com- prehension and stimulating their interest ( (Yao al., 2022; Zhang et al., 2022a). Early work in this area focused on generating questions of multiple choice word cloze from children's 's Books, targeting named entities, nouns, verbs and prepositions (Hill et al., 2016). Other studies designed commonsense- related questions from narratives in simulated world (Labutov al., 2018). To enhance chil- dren's 's learning experiences, the FairytaleQA (Xu et al., 2022) dataset was created for question gener- ation (QG) tasks, covering seven types of narrative elements or relations. The dataset was used in the work of (Yao et al., 2022), which employed a
The dataset further provided supporting snippets from
abstractive summaries and the

---

pipeline approach comprising beuristics-basee an- swer generation, BART-based question generation, and DistilBERT--bsed candidate ranking. Zhao et al. (2022b) experimented with a question gener- ation model, which incorporated a summarisation module, using the same FairytaleQA dataset. Ad- ditionally, the StoryBuddy (Zhang et al., 2022a) system provided an interactive interface for parents to participate in the process of generating question- answer pairs, serving an educational purpose. major taxonomical concern tasks or datasets employing similar input-output schemes, particu- larly NarrativeQA (Kočiský et al., 2018) with the objective of answering questions relating to narra- tive consistency, is the potential overlap between finely finely-ettilee tasks as defined in our taxonomy. For example, depending on the instructions and prompts given, the scope of QA can encompass more specific tasks, including narrative summariza- tion (Who is Charlie?), and narrative generation (What would happen after the Princess marries the Prince?). Despite some similarities, the majority of instruction templates are indistinguishable (Sanh et al.. 2022). Therefore, they are considered as equivalent tasks.
understanding complex narratives found in books and novels. Due high annotation costs, there is a lack of sufficiently annotated datasets for these types of materials (Zhu et al., 2015; Bandy and Vin- cent, 2.21). Existing work (Frermann et al., 2018; Chaudhury et al., 2020: Kryscinski et al., 2022) employed available summaries and diverse meta- information from books, movies, and TV episodes to generate sizable, high- datasets. Addi- tionally, efficient data collection strategies, such as game design (Yu et al., 2023a), character-actor linking for movies (Zhang et al., 2019b), and lever- aging online reading notes (Yu et al., 2023b), can be explored to facilitate the creation of datasets.
Data Reuse Despite the availability of high- quality annotated datasets for various narrative un- derstanding tasks, there is limited reuse of these datasets the field. Researchers often face chal- lenges in finding suitable data for their specific tasks, which leads to the creatation of their own new and costly datasets. Some chose to build a large, general dataset (Zhu et al., 2015; Mostafazadeh et al., 2020), while others chose to gradually an- notate the same corpus over time (Frermann et al., 2018). Some made use of platforms such Hug- gingFace for data sharing (Huang et al., 2019), or provided dedicated interfaces for public access (Koupace and Wang, 2018). To facilitate data reuse and address the challenges associated with find- ing relevant data, the establishment of an online repository could prove beneficial.
3 Dataset
We have conducted a review of datasets (Table 1-3 in Appendix) that are either designed for, or appli- cable to, tasks related to narrative understanding. There three major concerns:
Data Source Due to copyright restrictions, most datasets focus on public domain works. This limi- tation has rendered valuable resources, such as the Toronto Book Corpus (Zhu et al., 2015), unavail- able. To overcome this challenge, diverse sources have been explored, such plot deseriptions from movies and TV episodes (Frermann al., 2018; Zhao et al., 2022a). However, there remains a need for datasets that cover specialized knowledge bases for specific worldviews narratives, such as magi- cal worlds, post-apocalyptic wastelands, and futur- istic settings. One potential data source that could be utilized for this purpose is TVTropes which provides extensive descriptions of character traits and actions.
Evaluation Methods
For classification tasks, such as multiple-choice QA and next sentence prediction, accuracy, pre- cision, recall, and the Fl score serve as the most suitable evaluation metrics. Here we mainly dis- cuss evaluation metries for free-form or generative tasks, crucial yet challenging for narrative under- standing. Traditional metrics based on N-gram overlap like BLEU (Papineni et al., 2002), ROUGE (Lin and Hovy, 2003), METEOR (Lavie and Agar- wal, 2007), CIDER (Vedantam et al., 2015), WMD (Kusner et al., 2015), and their variants, such as NIST (Doddington, 2002), BLEUS (Lin and Och, 2004), SacreBLEU (Post, 2018) remain popular due their reproducibilityy However, their correla- tion with human judgment is weak for certain tasks (Novikova al., 2017; 7: Gatt and Krahmer, 2018). Alternative content overlap measures have been proposed, such as PEAK (Yang et al., 2016), which
Data Annotation The majority of existing datasets contain short stories, consisting of only a few sentences, which limits their usefulness for
hhttp://tvtropes.org

---

compares weighted Summarization Content Units, and SPICE (Anderson et al., 2016), which eval- uates overlap by parsing candidate and reference texts into scene graphs representing objects and relations. The advent of BERT introduced a new approach to evaluation, based on contextualized embeddings, as seen in metrics such as BERTScore (Zhang et al., 2019a), MoverScore (Zhao et al., 2019), and BARTScore (Yuan et al., 2021). Task- specific evaluations using BERT-based classifiers, such FactCC (Kryscinski et al., 2020) for sum- mary consistency and QAGS (Wang et al., 2020) to determine answers are sourced from the docu- ment, have been explored.
quality w.r.t the development of ideas and ex- emplification. Somasundaran et al. (2018) anno- tates multiple dimensions of narrative quality, such as narrative development and narrative organiza- tion, to combat the scarcity of scored essays. Such narrativity is also evaluated in (Steg et al., 2022) with focus on detecting the cognitive aspects, i.e., suspense, curiosity, and surprise. Other sub-tasks, such as comment generation, are studied in (Lchr et al.. 2013; Zhang al., 2022b).
Story Infilling As mentioned in Section 2.1, the story cloze task selects a more coherent ending based on the story context. The story infilling com- pletes a story in a similar way that sequences of words are removed from the text to create blanks for replacement (Ippolito et al., 2019). Mori et al. (2020) makes step forward in detecting the missing flawed part of story. The proposed method predicts the positions and provides alterna- tive wordings, which serves as a writing assistant. It is worth mentioning that narrative generation intersects with this task with the key difference that story infilling aims to comprehend the nar- ratives and generate minor parts to complete the story. Wider applications of this task are auxiliary writing systems such as an educational question de- signer (Zhang et al., 2022a) and creative writing supporter (Roemmele and Gordon, 2018b).
Recently, LLMs have also been employed for evaluation tasks. For instance, CANarEx (Anan- tharama et al., 2022) assesses time-series clus- ter recovery from GPT-3 generated synthetic data. UniEval (Zhong et al., 2022) uses a pretrained T5 model for evaluation score computation. GPTScore (Fu et al., 2023) leverages zero-shot capabilities of ChatGPT for text scoring, while G-Eval (Liu et al., 2023) applies LLMs within Shain-of-thought framework and form-filling paradigm for output quality assessment.
To facilitate tracking the research progress, we list the performance of current SoTA models on representative benchmarks in Table 4 in the Ap- pendix. These benchmarks have been categorized into specific tasks based on the instruction tem- plates outlined in Section 2. It is worth noting that methooooogical development or result analysis is not the focus of this survey. Sections 2.1-2.3 are in- tended to serve as timelines depicting the evolution of different research directions and task compar- isons. Further, Section 3 offers a review of the diverse datasets and their respective formats.
Narrative Understanding vs. Narrative Genera- tion The main distinction we make between nar- rative understanding and generation that the latter aims to produce longer sequences conditioned on prototypical story snippets. An epitome is story generation conditioned upon prompts (Fan et al., 2019). where the prompts draft the action plan and reserve the placeholders for generative models to complete. Unlike Story Infilling, the out-of- distribution narrative elements and their relations, e.g., plot structures (Goldfarb- -arrrat et al., 2020), novel plots (Ammanabrolu et al., 2019), creative interactions (DeLucia et al., 2021), and interesting endings (Gupta et al., 2019), are to be generated, which poses the main challenge in story genera- tion (Goldfarb- -arrrattt et al., 2019). Other litera- ture focuses the significant enrichment of details to brief story skeleton (Zhai et al., 2020). Latent discrete plans illustrated by thematic keywords are posited (Peng et al., 201 8), and latent variable mod- els are leveraged to steer the generation (Jhamtani and Bang-Kirkpatrick, 2020). Commonsense rea- soning also needs to be pondered for storytelling
5 Applications
Narrative Assessment The "good at language -> good at thought" fallacy (Mahowald et al., 2023) has spurred the application of narrative understand- ing to the automatic assessment of student essays. Studies on automatic story evaluation (Chen et al., 2022; Chhun et al., 2022) reveal that the referenced metrics, e.g., BLEU and ROUGE scores, deviate from human preferences such aesthetic and in- trigue. This calls for the identification of narrative elements and their relations. For example, Soma- sundaran et al. (2016) builds a graph of discourse proximity of essay concepts to predict the essay

---

everyday scenarios (Mao et al., 2019; Xu et al., 2020). Evaluation criteria are adjusted accordingly to measure the aesthetic merit and correlate with hu- man preferences (Akoury et al., 2020; Chhun et al., 2022). In this sense, narrative generation is the opposite task of narrative understanding with a key emphasis on generating intriguing plots and rich details based on a small corpus. Despite being the opposite in the model-theoretic view, narrative gen- eration needs comply with certain restrictions, e.g.. pragmatics, coreference consistency (Clark al., 2018), long-range cohesion (Zhai et al., 2019; Condfarb-Tarran et al., 2020), and adherence to genres (Alabdulkarim et al., 2021), 1), to name a few.
forts have formulated the objective as predicting the narrative elements or the narrative structure (as discussed in Section 2), given the intractabil- ity of the likelihood of generating the narrative, Pe (narrative|ssketch, task). It is also viable, how- ever, to probe and optimize sketch directly, con- sidering the ability of LLMs to compute the error in close proximation. Hence, Ps (narraativessketclll could be derived as the marginal likelihood task|sketch), and its expecta- tion form Bptttssk(skttch)) narrative|sketee task) can be approximated by sampling an appropriate task. The dependence between task and sketch aligns well with the composition practice of choos- ing suitable genre to match the author's 's creative intent. In more complex cases where the narra- tor is known to have employed particular narratol- ogy techniques (c.g., Flashback (Han et al., 2022)), task and sketch can be parameterized by generative models (e.g., LM-driven prompt engineers (Zhou et al., 2023)), which can be tuned by prompt base through gradient descent optimization.
EraskP
6 Challenges and Future Directions
Prompt Tuning and Author Prompt Recapit- ulation In lieu of the prompt-driven nature of LLMs that clicits tasks with task deseriptions or few-shot examples rather than task-specific fine- tuning (Brown et al., 2020; Sanh et al., 2022; Tou- vron et al., 2023; Tay et al., 2023; Anil et al., 2023), we propose a unified approach for nar- rative understanding. This approach involves a single supervised training process, denoted as (3 [Ti:, context), where context represents a prompt of task description or few-shot examples, and Vı.N denotes predictable annotations produced by crowd-sourced platforms such as Amazon Me- chanical Turk (Mostafazadeh et al., 2016; Papalam- pidi et al., 2020; Lal al., 2021). While the frame- work is universally applicable, the practice of re- lying on annotators to infer authors' thoughts or construct skeleton prompts is inefficient, inaccu- rate and unreliable (Mahowald et al., 2023), Direct consultation with the authors is often impractical, particularly for posthumous masterworks. As sug- gested in the model-thcoretic view (Castricato et al., 2021), gap needs to be closed between the nar- rator and the audience overcome the reader's s uncertainty and other environmental limitations.
Interactive Narrative LLMs, with their ability to carry out numerous language processing tasks ef- fectively in a zero-shot manner (Shen et al., 2023), are paving the way towards a future of immersive and interactive narrative environments. These en- vironments could resemble the dynamic storylines experienced by individuals, as depicted in the TV series "Westworld".
Agent Recent studies (Park et al., 2023; Auto- GPT, 2023) have shown promising results in us- ing database to store an agent's 's experiences and thought processes, effectively serving as a person- ality repository. By retrieving relevant memories from this database and incorporating them into prompts, LLMs can be guided to predict behaviors, thereby producing human-like responses. How- ever, extracting comprchensive character-centric memory from narratives, encompassing aspects such "Who", "When", "Where", "Action", "Feel- ing". "Causal relation", "Outcome", and "Predic- tion" (Xu et al., 2022), remain largely unexplored. Current studies primarily focus on simple corpora and there ample room for further investigation in this area.
In this regard, we envisage a Bayesian perspec- tive (Lyle et al., 2020) for the recovery of the author's 's thoughts. Let (o(marrativelsketch, task) denote the oracle model that simulates the au- thor's composition process, where the author fills in the conceived skeleton out of their own initia- tive (where sketch is the skeleton prompts and task is the task description). The recapitulation of the author's prompt can then be expressed as argmax Pa (naerratiieeskettc)) Previous ef-
Environment Creating immersive and interactive environments for users and agents presents several challenges, primarily due to three key factors: (1) Environment Extraction. Character locations and environments, often vaguely defined unless crucial

---

the plot, have to be clarified. Most works rely on pre-built sandbox environments (Côté et al., 2018; Hausknecht et al., 2020; Park et al., 2023) to ad- dress this issue. However, challenges remain in extracting and representing the environment accu- rately. (2) Environment Generation. Interactive narratives aim to provide users with greater free- dom, but this poses the challenge of automatically generating reasonable and coherent details within the narrative's world. is crucial maintain con- sistency and engagement in storytelling, despite varying user inputs and directions. (3) Environ- ment Update. Agents' text commands may change the state of the world, requiring accurate and cost- effective updates. Current systems update envi- ronment states using predefined rules (Côté et al., 2018: Wang et al., 2022). However, using LLMs to derive and generate narrative environments chal- lenges the use of predefined rules, making efficient and large-scale environment updates future re- search direction.
ment Learning with Human Feedback) (Ouyang et al., 2022) strategy could be applied to the open- world system, which enables the iterative training of the narrative understanding process.
Conclusion
In this paper, we have systematically examined the emerging field of narrative understanding, cata- loguing the approaches and highlighting the unique challenges it poses along with the potential solu- tions that have emerged. We have emphasized the crucial role of LLMs in advancing narrative un- derstanding. Our intention is for this survey serve as a thorough guide for researchers navigat- ing this intricate domain, drawing attention to both the commonalities and unique aspects of narrative understanding in relation to other NLP research paradigms. We aspire to bridge the gap between existing works and potential avenues for further development, thus inspiring meaningful and inno- vative progress in this fascinating field.
Open World Knowledge Incorporating external knowledge and commonsense has been long- standing challenge in both dataset construction and model design (Zellers al., 2019; Wanzare al., 2019; Mikhalkova et al., 2020; Ashida and Sug- awara, 2022). Efforts to address this challenge have been made over the years, with the emer- gence of LLMs providing source potential of commonsense knowledge (Bosselut et al.. 2019; Petroni et al., 2019). Notably, the Text World The- ory (TWT) (Werth, 1999; Gavins, 2007) has been leveraged to provide world knowledge relevant to everyday life, which is simulated through natu- ral language descriptions (Labutov et al., 2018). Similarly, in (Mikhalkova et al., 2020), a text world framework is established, in which the world- building elements (c.g., characters, time and space) are annotated to enhance readers' perception.
Limitations
This paper provides overview of narrative un- derstanding tasks, drawing inspiration from com- putational narratology (Matthews al., 2003) and exploring potential new directions. However. does not delve into broader concepts of cog- nitive computational theory, such as the theory of mind (Happé. 1994), the philosophy of read- ing (Mathies, 2020) and pedagogics (Nicolopoulou and Richner, 2007). Therefore, this survey does not incorporate angnitive-theoreti insights into the un- derlying mechanisms that contribute to the success of models. Another major limitation is the lack of discussion of methodological improvements, as the focus of the research progression is primarily centered around the tasks. Additionally, this survey does not explore narrative generation tasks.
The text world entails nuanced knowledge de- rived from the interplay of various elements. How- ever, such supervision provided by the static world is somewhat limited, as the reward is implicit and the model needs to extrapolate from the annota- tions to exploit the world knowledge. In contrast, the open world (Raistrick et al., 2023) presents an ideal source of supervisor, where the model can be rewarded with incentives derived from world mech- anisms (Assran et al., 2023) that synergistically complements the audience model with the everyday commonsense. To this end, the RLHF (Reinforce-
Acknowledgments
We would like to thank the anonymous review- ers for their insightful comments and helpful sug- gestions. This work was funded by the UK En- gineering and Physical Sciences Research Coun- cil (grant no. EP/T017112/1, 1P/T017112/2, EP/VO48597/1). YH is supported by a Turing AI Fellowship funded by the UK Research and Inno- vation (grant no. EP/V020579/1. EP/V020579/2).

---

References
Sneha Kudugunta, Chang Lan, Katherine Lee, Ben- jamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li. Jian Li, Hyeontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nys- trom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif. Bryan Richter, Parker Riley, Alex Castro Ros, Au- rko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang. John Wiet- ing, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng. Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit lyyer. 2020. STO- RIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6470-6484, Online. Association for Computational Linguistics.
Amal Alabdulkarim, Siyan Li, and Xiangyu Peng. 2021. Automatic story generation: Challenges and attempts. In Proceedings of the Third Workshop on Narrative Understanding, pages 72-83, Virtual. Association for Computational Linguistics.
Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara Martin, and Mark Riedl. 2019. Guided neural language generation for automated storytelling. In Proceedings of the Second Workshop on Storytelling, pages 46-55, Florence, Italy, Association for Computational Linguisties.
Jeffery Ansah, Lin Liu, Wei Kang, Selasie Kwashie, Jixue Li, and Jiuyong Li. 2019. A graph is worth thousand words: Telling event stories using time- line summarization graphs. In Proceedings of the World Wide Web Conference, page 2565-2571, 1, San Francisco, CA, USA. Association for Computing Ma- chinery.
Nandini Anantharama, Simon Angus, and Lachlan O'Neill. 2022. CANarEx: Contextually aware nar- rative extraction for semantically rich text-as-data applications. In Findings of the Association for Com- putarional Linguistics: EMNLP 2022, pages 3551- 3564, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mana Ashida and Saku Sugawara. 2022. Possible sto- ries: Evaluating situated commonsense reasoning under multiple possible scenarios. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3606-3630, Gycongju, Republic of Korea. International Committee on Computational Linguisties.
Peter Anderson, Basura Feraando, Mark Johnson, and Stephen Gould, 2016. Spice: Semantic propositional image caption evaluation. Proceedings of Ma- chine Translation Summit IX: Papers, Amsterdam, The Netherlands. Springer.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learning from images with a joint-embedding predic- tive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pages 15619-15629, Vancouver, Canada. IEEE Computer Society.
Stefanos Angelidis, Lea Frermann, Diego Marcheggiani, Roi Blanco, and Lluís Marquez. 2019. Book QA: Sto- Proceedings
ries of challenges and opportunities. of the 2nd Workshop on Machine Reading for Ques- tion Answering. pages 78 85, Hong Kong, China. Association for Computational Linguistics.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Hellster,,,,, Gau- rav Mishra, Erica Moreira. Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay. Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury. Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo Chowdhery. Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev. Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Gar- cia. Sebastian Gehrmann, Lucas Gonzalez, Guy Gur- Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hur- witz, Michael Isard, Abe Ittycheriah, Matthew Jagiel- ski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun,
AutoGPT. 2023. Auto-gpt: An autonomous gpt-4 ex- periment.
Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2017. Embracing data abundance: Booktest datasct for reading comprehension.
David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 44-54, Marseille, France. European Language Resources Association.
Aakanksha
David Bamman, Brendan O' and Noah A. Smith. 2013. Learning latent personas of film char- acters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, page 352-361, Sofia, Bulgaria. Association for Computa- tional Linguistics.

---

Jack Bandy and Nicholas Vincent. 2021. Addressing "documentation debt" in machine learning: A ret- rospective datasheet for bookcorpus. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks, online. Curran Associates, Inc.
Third Workshop on Narrative Understanding, pages 95-104, Virtual. Association for Computational Lin- guistics.
Nathanael Chambers and Dan Jurafsky. 2008, Unsuper- vised learning of narrative event chains. In Proceed- ings of ACI-088 HIT., pages 789-797, Columbus, Ohio. Association for Computational I Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap. Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi.
2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy, Association for Computational Linguis- tics.
Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 1603-1614, Copenhagen, Denmark. Associa- tion for Computational Linguisties.
Facze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. "let your characters tell their story": A dataset for ceeracter-centric narrative understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1734-1752, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Snigdha Chaturvedi, Shashank Srivastava, and Dan Roth. 2018. Where have heard this story before? identifying narrative similarity in movie remakes. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 673 678, New Orleans, Louisiana. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Atef Chaudhury, Makarand Tapaswi, Seung Wook Kim, and Sanja Fidler. 2020. The shmoop corpus: A dataset of stories with loosely aligned summaries.
Michael W.L. Chee, 2006. Dissociating language and word meaning in the bilingual brain. Trends in Cog- nitive Sciences, 10(12):577-529
Hong Chen, Duc Vo, Hiroya Takamura, Yusuke Miyao, and Hideki Nakayama. 2022. StoryER: Automatic story evaluation via ranking, rating and reasoning. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1739-1753, Abu Dhabi, United Arah Emirates. Asso- ciation for Computational Linguistics.
Pengshan Cai, Mo Yu, Fei Liu, and Hong Yu. 2022. Generating coherent narratives with subtopic plan- ning to answer how-to questions. In Proceedings of the 2nd Workshop on Natural Language Gener- ation, Evaluation, and Metrics, pages 26-42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Cyril Chhun, Pierre Colombo, Fabian M. Suchanek, and Chloé Clavel. 2022. Of human criteria and au- tomatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 5794-5836, Gyeongju. Republic of Korca. In- ternttional Committee on Computational Linguistics.
Zheng Cai. Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the endingsstrong neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers). pages 616-622, Vancouver, Canada. Association for Computational Linguistics.
Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. Neural text generation in stories using entity repre- sentations as context. In Proceedings of the 2018 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). pages 225022200000 2260, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Tommaso Caselli and Piek Vossen. 2021. The event
storyline corpus: A new benchmark for causal and temporal relation extraction. Findings of the Asso- ciation for Computational Linguistics: ACL-IJCNLP. page 77-86, Vancouver, Canada. Association for Computational Linguistics.
Marc- Alee-Ale Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla EI Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for text- based games. In Proceedings the Computer Games Workshop at IJCAI 2018 Stockholm, Sweden. IJCAI Press.
Louis Castricato, Stella Biderman, David Thue, and Rogelio Cardona- Rivera. 2021. Towards model- theoretic view of narratives. In Proceedings of the

---

James E Cutting, 2016. Narrative theory and the dy- namics of popular movies. Psychonomic Bulletin & Review, 23(1):1113-1443.
Gérard Genette. 1988. Narrative Discourse Revisited. Cornell University Press.
Seraphina Goldfarb-Tarrant Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020, Content plan- ning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4319-4333 Online. Association for Computational Linguistics.
Barbara Czamiawska. 2004. Narratives in social science research.
Alexandra DeLucia, Aaron Mueller, Xiang Lisa Li, and João Sedoc. 2021. Decoding methods for neural narrative generation. In Proceedings of the 1st Work- shop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 186-185, Online. Association for Computational Linguistics.
Seraphina Goldfarb-Taran Haining Feng, and Nanyun Peng. 2019. Plan, write, and revise: an interac- tive system for open-domain story generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics (Demonstraations) pages 89-97, Minneapolis. Minnesota. Association for Computa- tional Linguistics.
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated- attention readers for text comprehension. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1832-1846, Vancouver, Canada. Association for Computational I Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett, 2022. SNaC: Coherence error detection for narrative sum- marization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 444-663, 463, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
George Doddington. 2002. Automatic evaluation of ma- chine translation quality using n- n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Re- search, page 138-145, San Francisco, United States. Morgan Kaufmann Publishers Inc.
Mark Granroth-Wiide and Stephen Clark. 2016. What happens next? event prediction using a composi- tional neural network model. In Proceedings of the Thirtieth AAA/ Conference Artificial Intelligence, AAAI' 16, page 2727-2733. AAAI Press.
Denis Emelin, Ronan Le Bras, Jena D. Hwang. Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situ- ated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Prakhar Gupta, Vinayshekhar Bannihatti Kumar, Mukul Bhutani, and Alan W Black. 2019. WriterForcing: Generating more interesting story endings. In Pro- ceedings of the Second Workshop on Storytelling, pages 117-126, Florence, Italy, Association for Com- putational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 2650- 2660, Florence, Italy, Association for Computational Linguistics.
Rujun Han. Hong Chen, Yufei Tian, and Nanyun Peng. 2022. Go back in time: Generating flashbacks in stories with event temporal prompts. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1450-1470, Seattle, United States. Association for Computational Linguistics.
Lea Frermann. 2019. Extractive NarrativeQA with heuristic pre-training. In Proceedings of the 2nd Workshop on Machine Reading for Question Answer- ing. pages 172-182, Hong Kong, China. Association for Computational Linguistics.
Francesca G. E. Happé. 1994. An advanced test of theory of mind: Understanding of story characters' thoughts and feelings by able autistic, mentally hand- icapped, and normal children and adults. Journal of Autism and Developmental Disorders, 24(2):199 154.
Lea Frermann, Shay B. Cohen, and Mirella Lapata. 2018. Whodunnit? crime drama as a case for nat- ural language understanding. Transactions of the Association for Computational Linguistics, 6:1-15.
Jinlan Fu, See- Ng, Zhengbao Jiang, and Pengfei Liu. 2023, Gptscore: Evaluate as you desire.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare- Alexandre Côté. and Xingdi Yuan. 2020. Interac- tive fiction games: A colossal adventure. In Pro- ceedings of the AAAI Conference on Artificial Intel- ligence, pages 7903-7910, New York, New York. USA. AAAI Press.
Albert Gatt and Emiel Krahmer. 2018 Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Al Research, 61(1):65-170.
Marti A Hearst. 1997. Texttiling: Segmenting text into multi pattagaar subtopic passages. Computational linguistics, 23( 1):3-64.
Joanna Gavins. 2007. Text World Theory: An Introduc- tion. Edinburgh University Press.

---

Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. In Proceedings of the Sth International Conference on Learning Rep- rexentations, Toulon, France.
You Jin Kim, Yun Gyung Cheong, and Jung Hoon Lee. 2019. Prediction of movie's ss success from plot sum- maries using deep learning models. In Proceedings of the Second Workshop Storytelling, pages 127- 135, Florence, Italy. Association for Computational Linguisties.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks prineiple: Reading children's 's books with explicit memory representa- tions. In 4th International Conference on Learning Representations, San Juan, Puerto Rico.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The Narrative reading comprehension challenge. Transactions of the Asso- ciation Computational Linguistics, 6:317-328.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading com- prehension with contextual commonsense reasoning. Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, page 2391 22001, Hong Kong, China, Association for Computational Linguistics.
Oleksandr Kolomiyets, Steven Bethard, and Marie- Francine Moens. 2012. Extracting narrative time- lines temporal dependency structures. In Proceed- ings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 88-97, Jeju Island, Korea. Association for Computational Linguistics.
Peter Hühn, Jan Christoph Meister, John Pier, and Wolf Schmid, editors. 2014. Handbook of Narratology. De Gruyter, Berlin, München, Boston.
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged large language models.
Mahnaz Koupaee and William Yang Wang. 2018. Wiki- how: A large scale text summarization dataset.
Daphne Ippolito. David Grangier. Chris Callison-Buu -urchh, and Douglas Bek. 2019. Unsupervised hierarchical story infillin. In Proceedings of the First Work- shop on Narrative Understanding, pages 37-43, Min- neapolis, Minnesota. Association for Computational Linguistics.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). page 3332-9346, Online. Association for Computa- tional Linguistics.
Daphne Ippolito, David Grangier, Douglas Eck, and Chris Callison-Burch. 2020. Toward better storylines with sentence-level language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7472-7478, Online. Association for Computational Linguisties.
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 6536-6558, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Labiba Jahan and Mark Finlayson. 2019. Character identification refined: A proposal. In Proceedings of the First Workshop on Narrative Understanding, pages 12-18, Minneapolis, Minnesota. Association for Computational I Linguistics.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference International Conference Machine Learning, page 957-966, Lille, France. JoLR.org.
Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2020.
Narrative text generation with a latent discrete plan. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 3637-3650, Online. Association for Computational Linguistics.
Igor Labutov, Bishan Yang. Anusha Prakash, and Amos Azaria. 2018 Multi-relational question answering from narratives: Machine reading and reasoning in simulated worlds. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume I: Long Papers). pages 333-844, Melbourne, Australia. Association for Computational Linguistics.
Emil Kalbaliyev and Kairit Sirts. 2022. Narrative why- question answering: A review of challenges and datasets. In Proceedings of the 2nd Workshop on Nat- ural Language Generation, Evaluation, and Metrics, pages 520 530. Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Sudipta Kar, Gustavo Aguilar, Mirella Lapata, and Thamar Solorio. 2020. Multi-view story charac- terization from movie plot synopses and reviews. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing, pages 6629-5646, Online. Association for Computational Linguistics.
Faisal Ladhak, Bryan Li. Yaser Al- Al-Onaizan, and Kath- leen McKeown. 2020. Exploring content selection in summarization of novel chapters. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 5043-5054 Online. Association for Computational Linguistics.

---

Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. TellMeWhy: A dataset for answering why-questions narratives. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 596-610, Online. Association for Computational Lin- guistics.
statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, page 150-157. Association for Computa- tional Linguistics.
Chin-Yew Lin and Franz Josef Och. 2004. Orange: method for evaluating automatic evaluation metrics for machine translation. In Proceedings of the 20th International Conference on Computational Linguis- rics, page 501-507, Geneva, Switzerland. Interna- tional Committee on Computational Linguistics.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metrie for mt evaluation with high levels of correlation with human judgments. In Proceed- ings of the Second Workshop on Statistical Machine Translation, page 228-231, Prague, Czech Republic. Association for Computational Linguistics.
Fei Liu, Trevor Cohn, and Timothy Baldwin. 2018. Narrative modeling with memory chains and seman- tic supervision. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages Melbourne, Australia. Association for Computational Linguisties.
I-Ta Lee, Maria Leonor Pacheco, and Dan Goldwasser. 2021. Modeling human mental states with an entity- based narrative graph. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4916-4926, Online. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human align- ment.
Yoonjoo Lee, Tae Soo Kim, Minsuk Chang, and Juho Kim. 2022. Interactive children's 's story rewriting through parent-children interaction. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants, pages 62-71, Dublin, Ireland. As- sociation for Computational Linguistics.
Clare Lyle, Lisa Schut, Robin Ru, Yarin Gal, and Mark van der Wilk. 2020. A bayesian perspective training speed and model selection. In Advances in Neural Information Processing Systems, volume 33, pages 10396-10408. Curran Associates, Inc.
Maider Lehr, Izhak Shafran, Emily
and Brian Roark. 2013. Discriminative joint mod- eling of lexical variation and acoustic confusion for automated narrative retelling assessment. In Proceed- ings of the 2013 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human anguage Technologies, pages 21 1- 220, Atlanta, Georgia, Association for Computational Linguistics.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective.
Huanru Henry Mao, Bodhisattwa Prasad Majumder. Julian McAuley, and Garrison Cottrell. 2019. Im- proving neural story generation by targeted common sense grounding. In Proceedings of the 2019 Confer- ence on Empiricul Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, pages Hong Kong, China. Association for Computational Linguistics.
Blfi Levi, Guy Mor, Tamir Sheafer, and Shaul Shenhay. 2022. Detecting narrative elements in informational text. In Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 1755-1765, Seattle, United States. Association for Computational Linguistics.
Boyang Li, Beth Cardier, Tong Wang, and Florian Metze. 2018a. Annotating high-level structures of short stories and personal anecdotes. In Proceedings of the Eleventh International Conference on Lan- guage Resources and Evaluation, Miyazaki. Japan. European Language Resources Association (ELRA).
Susanne Mathies, 2020. The simulated self - fic- tion reading and narrative identity. Philosophia, 48(1 18(1))325-345.
Alastair Matthews, Jan Christoph Meister, and Marie- Laure Ryan. 2003. Computing Action: Narratolog- ical Approach.
Qian Li, Ziwei Li, Jin-Mao Wei, Yanhui Gu, Adam Ja- towt, and Zhenglu Yang. 2018b. A based neural network with external knowledge for story ending predicting task. In Proceedings of the 27th International Conference Computational Lin- guistics, pages 1754-1762, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Todor Mihaylov and Anette Frank. 2019. Discourse- aware semantic self-attention for narrative reading comprehension. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing, pages 2541-2552, Hong Kong, China, Association for Computational Linguistics.
Chin-Yes Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence

---

Elena Mikhalkova, Timofei Protasov, Polina Sokolova, Anastasíia Bashmakova, and Anastasiaa Drozdova. 2020. Modelling narrative elements in short story: A study on annotation schemes and guidelines. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 126-132, Marseille, France. European Language Resources Association.
Ageliki Nicolopoulou and Elizabeth S. Richner. 2007. From actors to agents to persons: The development of character representation in young children's 's narra- tives. Child Development, 78(2)412-422,
Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazu- toshi Shinoda, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2019, Multi-style generative reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2274-2284, Florence, Italy, Associa- tion for Computational Linguistics.
Ashutosh Modi, Tatjana Anikina, Simon Ostermann, and Manfred Pinkal. 2016. Inseript: Narrative texts annotated with script information. In Proceedings of the Tenth Interational Conference on Language Re- sources and Evaluation, page 3485-3493, Portorož, Slovenia. Association for Computational Linguistics.
Brian Felipe Keith Norambuena and Tanushree Mitra. 2021. Narrative maps: An algorithmic approach to represent and extract information narratives. Pro- ceedings of the ACM on Human-Computer Interac- rion, 4(228):1-333.
Yusuke Mori, Hiroaki Yamane, Yusuke Mukuta, and Tatsuya Harada. 2020. Finding and generating a missing part for story completion. In Proceedings of the The 4th Joint SIGHUM Workshop on Com- putational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 156- Online. International Committee on Computational Linguistics.
Jekaterina Novikova, Ond rej Dus ek, Amanda Cer- cas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nig. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, page 2241-2252, Copenhagen, Denmark. Association for Computational Linguis- tics.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A cor- pus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 839-849, San Diego California. Association for Computational Linguisties.
Carolin Odebrecht, Lou Burnard, and Christof Schöch. 2021. [link].
OpenAL. 2023. Gpt-4 technical report.
Simon Ostermann, Michael Roth, and Manfred Pinkal. 2019. Mcscript2.0: A machine comprehension cor- pus focused on script events and participants. In Pro- ceedings of the Eighth Joint Conference on Lexical and Computational Semantics, page 103-117, Min- neapolis, Minnesota, Association for Computational Linguistics.
Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll. 2020. GLUCOSE: General.- ized and COntextualized story explanations. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing, pages 4569- 4586, Online. Association for Computational Lin- guistics.
Takaki Otake, Sho Yokoi, Naoya Inoue, Ryo Takahashi, Tatsuki Kuribayashi, and Kentaro Inui. 2020. Mod- eling event salience in narratives via barthes' car- dinal functions. In Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 1784- 1794, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James F. Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, page 46-51, Valencia, Spain. Association for Com- putational Linguistics.
Jessica Ouyang, Serina Chang, and Kathy McKeown. 2017. Crowd-sourced iterative annotation for narra- tive summarization corpora. Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 2, Short Papers, pages 46-51, Valencia, Spain. Associa- tion for Computational Linguistics.
Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open- domain QA techniques: A comprehensive study. Transactions of the Association for Computational Linguistics, +1002-1046.
Jessica Ouyang and Kathleen McKeown. 2015. Mod- cling reportable events as turning points in narrative. In Proceedings of the 2015 Conference Empiri- cal Methods in Natural Language Processing, pages 1149-2158, Lisbon, Portugal. Association for Com- putational Linguistics.
Xiangyang Mou, Mo Yu, Bingsheng Yao, Chenghao Yang, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2020. Frustratingly hard evidence retrieval for QA over books. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 108-113, Online. Association for Computa- tional Linguistics.
Jessica Ouyang and Kathy McKeown. 2014. Towards automatic detection of narrative structure. In Pro- ceedings of the 9th International Conference on Lan- guage Resources and Evaluation, pages 4631-

---

Reykjavik, Iceland. European Language Resources Association.
on Natural Language Processing (EMNLP-IJCNLP), pages 2463 2663-2473, Hong Kong, China. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, New Orleans, LA, USA. Curran Associates, Inc.
Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Processing, pages 298-311, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Matt Post. 2018. call for clarity in reporting bleu scores. In Proceedings of the Third Conference Machine Translation: Research Papers, page 186-191, Belgium, Brussels. Association for Compu- tational Linguistics.
Pinelopi Papalampidi, Frank Keller, Lea Frermann, and Mirella Lapata. 2020. Screepplay summarization us- ing latent narrative structure. In Proceedings of the 58th Annual Meeting of the Association for Compu-
tational Linguistics, pages 1920-1933, Online, Asso- ciation for Computational Linguistics.
James Pustejovsky and Amber Stubbs. 2011. Increasing informativeness in temporal annotation. In Proceed- ings of the 5th Linguistic Annotation Workshop, page 152-160, Portland, Oregon, USA. Association for Computational Linguistics.
Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identi- fication. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joini Conference on Nat- ural Language Processing, page 1707-1717, Hong Kong, China. Association for Computational Linguis- tics.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Blizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 0043, Hong Kong, China. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 311-318, Philadelphia Pennsylvania. Association for Computational Lin- guistics.
Elahe Rahimtoroghi, Jiaqi Wu, Ruimin Wang, Pranav Anand, and Marilyn Walker. 2017. Modelling pro- tagonist goals and desires in first-person narrative. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 360-369, Saar- brücken, Germany. Association for Computational Linguistics.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simu- lacra of human hehavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA. Association for Computing Machinery.
Alexander Raistrick. Lahav Lipson, Zeyu Ma. Lingjie Mei, Mingzhe Wang. Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang. Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, and Jia Deng. 2023. Infinite photorealistic worlds using pro- cedural generation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recog- nition, pages 12630-1264) IEEE Computer Society.
Sangameshwar Patil, Sachin Pawar, Swapnil Hingmire, Girish Palshikar, Vasudeva Varma, and Pushpak Bhat- tacharyya. 2018. Identification of alias links among participants in narratives. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. pages 63 68, Melbourne, Australia. As- sociation for Computational Linguistics.
Michael Raring, Malte Ostendorff, and Georg Rehm. 2022. Semantic relations between text segments for semantic storytelling: Annotation tool dataset evaluation. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 4923 4932, Marseille, France. European Language Resources Association.
Nanyun Peng. Marjan Ghazvininejad, Jonathan May. and Kevin Knight. 2018 Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49, New Orleans, Louisiana. Association for Computational Linguistics.
Hannah Rashkin, Antoine Bosselut, Maarten Sap. Kevin Knight, and Yejin Choi. 2018. Modeling naive psy- chology of characters in simple commonsense sto- ries. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2289-2299, Melbourne, Australia. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel. Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference

---

Hannah Rashkin, Asli Celikyilmaz. Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline- conditioned generation with dynamic plot state track- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing, pages 4274 4295, Online. Association for Computa- tional Linguisties.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4267-4287, Seattle, United States. Association for Computational Linguistics.
Victor Sanh, Albert Wehson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja. Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In Proceedings of the Tenth International Conference on Learning Representations, Virtual. ICLR Press.
Ashwathy T. Revi, Stuart E. Middleton, and David E. Millard. 2020. Idn-sum: A new dataset for interac- tive digital narrative extractive text summarisation. In Proceedings of The Workshop on Automatic Summa- rization for Creative Writing, page 1-12, Gyeongju. Republic of Korea. Association for Computational Linguisties.
Ashwathy T. Revi, Stuart E. Middleton, and David E. Millard. 2021. Timeline summarization based on event graph compression via time-aware optimal transport. In Proceedings of the 2021 Conference on Empirical Methods in Natural Langwage Processing, page 4443-655.. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Maarten Sap, Eric Horvitz, Yejin Choi, Noah A. Smith, and James Pennebaker. 2020. Recollection versus imagination: Exploring human memory and cogni- tion via neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1970-1978, On- line. Association for Computational Linguistics.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA. Association for Computational Linguistics.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang, 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-99 95. AAAI Press.
Swapna Somasundaran, Michael Flor, Martin Chodorow, Hillary Molloy, Binod Gyawali, and Laura McCulla. 2018. Towards evaluating nar- rative quality in student writing. Transactions of the Association for Computational Linguistics, 6:91-106.
Melissa Roemmele and Andrew Gordon. 2018a. An encoder-decoder approach to predicting causal rela- tions in stories. In Proceedings of the First Work- shop on Storytelling, pages 50-59, New Orleans, Louisiana. Association for Computational Linguis- tics.
Swapna Somasundaran, Brian Riordan, Binod Gyawali, and Su-Youn Youn Yoon. 2016. Evaluating argumentative and narrative essays using graphs. In Proceedings of the 26th International Conference on Computa- tional Linguistics: Technical Papers, pages 1568- 1578, Osaka, Japan. The COLING 2016 Organizing Committee.
Melissa Roemmele and Andrew Gordon. 2018h. Lin- guistic features of helpfulness in automated support for creative writing. In Proceedings of the First Work- shop on Storytelling, pages 14-19, New Orleans, Louisiana. Association for Computational Linguis- tics.
Siddarth Srinivasan, Richa Arora, and Mark Riedl. 2018. A simple and effective approach to the story cloze test. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putcationll Linguistics: Human Lunguage Technolo- gies, Volume 2 (Short Pupers), pages 92-96, New Orleans, Louisiana. Association for Computational Linguistics.
Melissa Roemmele, Kyle Shaffer, Katrina Olsen, Yiyi Wang, and Steve DeNeefe. 2023. Ablit: A resource for analyzing and generating abridged versions of english literature. In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. Association for Computational Linguistics.
Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Au- tomated extraction of character roles without training data. In Proceedings of the 4th Workshop of Nar- rative Understanding, pages 47-56, Seattle, United States. Association for Computational Linguistics.
Visi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022. TVShowGuess: Char- acter comprehension in stories as speaker guessing. In Proceedings of the 2022 Conference of the North

---

Max Steg, Karlo Slot, and Federico Pianzola. 2022. Computational detection of narrativity: A compar- ison using textual features and reader response. In Proceedings of the 6th Joint SIGHUM Workshop on Compulational Linguistics for Caltural Heritage, So- cial Sciences, Humanities and Literature, pages 105- 114, Gyeongju, Republic of Korea, International Con- ference on Computational Linguistics.
58th Annual Meeting of the Association for Compu- tational Linguistics, page 5008-5020, Online. Asso- ciation for Computational Linguistics.
Jing Wang. Yufang Ho,, Zhijie Xu. Dan Mcintyre, and Jane Lugea. 2016. The visualisation of cognitive structures in forensic statements. In 2016 20th Inter- national Conference Information Visualisation (IV), pages 106-111, Lisbon, Portugal. IEEE Computer Society.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. IEEE Computer Society.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279 11298, Abu Dhabi, United Arab Emirates, Association for Computational Linguistics.
Yi Tay, Mostafa Dehghami, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng. Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations, Kigali, Rwanda.
Shichao Wang, Xiangrui Cai, HongBin Wang. and Xi- aojie Yuan. 2021. Incorporating circumstances into narrative event prediction. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 4440-48449 Punta Cana, Dominican Republic. Association for Computational Linguisties.
Tay, Shuohang Wang, Anh Tuan Luu, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. 2019. Simple and effective cur- riculum sinter-generatoor networks for reading com- prehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4922 0922-991, Florence. Italy, Association for Computational Linguistics.
Lilian Diana Awuor Wanzare, Michael Roth, and Man- fred Pinkal. 2019. Detecting everyday scenarios in narrative texts. Proceedings of the Second Work- shop on Storytelling, pages 90-106, Florence, Italy. Association for Computational Linguisties
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Zhixing Tian, Yuanzhe Zhang, Kang Liu, Jun Zhao, Yantao Jia, and Zhicheng Sheng. 2020. Scene restor- ing for narrative machine reading comprehension. In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing, pages 3063-3073, Online. Association for Computational Linguistics.
Paul Werth. 1999. Text Worlds: Representing Concep- tual Space in Discourse. Pearson Education Limited, New York. USA. 2010.
Michael Regan William Croft, Pavlína Pešková. 2017. Integrating decompositional event structures into sto- rylines. In Proceedings of the Events and Stories in the News Workshop, page 98 18-109, Vancouver. Canada. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume. Lample. 2023. Llama: Open and efficient foundation language models.
David Wilmot and Frank Keller. 2021. Memory and knowledge augmented language models for inferring salience in long-form stories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 851-865, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alex Tozzo, Dejan Jovanović, and Mohamed Amer. 2018. Neural event extraction from movies descrip- tion. In Proceedings of the First Workshop on Story- lelling, pages 60 666, New Orleans, Louisiana. Asso- ciation for Computational Linguistics.
Sean Wilner, Daniel Woolridge, and Madeleine Glick. 2021. Narrative embedding; Re- Re-Connextualizztion through attention. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1393-1405, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the 2015 IEÉE Conference Computer Vision and Pattern Recognition, Boston, USA. IEEE Computer Society.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tul consistency of summaries. In Proceedings of the
Sarah E Worth. 2004. Narrative understanding and understanding narrative. Contemporary Aesthetics. 2.

---

Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen tau Yih. 2022. Adapt- ing pretrained text-to-text models for long text se- quences.
Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Gar- cia, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2023a. Aggregating crowdsourced and au- tomatic judgments to scale up corpus of anaphoric reference for fiction and Wikipedia texts. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 767-781, Dubrovnik, Croatia, Association for Computational Linguistics.
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2020. MEGATRON- : Control- lable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2845, Online. Association for Computational Linguistics.
Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xi- aochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023b. Personality understanding of fictional characters during book reading. In Proceedings of 61st Annual Meeting of the Association for Com- putational Linguistics, page 14784-14802, Toronto, Canada. Association for Computational Linguisties.
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bing- sheng Yao. Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang. Nanyun Peng. Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA - an authentic dataset for narrative comprehension. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics, pages 447- 460, Dublin, Ireland. Association for Computational Linguistics.
Weizhe Yuan, Graham Neubig. and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems. Curran Associates, Inc.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
Xinru Yan, Aakanksha Naik, Yohan Jo, and Carolyn Rose. 2019. Using functional schemas to understand social media narratives. In Proceedings of the Second Workshop on Storytelling, pages 22-33, Florence, Italy, Association for Computational Linguistics.
Fangzhou Zhai, Vera Demberg, and Alexander Koller. 2020. Story generation with rich details. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 2346-2351, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Zhihua Yan and Xijin Tang. 2023. Narrative graph: Telling evolving stories based on event-centric tem- poral knowledge graph. Journal of Systems Science and Systems Engineering, 22:2.6-221.
Qian Yang. Rebecca J. Passonneau, and Gerard de Melo. 2016. Peak: Pyramid evaluation via automated knowledge extraction. In Proceedings of the Thirti- eth AAAI Conference on Artificial Intelligence, page 2673-2680, Phoenix, Arizona, AAAI Press.
Fangzhou Zhai, Vera Demberg, Pavel Shkadzko, Wei Shi, and Asad Sayeed. 2019. A hybrid model for globally coherent story generation. In Proceedings of the Second Workshop on Storytelling, pages 34- 45, Florence, Italy. Association for Computational Linguistics.
Chi yang Hsu, Yun-Wei Chu, Ting-Hao Huang, and Lun- Wei Ku. 2021. Plot and rework: Modeling storylines for visual storytelling. In Findings of the Associa- tion for Computational Linguistics, page 4443-4453, Bangkok, Thailand. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artziv, 2019a. Bertscore: Eval- uating text generation with bert. In Proceedings of the Seventh International Conference on Learning Representations, New Orleans, United States. ICLR Press.
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Li, Mo Yu, and Ying Xu. 2022. It Al's turn to ask humans a question: Question- answer pair generation for children's story books. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Weiwei Zhang, Jackie Chi Kit Cheung, and Joel Oren. 2019b. Generating character descriptions for auto- matic summarization of fiction. In Proceedings of the Thirt AAAI Conference on Artificial Intel- ligence, Honolulu, Hawaii, USA. AAAI Press.
1: Long Papers), pages 731-744, Dublin, Ireland. Association for Computational Linguistics.
Xiyang Zhang, Muhao Chen, and Jonathan May. 2021. Salience-avee -wwre event chain modeling for narrative understanding. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1418-1428, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenlin Yao and Ruihong Huang, 2018. Temporal event knowledge acquisition via identifying narratives. In Proceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics, pages 537-547, Melbourne, Australia. Association for Computational Linguistics.

---

Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022a. Storybuddy: A human-ai collaborative chatbot for parent-child inter- active storytelling with flexible parental involvement. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, CHI '22, New York, NY, USA. Association for Computing Machin- ery.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE International Conference on Computer Vision, pages 19-27, Santiago, Chile. IEEE Computer Society.
Zhexin Zhang, Jian Guan, Guowei Xu, Yixiang Tian, and Minlie Huang. 2022b. Automatic comment gen- eration for Chinese student narrative essays. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, pages 214-223, Abu Dhabi, UAE, Associa- tion for Computational Linguistics.
Chao Zhao, Faeze Brahman, Kaiqiang Song. Wenlin Yao, Dian Yu, and Snigdha Chaturvedi. 2022a. Nar- raSum: A large-scale dataset for abstractive narra- tive summarization. In Findings the Association for Computational Linguistics: EMNLP 2022. pages 182-197, Abu Dhabi, United Arab Emirates. Associ- ation for Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing, page 563-578, Hong Kong, China. Association for Computational Linguistics.
Zhenjie Zhao, Yufang Hou, Dakuo Wang. Mo Yu, Chengzhong Liu, and Xiaojuan Ma. 2022b. Educa- tional question generation of children storybooks via question type distribution learning and event-centric summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics, pages 5073-5085, Dublin, Ireland. Associa- tion for Computational Linguistics.
Ming Zhong, Yang Liu. Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, Kigali, Rwanda.
Xianshu Zhu and Tim Oates. 2012. Finding story chains in newswire articles. In Proceedings of the IEEE International Conference Information Reuse and Integration, page 93-100. Las Vegas, Nevada, USA. IEEE Computer Society.

---

Setting Datasets Related To Narrative Understanding Tasks
Datsset
Reference
Task
Evaluation
Toresto Book Corpus
Zhu al. (2015)
Narrative Understanding
Coop.orr
Bandy and Vincent (2021)
Narratvv Understanding
FI.TcC
Odchrecht et (2021)
Namative Understanding
GLUCOSE
Mostafazadeh et al. (2020)
Reading Comprehension
BLEU scores Human
ROCStories
Mostafazadeh et (2016)
Consistency Checking: Event-Centric Centric
Story Cloze Test
ROCStories 2017
Mostafazadkh (2017)
Consistoncy Checking: Event-Ceutric
Story Cloze Test
Possible Stories
Ashida and Sugawara (2022)
Consistency Checking: Event-Centric
Accuracy
COPA
Roemmele al. (2011)
Consistency Checking: Plausible Alterativess
Accuracy
TIMETRAVEL
Qin (2019)
Counterfactual
Similarity Metrics Human !1
HellaSwag
Zellers (2019)
Counterfactual
Accuracy
StoryCommonsemse
Rashkin et al. (2018)
Consistency Checking: Character-Centric
F-score + Explanation somelfd
TVShowGness
Sang et (2022)
Consistency Checking: Character-Cemte Centric
Accuracy
PERSONET
Yu (20236)
Consistoncy Checking: Charactes-
Accuracy
LitBank
Bamman al. (2020)
Coreference Edentification
F-scure
Phrase Detectives
Yu al. (2023a)
Coreference Ideutification
Accuracy
Wanzare (2019)
Consistency Checking:
F-score
SNaC
Goyal (2022)
Consistency Checking: Multiple Elements
Fiscore
Pustejovsky and Subbs (2011)
Stractural Analysis
Accuracy
InSeript
Modi al. (2016)
Structural Analysis
Hippocorpes
Sap (2020)
Stractural Analysis
Narative Flow Event contains
ESC v0.9
Caselli and Vossen (2021)
Stractural Analysis
F-score
DesireDB
Rahimtoroghi et al. (2017)
Event Relation Extraction
F-score
Moral
Emelin (2021)
Pvent-Relation
F-score Similarity Metricc
CSI
Fremmann et (2018)
Corpu-Levy Summarisation
F-scure
Shanoop
Chaudhury et al. (2020)
CorpusLevel Summarisatioo
Accuracy
NovelChapter
Ladhak al. (2020)
Corpus-Lovel Summurisstion
Simularity Metriss
BookSum
Kryscinski (2022)
Corpus: Level Summarisation
ROUGE- BERTS SummaQA
NARRASUM
Zhao et al. (2022a)
Corpus Level Summarisation
ROUGE n. SummaC
IDN Sum
Revi et (2020)
Corpus Level Summarisation
ROUGE I + Sccoree
Roemmele
(2023)
Corpus Summarisation
score
CMU Movie Summary
Bamman al. (2013)
Character Centric Summarisation
Variation of information Purily
Zhang et al. (2019h)
Charattr-CCanic Summarisation
Recall &
et (2021)
Charatte--
Accuracy
BookTest
al. (2017)
Story Clore
Accuracy
MCTest
Richardsea et al. (2013)
Answer Generation
Accuracy
Children's Book Test
Hill et (2016)
Answer Generation
Accuracy
MovicQA
Tapaswi (2016)
Generation
Accuracy
WikiHow
Koupaee and Wang (2018)
Answer Generation Summarisation
METEOR
MCScripp2.0
Ostermann al. (2019)
Answer Generation
Accuracy
Cosmos
Huang (2019)
Generation
Accuracy
NarrativQAA
Kočisky et al. (2018)
Narrative Question Answering
Similarity Metrics
TelMMWhhy
Lal et al. (2021)
Narrative Question Answering
Similarity Metrics
FairytaleQA
Xu al. (2022)
Narrtive Question Answering
ROUGE-L score
Table Settings for datasets related to narrative understanding tasks, including references, the tasks the dataset created for, and the evaluation methods. Below are some supplementary information that cannot fit in the table: [1] Similarity Metries (e.g. BLUE-4, ROUGE-L, BERT, BERT-FT, Word Mover's Similarity. Sentence Word Mover's Similarity), complemented by human evaluation using Likert scale scores; [2] F-score for category labels, Vector average and extrema score for annotation explanations; [8] Accuracy and F1 score for classification, as well as similarity metrics for generation tasks; [4] ROUGE-I precision hetween spans and F-score for sentence labels.

---

Domain Information On Datasets Related To Narrative Understanding Tasks
Domain
Dataset Size
Average Longth
Language
Toronto Book Carpus
Romance, Historical, Adventure, etc.
11,038 books
-6,704 sentences
English
BockCorpusOOeen
Romance, Historical, Adventure, etc.
17,868 books
Eaglish
ELTeC
1,250 novels
8 Langunges!!
GLUCOSE
Commmemennse stoties (ROCStories)
4,881 stories
5 sentences
English
ROCStories
Commonsense Stories
49,255 stories
5 sentences
English
ROCStories Winter 2017
Commonsense Stories
98,159 sturies
5 sentences
English
Possible Stories
Short story with multiple andings
1,313 passages
tokens
English
COPA
Choice Of Plausible Allematives
IK questions
I sentenee
English
TIMETRAVEL
Commonsense stories (ROCSSooris)
story sowritings
sentenees
English
HellsSwag
Commonsense stories (SWAG)
70K passage
I sentence
English
Commomenne
Commonsense stories (ROCStories)
stories
sentences
English
TVShow
Soripts of TV series
318 charneters
137,568 tokens
English
PERSONET
Novel
books
sentences
English, Chinese
LitRank
Fiction
100 fictions
2,105. 3 tokens
English
Phruse Detectives
Fiction and Wikipedia
805 decuments
1,712.4 tokens
English
Wanzure al. (2019)
Blog (Spinn3e)
504 stories
35.74 sentences
English
SNaC
1.Mss generated book/movie summaries
150 books
41 sentences
English
Pustejavsky and Stubbs (2011)
183 articles
English
InScript
Commonsense stories (given secnarios)
910 stories
12.4 sentences
English
Hippocorpus
Stoies of imsgodrecallled ovents
6,854 Stories
17.6 senteences
English
ESC v0.9
BCB+ corpus|2)
258 documents
English
DesireDB
Blog (Spinn3e)
3,680 instances
English
Moral Stories
Social Norms, Marality/Phhies
12K stories
Eaglish
CSI
Crime Drama
episodes (59 cases)
689 sentences per case
English
Shmoop
Novels, plays, shoet stories!!
231 stories
112,060 tokens
English
NovelChapter
Novel
4, 3,383 chapters
5,165 words
English
RookSSun
Plays, short storles, novels (Gutenbergg)
405 books
112,885,15 tokens
English
NARRASUM
Plor descriptions of Movic/TV qpisedes
122K marraives
786 takons
English
Narrative gume scripts
IDN episodes
sentences
English
ABLIT
Novels (Gutenberg)
868 chapters
154.1 sentences
English
CMU Movic Summary
Movic plot summarics
42. 306 movies
176 words
English
Zhang al. (2019b)
Romance, Werewolf, etc. (Watipe))
1,036,96 965 stories
15,600 words
English
LISCU
Educational stories
1,220 books
1431.2 takens
English
BookTest
Books (Gutenberg)
14,1 14,.40,82 questions
522 takens
English
MCTest
Books (Gutenberg)
500 stories 2,000 questions
212 words
English
Children's Book Test
Books (Gutenberg)
books 687,343 questions
462.7 words
English
MovieQA
Movie scripts
14,944 guestions
9.3 words
English
WikiHow
How website
230,843 articles
tokens
English
MCScrip12.0
Short stories around
scenurios
3,487 lexts 19,821 questions
164.4/ tokens
English
Cosmas QA
Paragraph+ Questions ["]
35,600 (paragraphs questions)
69.4/10.3 tokens
English
NarrariveQA
Books (Gutenberg), movie scripts
1572 drenments 46,765 questions
61.472/ 9.8 takens
English
TelloWhy
Commonsense stories (ROCSooiee)
9.636 stories + 30,519 questions
5 sentences
English
FairytaloQA
Classic fairytale stories
278
questions
1401.3 0/0.3/3..0 tokens
English
Table 2: Domain information on datasets related to narrative understanding tasks, including data domain, dataset size, average text length (per narrative or character), and the language used. Below are some supplementary information that cannot fit in the table: [1] 8 Europoean Language including Czech, German, English, French, Hungarian, Polish, Portuguese and Slovenian; [2] ECCB+ corpus focuses on calamity events, such as shooting and accidents; [8] Short story sources include Shmoop website and Gutenberg; [4] The median length is 176 words since no average text length is provided; [5] The length pertains to book summaries; [6] Text length ranges from 150-300 words; [?] Questions relate to causes, effects, facts, and counterfactuals.

---

Annutation Information On Dutasets Related To Narrative Understanding Tusks
Dataset
Total Number of Amotations
Annotation Type
Annotation Procedure
Torceso Book Corpus
BookCoppsOpen
ELTcC
GLUCOSE
~670K
Commonsense Causal Knowlodge
Humon- Crowdbouredd (MTurk)
ROCStories
Causal + Temporal Span
Humon- Crowdooured (MTuk)
ROCSSooies Winter 2017
Causal Teempraa Span
Human- Crondsscoddd (urk)
Possible Stories
8,885 ending + 4,533 questions
Alternative Fading Cansal Question the more Plansibic Alteriatives Countertactual Rewritings
Mumon-Croodsonngee (MTurk)
COPA
2 Altematives for each question
Human
TIMETRAVEL
81,407 countefaactall brunch 70K Answens! !!
Human Crowdsourced (MTukk)
HellaSwag
Counterfactual Ressoning
Machine-gonerstod
MleryCommenecmme
55,747 wimoliv 104,950 w/emot
Motivations Emotional Reactions
Humon
GVess
12,413 scene
Character Facts
Human (2 expents)
PERSONET
140,268 X
Personalitis Traits
Automatic collection Human
LaRank
105 montions
Anaphoric Reference + Futity Categrey
Human (3 experts)
Phrase Detectives
282.558 mensions
Anaphoric Reference
Human Crowdbourced
Wanzare al. (2019)
10.754 sentences
Scenatios Segmeniazion
Humaníé studeni assistants) Humann
SNaC
Span
Caheresez Enor Span Type
Pustejovsky and Stublss (201 1)
Temporal Syon
Humon (3 studenis)
InScript
62,062
Scipt Structure
Hluran-Crowdseee (MTurk)
Hippocorpus
Human Recalled Events
Humonn-Cowdoonnee (MTuk)
ESC
9169 relations/1
Event Relation Temporal Span
Human (2 studems)
DesireDB
3,680
Desire Expressione!!
Human Cumann-Croweer Crowdsourced (MTurk)
Moral
24K action 48K consequence
Surr Segment Sentence Categories
Human Crowdsoureed (MTuk)
CSI
Story Segment Sentenoe Caregories
Factmal/Simittur Metadata"
Human (3 studens)
Shunoop
7,234 summaries 7,254 chapters
Segmentation chapter-level sanmaries
Automatic collection [w]]
NovelChupter
8,088 chupten'ammarry puirs
Chapter level summaries
Automatic collection Human written '41
BankSum
405 summuries
Paragruph, chapter, book-level summaries Book-Jovel summarics
Automatic aligmment Human inspection
NARRASUM
122K summarics
Automatic alignment Homan inspaction
IDN Sum
10k summuries for 10k documents
Interactive nurnaivvs summuries
Automatic collection [44
ABLIT
808
Paragruph level abridged texts
Automatic alignment Human writen [X]
CMU Movic Summary
29,902 charackes
Character mctadata
Automatic matching
Zhang al. (20196)
18,100 characters
Character Metadata- Troppe
Automatic extraction- Humon DH
LISCU
9459
Character Deseription
Automaric creation Human evaluation!?]
BookTest
141,408,250 options
Cloze- form
Automatic creation
MCTest
8000
Multigle-choice
Human- coann--roodooreeedd (MTurk)
Children's Rook Test
choices each question
Multigle-hhice
Automatic creation
MovieQA
74,720 answers
Multiple choice
Human
WikiHow
230,843 summuries
Subtopics form Answer
Automatic collection
MCScript2.0
2 choices for each question choices each question
Answer Generazion Multiie--hoice
Human Crowdsourced (MTurk)
Cosmos QA
Mulliple-choice
Human Crowdsoureed (MTuk)
NaraaiivQA
46.765 answers
fure-frmm& Answer
Human Crowdbourced (MTuk)
Why
answers for question 10,580
form Answer
Crowdsoureed (MTuk)
FairytleeQA
Answer + Ground truth Question Pairs
Human posigraduate students
Table 3: Annotation information on datasets related to narrative understanding tasks, including the total number of annotations, annotation type, and annotation procedure (expert vs. crowdsourced, human vs. automatic). Below are some supplementary information that cannot fit in the table: [1] Adversarial wrong answers for each passage; [2] 140,268 traits derived from 110, I 14 notes, which were automatically collected from reading apps. These notes were written by the app's users; [3] The data was sourced via a haame-with---uupee
approach; Annotators labelled total 504 documents, which comprised 10,754 sentences. A label for a scenario could be assigned from one of the 200 predefined scenarios marked as "None" for sentences that didn't fit any scenario; [5] Both expert evaluators (3 experts) and human crowdsourcing through MTurk were used for annotation; [6] Stories were annotated across 10 distinct scenarios. Verbs and noun phrases were labelled with event and participant types, respectively. The text also includes coreference annotations. [7] The dataset includes 6,904 temporal relations and 2,265 explanatory relations.; [M] It has gold standard labels for identifying statements of desire, spans of evidence supporting the fulfillment of the desire, and annotations indicating whether the stated desire is fulfilled based on the narrative context; (9) References to the mentioned perpetrator and relation to previous cases; [10] Paired summaries and narrative texts sourced from websites; [11] Summaries and chapter pairs were automatically collected from online study guides, written by experts: [12] Paired summaries and narrative texts sourced from websites; [13] The dataset has an automatically aligned abridged version, which is written by single human author;; [14] Characters are matched to actors with a public date of birth; [15] Annotations are collected through questionnaires to 100 authors; [16] 1708 literature summaries and 9499 character descriptions; [17] 3 judges were asked to evaluate the quality of the description, focusing on fact coverage and task difficulty; (18) There are 4 questions associated with each story. and each question offers 4 answer choices.

---

Task
Representative Dataset
Best Model
Result
StoryCloze
StoryCloze (Mostafazadeh et al., 2016)
FLAN 137B zero-shot (Wei al., 2022)
93.4 Accuracy
CounterfatucllRessninng
Hellaswag (Zellers al., 2019)
GPT-4 (OpenAI, 2023)
95.3 Accuracy
NarraiveSunmarization
BookSum (Kryscinski
2022)
BART-LS (Xiong et al., 2022)
38.5 Rouge-1
NarrativeQA
NarrativeQA (Kočiský et al., 8)
Masque (Nishida al., 2019)
59.87 Rouge-I.
NarrativeQA
Children's Test (Hill et al., 2016)
NSE (Dhingra et al., 2017)
Accuracy
Table 4: Performance of the most recent models on representative datasets. The results are extracted from their respective papers.


Reviews:
[{"review_id": "FIMGy6nYZV", "reviewer": "Reviewer_Bkvf", "summary": "", "questions": "A. In Fig1, should the citation for narrative summarization->event be zhao 2022a instead of b?\n\nB. When building the task taxonomy, why is reading comprehension considered to have categorical labels while question answering is treated as an isolated branch instead of a reading comprehension task?\n\nC. Should some narrative-specific NLP tasks such as novel NER also be considered understanding tasks or do you prefer to regard them as low-level processing tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! To address your specific concerns:\n> 1.  For future directions, it would be valuable to have more discussions about the intrinsic challenges of narrative understanding, not necessarily limited to the capabilities of LLMs.\n\nNarrative understanding presents many challenges due to the complex nature of human storytelling. Some of the challenges include: (1) Character identification and event tracking.  Character identification needs to deal with the challenges of co-referencing and entity linking, while unsupervised event detection approaches need to be developed to recognise main events with an arbitrary set of arguments from text. (2) Dynamic character network construction. The relationship between characters may not be static, but evolving over time. Also the relation types cannot be pre-defined. (3) Plot structure analysis. Traditionally, an event storyline usually organises related events in a chronological order. However, in narratives, stories are complex and the narrative could be structured in many different ways including flashbacks, multiple stories happening simultaneously, many smaller stories taking place within a framework of a bigger plot, or even intertwining stories. Developing methods to automatically identify plot structured elements is challenging. More discussions will be added in the revised version,\n\n> 2. In Fig1, should the citation for narrative summarization->event be zhao 2022a instead of b?\n\nYes, the correct citation should be 'Zhao 2022a' instead of 'Zhao 2022b.' The '2022a' paper focuses on events, while the '2022b' paper is centred on NarrativeQA.\n\n> 3. When building the task taxonomy, why is reading comprehension considered to have categorical labels while question answering is treated as an isolated branch instead of a reading comprehension task?\n\nThis is precisely the issue we considered when building the task taxonomy: Question answering is one of the most popular ways for evaluating reading comprehension tasks. Reading comprehension takes the form of QA in datasets like NarrativeQA, TellMeWhy, and FairytaleQA. With the advancement of Large Language Models (LLMs), the scope of generative-based question answering has significantly widened. Depending on the specific questions posed, its scope can encompass other more fine-grained tasks defined in our taxonomy, including narrative summarization (Who is Charlie?), and narrative generation (What would happen after the Princess marries the Prince?). Therefore, we decided to treat it as an isolated branch.\n\n> 4. Should some narrative-specific NLP tasks such as novel NER also be considered understanding tasks or do you prefer to regard them as low-level processing tasks?\n\nSome narrative-specific NLP tasks such as Named Entity Recognition (NER) might be perceived as understanding-related tasks when determining whether designating 'a talking cup' as a 'person' is appropriate. We will add a section on 'future work' to discuss the modification needed for traditional NLP tasks to be effectively applied to narratives.\n\n> 5. I suggest adding this dataset to Table A1: Ladhak, Faisal, et al. \"Exploring Content Selection in Summarization of Novel Chapters.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\nThanks for the suggestion, we will add this dataset to Table A1.\n\n|Dataset | Reference | Task | Language | Domain | Evaluation |\n| :------------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| NovelChapter| [Ladhak et al., 2021](https://aclanthology.org/2020.acl-main.453.pdf)  | Corpus-Level Summarisation | English | Novel | Similarity Metrics |\n\n|Dataset | Dataset Size | Text Length (Average) | Number of Annotations (Total) | Annotation Type | Annotation Procedure |\n| :------------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| NovelChapter |  4,383 chapters | 5,165 words | 8,088 chapter/summary pairs | Chapter-level summaries | Automatic collection (summary-chapter pairs online study guides) + Human written (expert) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a comprehensive survey and taxonomy of narrative understanding tasks.", "reasons_to_reject": "I didn't find significant weaknesses. While it would be nice to include narrative generation in the survey, I understand that it may exceed the scope of this submission. For future directions, it would be valuable to have more discussions about the intrinsic challenges of narrative understanding, not necessarily limited to the capabilities of LLMs.", "questions_to_authors": "A. In Fig1, should the citation for narrative summarization->event be zhao 2022a instead of b?\n\nB. When building the task taxonomy, why is reading comprehension considered to have categorical labels while question answering is treated as an isolated branch instead of a reading comprehension task?\n\nC. Should some narrative-specific NLP tasks such as novel NER also be considered understanding tasks or do you prefer to regard them as low-level processing tasks?", "paper_topic_and_main_contributions": "This paper conducts a comprehensive survey and taxonomy of narrative understanding tasks, including reading comprehension, summarization, and question answering, along with their associated datasets and evaluation measures. The final section explores the potential of extending the capabilities of modularized LLMs to tackle novel narrative understanding tasks.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I suggest adding this dataset to Table A1:\nLadhak, Faisal, et al. \"Exploring Content Selection in Summarization of Novel Chapters.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020."}}, {"review_id": "dX942iQYoU", "reviewer": "Reviewer_xuKr", "summary": "", "questions": "No detailed questions as most sections are about high-level introduction to the area. More suggestions are shared in the presentation improvement section below.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. First, to address the concerns raised in your Reasons To Reject of the review:\n\n> 1. Narrative understanding could be too broad to be discussed in a 8-page conference paper. More topics and works can only be discussed with a light touch. Might consider narrowing the focus area (such as only keeping Narrative Consistency and Structural Analysis for Events), or submit as a comprehensive journal paper.\n\nWe’ve seen survey papers published in conferences like EMNLP ([Ali and Hassan, 2022]( https://aclanthology.org/2022.emnlp-main.633.pdf )) and NAACL ([Wang et al., 2022]( https://aclanthology.org/2022.naacl-main.339.pdf )). These papers primarily focus on task or dataset taxonomies rather than serving as an exhaustive compendium of benchmark details and model performance. In our case, the field of narrative understanding is fast evolving, as evidenced by datasets like StoryCloze and HellaSwag for LLM evaluation ([Brown et al., 2020]( https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf ); [Sanh et al., 2022](https://arxiv.org/pdf/2110.08207.pdf)). Given fast development of this field, concise surveys might be better suited to track the swift shifts in dataset exploration trends. With respect to narrowing focus area, we respectfully argue that 1) \"narrative understanding\" is a self-contained area, with many datasets shared across various tasks. Streamlining this landscape could impede a comprehensive understanding of their overlaps and differences. 2) Present research trends often encompass a range of applications rather than focusing on a few. In our approach, we organised dispersed datasets into well-defined tasks such as reading comprehension and summarization, all consolidated within the frameworks of LLM paradigms, such as multiple choice formats and free-form text generation guided by instructions. This approach fosters both unified strategies and specialized methodologies.\n\n> 2. Lack any quantitative or qualitative studies to offer objective comparisons among works.\n\nWe agree with you to make quantitative comparisons and will report model performance on representative benchmarks in the Appendix:\n|Task | Representative Dataset | Best Model | Result |\n| :------------: | :-----------: | :-----------: | :-----------: |\n| StoryCloze | StoryCloze ([Mostafazadeh et al., 2016](https://aclanthology.org/N16-1098 )) | FLAN 137B zero-shot ([Wei et al., 2021](https://openreview.net/pdf?id=gEZrGCozdqR)) | 93.4 Acc. | \n| CounterfactualReasoning | Hellaswag ([Zellers et al., 2019](https://aclanthology.org/P19-1472 )) | GPT-4 ([GPT-4 Technical Report](https://arxiv.org/abs/2303.08774 )) | 95.3 Acc.|\n|NarrativeSummarization| BookSum ([Kryscinski et al., 2022](https://aclanthology.org/2022.findings-emnlp.488.pdf )) | BART-LS ([Xiong et al., 2022](https://arxiv.org/pdf/2209.10052.pdf )) | 38.5 Rouge |\n| NarrativeQA | NarrativeQA ([Kocisky et al., 2018](https://aclanthology.org/Q18-1023 )) | Masque ([Nishida et al., 2019](https://aclanthology.org/P19-1220.pdf )) | 59.87 Rouge-L |\n| NarrativeQA | Children’s Book Test ([Hill et al., 2016](http://arxiv.org/abs/1511.02301 )) | NSE ([Dhingra et al., 2017](https://aclanthology.org/P17-1168.pdf )) | 71.9 Acc. | \n\nBut because we are dedicated to offering an overview of the tasks, analyzing the performances of the models from the [leaderboards](https://paperswithcode.com/sota/sentence-completion-on-hellaswag ) will reduce the number of datasets we can cover, and is uncommon in task-oriented surveys ([Ali and Hassan, 2022]( https://aclanthology.org/2022.emnlp-main.633.pdf ); [Wang et al., 2022]( https://aclanthology.org/2022.naacl-main.339.pdf ); [Dzendzik et al., 2021]( https://aclanthology.org/2021.emnlp-main.693.pdf )). As for the qualitative comparisons, we consider Sections 2.1-2.3 as timelines of strands of works and comparisons between them when we review the developments.\n\n> 3. Lack of evaluations for the proposed framework.\n\nWe will provide running examples of the author’s prompt recapitulation as soon as time and computational resources permit. But it is typically not expected that the generalized definition will be evaluated within the scope of a survey, nor are the frameworks suggested in Future Directions.\n\nWe are grateful for your suggestions on presentation improvements:\n> a) Line 14: it is not very clear to me whether this work suggests a new method about LLM and narrative understanding. I could guess it might be related to the paragraph of Line 597. If this is one main contribution of the paper, it will be good to make this section more explicit.\n\nWe will explain in more detail in the revised version. This work firstly aligns disparate tasks with the  LLM paradigms denoted as $p_θ(y|x, \\mathsf{context})$, and categorizes  them based on the selection of $\\mathsf{context}$ and the structure of $y$ (see lines 91-115, 597-612). Subsequently, it introduces Bayesian prompt selection as an alternative approach to define the task of narrative understanding (see lines 612-653).\n> b) Figure 1: the discussions in the paper do not connect to this topology very well. Suggest adding references to a specific node of the topology when discussing it in the paper. c) Line 678: “and interactive”\n\nWe appreciate your review and references will be added to the nodes. The typo will be fixed as well.\n> d) Line 704: for world knowledge, it is crucial to include related works about Knowledge Graph, starting from early works such as ConceptNet [1], FrameNet [2], to more recent versions that connect to LLM.\n\nWe cited LAMA ([Petroni et al., 2019](https://aclanthology.org/D19-1250.pdf )) and COMET ([Bosselut et al., 2019](https://aclanthology.org/P19-1470.pdf )) as they establish connections between ATOMIC ([Sap et al., 2019](https://ojs.aaai.org/index.php/AAAI/article/view/4160 )) and GPT, forming a foundational  static world. Additional references will be added in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a) Survey papers dedicated to discussing narrative events are not common. The presentation can benefit researchers who are new to this area.\n\nb) The write up is clear and easy to follow.", "reasons_to_reject": "a) Narrative understanding could be too broad to be discussed in a 8-page conference paper. More topics and works can only be discussed with a light touch. Might consider narrowing the focus area (such as only keeping Narrative Consistency and Structural Analysis for Events), or submit as a comprehensive journal paper.\n\nb) Lack any quantitative or qualitative studies to offer objective comparisons among works.\n\nc) Lack of evaluations for the proposed framework.", "questions_to_authors": "No detailed questions as most sections are about high-level introduction to the area. More suggestions are shared in the presentation improvement section below.", "paper_topic_and_main_contributions": "This work conducts a survey about an overview of recent narrative understanding works.The authors examine a broad categories of narrative understanding, such as taxonomy, key characteristics, datasets, evaluations, and applications. In addition, the authors propose a general framework about leveraging LLM to conduct narrative understanding, which has broad applicability in Narrative Understanding.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. \"ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.\" In proceedings of AAAI 31.\n[2] Baker, Collin F., Charles J. Fillmore, and John B. Lowe. \"The berkeley framenet project.\" In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics. 1998.", "Typos_Grammar_Style_and_Presentation_Improvements": "a) Line 14: it is not very clear to me whether this work suggests a new method about LLM and narrative understanding. I could guess it might be related to the paragraph of Line 597. If this is one main contribution of the paper, it will be good to make this section more explicit.\n\nb) Figure 1: the discussions in the paper do not connect to this topology very well. Suggest adding references to a specific node of the topology when discussing it in the paper. \nc) Line 678: “and interactive”\n\nd) Line 704: for world knowledge, it is crucial to include related works about Knowledge Graph, starting from early works such as ConceptNet [1], FrameNet [2], to more recent versions that connect to LLM."}}, {"review_id": "8QhClehrxW", "reviewer": "Reviewer_ZCLn", "summary": "", "questions": "Line 34-37: \"This is supported ... English speaker\".\nHow does this result from cognitive research support the support the previous statement about LLMs lacking the ability to capture the author's thoughts?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review! In response to the weaknesses you’ve pointed out:\n> 1. Being a survey paper, the authors could have strengthened their contribution with a more fine-grained categorization of the tasks and datasets. For example, dataset size, narrative text length, annotation procedure (expert vs crowdsourced, human vs automatic), domain, period, author count, language, evaluation, etc.\n\nThe original table for the dataset can be found in Appendix Table A1. Based on the suggestions, we have added more detailed information about those datasets, which is shown in the two tables below. The first table provides information and settings for each dataset, including references, the tasks the dataset was created for, the language used, the data domain, and the evaluation methods. The second table includes statistical and annotation information about each dataset, such as dataset size, average text length (per narrative or character), total number of annotations, annotation type, and annotation procedure (expert vs. crowdsourced, human vs. automatic).\n\n|Dataset | Reference | Task | Language | Domain | Evaluation |\n| :------------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| Toronto Book Corpus |  Zhu et al., 2015 | Narrative Understanding | English | Romance, Historical, Adventure, etc. | - |\n| BookCorpusOpen |  Bandy and Vincent, 2021 | Narrative Understanding | English | Romance, Historical, Adventure, etc. | - | \n| ELTeC |  Odebrecht et al., 2021 | Narrative Understanding | 8 European Languages $^{\\mathrm{[1]}}$ | - | - |\n| GLUCOSE |  Mostafazadeh et al., 2020 | Reading Comprehension | English | Commonsense stories (ROCStories) | BLEU scores + Human Evaluation | \n| ROCStories |  Mostafazadeh et al., 2016 | Consistency Checking: Event-Centric | English | Commonsense stories | Story Cloze Test  |\n| ROCStories Winter 2017 |  Mostafazadeh et al., 2017 | Consistency Checking: Event-Centric | English | Commonsense stories | Story Cloze Test  |\n| Possible Stories | Ashida and Sugawara, 2022 | Consistency Checking: Event-Centric | English |Short story with multiple endings | Accuracy |\n| COPA | Roemmele et al., 2011 | | English | Choice Of Plausible Alternatives | Accuracy |\n| TIMETRAVEL | Qin et al., 2019  | Counterfactual | English | Commonsense stories (ROCStories) | Similarity Metrics $^{\\mathrm{[2]}}$ + Human Evaluation (Likert scale scores) |\n| HellaSwag | Zellers et al., 2019  | Counterfactual | English | Commonsense stories (SWAG) | Accuracy |\n| StoryCommonsense | Rashkin et al., 2018 | Consistency Checking: Character-Centric | English | Commonsense stories (ROCStories)  | F1 score ( for category labels) + Vector average and extrema score (for annotation explanations) |\n| TVShowGuess | Sang et al., 2022 | Consistency Checking: Character-Centric | English | scripts of TV series | Accuracy |\n| PERSONET | Yu et al., 2023b | Consistency Checking: Character-Centric | English, Chinese | Novel| Accuracy |\n| LitBank | Bamman et al., 2020 | Coreference Identification | English | Fiction | F-score |\n| Phrase Detectives | Yu et al., 2023a | Coreference Identification | English | Fiction and Wikipedia | Accuracy |\n| - | Wanzare et al., 2019 | Consistency Checking: Other Elements | English | Blog (Spinn3r) | F-score |\n| SNaC | Goyal et al., 2022 | Consistency Checking: Multiple Elements | English | LLMs generated book/movie summaries | F-score  |\n| - | Pustejovsky and Stubbs, 2011 | Structural Analysis | English | - | Accuracy |\n| InScript | Modi et al., 2016 | Structural Analysis | English | Commonsense stories under given scenarios | - |\n| Hippocorpus | Sap et al., 2020 | Structural Analysis | English | Stories of imaged/recalled events | Narrative Flow + Event contains |\n| ESC v0.9 | Caselli and Vossen, 2021 | Structural Analysis | English | ECB+ corpus concerning calamity events (shooting, accidents, etc.) | F-score |\n| DesireDB | Rahimtoroghi et al., 2017 | Event-Relation Extraction | English | Blog (Spinn3r) | F-score |\n| Moral Stories | Modi et al., 2016 | Event-Relation Extraction | English | Social Norms, Morality/Ethics | Classification (Accuracy, F1 score) + Generation (Similarity Metrics) |\n| CSI | Frermann et al., 2018 | Corpus-Level Summarisation | English | Crime Drama | F-score |\n| Shmoop | Chaudhury et al., 2020 | Corpus-Level Summarisation | English | Novels, plays and short stories (Shmoop website & Gutenberg) | Accuracy |\n| BookSum | Kryściński et al., 2022 | Corpus-Level Summarisation | English | Plays, short stories, and novels (Gutenberg) | ROUGE-n, BERTScore, SummaQA |\n| NARRASUM | Zhao et al., 2022a | Corpus-Level Summarisation | English | Plot descriptions of Movie/TV episodes |  ROUGE-n, SummaC |\n| IDN-Sum | Revi et al., 2020 | Corpus-Level Summarisation | English | Narrative game scripts | ROUGE-1 F1 |\n| ABLIT | Roemmele et al., 2023 | Corpus-Level Summarisation | English | Novel (Gutenberg) | ROUGE-1 precision score (between spans) + F-score (of sentence labels) |\n| CMU Movie Summary | Bamman et al., 2013 | Character-Centric Summarisation | English | Movie plot summaries | Variation of information + Purity score |\n| - | Zhang et al., 2019b | Character-Centric Summarisation | English | Romance, Werewolf, etc. (Wattpad) | Recall@K |\n| LiSCU | Brahman et al., 2021 | Character-Centric Summarisation | English | Educational stories | Accuracy|\n| BookTest | Bajgar et al., 2017 | Story Cloze | English | Books (Gutenberg) | Accuracy |\n| MCTest | Richardson et al., 2013 | Answer Generation | English | Short fictional stories| Accuracy|\n| Children’s Book Test  | Hill et al., 2016 | Answer Generation | English| Books (Gutenberg) | Accuracy |\n| MovieQA | Tapaswi et al., 2016 | Answer Generation | English | Movie scripts | Accuracy |\n| WikiHow | Koupaee and Wang, 2018 | Answer Generation & Summarisation | English | HowWiki website | METEOR |\n| MCScript2.0 | Ostermann et al., 2019 | Answer Generation | English| Short stories around everyday scenarios | Accuracy |\n| Cosmos QA | Huang et al., 2019 | Answer Generation | English| Paragraph & questions relates to causes, effects, facts, counterfactuals | Accuracy |\n| NarrativeQA | Kočiský, et al., 2018 | Narrative Question Answering | English| Books (Gutenberg), movie scripts | Similarity Metrics |\n| TellMeWhy | Lal et al., 2021 | Narrative Question Answering | English| Commonsense stories (ROCStories) |Similarity Metrics |\n| FairytaleQA | Xu et al., 2022 | Narrative Question Answering | English| Classic fairytale stories | ROUGE-L F1 score |\n\n[1] Czech, German, English, French, Hungarian, Polish, Portuguese and Slovenian\n\n[2] BLUE-4, ROUGE-L, BERT, BERT-FT, Word Mover’s Similarity (WMS), Sentence + Word Mover’s Similarity (S+WMS)\n\n|Dataset | Dataset Size | Text Length (Average) | Number of Annotations (Total) | Annotation Type | Annotation Procedure |\n| :------------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| Toronto Book Corpus |  11,038 books | ~ 6,704 sentences | - | - |\n| BookCorpusOpen |  17,868 books | - | - | - | - |\n| ELTeC |  1,250 novels | - | - | - | - |\n| GLUCOSE |  4,881 stories | 5 sentences |  ~670K | Commonsense Causal Knowledge | Human-Crowdsourced (MTurk) | \n| ROCStories |  49,255 stories | 5 sentences | - | Causal & Temporal Span | Human-Crowdsourced (MTurk)  |\n| ROCStories Winter 2017 |  98,159 stories | 5 sentences | - | Causal & Temporal Span | Human-Crowdsourced (MTurk) |\n| Possible Stories | 1,313 passages  | 46.3 tokens | 8,885 ending & 4,533 questions | Alternative Ending & Causal Question  | Human-Crowdsourced (MTurk) |\n| COPA | 1K questions  | 1 sentence | 2 Alternatives for each question | the more Plausible Alternatives  | Human |\n| TIMETRAVEL |  29,849 story rewritings | 5 sentences | 81,407 counterfactual branch | Counterfactual Rewritings | Human-Crowdsourced (MTurk) |\n| HellaSwag | 70K passage  | 1 sentence | adversarial wrong answers for each passage | Counterfactual Reasoning | Machine-generated  |\n| StoryCommonsense | 15K stories | 5 sentences | 55,747 w/motiv & 104,930 w/emot| Motivations & Emotional Reactions | Human |\n| TVShowGuess | 318 characters | 137,568 tokens | 12,413 scene | Character Facts | Human-2 experts |\n| PERSONET | 33 books | 11,876 sentences |110,114 notes that contain 140,268 traits | Personalities | Automatic collection (notes from reading apps) + Human written (Crowdsourced)  |\n| LitBank | 100 fictions | 2,105.3 tokens | 29,103 mentions | Anaphoric Reference & Entity Category| Human-3 experts |\n| Phrase Detectives | 805 documents  | 1,712.4 tokens | 282,558 mentions | Anaphoric Reference | Human-Crowdsourced (game-with-a-purpose) |\n| Wanzare et al. (2019) | 504 stories | 35.74 sentences | label each sentence from 200 scenarios or none | Scenarios & Segmentation | Human-4 student assistants| \n| SNaC | 150 books | 41 sentences | 9.6K Span | Coherence Error Span & Type| Human-3 experts + Human-Crowdsourced (MTurk) |\n| Pustejovsky and Stubbs (2011) | 183 articles | - | - | Temporal Span | Human-3 students |\n| InScript | 910 stories | 12.4 sentences | 62,062 annotation for event & participant over 10 scenarios | Script Structure | Human-Crowdsourced (MTurk) |\n| Hippocorpus | 6,854 Stories |17.6 sentences | -  |Human Recalled Events | Human-Crowdsourced (MTurk) |\n| ESC v0.9 | 258 documents| - | 6,904 temporal relations & 2,265 explanatory relations | Event Relation & Temporal Span | Human-2 experts |\n| DesireDB | 3,680 instances | - | 3,680 | Fulfilment status of the desire expression sentence & Evidence span| Human-Crowdsourced (MTurk) |\n| Moral Stories | 12K stories | - |24K action + 48K consequence | Story Segment & Sentence Categories | Human-Crowdsourced (MTurk) |\n| CSI | 39 episodes (59 cases) |689 sentences per case | -  | Factual/Structural Metadata (perpetrator mentioned + relation to previous cases) | Human-3 students |\n| Shmoop | 231 stories | 112,080 tokens | 7,234 summaries for 7,234 chapters | Segmentation & chapter-level summaries  | Automatic collection (paired summaries and narrative texts from websites) |\n| BookSum | 405 books | 112,885.15 tokens | 405 summaries | Paragraph, chapter, book-level summaries  | Automatic alignment + Human inspection |\n| NARRASUM | 122K narratives | 786 tokens | 122K summaries | Book-level summaries  | Automatic alignment + Human inspection |\n| IDN-Sum | 8 IDN episodes | 3250 sentences | 10k summaries for 10k documents |  Interactive narratives summaries  | Automatic collection (paired summaries and narrative texts from websites) |\n| ABLIT | 868 chapters |154.1 sentences | 868 | Paragraph-level abridged texts  | Automatic alignment (abridged version available) + Human written (1 single author) |\n| CMU Movie Summary | 42,306 movies |176 words (median length) | 29,802 characters matched to actors with public date of birth | Character metadata  | Automatic matching |\n| Zhang et al. (2019b) | 1,036,965 stories | 15,600 words | 29,802 characters |  Character Metadata  & Tropes  | Automatic extraction + Human-100 experts (questionnaires to authors) |\n| LiSCU | 1,220 books | 1431.2 tokens (summary) | 1708 literature summaries + 9499 character descriptions | Character Description  | Automatic creation + Human evaluation (fact coverage, task difficulty)|\n| BookTest | 14,140,82 questions | 522 tokens |  141,408,250 options |  Cloze-form  | Automatic creation |\n| MCTest | 500 stories + 2000 questions | 212 words (150-300 words) | 8000 (4 questions for each story, 4 choices for each question) |  Multiple-choice  | Human-Crowdsourced (MTurk) |\n| Children’s Book Test | 108 books + 687,343 questions | 462.7 /30.7 words | 10 choices for each question | Multiple-choice | Automatic creation |\n| MovieQA | 14,944 questions | 9.3 words | 74,720 answers | Multiple-choice  | Human  |\n| WikiHow | 230,843 articles |579.8 tokens |230,843 summaries |  Subtopics & Free-form Answer  | Automatic collection |\n| MCScript2.0 | 3,487 texts + 19,821 questions |  164.4 / 8.2 tokens | 2 choices for each question |Answer Generation & Multiple-choice | Human-Crowdsourced (MTurk) |\n| Cosmos QA |35,600 paragraphs & questions | 69.4 /10.3 tokens | 4 choices for each question | Multiple-choice | Human-Crowdsourced (MTurk) |\n| NarrativeQA | 1572 documents + 46,765 questions | 61,472 / 9.8 tokens  | 46,765 answers | Free-form Answer | Human-Crowdsourced (MTurk)  |\n| TellMeWhy | 9,636 stories + 30,519 questions | 5 sentences | 3 answers for each question | Free-form Answer | Human-Crowdsourced (MTurk) |\n| FairytaleQA | 278 stories + 10,580 questions | 1401.3 / 3.3 tokens |10,580 answer & ground-truth question pairs | Free-form Answer | Human-5 postgraduate students|\n\n> 2. A task-specific example for the Bayesian estimation of the author's thoughts is necessary. The corresponding discussion was interesting but difficult to understand without being able to trace to a particular task.\n\nThank you for the suggestion. We will provide a running example as follows:\n|Task prompt | Author prompt | Narration | BLEU-1 |\n| :------------: | :-----------: | :-----------: | :-----------: |\n| You can write plots, stories, and novels. You are an expert in tragedy and modernism. You had a life-long obsession with a girl. | The first plot is ‘Nick is invited to Gatsby’s party’. The second plot is ‘Gatsby has an affair with Daisy’. | Nick Carraway, a young man recently moved to New York City to learn about bonds, receives an unexpected invitation. It’s a lavish, handwritten note inviting him to a party at his neighbor's massive mansion. This isn't just any neighbor but Jay Gatsby, a man whose reputation for opulence precedes him… He discovers that Gatsby's ostentatious lifestyle, these grand parties, are designed to attract one particular guest - Daisy Buchanan, Nick's cousin and Gatsby's long-lost love… | 31.4 |\n| You can write plots, stories, and novels. You are an expert in fantasy fiction. You spent your childhood in Derbyshire, UK. | The first plot is ‘Nick is invited to Gatsby’s party’. The second plot is ‘Gatsby has an affair with Daisy’. | In the tranquil town of Lexington, known for its enchanting beauty and subtle charm, resides a mysterious man named Gatsby… everyone except for Nick, a humble middle-class man, who has always viewed Gatsby's world as an outsider. Nick, despite living in the same city, leads an ordinary life until one fateful day when an invitation to Gatsby's party lands on his doorstep… At the party, he meets Daisy, Gatsby's old flame, who is known for her ethereal beauty and elegance. Little does Nick know, Gatsby has been having a secret affair with Daisy, rekindling their past love. Their affair is the best-kept secret in Lexington, concealed beneath the glamour and glitz of Gatsby's extravagant lifestyle. | 27.1 |\n| You are an expert in fantasy fiction. You spent your childhood in Derbyshire, UK. You can write plots, stories, and novels.| The first plot is 'Charlie’s grandpa introduced a chocolate factory to him’. The second plot is ‘The factory owner invites Charlie to visit the factory’.|  Living in a small town of England, young Charlie Bucket's world is filled with modesty. His only escape from everyday life is through the stories told by his Grandpa Joe about an enchanting chocolate factory that once stood as the heart and soul of their town… one day the eccentric chocolatier reveals that he has decided to open up the factory once again. More incredibly, he invites Grandma Josephine to return as his assistant because of her exceptional talent and experience. However, age has taken a toll on Grandma Josephine's health. So she proposes Charlie to take her place instead… | 29.3 |\n\nHere, the narration is generated by an LLM model that simulates the author’s composition process. Task and sketch (refer to line 642) are parameterized by generative models. It could be inferred that the first line of prompts could generate the plots of The Great Gatsby and the third line of prompts correspond to Charlie and the Chocolate Factory.\n\n> 3. Line 34-37: \"This is supported ... English speaker\". How does this result from cognitive research support the previous statement about LLMs lacking the ability to capture the author's thoughts?\n\nThis indicates a dissociation between language proficiency and thoughts in the human brain. Consequently, while LLMs might excel in adhering to linguistic rules, their competence might be limited in cognitive aspects such as logical reasoning and situation modelling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of this paper is the discussion surrounding the future directions for narrative understanding tasks, especially the proposal of a Bayesian estimation for the author's thoughts. Additionally, this paper can serve as a good resource for researchers to find an index of narrative understanding datasets.", "reasons_to_reject": "Being a survey paper, the authors could have strengthened their contribution with a more fine-grained categorization of the tasks and datasets. For example, dataset size, narrative text length, annotation procedure (expert vs crowdsourced, human vs automatic), domain, period, author count, language, evaluation, etc.\n\nA task-specific example for the Bayesian estimation of the author's thoughts is necessary. The corresponding discussion was interesting but difficult to understand without being able to trace to a particular task.", "questions_to_authors": "Line 34-37: \"This is supported ... English speaker\".\nHow does this result from cognitive research support the support the previous statement about LLMs lacking the ability to capture the author's thoughts?", "paper_topic_and_main_contributions": "The paper surveys the narrative understanding task along the lines of reading comprehension, summarization and question answering. The authors postulate that narrative understanding is the task of finding the author's latent thoughts that generated the story. The work suggests that we can estimate these latent thoughts with task descriptions and prompts.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 305: recapitulation -> recapitulating"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers collectively see merit in this comprehensive survey, both in terms of the content and style of the paper. The main issues seem to be related to the 8-page length restriction. Because of this restriction, more detailed descriptions, analyses, and examples are lacking, which may be added if the paper is revised as a journal article.

Paper Topic And Main Contributions:

This paper provides a comprehensive survey of narrative understanding tasks, encompassing reading comprehension, summarization, and question answering. It explores various aspects of narrative understanding, including taxonomy, datasets, evaluation measures, and potential applications. Additionally, the authors propose a novel framework leveraging Large Language Models (LLMs) to tackle narrative understanding tasks, emphasizing the estimation of latent author thoughts using task descriptions and prompts.

Reasons To Accept:

* Comprehensive Survey, Useful Resource, and Insights for Future Directions
    * The paper provides an in-depth and comprehensive survey of narrative understanding tasks, making it a valuable resource for both new and experienced researchers in the field.
    * The paper serves as a useful resource for researchers by providing an index of narrative understanding datasets, facilitating easy access to essential research materials in the domain.
    * The paper offers valuable insights into the future directions of narrative understanding tasks, including the proposal of a Bayesian estimation approach for understanding the author's thoughts. This forward-looking perspective adds significant value to the paper.
* Clarity and Accessibility
    * The write-up is clear and easy to follow, enhancing its accessibility to a wide range of readers.

Reasons To Reject:
* Maybe Better Suited as a Journal Article: 
    * Because the paper attempts to cover a broad range of narrative understanding topics, this exceeds the depth that can be provided in an 8-page conference paper. This may suggest the need for a narrower focus or a submission as a comprehensive journal paper.
* Issues with the Proposed Framework: 
    * The paper introduces a framework for narrative understanding but does not provide any evaluations or practical implementations of this framework, which leaves its effectiveness unproven.
    * The authors could have strengthened the paper by providing a more fine-grained categorization of tasks and datasets, considering factors such as dataset size, annotation procedure, domain, period, language, etc., which would have added depth to the survey.
* Need for More Analyses and Examples: 
    * The paper lacks quantitative and qualitative studies that would allow for objective comparisons among the works discussed. This absence of empirical evaluation or analysis reduces the paper's contribution.
    * While the paper discusses the Bayesian estimation of the author's thoughts, it lacks a concrete task-specific example, making it challenging for readers to understand how this concept can be applied in practice.
Decision: Accept-Findings