SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Reisberg. 2013. The Oxford handbook of cognitive psychology. Oxford University Press.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 193-203.
Mark O Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. Journal of Artificial Intelligence Research, 39:217-268.
Matthew Sims, Jong Ho Park, and David Bamman. 2019. Literary event detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3623-3634.
Endel Tulving. 2002. Episodic memory: From mind to brain. Annual review of psychology, 53(1):1-25.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Roektaschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094.
Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med, 37(5):360-363.
Alessandro Vinciarelli and Gelareh Mohammadi. 2014. A survey of personality computing. IEEE Transactions on Affective Computing, 5(3):273-291.
Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hirst. 2019. Are fictional voices distinguishable? classifying character voices in modern drama. In Proceedings of the 3rd joint SIGHUM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature, pages 29-34.
Marilyn A Walker, Grace I Lin, Jennifer Sawyer, et al. 2012. An annotated corpus of film dialogue for learning and characterizing character style. In LREC, pages 1373-1378.
Martin Warren. 2006. Features of naturalness in conversation, volume 152. John Benjamins Publishing.
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, et al. 2022. Fantastic questions and where to find them: Fairy Taleqa-an authentic dataset for narrative comprehension. arXiv preprint arXiv:2203.13947.
Zhengzhe Yang and Jinho D Choi. 2019. *Friendsqa: Open-domain question answering on tv show transcripts*. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 188-197.
Shunyu Yao, Karthik Narasimhan, and Matthew Hausknecht. 2021. Reading and acting while blindfolded: The need for semantics in text game agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3097-3102.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxi Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9693-9700.
A A Detailed Survey of Related Work
In this section, we first give an in-depth analysis on the difference between narrative and synopsis, from both the empirical challenges in NLP studies and the linguistic theory from (Morrow, 1985). Then we provide detailed discussion on how we summarize related work in Table 8.
A.1 Background: Narrative versus Synopsis
As our work focuses on narrative comprehension, following setups like (Kočisky et al., 2018; Kryscinski et al., 2021; Chen et al., 2021), it is necessary to distinguish between comprehension of original narrative stories versus comprehension of their synopses (the human-written plot summaries), e.g., from the story's Wikipedia page.
Narrative stories are told by creating scenes, with the goal of helping readers experience events as they occur in the plot, and empathize with the story characters in relation to their own experiences. To engage readers, story writers often use complex narrative clues (e.g., character activities, event development, scenery changes); variable narrative sequences (e.g., narrative, flashback, interpolation); and various linguistic expressions (e.g., argument, lyricism, narrative, description, illustration). By comparison, a synopsis is a descriptive summary of the main idea of a story in plain language. It contains only the main characters, time, place, important plot points, and resolution, rather than allowing the story to unfold through the actions of the characters. The goal of the synopsis is to inform readers what happened with little or no original material from original story.
Therefore, comprehension of narrative stories requires more sophisticated skills to understand the complex chain of clues and expressions, in order to finally build a complete narrative representation from a sequence of individual scene comprehensions along with a developed understanding of characters' mental models (Morrow, 1985). In this light, a synopsis represents "processed results" from the application of these comprehension skills by a (experienced) human reader.
A.2 Background: Dialogue in Stories vs. Real-Life Conversation
Fictional dialogue canonically serves a purpose in a narrative. Either i contributes to the develop of a character or advances the plot (McHale, 2004). Nash proposed three categories of fictional dia
logue: (1) confrontational dialogue which “includes challenges, quarrels, disputes, interviews, and any kind of personal encounter in which the participants are in covert or overt opposition to each other”; (2) instructional dialogue, which “conveys information about matters of science, technology, politics, world events, etc, some knowledge of which is essential to understanding the plot”; (3) collaborative dialogue that consists of “a series of exchanges which cumulatively present, for the reader’s benefit, a picture of events, histories, personalities, and relationships” (Nash, 1990). Authors have total control of the fictional dialogue, and ideally it functions according to the author’s intention at every part of the story (De Haan, 1996). However, real-life conversation is a joint action and is natural. There is no individual has whole control and the conversation goal of each party may be very different. Additionally, real-life conversations are temporally linear, such that communicators cannot revise earlier speech to fit a story. Moreover, real-life conversations tend to be highly implicit because spoken language derives much of its meaning from context (Warren, 2006).
A.3 Assessment of Narrative Comprehension
We summarize the related tasks people use for assessment of general narrative comprehension skills.
Cloze Test Cloze tests take a snippet of the original text with some pieces (usually entities) masked as blanks, with the goal of filling these blanks from a list of candidates. Cloze tests can be automatically constructed, resulting in an advantage in creation of large scale datasets. Examples of cloze tests for narrative comprehension assessments are BookTest (Bajgar et al., 2016) and (Ma et al., 2018). Both datasets are based on excerpts of books or scenes of TV shows. As the input consists only of short paragraphs, there is not sufficient information to infer complex character set via reading the stories. Therefore, these datasets address few questions related to the understanding of characters. $^{8}$
Moreover, when built on short snippets, cloze tests are known to prone to mostly local inference but not much reasoning and commonsense
There may be a possible confusion of these tasks and ours, as cloze tests also include filling in anonymous character names in the blanks. However, in these tasks, the required answers are also anonymized character IDs that appear in the inputs, and the IDs for the same character are random across different scenes. Therefore the character's information is not available for learning by design. In other words, their design of tasks deliberately prevent the task of character understanding.
Dataset	Task Format	Narrative Type Source	Length	Assessed Narrative Comprehension Skills	Assessed Commonsense Knowledge
Plot Structures	Character Facts	Character ToMs	Concepts	Events/States	Story Flows
MCTest	Multi-choice QA	Short fiction (Children stories)	~20*	✓	✓	✓	✓
BookTest	Cloze test	Literature (Excerpt)	-	✓
(Ma et al., 2018)	Cloze test	TV show transcripts (Scenes)	~20	✓
NarrativeQA	Generative QA	Movie Scripts, Literature (Full stories)	~11K*	✓	✓	✓
FriendsQA	Extractive QA	TV show transcripts (Scenes)	~20*	✓	✓
TellMeWhy	Multi-choice QA	Short fiction (ROCStories)	5	✓
NovelChapters/BookSum	Summarization	Literature (Chapters or Full stories)	~4K	✓	✓
SummScreen	Summarization	TV show transcripts (Scenes)	~330	✓	✓
(Chen and Choi, 2016) / (Chen et al., 2017b)	Coref Resolution	TV show transcripts (Episodes or scenes)	~20/260†	✓	✓	✓	✓
(Flekova and Gurevych, 2015)	Classification	Literature (Full stories)	~22K	✓
TVSHOWGUESS	Multi-choice	TV show transcripts (Full stories)	~50K	✓ (indirect)	✓	✓	✓	✓	✓
Table 8: Properties of existing narrative comprehension datasets compared to TVSHOWGUESS. We organize the datasets according to the following dimensions related to narrative understanding: Source of the texts for reading comprehension; Length of the texts from the source that makes the task solvable, we report the numbers of sentences or utterances for books and scripts respectively; whether the task assesses the ability of understanding plot structures in the stories; whether the task assesses the ability of understanding basic character facts like personality, profession, etc; whether the task assesses the ability of building character theory-of-mind (ToM); whether the task assesses the commonsense knowledge of concepts, events and states; and whether the task assesses the additional commonsense about the narrative development, including the knowledge about the coherence among non-verbal narratives and dialogues, and how they form the story/plot flow. * Numbers are not reported in the original paper so we calculated them from the dataset. †(Chen et al., 2017b) proposes two settings with single scene and the whole episode as inputs respectively. Different from ours, their include of episode is not to support the in-scene prediction with necessary history, but mostly increase the difficulty level of the co-ref task.
knowledge, as studies in the NLP community suggested (Chen et al., 2016). On the other hand, although our task also has form similar to cloze, it requires information about the characters from previous scenes, which is not only about understanding the characters, but also requires global inference across features of the story (see Figure 1).
Question Answering The most popular form of narrative comprehension evaluation is through question answering, starting from the early work of MCTest (Richardson et al., 2013), to the more recent crowd-sourced tasks like NarrativeQA (Kočisky et al., 2018), FriendsQA (Yang and Choi, 2019), TellMeWhy (Lal et al., 2021) and Fairy Tale QA (Xu et al., 2022).
Among them, the MCTest and TellMeWhy conduct multi-choice question answering on short stories. As above, te input consists of short paragraphs, so there is not sufficient information to infer complex character facts via reading the stories. Therefore, these datasets also cover few questions assessing the understanding of characters. The TellMe-Why has a specific focus on why-questions assessing the causal knowledge between states and events. The inputs are short stories from the ROCStories dataset (Mostafazadeh et al., 2016). MCTest covers a much wider set of reading skills, as it is based on complete stories, and generates questions with the goal of assessing children's reading comprehension
over both story plots and commonsense.
NarrativeQA and FriendsQA conduct natural question answering tasks. NarrativeQA aims to infer free-form answers to questions about a specific book or movie script. According to the human study from (Mou et al., 2021), the major part of the dataset is event-centric questions, which queries the explicit plots from the original books thus do not require a significant amount of commonsense reasoning. The study also reveals that NarrativeQA consists of a small portion of character-related questions. These questions mainly query the simple facts of characters, such as age and profession. The more complexity character persona types, like personality, emotional/psychological status and history experience studied in our work, are not covered. Similar to NarrativeQA, FriendsQA is a QA task over TV show scripts. The dataset consists of six types of questions: who, what, when, where, why, and how. The who questions target on asking speaker names of utterance contents or participants of events, therefore are mainly assessing understanding of plot structures (i.e., participant arguments of events).
Both NarrativeQA and FriendsQA have human-written questions with a reference of the plot summary, which require evidence explicitly exists in the original story texts, and thus do not have much requirement of reasoning. The FriendsQA ques
tions are based on scene summaries, and thus require mostly local evidence; the NarrativeQA questions are based on the book-level summary, and thus sometimes require the ability to bridge the gap between coarse-grained and fine-grained event descriptions (i.e., commonsense of sub-events).
Summarization There is a recent trend to evaluate model's understanding of stories via summarization, including NovelChapters (Ladhak et al., 2020), BookSum (Krysciński et al., 2021) and ScreenSum (Chen et al., 2021). These works provide a good research opportunity to future story reading research, by showing that book-level or chapter-level summarization is challenging to existing machine reading models. However, it is more difficulty to identify the specific required reading skills by these tasks, as there exist many factors beyond reading skills to generate a good summary, such as encoding and generating long narrative texts. Intuitively, story summarization is largely plot-related instead of character-related; and requires the knowledge to understand the story flow.
Interactive Fiction Game Playing Interactive fiction (IF) games (Hausknecht et al., 2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories as environment observations. Research work has successfully demonstrated that reading comprehension can provide helpful inductive biases for efficient policy learning (Guo et al., 2020); while Yao et al. (2021) also reveal the shortcomings of these games. The debate calls for future investigations to understand the necessary narrative elements and the roles they play in the IF games.
A.4 Character-Centric Prediction over Narratives
Our task can be seen as a character-centered understanding of the narrative, where the understanding of the character deepens the understanding of the story and makes the narrative engaging. There are limited studies on understanding characters' persona from reading stories. In this section we review some existing character-centric prediction tasks over narrative texts, and discuss the relations and differences.
Character Name Linking The task of coreference resolution for story characters (Chen and Choi, 2016; Chen et al., 2017b) is closely related to our TVSHOWGUESS. These coreference resolution fo
cuses on identifying the characters mentioned in multiparty conversations from TV shows scripts. The goal of these tasks is to resolve the coreference of pronouns and character-indicating nominals (e.g., you and Mom) in dialogues of the character names that appear in the local context. It also covers linking a named entity (e.g., Ross) to the character, which is more on name matching instead of character understanding.
The task form of coreference resolution mainly requires the understanding of discourse relations. It does not assess the modeling of character theory-of-mind, especially the character's memories, as there are no predictions of character behaviors involved. The major character persona type it assesses is character facts, since the resolution of nominals requires the understanding of the target characters' occupations and relationships.
The lack of ToM modeling and complex reasoning of the coreference resolution task also makes it relatively easier - on Friends and The Big Bang Theory, a CNN model gives a  $>90\%$  average accuracy. By comparison, our task, although solvable by humans with a  $\sim 95\%$  accuracy, is challenging to neural models as the best BERT-based model gives a  $\sim 65\%$  average accuracy on the same two shows with even smaller candidate sets.
Personality Prediction Our work is also related to the prediction of fiction characters' personality types by reading the stories (Flekova and Gurevych, 2015). Specifically, the tasks require to predict a fiction character's MBTI personality types (Myers and McCauley, 1988) rooted in Jung's theory, based on the character's verbal and non-verbal narratives in the original stories. Compared to the aforementioned character-centric prediction tasks, these studies require to read and comprehend the original long stories, but the prediction task are relatively simpler since they only focus on personality which is a single perspective of persona.
A.5 Character-Centric Prediction over Non-Narratives
Character name linking between story synopses  
Recently Brahman et al. (2021) propose the LiSCU, which is a novel textual entailment task linking an anonymous summative descriptions of story character to the name appearing in the story's plot summary. Similarly to (Chen and Choi, 2016), the task assess the resolution of names and events instead of the ToM modeling. This is because the
task does not involve much explicit behavior predictions, since the task form is entailment between two given statements rather than predicting the possibility of new contents. The usage of synopses over original stories reduces the challenges in narrative understanding; and further prevents the character comprehension from stories, as pointed out by (Kočisky et al., 2018), the summaries themselves are humans' comprehension results of the stories.
Personalized Dialogue Generation Our work is also related to personalized dialogue generation, for which datasets (Mairesse and Walker, 2007; Walker et al., 2012; Zhang et al., 2018; Li et al., 2020) and models (Li et al., 2016; Mazaré et al., 2018; Qian et al., 2018; Zheng et al., 2020) are proposed for generating dialogues for speakers with persona features. These benchmarks usually cover a single aspect of the multidimensional persona (Moore et al., 2017). For example, PERSONA-CHAT (Zhang et al., 2018) focuses on personal facts such as "I'm a writer" and "I live in Springfield"; other works mainly focus on learning language styles from speakers' personality types, such as the Big Five traits of the extraversion personality in PERSONAGE (Mairesse and Walker, 2007), and the personality types derived from TV tropes (e.g. jealous girlfriend, book doom, anti hero) in ALOHA (Li et al., 2020).
LIGHT (Urbanek et al., 2019) is a crowdsourced dataset for text game adventure research. It includes natural language descriptions of fantasy locations, objects and their affordances, characters and their personalities, dialogue and actions of the characters. The biggest difference between ours and LIGHT is that LIGHT is based on the local environment of the conversation, rather than on a story. Examples from the LIGHT dataset are independent conversations and the context in which they occur. Crowd workers created the dialogues of characters by a given setting and a persona. The persona is modeled by the Persona-Chat dataset which is defined as a set of three to five profile sentences describing their personal facts such as "I am a part of a group of travelers" and "I go from town to town selling food to the locals".
To the best of our knowledge, none of the existing studies cover a comprehensive multidimensional persona similar to our work, especially with respect to how a character's past experience builds her ToM.
Authorship Attribution Finally, authorship attribution has parallel ideas to our task, insofar as it aims at guessing author identities from the texts they wrote (Ni et al., 2019; Andrews and Bishop, 2019; Bevendorff et al., 2020) and thus requires a certain degree of author profiling. These tasks differ from ours because the reviews, tweets or fandoms under the same authors do not usually form consecutive plotlines. Therefore, the tasks mainly require to understand the authors' writing styles rather than building mental models from the their past experiences. From this perspective, this direction is in fact more closely related to stylistic analysis in narrative understanding (Vishnubhotla et al., 2019), rather than character understanding.
B Supplementary for the Dataset Analysis
B.1 Summary of the Annotation Schema
We include a summary of our annotation schema in Figure 6.
B.2 Examples of Each Evidence Types
Linguistic Style
Background: (from TBBT) [Amy's car]
Candidates: {Leonard, Penny, Sheldon, Amy}
P0: Whatever. You can't even go on a date without checking your relationship agreement.
P1: If you've got a problem basing a relationship on a contract, I'd like to tell you about 13 plucky colonies that entered a relationship agreement called the U.S. Constitution. And it may not be cool to say so, but I think that love affair is still pretty hot today.
Answer: P1  $\rightarrow$  Leonard
Rationale: (Shelton's language is characterized by the use of long, difficult sentences and references to historical stories.)
Background: (from TBBT) [The cafeteria]
Candidates: {Leonard, Howard, Sheldon, Raj}
P0: And you love the sound of your own voice.
P1: Yeah, well, of course I do. Listen to it. It's like an earful of melted caramel.
Answer: P1  $\rightarrow$  Sheldon
Rationale: (Sheldon is a self-centered person so he will praise his own voice.)
P0: There's something I wanted to run past you.
P1: What's up?
P0: Mm, the guys and I were thinking about investing in Stuart's comic book store. Is that okay?
P1: Why are you asking me?
Rationale: (In a previous scene, Leonard and his friends discussed about investing in Stuart's store, so he is the only one between the two who has this memory.)
Evidence Type	Description
Linguistic style	Linguistic style refers to a character's individualized speech pattern. It consists of a selection of linguistic features such as vocabulary, syntactic patterns, rhythm, and tone. It also includes the use of elements such as direct or indirect, metaphor and irony.
Personality	Personality is a person's stable attitude toward objective facts and the habitual way of behavior that is compatible with it. We adopt a wider definition of personal traits as in (Li et al., 2020).
Fact	Attributes	Fact of a character's attributes in the TV series setting, such as race, profession, education level etc.
Relations	A character's relationship with others that truly exist in the TV series setting, including both social relations and drama role relations.
Status	Facts of a character's temporal emotional or psychological status in the time period when the scene happens.
Memory	The episodic memory about history events a character has in the previous show scenes. This also includes a rare case of a knowledge fact (i.e. the semantic memory) a character acquires from history scenes, which cannot be inferred from the facts of the character.
Inside-scene	Background	The character's identity can be inferred from the background introduction of scene, or from the description of the other characters' words.
Mention	The character's name or alias is called by the other people.
Exclusion	The character's identity can be determined from the presence of characters in the scene and the other resolved characters.
Figure 6: The definitions of evidence types.
- Attribute
Background: (from TBBT) [Amy's lab]
Candidates: {Amy, Penny}
P0: Hey. Ready to go to lunch?
P1: Just give me a minute. I'm stimulating the pleasure cells of this starfish. I just need to turn it off.