SlowGuess/NAACL_2022_OCR huggingface dataset:
TVSHOWGUESS: Character Comprehension in Stories as Speaker Guessing
Yisi Sang $^{1*}$  Xiangyang Mou $^{2*}$  Mo Yu $^{3*}$  Shunyu Yao $^{4}$  Jing Li $^{5}$  Jeffrey Stanton $^{1}$
$^{1}$ Syracuse University  $^{2}$ Rensselaer Polytechnic Institute  $^{3}$ Pattern Recognition Center, WeChat AI
$^{4}$ Princeton University  $^{5}$ New Jersey Institute of Technology
yisang@syr.edu moux4@rpi.edu moyumyu@tencent.com
We propose a new task for assessing machines' ability to understand fictional characters in narrative stories. The task, TVSHOWGUESS, builds on the scripts of TV series and takes the form of guessing the anonymous main characters based on the backgrounds of the scenes and dialogues. Our human study supports that this form of task covers comprehension of multiple types of character persona, including understanding characters' personalities, facts and memories of personal experience, which are well aligned with the psychological and literary theories about the theory of mind (ToM) of human beings on understanding fictional characters during reading. We further propose new model architectures to support the contextualized encoding of long scene texts. Experiments show that our proposed approaches significantly outperform baselines, yet still largely lag behind the (nearly perfect) human performance. Our work serves as a first step toward the goal of narrative character comprehension.
Stories have two essential elements, plots and characters (McKee, 1997). Character comprehension has been widely recognized as key to understanding stories in psychological, literary and educational research (Bower and Morrow, 1990; Kennedy et al., 2013; Currie, 2009; Paris and Paris, 2003). When reading stories, humans build mental models for characters based on their personae, which helps people to explain a character's emotional status (Gernsbacher et al., 1998), identity, understand future behaviors (Mead, 1990), and even make counterfactual inferences for stories about that character (Fiske et al., 1979).
The ultimate goal of creating a character comprehension system is to equip a machine with prac
Figure 1: A scene example from TVSHOWGUESS. The character Amy can be determined within the scene or with the fact of her relationship; while guessing Sheldon would require memory of the character from previous episodes.
tical capabilities that emulate what humans can accomplish. For example, understanding personae can facilitate story memorization and generation of new statements consistent with the story (Riedl and Young, 2010). Such capabilities could be valuable in the construction of dialog engines that help people address practical problems such as those experienced in customer service encounters (Mairesse and Walker, 2007; Zhang et al., 2018; Urbanek et al., 2019). More importantly, understanding the persona of a particular person can help chatbots to understand the intention behind a human user's language (Bender and Koller, 2020), which can lead to better services and ultimately give systems the ability to demonstrate behaviors that users interpret as empathetic. For instance, Amy's last sentence in Figure 1 is a joking braggadocio to remind her boyfriend to value her more. Only when Sheldon understood the facts of their relationship as a couple and Amy's temporary show-off mentality could he see her true intentions.[2]
Despite the importance of this capability, there
*Authors contributed equally to this paper. Mo Yu is the corresponding author.
1Our code and data are released at https://github. com/YisiSang/TVSHOWGUESS.
2But he cannot
Human Language Technologies, pages 4267 - 4287
has been limited attention to modeling characters in stories in the natural language processing (NLP) community. $^3$  Most existing character-centric prediction tasks have the input sources in expository text such as synopsis (summaries) of stories (Brahman et al., 2021) or non-narrative dialogues (Zhang et al., 2018; Urbanek et al., 2019; Li et al., 2020). A few exceptions work on stories, but focus on limited aspects of personae, such as facts for coreference resolution (Chen and Choi, 2016), personality (Bamman et al., 2013; Flekova and Gurevych, 2015) and character relationships (Iyyer et al., 2016), with only a few Chen and Choi (2016); Flekova and Gurevych (2015) providing evaluation benchmarks. Besides the limited persona aspect coverage, these models also lack the ability to take into account a theory of mind (ToM) which is the knowledge of epistemic mental states that humans use to describe, predict, and explain behavior (Baron-Cohen, 1997).
In this work, we propose the first task on character comprehension in stories to assess the ability of mental model construction in NLP. A character's words are her direct reflection of the contexts, conditioned on her character model (Holtgraves, 2010). Our task, TVSHOWGUESS (TVSG), aims to guess anonymous speakers using dialogues, scene descriptions and historical scenes, which requires models to interpret the behavior of characters in the form of dialogues, which meets the requirements for the evaluation of ToMs.
Through experiments and human studies we found the following results. First, human performance was nearly perfect, while the model performed poorly. Second, although our TVSG has a simple task setup, it has a surprisingly wide coverage of persona understanding skills including the linguistic styles, personality types, factoids, personal relations, and the memories of characters' previous experience. Third, most of the cases  $(>60\%)$  require identification and understanding of characters' historical experiences to resolve. Among them, many rely on facts of characters that are not explicitly described in texts but need to be inferred from event history. The wide persona coverage and heavy dependency on history present challenges to existing NLP techniques. These challenges lead to more than  $20\%$  accuracy gap between our baselines and humans.
Specifically, we make the following contributions. (1) We propose the research direction of character comprehension in stories; with an extended survey (Section 2 and Appendix A) discussing the differences and challenges compared to related work. (2) We propose the first task and dataset for multi-aspect persona (especially ToM) understanding in stories (Section 3). (3) We propose a new schema to analyze the required evidence for character understanding; and conduct human studies to analyze the required skills of our task (Section 4 and Appendix C). (4) We propose new model architectures as the initial step of this direction; and conduct comprehensive experiments to provide insights to future work (Section 5 and 6).
We discuss and compare the following related areas: the assessment benchmarks for general narrative comprehension skills and the tasks specifically designed for character-centered predictions over narratives and non-narratives. Table 1 gives a summary of these narrative comprehension tasks, associated with their required comprehension skills. We also reviewed studies on character-centered tasks over non-narrative texts like synopses and chit-chat (i.e., not story-related) conversations. Appendix A discusses detailed rationales of their required skills.
Assessment of Narrative Comprehension There are many forms of reading comprehension tasks such as cloze tests (Bajgar et al., 2016; Ma et al., 2018), question answering (Richardson et al., 2013; Kočisky et al., 2018; Yang and Choi, 2019; Lal et al., 2021; Xu et al., 2022), and text summarization (Ladhak et al., 2020; Krysciński et al., 2021; Chen et al., 2021). Most of these tasks are built on very short stories or can be solved in segments of a story, thus presenting limited challenges to understanding the elements, especially the characters, of the story. The exceptions are NarrativeQA (Kočisky et al., 2018) and the three summarization tasks, which are mainly event-centric tasks focusing on understanding the plot structures in stories. The NarrativeQA consists of a small portion of character-related questions according to the human study in (Mou et al., 2021), but mainly about simple facts of characters like age, place of birth and profession. Finally, text games (Hausknecht et al., 2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories. Studies have been conducted (Guo
3In contrast, plot comprehension is a popular NLP topic, especially on event structures (Finlayson, 2012; Elsner, 2012; Sims et al., 2019; Lal et al., 2021; Han et al., 2021).
Dataset	Task Format	Narrative Type	Assessed Narrative Comprehension Skills
Source	Length	Plot Structures	Character Facts	Character ToMs
MCTest	Multi-choice QA	Short fiction (Children stories)	~20*	✓
BookTest	Cloze test	Literature (Excerpt)	-	✓
(Ma et al., 2018)	Cloze test	TV show transcripts (Scenes)	~20	✓
NarrativeQA	Generative QA	Movie Scripts, Literature (Full stories)	~11K*	✓	✓
FriendsQA	Extractive QA	TV show transcripts (Scenes)	~20*	✓	✓
NovelChapters/BookSum	Summarization	Literature (Chapters or Full stories)	~4K	✓
SummScreen	Summarization	TV show transcripts (Scenes)	~330	✓
(Chen and Choi, 2016)/ (Chen et al., 2017b)	Coref Resolution	TV show transcripts (Episodes or scenes)	~20/260†	✓	✓
(Flekova and Gurevych, 2015)	Classification	Literature (Full stories)	~22K	✓
TVSHOWGUESS	Multi-choice	TV show transcripts (Full stories)	~50K	✓‡	✓	✓
Table 1: Properties of existing narrative comprehension datasets compared to TVSHOWGUEss. * Numbers are not reported in the original paper so we calculated them from the dataset. †(Chen et al., 2017b) proposes two settings: single scene and the whole episode. ‡Our task requires reasoning based on history scenes, which is a form of plot understanding.
et al., 2020; Yao et al., 2021) to investigate the roles reading comprehension plays in these games.
Character-Centric Prediction over Narratives The task of coreference resolution of story characters (Chen and Choi, 2016; Chen et al., 2017a) is most closely related to our TVSHOWGUESS. These tasks focus on identifying the characters mentioned in multiparty conversations, which mainly requires the understanding of discourse relations and assessment of personal facts. However, coreference does not assess the modeling of the character's theory-of-mind, especially the character's memories, as there are no predictions of character behaviors involved. The prediction of fictional characters' personality types by reading the original stories (Flekova and Gurevych, 2015) is another character-centric task related to the present work. The work covers only the character's personality such as the big five and the MBTI types, also a perspective of the persona our work considers.
Character-Centric Prediction over Non-Narratives Many tasks use the story summary instead of the original story. The textual entailment task LiSCU (Brahman et al., 2021) links an anonymous character summary to the name in the story summary. Using summaries precludes ToM modeling, as discussed in Appendix A.1. Personalized dialogue generation benchmarks (Mairesse and Walker, 2007; Walker et al., 2012; Zhang et al., 2018; Urbanek et al., 2019; Li et al., 2020) are based on daily chit-chats. They usually cover a single facet of multi-dimensional personae (Moore et al., 2017), e.g., personal facts (Zhang et al., 2018) or personality types (Mairesse and Walker, 2007; Li et al., 2020). The LIGHT environment (Urbanek et al., 2019) covers both facts and personalities. None of them covers a comprehensive persona like ours, especially how a character's past experience builds her ToM.
Authorship attribution has a parallel goal to ours, insofar as it aims at guessing author identities from the texts they wrote (Ni et al., 2019; Andrews and Bishop, 2019; Bevendorff et al., 2020). These tasks differ from ours in two respects: first, multiple prose examples generated by the same author do not usually form consecutive plot lines, and second, they rarely model the event history of depicted characters. On this basis, they mainly require understanding authors' writing styles rather than building mental models of facts, events, and experiences.
3 Our TVSHOWGUESS Benchmark
TVSG adopts a multi-choice setting. The goal is to guess the anonymous speakers who are the main characters (maximum number of 6 for each show) in the scene. The models are provided with an anonymous scene that consists of  $n$  lines  $\tilde{S}^{(t)} = \{\tilde{s}_1^{(t)},\tilde{s}_2^{(t)},\dots,\tilde{s}_n^{(t)}\}$  ( $t$  stands for the  $t$ -th scene in the entire show). Each line  $\tilde{s}_i$  can either be a dialogue turn or a background description. When the line is a dialogue turn, it is associated with an anonymous speaker ID (with the form of  $P_x$ ,  $1 \leq x \leq 6$ ) of a main character, or the real name of a supporting character. Similarly, we introduce the notation of the standard scene  $S^{(t)} = \{s_1^{(t)},s_2^{(t)},\dots,s_n^{(t)}\}$ , which has the same definition as the anonymous scenes, with the only difference that the dialogue turns always have their real names of speakers associated.
The anonymous scene  $\tilde{S}^{(t)}$  is associated with a candidate set  $\mathcal{C}^{(t)} = c_1^{(t)},\dots,c_k^{(t)},k\leq 6$  where each character  $c_{j}^{(t)}$  is a main character who appears in  $\mathcal{S}$ . The goal is thus predicting each  $\mathrm{P}_x$ 's actual role  $c_{j}^{(t)}$ , i.e., a match  $\pi (\cdot)$  from the anonymous IDs to the real characters, conditioned on the scene  $\tilde{S}^{(t)}$  and all previous scenes  $S^{(1:t - 1)}$ :
$$
P \left(\mathrm {P} _ {x} = c _ {j} ^ {(t)} \mid \tilde {S} ^ {(t)}, S ^ {(1: t - 1)}\right). \tag {1}
$$
We use community contributed transcripts from The TV MegaSite (TMS)<sup>4</sup> like (Chen et al., 2021). Our scenes are from the scripts of five popular TV series: Friends, The Big Bang Theory (TBBT), The Office, Frazier and Gilmore Girls.
Data Cleaning Our data consists of character dialogues and background descriptions. The dialogues start with the characters' names. One or more turns of dialogue between characters comprise a scene. Scenes are separated by short background cus that begin with markers such as location (e.g. "Howard's car", "Kingman Police Station"), special words (e.g., "Scene", "Cut"), or symbols (e.g. "[ J"). We created a rule-based parser which splits the content of an episode into multiple independent scenes using scene separation markers.
Character Recognition and Anonymization We used the main characters' names to identify their dialogues within each scene and randomly labeled them with speaker IDs (e.g., P0, P1). Since the different names of the characters, such as nicknames, first names, and last names, mark the dialogue in a mixed way, in order to match the lines to the correct speaker, we first identified the main characters in each TV series by consulting Fandom's cast list. Then, we calculated the frequency of speech to find references to the same main character's name.
4 Analysis of Our Benchmark
We propose a comprehensive schema of persona types for the machine narrative comprehension. The schema facilitates the analysis of the challenges in our task; and provides insights into the deficiencies in current narrative comprehension models, by allowing a decomposition of model performance to the dimensions of categories (Section 6).
4.1 Our Annotation Schema for Human Study
Two researchers with backgrounds in psychology, linguistics, NLP, and education developed and tested an inductive coding method based on the methods of grounded theory (Glaser and Strauss, 2017). They conducted three rounds of independent annotation and discussion of the evidence needed to identify the characters, using 10 randomly selected scenes for each round. After each discussion, they updated the codebook accordingly.
Modifications to the codebook led to the achievement of saturation during the process. Then the two researchers coded a total of 318 characters from 105 scenes of Friends and The Big Bang Theory. The annotation interface appears in Appendix B.
This schema categorizes the required evidence to resolve the task into four persona data types: linguistic style, personality, fact, memory. Table 4 reports inter-rater reliability calculated by Cohen's Kappa (Cohen, 1960). The kappa values range from 0.76 to 0.87 and would all be considered satisfactory (Viera et al., 2005), reflecting the success of our codebook and process.
We have one additional type, inside-scene, referring to tasks that can be resolved purely within local contexts, thus not requiring persona understanding. To better depict how these pieces of evidence are used in human rationales, we added two complementary categories: (1) how the task instance relies on the history scenes, and (2) when there are multiple pieces of evidence required, what types of reasoning skills are used to derive the answer from the evidence (see Section C). Table 6 shows the definitions of each evidence type. We provide examples of each evidence type in Section B.2.
4.1.1 Major Evidence Types
Linguistic Style Personalized language patterns that reflect individual differences in self-expression and are consistently reliable over time and situations (Pennebaker and King, 1999).
Personality Stable individual characteristics (Vinciarelli and Mohammadi, 2014) that can distinguish "internal properties of the person from overt behaviors" (Matthews et al., 2003).
Memory A character's episodic memory of events from previous episodes and the semantic memory<sup>5</sup> inferred from events. Note here we want to model the memory of a particular character, i.e., the historical scenes experienced by the particular character instead of from the audiences' perspective. A character's memory is crucial for humans to build her ToM, but is largely ignored as a part of persona in previous research.
Fact The truth about characters as opposed to interpretation, which can usually be represented as knowledge triples.
5Semantic memory is the characters' general world knowledge that they accumulate over time (Reisberg, 2013). Episodic memory, on the other hand, is the characters' memory of specific experiences in their lives (Tulving, 2002)
<sup>4</sup>http://tvmegasite.net.
Show	train	dev	test	#tokens per utterance	#tokens per scene	#tokens per character
avg	max	avg	max	avg	max
Friends	2,418	210	211	21	350	862	6,817	190,932	516,191
TBBT	1,791	130	130	19	364	414	6,051	167,027	183,748
Frasier	1,368	140	141	16	363	812	14,276	165,483	475,372
Gilmore_Girls	1,495	141	142	19	336	360	4,572	105,723	214,779
The_Office	3,699	198	199	19	338	123	1,660	58,676	132,992
total	10,771	819	823	18	364	371	14,276	137,568	516,191
Table 2: Statistics of our TVSHOWGUESS. The numbers in the first 3 columns refer to the total numbers of scenes.
- Attribute All explicitly provided factual character identity information in the TV series setting, such as race, occupation, and education level.
- Relationship Relationship includes social relationships (e.g., husband and wife) and dramatic relationships (e.g., arch-enemy). When talking to people with different relationships, characters change their identity masks by using different words (Gergen, 1972).
- Status The emotional or psychological status of a character when facing a specific situation.
Inside-Scene The textual evidence inside the scene, independent from the character's persona.
- Background Background introduction and descriptions in other character dialogues.
- Mention The character's name or alias is called out by other characters. Although mention is persona-independent, it still presents challenging cases. In a multi-person multi-round dialog, because anaphora in the current sentence may not bound to an antecedent on the right frontier of the structure, common sense analysis of conversational coherence is needed to determine which speaker is being referred to.
Exclusion A guessing technique for elimination using a given list of characters which is neither evidence nor inference, but it depends on the character list provided within the scene, so we include it as a subcategory of inside-scene evidence.
4.1.2 Dependence on History
To understand how much humans rely on memory to identify a character, the annotators coded whether the evidence necessary to solve the task depends directly on historical events or indirectly on history by abstracting from historical events.
Direct Dependency Characters that can only be
identified through events that are explicitly expressed in previous episodes, for example:
Background: (from TBBT) [The stairwell]
Candidates: {Leonard, Penny}
P0: There's something I wanted to run past you.
P1: What's up?
P0: Mm, the guys and I were thinking about investing in Stuart's comic book store. Is that okay?
P1: Why are you asking me?
Answer: P0  $\rightarrow$  Leonard
Rationale: In a previous scene, Leonard and his friends discussed investing in Stuart's store. He is the only one between the two who has this memory.
Indirect Dependency Characters can only be identified using evidence not explicitly expressed in previous episodes, but inducible from previous events. For example, Personality can be inferred from the character's previous behavior.[7]
Background: (from Friends) [Central Perk]  
Candidates: {Joey, Rachel, Ross}
P0: Here you are (Hands Rachel a cup of coffee)
P1: Thank you Joey. You know what? I'm not even sure I can have caffeine.
P2: I went thru this with Ben and Carol. One cup of coffee won't affect your milk.
P1: Yeah. Just to be sure I'm gonna call Dr. Wiener.
Answer:  $\mathrm{P2} \rightarrow \mathrm{Ross}$
Rationale: There is no actual scene with Ross going through this with Carol; the answer is inferred based on Ross' known relations to Ben (parent-child) and Carol (ex-spouse). Thus the evidence about Ross has indirect dependency on scene history.
If the answer can be inferred within the scene, like answering  $\mathbf{P0} \rightarrow \mathbf{Joey}$  in the above example. We have a special rule on the Exclusion evidence type – If a character can only be inferred on the basis of other characters being eliminated, it should have dependency type labeled if any of the other character has a history dependency. In other words, when guessing the identity with Exclusion requires history dependency on another character, the dependency type is transitive.
<sup>6</sup>If a character can be identified with evidence of both Memory and Inside-Scene, it will be labeled as No-Dependency.
The annotation of indirect dependency is very subjective as different annotators may have memory of previous scenes and use different evidence to guess the character.
Evidence Type	Friends(%)	TBBT(%)
\( \widehat{\Xi } \)	Ling. Style	0.66	9.93
Personality	7.28	21.85
Fact	20.53	33.12
(Attribute)	2.65	8.61
(Relation)	16.56	22.52
(Status)	1.32	1.99
Memory	36.42	27.15
Inside-Background	33.11	12.58
Inside-Mention	15.23	15.23
Exclusion	8.61	22.52
Dependence of Hist.	Friends(%)	TBBT(%)
\( \widehat{\Xi } \)	No Dep.	53.64	32.45
Direct Dep.	26.49	36.42
Indirect Dep.	19.87	31.13
Table 3: Percentage of the required evidence types in the two TV shows, Friends and The Big Bang Theory.
Category	κ(%)
Evidence type
Coarse-grained types	81.53
Fine-grained types	80.99
Dependence of history
Direct dependence only	82.02
All dependency types	75.51
Reasoning Type†	87.21
Table 4: Annotation agreement. †: see our extended study in Appendix C. We list the number for reference.
Main Statistics Table 3 shows the proportions of the required evidence types and dependency of history. According to the statistics, history is an important factor in guessing the characters.  $46.36\%$  of the examples from Friends and  $67.55\%$  examples from the Big Bang Theory require history.
Human Performance Accuracy One annotator (who had not watched the seasons under evaluation) obtained nearly perfect accuracy in guessing the characters in FRIENDS  $(98.68\%)$ , and a lower but still good accuracy in TBBT  $(89.82\%)$ . A second annotator (who had watched all episodes and thus would be considered an expert) confirmed that most the error cases were unsolvable given only data extracted from scenes. We list the unsolvable cases and human mistakes in Appendix E.