Venue: NAACL 2022

# TVShowGuess: Character Comprehension in Stories

as Speaker Guessing

Yisi Sang\({}^{1}\) Xiangyang Mou\({}^{2*}\) Mo Yu\({}^{3*}\) Shunyu Yao\({}^{4}\) Jing Li\({}^{5}\) Jeffrey Stanton\({}^{1}\)

###### Abstract

We propose a new task for assessing machines' ability to understand fictional characters in narrative stories. The task, TVShowGuess, builds on the scripts of TV series and takes the form of guessing the anonymous main characters based on the backgrounds of the scenes and dialogues. Our human study supports that this form of task covers comprehension of multiple types of character persona, including understanding characters' personalities, facts and memories of personal experience, which are well aligned with the psychological and literary theories about the theory of mind (ToM) of human beings on understanding fictional characters during reading. We further propose new model architectures to support the contextualized encoding of long scene texts. Experiments show that our proposed approaches significantly outperform baselines, yet still largely lag behind the (nearly perfect) human performance. Our work serves as a first step toward the goal of narrative character comprehension.1

Footnote 1: Our code and data are released at [https://github.com/YisiSang/TVSHOWGUESS](https://github.com/YisiSang/TVSHOWGUESS).

\begin{tabular}{c c}  & \\ Yisi Sang\({}^{1}\)+ Xiangyang Mou\({}^{2*}\) & Mo Yu\({}^{3*}\) & Shunyu Yao\({}^{4}\) Jing Li\({}^{5}\) Jeffrey Stanton\({}^{1}\) \\   \({}^{1}\)Syracuse University & \({}^{2}\)Rensselaer Polytechnic Institute & \({}^{3}\)Pattern Recognition Center, WeChat AI \\   \({}^{4}\)Princeton University & \({}^{5}\)New Jersey Institute of Technology \\   yissang@syr.edu moux4@rpi.edu moyumyu@tencent.com \\ \end{tabular}

Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4267 - 4287

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Stories have two essential elements, plots and characters McKee (1997). Character comprehension has been widely recognized as key to understanding stories in psychological, literary and educational research Bower and Morrow (1990); Kennedy et al. (2013); Currie (2009); Paris and Paris (2003). When reading stories, humans build mental models for characters based on their personae, which helps people to explain a character's emotional status Gernsbacher et al. (1998), identity, understand future behaviors Mead (1990), and even make counterfactual inferences for stories about that character Fiske et al. (1979).

The ultimate goal of creating a character comprehension system is to equip a machine with practical capabilities that emulate what humans can accomplish. For example, understanding personae can facilitate story memorization and generation of new statements consistent with the story Riedl and Young (2010). Such capabilities could be valuable in the construction of dialog engines that help people address practical problems such as those experienced in customer service encounters Mairesse and Walker (2007); Zhang et al. (2018); Urbanek et al. (2019). More importantly, understanding the persona of a particular person can help chatbots to understand the intention behind a human user's language Bender and Koller (2020), which can lead to better services and ultimately give systems the ability to demonstrate behaviors that users interpret as empathetic. For instance, _Amy_'s last sentence in Figure 1 is a joking braggedocio to remind her boyfriend to value her more. Only when _Sheldon_ understood the facts of their relationship as a couple and _Amv_'s temporary show-off mentality could he see her true intentions.2

Footnote 2: But he cannot #.

Despite the importance of this capability, there

Figure 1: A scene example from TVShowGuess. The character _Any_ can be determined within the scene or with the fact of her relationship; while guessing _Sheldon_ would require memory of the character from previous episodes.



has been limited attention to modeling characters in stories in the natural language processing (NLP) community.3 Most existing character-centric prediction tasks have the input sources in expository text such as synopsis (summaries) of stories Brahman et al. (2021) or non-narrative dialogues Zhang et al. (2018, 2019); Urbanek et al. (2019); Li et al. (2020). A few exceptions work on stories, but focus on limited aspects of personae, such as facts for coreference resolution Chen and Choi (2016), personality Bamman et al. (2013); Flekova and Gurevych (2015) and character relationships Iyyer et al. (2016), with only a few Chen and Choi (2016); Flekova and Gurevych (2015) providing evaluation benchmarks. Besides the limited persona aspect coverage, these models also lack the ability to take into account a theory of mind (**ToM**) which is the knowledge of epistemic mental states that humans use to describe, predict, and explain behavior Baron-Cohen (1997).

Footnote 3: In contrast, plot comprehension is a popular NLP topic, especially on event structures Finlayson (2012); Elsner (2012); Sims et al. (2019); Lal et al. (2021); Han et al. (2021).

In this work, we propose the first task on character comprehension in stories to assess the ability of mental model construction in NLP. A character's words are her direct reflection of the contexts, conditioned on her character model Holtgraves (2010). Our task, **TVShowGuess (TVSG)**, aims to guess anonymous speakers using dialogues, scene descriptions and historical scenes, which requires models to interpret the behavior of characters in the form of dialogues, which meets the requirements for the evaluation of ToMs.

Through experiments and human studies we found the following results. First, human performance was nearly perfect, while the model performed poorly. Second, although our TVSG has a simple task setup, it has a surprisingly _wide coverage of persona understanding skills_ including the linguistic styles, personality types, factoids, personal relations, and the memories of characters' previous experience. Third, most of the cases (\(>\)60%) require _identification and understanding of characters' historical experiences_ to resolve. Among them, many rely on facts of characters that are not explicitly described in texts but need to be inferred from event history. The wide persona coverage and heavy dependency on history present challenges to existing NLP techniques. These challenges lead to more than 20% accuracy gap between our baselines and humans.

Specifically, we make the following contributions. (1) We propose the research direction of character comprehension in stories; with an extended survey (Section 2 and Appendix A) discussing the differences and challenges compared to related work. (2) We propose the first task and dataset for multi-aspect persona (especially ToM) understanding in stories (Section 3). (3) We propose a new schema to analyze the required evidence for character understanding; and conduct human studies to analyze the required skills of our task (Section 4 and Appendix C). (4) We propose new model architectures as the initial step of this direction; and conduct comprehensive experiments to provide insights to future work (Section 5 and 6).

## 2 Related Work

We discuss and compare the following related areas: the assessment benchmarks for general narrative comprehension skills and the tasks specifically designed for character-centered predictions over narratives and non-narratives. Table 1 gives a summary of these narrative comprehension tasks, associated with their required comprehension skills. We also reviewed studies on character-centered tasks over non-narrative texts like synopses and chit-chat (_i.e._, not story-related) conversations. Appendix A discusses detailed rationales of their required skills.

#### Assessment of Narrative Comprehension

There are many forms of reading comprehension tasks such as cloze tests Bajgar et al. (2016); Ma et al. (2018), question answering Richardson et al. (2013); Kocisky et al. (2018); Yang and Choi (2019); Lal et al. (2021); Xu et al. (2022), and text summarization Ladhak et al. (2020); Kryscinski et al. (2021); Chen et al. (2021). Most of these tasks are built on very short stories or can be solved in segments of a story, thus presenting limited challenges to understanding the elements, especially the characters, of the story. The exceptions are NarrativeQA Kocisky et al. (2018) and the three summarization tasks, which are mainly event-centric tasks focusing on understanding the plot structures in stories. The NarrativeQA consists of a small portion of character-related questions according to the human study in Mou et al. (2021), but mainly about simple facts of characters like age, place of birth and profession. Finally, text games Hausknecht et al. (2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories. Studies have been conducted Guo 

et al., 2020; Yao et al., 2021) to investigate the roles reading comprehension plays in these games.

**Character-Centric Prediction over Narratives** The task of coreference resolution of story characters Chen and Choi (2016); Chen et al. (2017) is most closely related to our TVShowGuess. These tasks focus on identifying the characters mentioned in multiparty conversations, which mainly requires the understanding of discourse relations and assessment of personal facts. However, coreference does not assess the modeling of the character's theory-of-mind, especially the character's memories, as there are no predictions of character behaviors involved. The prediction of fictional characters' personality types by reading the original stories Flekova and Gurevych (2015) is another character-centric task related to the present work. The work covers only the character's personality such as the big five and the MBTI types, also a perspective of the persona our work considers.

**Character-Centric Prediction over NonNarratives** Many tasks use the story summary instead of the original story. The textual entailment task LiSCU Brahman et al. (2021) links an anonymous character summary to the name in the story summary. Using summaries precludes ToM modeling, as discussed in Appendix A.1 Personalized dialogue generation benchmarks Mairesse and Walker (2007); Walker et al. (2012); Zhang et al. (2018); Urbanek et al. (2019); Li et al. (2020) are based on daily chit-chats. They usually cover a single facet of multi-dimensional personae Moore et al. (2017), _e.g._, personal facts Zhang et al. (2018) or personality types Mairesse and Walker (2007); Li et al. (2020). The LIGHT environment Urbanek et al. (2019) covers both facts and personalities. None of them covers a comprehensive persona like ours, especially how a character's past experience builds her ToM.

Authorship attribution has a parallel goal to ours, insofar as it aims at guessing author identities from the texts they wrote Ni et al. (2019); Andrews and Bishop (2019); Bevendorff et al. (2020). These tasks differ from ours in two respects: first, multiple prose examples generated by the same author do not usually form consecutive plot lines, and second, they rarely model the event history of depicted characters. On this basis, they mainly require understanding authors' writing styles rather than building mental models of facts, events, and experiences.

## 3 Our TVShowGuess Benchmark

### Task Definition

TVSG adopts a multi-choice setting. The goal is to guess the anonymous speakers who are the main characters (maximum number of 6 for each show) in the scene. The models are provided with an anonymous scene that consists of \(n\) lines \(\tilde{\mathcal{S}}^{(t)}=\{\tilde{s}_{1}^{(t)},\tilde{s}_{2}^{(t)},...,\tilde {s}_{n}^{(t)}\}\) (\(t\) stands for the \(t\)-th scene in the entire show). Each line \(\tilde{s}_{i}\) can either be a dialogue turn or a background description. When the line is a dialogue turn, it is associated with an anonymous speaker ID (with the form of P\({}_{x}\), \(1\leq x\leq 6\)) of a main character, or the real name of a supporting character. Similarly, we introduce the notation of the standard scene \(\mathcal{S}^{(t)}=\{s_{1}^{(t)},s_{2}^{(t)},...,s_{n}^{(t)}\}\), which has the same definition as the anonymous scenes, with the only difference that the dialogue turns always have their real names of speakers associated.

The anonymous scene \(\tilde{\mathcal{S}}^{(t)}\) is associated with a candidate set \(\mathcal{C}^{(t)}=c_{1}^{(t)},...,c_{k}^{(t)}\), \(k\leq 6\), where each character \(c_{j}^{(t)}\) is a main character who appears in \(\mathcal{S}\). The goal is thus predicting each P\({}_{x}\)'s actual role \(c_{j}^{(t)}\), _i.e._, a match \(\pi(\cdot)\) from the anonymous IDs to the real characters, conditioned on the scene \(\tilde{\mathcal{S}}^{(t)}\) and all previous scenes \(S^{(1:t-1)}\):

\[P(\texttt{P}_{x}=c_{j}^{(t)}|\tilde{\mathcal{S}}^{(t)},S^{(1:t-1)}). \tag{1}\]

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Dataset** & **Task Format** & \multicolumn{4}{c}{**Narrative Type**} & \multicolumn{4}{c}{**Aressed Narrative Comprehension Skills**} \\  & & **Source** & **Length** & **Plot Structures** & **Character Facts** & **Character IoMs** \\ \hline
**MCTest** & Multi-choice QA & Sheet fiction (Chiang attention) & \(\sim\)20\({}^{*}\) & ✓ & & \\
**BookTest** & Cloze test & Literature (Escape) & – & ✓ & & \\
**Ala et al.** (2018) & Cloze test & TV Show transcripts (Stock) & \(\sim\)20 & ✓ & & \\
**NarrativeQA** & Generative QA & Movie Scripts, Literature (full sentence) & \(\sim\)11K\({}^{*}\) & ✓ & ✓ & \\
**FriendsQA** & Extractive QA & TV show transcripts (Scottes) & \(\sim\)20\({}^{*}\) & ✓ & ✓ & \\
**NovelChapy/BookSum** & Summarization & Literature (Geocentric Orders’ Mall Faces) & \(\sim\)4K & ✓ & & \\
**SummScreen** & Summarization & TV show transcripts (Seunes) & \(\sim\)330 & ✓ & & \\
**Chen and Choi (2016)** / & Corcell Resolution & TV show transcripts (Birolacek or scene) & \(\sim\)20260\({}^{*}\) & ✓ & ✓ & \\
**Glekova and Gurevych (2015)** & Classification & Literature (Full sentence) & \(\sim\)22K & & ✓ & \\ \hline
* & Multi-choice & TV show transcripts (Full sentence) & \(\sim\)50K & ✓\({}^{*}\) & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Properties of existing narrative comprehension datasets compared to TVShowGuess.
* Numbers are not reported in the original paper so we calculated them from the dataset. †(Chen et al., 2017) proposes two settings: single scene and the whole episode. ‡Our task requires reasoning based on history scenes, which is a form of plot understanding.
 



### Dataset Collection

We use community contributed transcripts from The TV MegaSite (TMS)4 like Chen et al. (2021). Our scenes are from the scripts of five popular TV series: _Friends_, _The Big Bang Theory (TBBT)_, _The Office_, _Frasier_ and _Gilmore Girls_.

Footnote 4: [http://tvmegasite.net](http://tvmegasite.net).

**Data Cleaning** Our data consists of character dialogues and background descriptions. The dialogues start with the characters' names. One or more turns of dialogue between characters comprise a scene. Scenes are separated by short background cues that begin with markers such as location (e.g. "_Howard's car_", "_Kingman Police Station_"), special words (e.g., "_Scene_", "\(Cut\)"), or symbols (e.g. "_[ I]_"). We created a rule-based parser which splits the content of an episode into multiple independent scenes using scene separation markers.

**Character Recognition and Anonymization** We used the main characters' names to identify their dialogues within each scene and randomly labeled them with speaker IDs (e.g., P0, P1). Since the different names of the characters, such as nicknames, first names, and last names, mark the dialogue in a mixed way, in order to match the lines to the correct speaker, we first identified the main characters in each TV series by consulting Fandom's cast list. Then, we calculated the frequency of speech to find references to the same main character's name.

## 4 Analysis of Our Benchmark

We propose a comprehensive **schema of persona types** for the machine narrative comprehension. The schema facilitates the analysis of the challenges in our task; and provides insights into the deficiencies in current narrative comprehension models, by allowing a decomposition of model performance to the dimensions of categories (Section 6).

### Our Annotation Schema for Human Study

Two researchers with backgrounds in psychology, linguistics, NLP, and education developed and tested an inductive coding method based on the methods of grounded theory Glaser and Strauss (2017). They conducted three rounds of independent annotation and discussion of the evidence needed to identify the characters, using 10 randomly selected scenes for each round. After each discussion, they updated the codebook accordingly. Modifications to the codebook led to the achievement of saturation during the process. Then the two researchers coded a total of 318 characters from 105 scenes of Friends and The Big Bang Theory. The annotation interface appears in Appendix B.

This schema **categories the required evidence to resolve the task** into four persona data types: _linguistic style_, _personality_, _fact_, _memory_. Table 4 reports inter-rater reliability calculated by Cohen's Kappa Cohen (1960). The kappa values range from 0.76 to 0.87 and would all be considered satisfactory Vera et al. (2005), reflecting the success of our codebook and process.

We have one additional type, _inside-scene_, referring to tasks that can be resolved purely within local contexts, thus not requiring persona understanding. To better depict how these pieces of evidence are used in human rationales, we added two complementary categories: (1) how the task instance **relies on the history scenes**, and (2) when there are multiple pieces of evidence required, what **types of reasoning skills** are used to derive the answer from the evidence (see Section C). Table 6 shows the definitions of each evidence type. We provide examples of each evidence type in Section B.2.

#### 4.1.1 Major Evidence Types

**Linguistic Style** Personalized language patterns that reflect individual differences in self-expression and are consistently reliable over time and situations Pennebaker and King (1999).

**Personality** Stable individual characteristics Vinciarelli and Mohammadi (2014) that can distinguish "internal properties of the person from overt behaviors" Matthews et al. (2003).

**Memory** A character's episodic memory of events from previous episodes and the semantic memory5 inferred from events. Note here we want to model the memory of a particular character, _i.e._, the historical scenes experienced by the particular character instead of from the audiences' perspective. A character's memory is crucial for humans to build her ToM, but is largely ignored as a part of persona in previous research.

Footnote 5: Semantic memory is the characters’ general world knowledge that they accumulate over time Reisberg (2013). Episodic memory, on the other hand, is the characters’ memory of specific experiences in their lives Tulving (2002)

**Fact** The truth about characters as opposed to interpretation, which can usually be represented

* **Attribute** All explicitly provided factual character identity information in the TV series setting, such as race, occupation, and education level.
* **Relationship** Relationship includes social relationships (e.g., husband and wife) and dramatic relationships (e.g., arch-enemy). When talking to people with different relationships, characters change their identity masks by using different words Gergen (1972).
* **Status** The emotional or psychological status of a character when facing a specific situation.

Inside-SceneThe textual evidence inside the scene, independent from the character's persona.

* **Background** Background introduction and descriptions in other character dialogues.
* **Mention** The character's name or alias is called out by other characters. Although mention is persona-independent, it still presents challenging cases. In a multi-person multi-round dialog, because anaphora in the current sentence may not bound to an antecedent on the right frontier of the structure, common sense analysis of conversational coherence is needed to determine which speaker is being referred to.

ExclusionA guessing technique for elimination using a given list of characters which is neither evidence nor inference, but it depends on the character list provided within the scene, so we include it as a subcategory of inside-scene evidence.

#### 4.1.2 Dependence on History

To understand how much humans rely on memory to identify a character, the annotators coded whether the evidence necessary to solve the task depends directly on historical events or indirectly on history by abstracting from historical events.

Direct DependencyCharacters that can only be identified through events that are explicitly expressed in previous episodes,6 for example:

Footnote 6: If a character can be identified with evidence of both _Memory_ and _Inside-Scene_, it will be labeled as _No-Dependency_.

\begin{tabular}{l l}  & **Background:** (from TBBT) [_The stairwell_] \\   **Candidates:** _(Leonard, Penny)_ \\ Po: _There's something I wanted to run past you._ \\ P1: _What's up?_ \\ Po: _Mn, the guys and I were thinking about investing in Stuart's comic book store. Is that okay?_ \\ P1: _Why are you asking me?_ \\
**Answer:** Po \(\rightarrow\)_Leonard_ \\
**Rationale:** In a previous scene, Leonard and his friends discussed investing in Stuart's store. He is the only one between the two who has this memory. \\ \end{tabular}

Indirect DependencyCharacters can only be identified using evidence not explicitly expressed in previous episodes, but inducible from previous events. For example, _Personality_ can be inferred from the character's previous behavior.7

Footnote 7: The annotation of indirect dependency is very subjective as different annotators may have memory of previous scenes and use different evidence to guess the character.

Background: (from Friends) [Central Perk]

Candidates: _(Joey, Rachel, Ross)_ \\ Po: _Here you are (Hands Rachel a cup of coffee)_ \\ P1: _Thank you Joye, You know what? I'm not even sure I can have caffeine._ \\ P2: _I went thr hits with Ben and Carol. One cup of coffee won’t affect your milk._ \\ P1: _Yeak. Just to be sure I'm gonna call Dr. Wiener._ \\
**Answer:** P2 \(\rightarrow\)_Ross_ \\
**Rationale:** There is no actual scene with Ross going through this with Carol; the answer is inferred based on Ross' known relations to Ben (parent-child) and Carol (ex-spouse). Thus the evidence about Ross has indirect dependency on scene history. \\ \end{tabular}

If the answer can be inferred within the scene, like answering P0 \(\rightarrow\)_Joey_ in the above example. We have a special rule on the _Exclusion_ evidence type - If a character can only be inferred on the basis of other characters being eliminated, it should have dependency type labeled if any of the other character has a history dependency. In other words, when guessing the identity with _Exclusion_ requires history dependency on another character, the dependency type is transitive.

 



### Analysis

**Main Statistics** Table 3 shows the proportions of the required evidence types and dependency of history. According to the statistics, history is an important factor in guessing the characters. 46.36% of the examples from Friends and 67.55% examples from the Big Bang Theory require history.

**Human Performance Accuracy** One annotator (who had not watched the seasons under evaluation) obtained nearly perfect accuracy in guessing the characters in FRIENDS (98.68%), and a lower but still good accuracy in TBBT (89.82%). A second annotator (who had watched all episodes and thus would be considered an expert) confirmed that most the error cases were unsolvable given only data extracted from scenes. We list the unsolvable cases and human mistakes in Appendix E.

**Correlation between Evidence Types and History Dependence** Figure 2 visualizes the flow from evidence types to the dependency of history. Most of them are correlated. Personality and history dependency are most closely related.

## 5 Methods

Inspired by the successes of pre-trained Transformers on reading comprehension tasks, we benchmarked our TVShowGuess by building baseline solutions on top of these pre-trained models. The key challenge of our task is that the prediction relies on how a character reacts to the scenario with her/his words, therefore the embedding of each utterance should be highly **context-aware**. This requires handling a whole scene as input, which is usually over the limits of these pre-trained models.

We propose two solutions. The first is to benefit from sparse attention, specifically, Longformer Beltagy et al. (2020). The second is to organize each utterance with its necessary history context as one **row**, and have a BERT model to encode each relatively short row independently. For both models, we finally conduct attentive pooling for each character over the contextualized embeddings of all her utterances for prediction.

Our baselines simplified the problem by (1) ignoring the historical scenes in Eq.(1); and (2) making independent prediction of characters within a scene. The former poses the challenge of handling longer contexts and the latter requires specific predictor design. We believe both are important to handle in future work.

### Transformers with Character-Pooling

Our first approach (the top in Figure 3) is denoted as Longformer-Pooling (or **Longformer-P**).

**Scene Encoding** The input \(\tilde{S}\) to the model includes the concatenation of all the utterances in an anonymous scene. Each utterance is prefixed by a speaker ID token and suffixed by a separation token, _i.e._,

\[T_{i}=[\text{P}_{x_{i}}]\oplus U_{i}\oplus[\text{SPLIT}] \tag{2}\]

\[\tilde{S}=T_{0}\oplus T_{1}\oplus...\oplus T_{N}, \tag{3}\]

where \(U_{i}\) is the \(i\)-th utterance and \([\text{P}_{x_{i}}]\) is its speaker ID (e.g., \([\text{P}_{0}]\) and \([\text{P}_{1}]\)). \([\text{SPLIT}]\) is a special token. \(\oplus\) denotes concatenation. We use a Longformer to encode the whole \(\tilde{S}\), to make the embedding of each utterance token _context-aware_, _i.e._, \(\mathbf{H}=\text{Longformer}(\tilde{S})\in\mathbb{R}^{L\times D}\).

\begin{table}
\begin{tabular}{l c} \hline \hline
**Category** & \(\kappa\)**(\%)** \\ \hline Evidence type & \\ Coarse-grained types & 81.53 \\ Fine-grained types & 80.99 \\ \hline Dependence of history & \\ Direct dependence only & 82.02 \\ All dependency types & 75.51 \\ \hline Reasoning Type & 87.21 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Annotation agreement. \(\dagger\): see our extended study in Appendix C. We list the number for reference.

Figure 2: Visualization of the flow from the required evidence types to their dependence of history.



character-Specific Attentive Pooling For each character ID P\({}_{x}\), we have a token-level mask \(M_{x}\in\mathbb{R}^{L\times 1}\) such that \(M_{x}[j]{=}1\) if the \(j\)-th word belongs to an utterance of P\({}_{x}\) and \(M_{x}[j]{=}0\) otherwise. For each P\({}_{x}\), we then collect the useful information from all her utterances selected by \(M_{x}\) as:

\[A =\text{Attention}(\mathbf{H}) \tag{4}\] \[\alpha_{x} =\text{Softmax}(A\odot M_{x}),\]

where Attention\((\cdot)\) is a one-layer feedforward network to compute the token-level attention weights. The character-specific attention \(\alpha_{x}\) is then used to pool the hidden states to summarize a character representation in the input scene \(\tilde{S}\) and make the prediction: \(P({\textrm{P}}_{x}=c|\tilde{S})=f_{k}(\mathbf{H}^{T}\alpha_{x})\). Here \(f_{k}:\mathbb{R}^{d\times 1}\rightarrow\mathbb{R}^{C\times 1}\) is the character classifier for the \(k\)-th TV show.

### Multi-Row BERT

The second approach (the bottom in Figure 3) is denoted as the multi-row BERT (**MR. BERT**). We split the long scene \(\tilde{S}\) into multiple segments \(\{\tilde{s}_{i}\}\). Encoding the segments reduces the overall complexity from \(O(L^{2})\) to \(O(RL_{s}^{2})\), where \(R\) is the number of segments and \(L_{s}\ll L\) is the maximum segment length. To construct each segment \(E_{i}\), we take an utterance \(T_{i}\) as in Eq. 2 and concatenate it with the nearest history utterances \(T_{i^{\prime}}(i^{\prime}<i)\) until arriving the maximum length \(L_{s}\). The \(R\) segments for each instance yield \(\{\tilde{s}_{i}\}\) as follows:

\[E_{i} =T_{t_{i}}\oplus\text{[SEP]}\oplus T_{t_{i}-1}\oplus T_{t_{i}-2}\cdots \tag{5}\] \[\{\tilde{s}_{i}\} =[E_{1};E_{2};\cdots;E_{R}].\]

Then we encode the \(\{\tilde{s}_{i}\}\) with a BERT encoder:

\[\mathbf{H}=\textsc{BERT}(\{\tilde{s}_{i}\})\in\mathbb{R}^{R\times L_{s}\times D}. \tag{6}\]

Different from Longformer-P, we have a segment-level mask \(M_{x}\in\mathbb{R}^{R}\) for each character ID such that \(M_{x}[j]=1\) if the first utterance in the \(j\)-th row (i.e., \(T_{t_{j}}\) in Eq. 5) is said by P\({}_{x}\). Applying the same attentive pooling technique to each segment following Eq. 4, we obtain \(R\) segment embeddings \(\{\textbf{E}_{i}\}_{1}^{R}\). We take the concatenation of these embeddings as the new input to the show-specific predictor and calculate the probability distribution of P\({}_{x}\) being each character, _i.e._,

\[P({\textrm{P}}_{x}=c|\tilde{S})=f_{k}([\textbf{E}_{1};\textbf{E}_{2};\cdots; \textbf{E}_{R}]). \tag{7}\]

Compared to Longformer-P, the MR. BERT model takes a smaller number of \(R\) utterances and benefits from their concatenated contextual utterances. To make the selection of \(R\) utterances representative, we applies two tricks: (1) _fill-empty_, which makes sure each P\({}_{x}\) has at least one segment selected; (2) _the reverse trick_, which selects the utterances starting from the end of scene to the start - as the utterances at the end have more histories, they cover more contents from the scene if selected.

## 6 Experiments

We evaluate the instance-level accuracy. An instance refers to a masked speaker in a scene.

### Baselines and Implementation Details

We compare with the vanilla pre-trained Transformer baseline, **Vanilla Longformer Classifier**. The model conducts direct classification over the concatenation of a character's utterances in the scene. It can be viewed as a discriminative language model of the characters' lines.

We include the implementation details of the baseline and our models in Appendix G.

### Results

**Overall Results** Table 5 compares different models on our TVShowGuess. The proposed architectures beat the vanilla character classifier with large margins (4-5%). However, human performance is

Figure 3: Our two proposed model architectures for the character prediction task.

 

significantly (21-26%) better than the best models. This shows that models are still far from reaching human level of character understanding.

Among all the shows, TBBT is the most challenging one, while Frasier and Gilmore Girls are relatively simpler. Given that there is no correlation between performance and scene lengths (Table 2), this shows that the difficulty of the task mainly comes from the persona modeling, inference and reasoning. Specifically, the _InsideScene_ evidence requires less persona understanding. Therefore, the relatively smaller amount of _InsideScene_ cases makes TBBT more difficult. Also the existing models are not good at resolving the related memory or facts from the history, thus the high ratio of _history dependent_ cases in TBBT also leads to lower performance.

### Analysis

**Learning Curves** We plots the learning curves of Friends and TBBT, with increasing number of shows used as training data (Figure 4). The curves become flat with all shows added, showing that our task has sufficiently data for training.

**Scene-Level Performance** Besides the instance-level accuracy, we further investigated the scene-level performance, _i.e._, the macro-average of instance-level accuracy in all the scenes. Table 6 shows the results, together with the decomposed results on scenes containing different numbers of speakers. The results show the more characters involved, the lower the accuracy is, even though our model is making independent predictions of each speaker. One possibility is that there is fewer available utterances per speaker. In addition, a larger set of speakers may make the logical structure of the conversation more complex.

**Character-Level Performance** Next, we examined whether our task is uniformly challenging for different characters, or whether there were certain characters that were more difficult to guess. Table 7 shows the results, where the characters are ranked by the accuracy of their guesses. There are clear discrepancies in accuracy by character.

**Impact of the Dependence on History** The bar charts in Figure 5 show the performance on different history dependence types. The performance of cases that require history supports is in general

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{3}{c}{**\#Speakers Contained**} \\  & Overall & 2 & 3 & 4 & 5 & 6 \\ \hline FRIENDS & 80.6 & 86.5 & 80.8 & 66.3 & 75.0 & 56.7 \\ TBBT & 67.7 & 77.0 & 66.7 & 57.0 & 55.0 & 47.9 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Scene-Level accuracy decomposition.

\begin{table}
\begin{tabular}{l r r l r l r l r l r l r l} \hline \hline \multirow{2}{*}{**System**} & \multicolumn{2}{c}{**FRIENDS**} & \multicolumn{2}{c}{**TBBT**} & \multicolumn{2}{c}{**Frasier**} & \multicolumn{2}{c}{**Gilmore\_Girls**} & \multicolumn{2}{c}{**The\_Office**} & \multicolumn{2}{c}{**Overall**} \\  & **dev** & **test** & **dev** & **test** & **dev** & **test** & **dev** & **test** & **dev** & **test** & **dev** & **test** \\ \hline Random & 35.23 & 31.59 & 33.08 & 37.79 & 34.74 & 31.61 & 36.43 & 38.90 & 44.30 & 46.71 & 36.79 & 36.59 \\ Vanilla Longformer & 67.79 & 60.63 & 61.58 & 63.95 & 85.11 & 82.06 & 79.84 & 74.52 & 70.92 & 71.60 & 72.55 & 69.72 \\ repl with BERT & 65.60 & 59.58 & 61.58 & 58.43 & 85.11 & 84.30 & 81.91 & 70.41 & 67.56 & 68.54 & 71.65 & 67.76 \\ \hline Our MR. BERT & **77.01** & **73.20** & 62.60 & 62.50 & 90.07 & 82.51 & **83.98** & **78.63** & 70.92 & 74.41 & 76.82 & 74.52 \\ - context & 62.92 & 57.19 & 59.54 & 63.95 & 81.64 & 76.23 & 74.42 & 67.12 & 66.00 & 67.37 & 68.33 & 65.54 \\ - reverse trick & 70.81 & 68.71 & 52.42 & 59.01 & 79.40 & 81.39 & 78.04 & 73.97 & 66.22 & 68.31 & 69.45 & 70.52 \\ - fill-empty trick & 74.33 & 68.56 & 58.27 & 63.37 & 86.10 & 78.48 & 72.87 & 69.86 & 68.90 & 73.71 & 72.28 & 70.92 \\ \hline Our Longformer-P & **77.01** & 69.91 & **63.87** & **66.57** & **90.32** & **87.67** & 82.17 & 75.07 & **71.81** & **76.29** & **76.95** & **74.97** \\ maxlen=1000 & 74.16 & 66.77 & 63.36 & 64.24 & 86.10 & 85.65 & 79.33 & 72.05 & 73.83 & 76.06 & 75.25 & 72.74 \\ repl with BERT & 68.12 & 58.83 & 61.32 & 63.95 & 82.63 & 76.91 & 68.48 & 65.75 & 72.48 & 71.83 & 70.49 & 66.79 \\ \hline Human\({}^{*}\) & 98.68 & – & 89.82 & – & – & – & – & – & – & – & \\ \hline \hline \end{tabular}
\end{table}
Table 5: Overall performance (%) on our TVShowGuess task. (*) Human evaluation was conducted on a subset of the dataset.

Figure 4: Learning curves of Friends and The Big Bang Theories with increasing training data from other shows.

 

harder for most of the models (\(\sim\)20% lower compared to the cases without dependency of history).

The results suggest that a more thorough extraction of historical events associated with each character is needed to make the model an improvement. This notion aligns with the theories that past experience is an key aspect of building characters' ToM, showing that our TVShowGuess may serve as a good benchmark for the in-depth study of character comprehension from stories.

Another finding is that the cases requiring indirect history dependence (usually _Personality_ and _Facts_) are even more challenging. The human mind develops detailed profiles of characters when reading stories. The neural models represent each character as a single vector (_i.e._, the weight vector in the output layer). This indicates a promising future direction of constructing structured persona representations (_e.g._, based on our schema of evidence) for more accurate character modeling.

Breakdown to Evidence TypesThe wind-rose charts (bottom) in Figure 5 provide performance breakdown onto our evidence categories. We omit the type of _Linguistic style_ because there are only two cases in Friends so the results are not stable.

As expected, the cases that can be resolved locally without character understanding (_Inside-Mention_) are relatively easier. All of _Personality_, _Fact_ and _Memory_ cases have much lower performance as they correspond to heavy dependency on the modeling of history.

The type _Exclusion_ gives the worst overall performance on the two shows. However, this does not indicate difficulty of character understanding - According to the definition, these cases cannot be directly resolved with the scene inputs, but require the model to have specific strategy to exclude some incorrect answers first.

It is surprising that the _Inside-Background_ type poses difficulties to models, because to human annotators it is similar to standard textual inference. We have identified two possible reasons: (1) As discussed in the introduction, some cases require pragmatic understanding from the surface form to intention, only after which textual inference can be performed; (2) The number of instances of this type is relatively small so the model may fail to recognize the required textual inference skills.

Effect of Scene ContextsFinally, the vanilla character classifier behaves differently compared to the other models. Because it cannot make use of contexts within scenes, there is a performance gap on the _Inside-Mention_ type (hence the drop on the _No Dep_ type). However, it does not suffer from significant differential on the other types. The gap appears because Longformer-P and/or MR. BERT perform considerably better on this type.

Challenges of History RetrievalOur experiments show that the history dependency presents serious challenges for existing models. Finding the evidence from history scenes is a retrieval task (but without groundtruth). We applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results. Our study shows that on our identified cases with _Direct Dependency_, the top-3 results (from in total 20 candidates) of a state-of-the-art semantic search model only give a recall of 35.5%. The result confirms that our task requires further advances on semantic retrieval. The detailed task setting and our discussions can be found in Appendix F.

## 7 Conclusion

We present the first task and dataset for evaluating machine reading comprehension models for understanding characters in narratives. Based on linguistic, educational, and psychological theories, we proposed a new schema and conducted human studies to analyze the types of evidence and reasoning required in understanding characters. We design a new model architecture and conduct comprehensive experiments as a basis for future studies.

### Acknowledgements

This research was supported, in part, by the NSF (USA) under Grant Numbers CNS-1948457.

Figure 5: Performance breakdown of Friends (the left column) and The Big Bang Theory (the right column).

 



## References

* [Andrews and Bishop2019] Nicholas Andrews and Marcus Bishop. 2019. Learning invariant representations of social media users. _arXiv preprint arXiv:1910.04979_.
* [Bajgar et al.2016] Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. _arXiv preprint arXiv:1610.00956_.
* [Bamman et al.2013] David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 352-361.
* [Baron-Cohen1997] Simon Baron-Cohen. 1997. _Mindblindness: An essay on autism and theory of mind_. MIT press.
* [Beltagy et al.2020] Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _arXiv preprint arXiv:2004.05150_.
* [Bender and Koller2020] Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5185-5198.
* [Bevendorff et al.2020] Janek Bevendorff, Bilal Ghanem, Anastasia Giachanou, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Rangel, Paolo Rosso, et al. 2020. Overview of pan 2020: Authorship verification, celebrity profiling, profiling fake news spreaders on twitter, and style change detection. In _International Conference of the Cross-Language Evaluation Forum for European Languages_, pages 372-383. Springer.
* [Bower and Morrow1990] Gordon H Bower and Daniel G Morrow. 1990. Mental models in narrative comprehension. _Science_, 247(4938):44-48.
* [Brahman et al.2021] Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Minmaya Sachan, and Snigdha Chaturvedi. 2021. " let your characters tell their story": A dataset for character-centric narrative understanding. _arXiv preprint arXiv:2109.05438_.
* [Chen et al.2016] Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. _arXiv preprint arXiv:1606.02858_.
* [Chen et al.2017a] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a. Reading wikipedia to answer open-domain questions. In _Proceedings of ACL 2017_, pages 1870-1879.
* [Chen et al.2017b] Henry Y Chen, Ethan Zhou, and Jinho D Choi. 2017b. Robust coreference resolution and entity linking on dialogues: Character identification on tv show transcripts. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 216-225.
* [Chen et al.2021] Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021. Summscreen: A dataset for abstractive screenplay summarization. _arXiv preprint arXiv:2104.07091_.
* [Chen and Choi2016] Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 90-100.
* [Cohen1960] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* [Currie2009] Gregory Currie. 2009. Narrative and the psychology of character. _The journal of aesthetics and art criticism_, 67(1):61-71.
* [De Haan1996] Pieter De Haan. 1996. More on the language of dialogue in fiction. _ICAME JOURNAL_, 20:23-40.
* [Elsner2012] Micha Elsner. 2012. Character-based kernels for novelistic plot structure. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 634-644.
* [Finkeyan2012] Mark Mark Alan Finlayson. 2012. _Learning narrative structure from annotated folktales_. Ph.D. thesis, Massachusetts Institute of Technology.
* [Fiske et al.1979] Susan T Fiske, Shelley E Taylor, Nancy L Etcoff, and Jessica K Laufer. 1979. Imaging, empathy, and causal attribution. _Journal of Experimental Social Psychology_, 15(4):356-377.
* [Elekova and Gurevych2015] Lucie Flekova and Iryna Gurevych. 2015. Personality profiling of fictional characters using sense-level links between lexical resources. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1805-1816.
* [Gergen1972] Kenneth J Gergen. 1972. Multiple identity: The healthy, happy human being wears many masks. _Psychology today_, 5(12):31-35.
* [Gernsbacher et al.1998] Morton Ann Gernsbacher, Brenda M Hallada, and Rachel RW Robertson. 1998. How automatically do readers infer fictional characters' emotional states? _Scientific studies of reading_, 2(3):271-300.
* [Glaser and Strauss2017] Barney G Glaser and Anselm L Strauss. 2017. _Discovery of grounded theory: Strategies for qualitative research_. Routledge.
* [Guo et al.2020] Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. 2020. Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. _arXiv preprint arXiv:2010.02386_.
* [Han et al.2021] Rujun Han, I Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, Nanyun Pen, et al. 2021. Ester: A machine reading comprehension dataset for event semantic relation reasoning. _arXiv preprint arXiv:2104.08350_

Matthew Hausknecht, Priithviraj Ammanabrolu, Marc-Alexandre Cote, and Xingdi Yuan. 2019. Interactive fiction games: A colossal adventure. _arXiv preprint arXiv:1909.05398_.
* Holtgraves (2010) Thomas Holtgraves. 2010. Social psychology and language: Words, utterances, and conversations.
* Iyyer et al. (2016) Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daume III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1534-1544.
* Kennedy et al. (2013) XJ Kennedy, Dana Gioia, and Dan Stone. 2013. _Literature: An introduction to fiction, poetry, drama, and writing_. Pearson.
* Kocisky et al. (2018) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. _Transactions of the Association for Computational Linguistics_, 6:317-328.
* Kryscinski et al. (2021) Wojciech Kryscinski, Nazneen Rajani, Divsmash Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. _arXiv preprint arXiv:2105.08209_.
* Ladhak et al. (2020) Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5043-5054.
* Lal et al. (2021) Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. Tellmewhy: A dataset for answering why-questions in narratives. _arXiv preprint arXiv:2106.06132_.
* Li et al. (2020) Aaron W Li, Veronica Jiang, Steven Y Feng, Julia Sprague, Wei Zhou, and Jesse Hoey. 2020. Aloha: Artificial learning of human attributes for dialogue agents. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8155-8163.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. _arXiv preprint arXiv:1603.06155_.
* Ma et al. (2018) Kaixin Ma, Tomasz Jurczyk, and Jinho D Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multiparty dialog. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2039-2048.
* Mairesse and Walker (2007) Francois Mairesse and Marilyn Walker. 2007. Personage: Personality generation for dialogue. In _Proceedings of the 45th annual meeting of the association of computational linguistics_, pages 496-503.
* Matthews et al. (2003) Gerald Matthews, Ian J Deary, and Martha C Whiteman. 2003. _Personality traits_. Cambridge University Press.
* Mazare et al. (2018) Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. _arXiv preprint arXiv:1809.01984_.
* McHale (2004) Brian McHale. 2004. Free indirect discourse. _Narrative Theory: Major issues in narrative theory_, 1:187.
* McKee (1997) Robert McKee. 1997. _Story: style, structure, substance, and the principles of screenwriting_. Harper Collins.
* Mead (1990) Gerald Mead. 1990. The representation of fictional character. _Style_, pages 440-452.
* Moore et al. (2017) Christopher Moore, Kim Barbour, and Katja Lee. 2017. Five dimensions of online persona.
* Morrow (1985) Daniel G Morrow. 1985. Prominent characters and events organize narrative understanding. _Journal of memory and language_, 24(3):304-319.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderweronde, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 839-849.
* Mou et al. (2021) Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain qa techniques: A comprehensive study. _arXiv preprint arXiv:2106.03826_.
* Myers and McCaulley (1988) Isabel Briggs Myers and Mary H McCaulley. 1988. _Myers-Briggs type indicator: MBTL_. Consulting Psychologists Press Palo Alto.
* Nash (1990) Walter Nash. 1990. _Language in popular fiction_. Routledge.
* Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 188-197.
* Paris and Paris (2003) Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. _Reading Research Quarterly_, 38(1):36-76.
* Pennebaker and King (1999) James W Pennebaker and Laura A King. 1999. Linguistic styles: language use as an individual difference. _Journal of personality and social psychology_, 77(6):1296.
* Qian et al. (2018) Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Assigning personality/profile to a chatting machine for coherent conversation generation. In _Ijcai_, pages 4279-4285 

* Reisberg (2013) Daniel Reisberg. 2013. _The Oxford handbook of cognitive psychology_. Oxford University Press.
* Richardson et al. (2013) Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 193-203.
* Riedl and Young (2010) Mark O Riedl and Robert Michael Young. 2010. Narrative planning: Balancing plot and character. _Journal of Artificial Intelligence Research_, 39:217-268.
* Sims et al. (2019) Matthew Sims, Jong Ho Park, and David Bamman. 2019. Literary event detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3623-3634.
* Tulving (2002) Endel Tulving. 2002. Episodic memory: From mind to brain. _Annual review of psychology_, 53(1):1-25.
* Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktaschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. _arXiv preprint arXiv:1903.03094_.
* Viera et al. (2005) Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. _Fam med_, 37(5):360-363.
* Vinciarelli and Mohammad (2014) Alessandro Vinciarelli and Gelareh Mohammadi. 2014. A survey of personality computing. _IEEE Transactions on Affective Computing_, 5(3):273-291.
* Vishnubhotla et al. (2019) Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hurst. 2019. Are fictional voices distinguishable? classifying character voices in modern drama. In _Proceedings of the 3rd joint SIGHM workshop on computational linguistics for cultural heritage, social sciences, humanities and literature_, pages 29-34.
* Walker et al. (2012) Marilyn A Walker, Grace I Lin, Jennifer Sawyer, et al. 2012. An annotated corpus of film dialogue for learning and characterizing character style. In _LREC_, pages 1373-1378.
* Warren (2006) Martin Warren. 2006. _Features of naturalness in conversation_, volume 152. John Benjamins Publishing.
* Xu et al. (2022) Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, et al. 2022. Fantastic questions and where to find them: Fairytaleqa- an authentic dataset for narrative comprehension. _arXiv preprint arXiv:2203.13947_.
* Yang and Choi (2019) Zhengzhez Yang and Jinho D Choi. 2019. Friendsqa: Open-domain question answering on tv show transcripts. In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_, pages 188-197.
* Yao et al. (2021) Shunyu Yao, Karthik Narasimhan, and Matthew Hausknecht. 2021. Reading and acting while blindfolded: The need for semantics in text game agents. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3097-3102.
* Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2204-2213.
* Zheng et al. (2020) Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxai Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 9693-9700.

 

A Detailed Survey of Related Work

In this section, we first give an in-depth analysis on the difference between narrative and synopsis, from both the empirical challenges in NLP studies and the linguistic theory from (Morrow, 1985). Then we provide detailed discussion on how we summarize related work in Table 8.

### Background: Narrative versus Synopsis

As our work focuses on narrative comprehension, following setups like Kocisky et al. (2018); Krysciinski et al. (2021); Chen et al. (2021), it is necessary to distinguish between comprehension of original narrative stories versus comprehension of their supges (the human-written plot summaries), _e.g._, from the story's Wikipedia page.

Narrative stories are told by creating scenes, with the goal of helping readers experience events as they occur in the plot, and empathize with the story characters in relation to their own experiences. To engage readers, story writers often use complex narrative clues (_e.g._, character activities, event development, scenery changes); variable narrative sequences (_e.g._, narrative, flashback, interpolation); and various linguistic expressions (_e.g._, argument, lyricism, narrative, description, illustration). By comparison, a synopsis is a descriptive summary of the main idea of a story in plain language. It contains only the main characters, time, place, important plot points, and resolution, rather than allowing the story to unfold through the actions of the characters. The goal of the synopsis is to inform readers what happened with little or no original material from original story.

Therefore, comprehension of narrative stories requires more sophisticated skills to understand the complex chain of clues and expressions, in order to finally build a complete narrative representation from a sequence of individual scene comprehensions along with a developed understanding of characters' mental models Morrow (1985). In this light, a synopsis represents "processed results" from the application of these comprehension skills by a (experienced) human reader.

### Background: Dialogue in Stories vs. Real-Life Conversation

Fictional dialogue canonically serves a purpose in a narrative. Either i contributes to the develop of a character or advances the plot McHale (2004). Nash proposed three categories of fictional dialogue: (1) confrontational dialogue which "includes challenges, quarrels, disputes, interviews, and any kind of personal encounter in which the participants are in covert or overt opposition to each other"; (2) instructional dialogue, which "conveys information about matters of science, technology, politics, world events, etc, some knowledge of which is essential to understanding the plot"; (3) collaborative dialogue that consists of "a series of exchanges which cumulatively present, for the reader's benefit, a picture of events, histories, personalities, and relationships" Nash (1990). Authors have total control of the fictional dialogue, and ideally it functions according to the author's intention at every part of the story De Haan (1996). However, real-life conversation is a joint action and is natural. There is no individual has whole control and the conversation goal of each party may be very different. Additionally, real-life conversations are temporally linear, such that communicators cannot revise earlier speech to fit a story. Moreover, real-life conversations tend to be highly implicit because spoken language derives much of its meaning from context Warren (2006).

### Assessment of Narrative Comprehension

We summarize the related tasks people use for assessment of general narrative comprehension skills.

#### Cloze Test

Cloze tests take a snippet of the original text with some pieces (usually entities) masked as blanks, with the goal of filling these blanks from a list of candidates. Cloze tests can be automatically constructed, resulting in an advantage in creation of large scale datasets. Examples of cloze tests for narrative comprehension assessments are BookTest Bajgar et al. (2016) and Ma et al. (2018). Both datasets are based on excerpts of books or scenes of TV shows. As the input consists only of short paragraphs, there is not sufficient information to infer complex character set via reading the stories. Therefore, these datasets address few questions related to the understanding of characters.8

Footnote 8: There may be a possible confusion of these tasks and ours, as cloze tests also include filling in anonymous character names in the blanks. However, in these tasks, the required answers are also anonymized character IDs that appear in the inputs, and the IDs for the same character are random across different scenes. Therefore the character’s information is not available for learning by design. In other words, their design of tasks deliberately prevent the task of character understanding.

Moreover, when built on short snippets, cloze tests are known to prone to mostly local inference but not m 

knowledge, as studies in the NLP community suggested (Chen et al., 2016). On the other hand, although our task also has form similar to cloze, it requires information about the characters from previous scenes, which is not only about understanding the characters, but also requires global inference across features of the story (see Figure 1).

Question AnsweringThe most popular form of narrative comprehension evaluation is through question answering, starting from the early work of MCTest Richardson et al. (2013), to the more recent crowd-sourced tasks like NarrativeQA Kocisky et al. (2018), FriendsQA Yang and Choi (2019), TellMeWhy Lal et al. (2021) and FairytaleQA Xu et al. (2022).

Among them, the MCTest and TellMeWhy conduct multi-choice question answering on short stories. As above, te input consists of short paragraphs, so there is not sufficient information to infer complex character facies via reading the stories. Therefore, these datasets also cover few questions assessing the understanding of characters. The TellMeWhy has a specific focus on _why_-questions assessing the causal knowledge between states and events. The inputs are short stories from the ROCStories dataset Mostafazadeh et al. (2016). MCTest covers a much wider set of reading skills, as it is based on complete stories, and generates questions with the goal of assessing children's reading comprehension over both story plots and commonsense.

NarrativeQA and FriendsQA conduct natural question answering tasks. NarrativeQA aims to infer free-form answers to questions about a specific book or movie script. According to the human study from Mou et al. (2021), the major part of the dataset is event-centric questions, which queries the explicit plots from the original books thus do not require a significant amount of commonsense reasoning. The study also reveals that NarrativeQA consists of a small portion of character-related questions. These questions mainly query the simple facts of characters, such as age and profession. The more complexity character persona types, like personality, emotional/psychological status and history experience studied in our work, are not covered. Similar to NarrativeQA, FriendsQA is a QA task over TV show scripts. The dataset consists of six types of questions: _who, what, when, where, why_, and _how_. The _who_ questions target on asking speaker names of utterance contents or participants of events, therefore are mainly assessing understanding of plot structures (_i.e._, participant arguments of events).

Both NarrativeQA and FriendsQA have humanwritten questions with a reference of the plot summary, which require evidence explicitly exists in the original story texts, and thus do not have much requirement of reasoning. The FriendsQA ques

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{4}{c|}{**Task Format Type**} & \multicolumn{2}{c}{**Assessed Narrative Comprehension Skills**} & \multicolumn{2}{c}{**Assessed Commonsense Knowledge**} \\  & & **Source** & **Length** & **Pite Structures** & **Character Forest** & **Character Tasks** & **Categories** & **Events/Status** & **Stery News** \\ \hline
**MCTest** & Multi-choice QA & Short fiction & \(<\)20\({}^{7}\) & ✓ & & & ✓ & ✓ & ✓ \\
* & ✓ & & & & & \\
**classory** & & & & & & & & \\
**diso** & Cizer test & Y/day transcripts & \(>\)20\({}^{5}\) & ✓ & & & & & \\
**NarratiusQA** & Generative QA & Movie Corpus, Literature & \(\sim\)11K\({}^{\prime}\) & ✓ & ✓ & & & ✓ & \\
**FriendQA** & Executive QA & Yes & ✓ & \(\sim\)20\({}^{7}\) & ✓ & & & & \\
**TelfMeWhy** & Multi-choice QA & Short fiction & 5 & & & & & ✓ & \\
**NotCobergetBeadSum** & Summationation & Latentize & \(<\)4K & ✓ & & & & & ✓ \\
**Summations** & Summarization & TV show transcripts & \(\sim\)300 & ✓ & & & & & ✓ \\
**Semanticven** & Summation & Text describing & \(\sim\)300 & ✓ & & & & & ✓ \\
**Classion and Calli (2016)** & Court Resolution & TV show transcripts & \(\sim\)202kgd\({}^{\dagger}\) & ✓ & ✓ & & & ✓ & ✓ \\
**Closarue and Gauroych (2015)** & Classification & Lectureate & \(\sim\)22k & & ✓ & & & & \\ \cline{2-8}
* & Multi-choice & & & & & & & \\ \hline \end{tabular}
\end{table}
Table 8: Properties of existing narrative comprehension datasets compared to TVShowGuess . We organize the datasets according to the following dimensions related to narrative understanding: **Source*
* of the texts for reading comprehension; **Length*
* of the texts from the source that makes the task solvable, we report the numbers of sentences or utterances for books and scripts respectively; whether the task assesses the ability of understanding **plot structures*
* in the stories; whether the task assesses the ability of understanding basic **character facts*
* like personality, profession, etc; whether the task assesses the ability of building **character theory-of-mind (ToM)**; whether the task assesses the commonsense knowledge of **concepts, events*
* and **states**; and whether the task assesses the additional commonsense about the **narrative development**, including the knowledge about the coherence among non-verbal narratives and dialogues, and how they form the story/plot flow.
* Numbers are not reported in the original paper so we calculated them from the dataset. †[2] proposes two settings with single scene and the whole episode as inputs respectively. Different from ours, their include of episode is not to support the in-scene prediction with necessary history, but mostly increase the difficulty level of the co-ref task.
 

tions are based on scene summaries, and thus require mostly local evidence; the NarrativeQA questions are based on the book-level summary, and thus sometimes require the ability to bridge the gap between coarse-grained and fine-grained event descriptions (_i.e._, commonsense of sub-events).

SummarizationThere is a recent trend to evaluate model's understanding of stories via summarization, including NovelChapters Ladhak et al. (2020), BookSum Kryscinski et al. (2021) and ScreenSum Chen et al. (2021). These works provide a good research opportunity to future story reading research, by showing that book-level or chapter-level summarization is challenging to existing machine reading models. However, it is more difficulty to identify the specific required reading skills by these tasks, as there exist many factors beyond reading skills to generate a good summary, such as encoding and generating long narrative texts. Intuitively, story summarization is largely plot-related instead of character-related; and requires the knowledge to understand the story flow.

Interactive Fiction Game PlayingInteractive fiction (IF) games Hausknecht et al. (2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories as environment observations. Research work has successfully demonstrated that reading comprehension can provide helpful inductive biases for efficient policy learning Guo et al. (2020); while Yao et al. (2021) also reveal the shortcomings of these games. The debate calls for future investigations to understand the necessary narrative elements and the roles they play in the IF games.

### Character-Centric Prediction over Narratives

Our task can be seen as a character-centered understanding of the narrative, where the understanding of the character deepens the understanding of the story and makes the narrative engaging. There are limited studies on understanding characters' persona from reading stories. In this section we review some existing character-centric prediction tasks over narrative texts, and discuss the relations and differences.

Character Name LinkingThe task of coreference resolution for story characters Chen and Choi (2016); Chen et al. (2017b) is closely related to our TVShowGuess. These coreference resolution focuses on identifying the characters mentioned in multiparty conversations from TV shows scripts. The goal of these tasks is to resolve the coreference of pronouns and character-indicating nominals (_e.g._, _you_ and _Mom_) **in dialogues** of the character names that appear in the local context. It also covers linking a named entity (_e.g._, _Ross_) to the character, which is more on name matching instead of character understanding.

The task form of coreference resolution mainly requires the understanding of discourse relations. It does not assess the modeling of character theory-of-mind, especially the character's memories, as there are no predictions of character behaviors involved. The major character persona type it assesses is character facts, since the resolution of nominals requires the understanding of the target characters' occupations and relationships.

The lack of ToM modeling and complex reasoning of the coreference resolution task also makes it relatively easier - on Friends and The Big Bang Theory, a CNN model gives a \(>\)90% average accuracy. By comparison, our task, although solvable by humans with a \(\sim\)95% accuracy, is challenging to neural models as the best BERT-based model gives a \(\sim\)65% average accuracy on the same two shows with even smaller candidate sets.

Personality PredictionOur work is also related to the prediction of fiction characters' personality types by reading the stories Flekova and Gurevych (2015). Specifically, the tasks require to predict a friction character's MBTI personality types Myers and McCaulley (1988) rooted in Jung's theory, based on the character's verbal and non-verbal narratives in the original stories. Compared to the aforementioned character-centric prediction tasks, these studies require to read and comprehend the original long stories, but the prediction task are relatively simpler since they only focus on personality which is a single perspective of persona.

### Character-Centric Prediction over Non-Narratives

Character name linking between story synopsesRecently Brahman et al. (2021) propose the LiSCU, which is a novel textual entailment task linking an anonymous summative descriptions of story character to the name appearing in the story's plot summary. Similarly to Chen and Choi (2016), the task assess the resolution of names and events instead of the ToM modeling. This is because the 

task does not involve much explicit behavior predictions, since the task form is entailment between two given statements rather than predicting the possibility of new contents. The usage of synopses over original stories reduces the challenges in narrative understanding; and further prevents the character comprehension from stories, as pointed out by Kocisky et al. (2018), the summaries themselves are humans' comprehension results of the stories.

Personalized Dialogue GenerationOur work is also related to personalized dialogue generation, for which datasets Mairesse and Walker (2007); Walker et al. (2012); Zhang et al. (2018); Li et al. (2020) and models Li et al. (2016); Mazare et al. (2018); Qian et al. (2018); Zheng et al. (2020) are proposed for generating dialogues for speakers with persona features. These benchmarks usually cover a single aspect of the multi-dimensional persona Moore et al. (2017). For example, PERSONA-CHATZhang et al. (2018) focuses on personal facts such as "_I'm a writer_" and "_I live in Springfield_"; other works mainly focus on learning language styles from speakers' personality types, such as the Big Five traits of the extraversion personality in PERSONAGE Mairesse and Walker (2007), and the personality types derived from TV tropes (e.g. _jealous girlfriend_, _book doom_, _anti hero_) in ALOHALi et al. (2020).

LIGHT Urbanek et al. (2019) is a crowdsourced dataset for text game adventure research. It includes natural language descriptions of fantasy locations, objects and their affordances, characters and their personalities, dialogue and actions of the characters. The biggest difference between ours and LIGHT is that LIGHT is based on the local environment of the conversation, rather than on a story. Examples from the LIGHT dataset are independent conversations and the context in which they occur. Crowd workers created the dialogues of characters by a given setting and a persona. The persona is modeled by the Persona-Chat dataset which is defined as a set of three to five profile sentences describing their personal facts such as "_I am a part of a group of travelers_" and "_I go from town to town selling food to the locals_".

To the best of our knowledge, none of the existing studies cover a comprehensive multidimensional persona similar to our work, especially with respect to how a character's past experience builds her ToM.

Authorship AttributionFinally, authorship attribution has parallel ideas to our task, insofar as it aims at guessing author identities from the texts they wrote Ni et al. (2019); Andrews and Bishop (2019); Bevendorff et al. (2020) and thus requires a certain degree of author profiling. These tasks differ from ours because the reviews, tweets or fandoms under the same authors do not usually form consecutive plotlines. Therefore, the tasks mainly require to understand the authors' writing styles rather than building mental models from the their past experiences. From this perspective, this direction is in fact more closely related to stylistic analysis in narrative understanding Vishnubhotla et al. (2019), rather than character understanding.

## Appendix B Supplementary for the Dataset Analysis

### Summary of the Annotation Schema

We include a summary of our annotation schema in Figure 6.

Examples of Each Evidence Types Linguistic Style Background: (from TBBT) [Any's car] Candidates: [Leonard, Penny, Sheldon, Amy] Pb: Whatever You can't even go on a date without checking your relationship agreement. P1: If you've got a problem basing a relationship on a contract, I'd like to tell you about 13 plucly colonies that entered a relationship agreement called the U.S. Constitution. And if may not be cool to say so, but I think that love affair is still pretty hot today. Answer:P1\(\to Leonard\) Rationalae: (Sheldon's language is characterized by the use of long, difficult sentences and references to historical stories.)

Personality Background: (from TBBT) [The cafeteria] Candidates: [Leonard, Howard, Sheldon, Raj] Pb: And you love the sound of your own voice. P1: Yeah, well, of course I do. Listen to it. It's like an earful of melted camel. Answer:P1\(\rightarrow\) Sheldon Rationalae: (Sheldon is a self-centered person so he will praise his own voice.)

Appendix C Memory Background: (from TBBT) [The stairawell] Candidates: [Leonard, Penny] Pb: There's something I wanted to run past you. P1: What s up? P0: Am, the guys and I were thinking about investing in Stuart's conbook store. Is that okay? P1: Why are you asking me? Answwer:P0\(\to Leonard\) Rationalae: (In a previous scene, Leonard and his friends discussed about investing in Stuart's store, so he is the only one between the two who has this memory.) 

* Attribute [command=
**Background:** (from TBBT) _[Anny's lab]_
**Candidates:** _[Any, Penny]_

P0:** _Hey. Ready to go to lunch?_

P1:_ Just give me a minute. I'm stimulating the pleasure cells of this starkfish. I just need to turn it off._

**Answer:** P1\(\rightarrow\) _Sheldon_

**Rationale:** (Sheldon is Amy's boyfriend. After identify PO is Amy, based on the relationship between Amy and Sheldon, P1 can be identified as Sheldon.)
* Relationship [command={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={},fontsize={}, fontsize={},fontsize 



**Background:** (from Friends) _[Scene: Outside the Junior's Closet, there are people having s*x and Mr. Geller is trying to give them some pamphlets.]_

**Candidates:** _[Monica, Chandler]_

**Mr. Geller: Kids, I spoke to a doctor and picked up this pamphlets on how to get pregnant. (He slides them under the door._

P0:___(walking by with Chandler.) Hey dad!_

P1:_Hey._

**Mr. Geller: _(pause) Sorry to bother you again, but could you pass my pamphlets back? (They do so.) Thank you._

**Answer:** P1\(\rightarrow\)_Chandler Rationale:_ (Monica is Mr. Geller's daughter. P0 called Mr. Geller had so she is Monica. There are only two candidate so the other one is Chandler)

## Appendix C Extended Study of Required Reasoning Types on our TVShowGuess

This section provides an in-depth analysis of the types of reasoning used to infer evidence when guessing characters.

### Our Annotation Schema of Reasoning Types

We define the following reasoning types with examples provided. A summary of our annotation schema of reasoning types can be found in Figure 7.

Multi-hop on CharacterstReasoning on the basis of other characters that have already been guessed. Using the already guessed character as a bridge, users can employ history event or the relationship between characters to make guesses about the target character.The difference between multi-hop character and exclusion is that after identifying the other characters, the exclusion technique relies only on the list of characters provided for guessing, however, multi-hop character reasoning requires additional evidence such as relationship to infer the target character.

```
Background:(from TBBT)[Angelis Stadium] Candidates:[Raj, Leonard, Sheldon, Amy] PS:Irew, I hear your re dermatologist. Emily:Uh, yeoh, I'm a resident at Huntington Hospital. ... P5:I have some odd freckles on my buttocks. Can I make an appointment for you to look at them? Emily: Um, okay, I guess. P0:I'm with him three years, nothing. She's with him two minutes, he's taking his pants off. Answer:P0\(\rightarrow\)Any Rationale:(Using P5 (Sheldon) as a bridge and the couple relationship between Amy and him, we can identify PD is Amy.)
```

Multi-hop on Textual EvidenceSome evidences are not directly presented in the scene but can be inferred from the descriptions of context and dialogues. Using the inferred evidences as bridges people can multihop over personality, or fact, or event inferred from the text to guess the characters.

```
Background:(fromTBBT)[The apartment] Candidates:[Amy, Leonard, Raj, Howard', Penny, Sheldon] Bernadete:I like your suit. P0:Oh, thanks. Got a couple new oufts for work. P1:How does it feel knowing your fiancee's job is to go out and flirt with doctors, looking like that, while you sit here, you know, looking like this? ... Answwer:P0\(\rightarrow\)Penny Rationale:(P0 has a new job can be inferred from the textual evidence "Got a couple new outtis for work". Plus we know that Penny has a new job, we can determine that P0 is Pennsylvania )
```

Commonsense of Concepts/EventsTask requires additional commonsense knowledge of attributes of daily concepts or social events, or their relations including causal/effect relations between an event and a social state or social relation. We restrict this category to be the aforementioned commonsense knowledge types, to distinguish from other relatively under-studied commonsense knowledge, such as the commonsense of dialogue flow required to work with our inside-scene evidence defined in Figure 6.

```
Background:(from TBBT)[Capital Comics] Candidates:[Howard, Sheldon]
```

P0:I know that if I had a wife or a fiancee, I'd ask her first before I invested money in a comic book store. P1:He's right. Answer:P1\(\rightarrow\)Howard Rationale:(A married or engaged person will answer "He's right". Howard is married. ) ```

Default ConjunctionA single piece of evidence will not solve this task; a combination between multiple pieces of evidence is needed to identify the person.

Analysis of the Human Annotation Correlation between the Human Annotated Schema CategoriesFigure 2 visualizes the flow between (a) evidence types and the dependency of history and (b) evidence types and the reasoning types. Most evidence types correlate with history dependency. Personality and history dependency are most closely related. Default conjunction is the reasoning type that accounts for the largest percentage.

### Experiments: Performance Decomposition on the Reasoning Types

We further studied the impact of the required reasoning types on the performance (the right column

in Figure 9). In general there is a clear gap (on average \({\sim}10\%\)) between cases that require complex reasoning and those that do not. The _Multihop-Textual_ type is most challenging, because it requires both deep understanding of what the texts implies and multihop reasoning. There is not a clear performance difference between _Multihop-Character_ and _Default Conjunction_, though the former is conceptually harder. We surmise this is because both types are beyond the reasoning ability of the model so the predictions largely rely on fuzzy matching of evidence - recall that we predict identities of main characters, so there can be a statistical bias of their context co-occurrence. The results on the _Commonsense_ type fluctuate due to the relatively smaller ratio.

## Appendix D Interface for the Human Study

Figure 10 shows the interfaces of the human study.

## Appendix E Examples of Human Errors

Table 11 provides an example of unsolvable cases and Table 12 provides an example of human mistakes. The human mislabeled characters are marked as red.

We further provide all the scene IDs on which our human tester makes incorrect predictions in Table 10.

## Appendix F Details of Human Study and Discussions on the Challenges of History Retrieval

Our experiments show that the history dependency challenges existing models. Finding the evidence

\begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{1}{r}{**\#Unsolvable**} & \multicolumn{1}{l}{**\#Human Mistakes**} \\ \cline{1-3} TBBT & Friends & TBBT & Friends \\ \hline
4882 & 2500 & 4921 & \\
4895 & & 4894 & \\
4907 & & 4910 & \\
4908 & & & \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 10: Human Errors.

Figure 8: Visualization of the flow from the required evidence types to their required reasoning types.

\begin{table}
\begin{tabular}{p{65.0pt} p{284.5pt}} \hline \hline
**Reasoning Type** & **Description** \\ \hline Default Conjunction & No single piece of evidence can solve the task, hence the conjunction among multiple pieces of evidence is required. This is the default reasoning type if there are multiple evidence types labeled but no other reasoning types are labeled. \\ \hline Multihop-Character & Task needs to be solved with the guessing results of other characters, then using the target person relation to or memory about the guessed ones to make the answer, _i.e._, multihop with guessed characters as bridges. \\ \hline Multihop-Textual & Task needs to be solved with the persona/fact/event not directly described in the scene but can be inferred from the context, _i.e._, multihop over persona/fact/event inferred from dialog and scene context.

\begin{tabular}{p{65.0pt} l} Commonsense & Task requires additional commonsense knowledge of attributes of daily concepts or social events, or their relations like causal relations between events. Those refer to the specific types of commonsense covered in ConceptNet- or Atomic-style KBs. \\ \hline \hline \end{tabular}
\end{table}
Table 9: Percentage of the required reasoning types in the two TV shows, Friends and The Big Bang Theory.

Figure 7: The definitions of reasoning types.

Figure 9: Performance breakdown according to our reasoning schema (left: Friends, right: The Big Bang Theory).

 



**Unsolvable Case**

\begin{tabular}{||l||} \hline
08x02 4882 \\
**Background:** (from TBBT) _[the Apartment]_ \\
**Candidates:** _[Howard, Sheldon, Raj, Amy, Leonard, Penny]_ \\ P0 : I recently read that during World War Two, Joseph Stalin had a research program to create supersoldiers by having women impregnated by gorillas. \\ P1 : What a sick use of science. \\ P2 : Hey, as long as the baby's healthy. \\ P3 : I wonder if Stalin considered any other animals. \\ P4 : Hippos are the deadliest creature. A half-human, half-hippo soldier would be pretty badass. \\ P1 : Yes, but when they're hungry-hungry, you can stop them with marbles. \\ P0 : Yeah, the correct animal for interspecies supersolider is koala. You would wind up with an army so cute it couldn't be attacked. \\ P2 : But half-man, half-owl could fly... \\ P0 : The answer is cuddly soldiers with big flat noses. Moving on. \\ P1 : So, Penny, when's the new job start? \\ P5 : Next Monday. \\ Bernadette : Did you get a chance to look over the materials I gave you? \\ P5 : Uh, not yet, but I will. \\ Bernadette : Great. When? \\ P5 : I said I'll get to it. \\ P0 : I’m sensing awkwardness, am I right? \\ P3 : Yes. \\ P0 : Swish. \\ Bernadette : I don't want to be pushy, but you've never done pharmaceutical sales before. It seems like you could use this time to get a head start. \\ P5 : Well, the first few weeks will be all training. They'll tell me everything I need to know. \\ Bernadette : But imagine how impressed they'd be if you showed up already familiar with the material. \\ P5 : Okay, so what, you want me to be like a teacher's pet? \\ Bernadette : Couldn't hurt. \\ P4 : Mm, I don't know. Who here has ever been hurt because they were the teacher's pet? \\ P0 : It was like the rest of the class wanted Ms. McDonald to forget the quiz. \\
**Answer: P0: Sheldon, P1: Howard, P2: Raj, P3: Amy, P4: Leonard, P5: Penny** \\ \end{tabular}

Table 11: Example of unsolvable case.

\begin{table}
\begin{tabular}{||l||} \hline
**Mistake** \\ \hline \hline
08x04 4921 \\
**Background:** (from TBBT) _[Penny’s partment]_ \\
**Candidates:** _[Raj, Penny]_ \\ P0 : I'm so glad we could work this all out. \\ P1 : Yeah, me, too. \\ Emily : You know, we should have dinner one night with you and Leonard. \\ P1 : Oh, we would love that. \\ P0 : Great. \\ background : (both chuckle) \\ P1 : Okay, good night, guys. \\ Emily : All right, night. \\ P1 : Bye. \\ Emily and Penny (simultaneously) : I hate her. \\
**Answer: P0: Raj, P1: Penny** \\ \end{tabular}
\end{table}
Table 12: Example of mistake.



history scenes for such cases is essentially a retrieval task (but without groundtruth). To see how it brings new challenges to existing semantic search, we applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results.

**Task**: We conduct the study on scenes in our human annotation sets that have the _Memory_ type labeled. With each scene as a query, we retrieve from a window of 20 previous scenes with a state-of-the-art model9 The window size is decided so as to guarantee that at least one required memory appears in the window, according to our human annotation process. The task of human study is to recognize whether the top-3 returned scenes contain at least one related history scene.

Footnote 9: We use all-mpnet-base-v2 model from [https://sbert.net/](https://sbert.net/) that reports the top-1 performance on 14 sentence embedding tasks and 6 semantic search tasks.

ResultsThe same annotators working on the study in Section 4 evaluated the retrieved scenes. Results show that the recall of the top-3 results from this state-of-the-art model is low (35.5%). This difficulty in scene retrieval may arise from: (1) the queries are scenes with structures, which leads to different query formats from standard IR tasks; (2) many relevant scenes are dissimilar to the query scenes in the semantic space, but associated with the query in specific aspects or even analogous to the query scene; (3) some scenes require multi-hop retrieval, especially when combined with ToM modeling (reasoning about what others know).

All these challenges are non-trivial, and calls for further studies on semantic search to address.

## Appendix G Model Checklist

We implement our baselines based on HuggingFace Transformers.10 We use the pre-trained allenai/longformer-base-4096 and bert-base-uncased models. We train all the models with the Adam optimizer.

Footnote 10: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

We train our model on a single V100 GPU. It takes around 1 hour and 40 minutes to train a Longformer-based model. It takes around 2 hour and 10 minutes to train a multi-row BERT model. For all the models, we train in total 40 epochs. But the models usually converge in less than 20 epochs.

HyperparametersWe set the number of rows in MR. BERT to 12, to maximize the usage of GPU memory. We set the maximum length of Longformer to 2000, which can handle the lengths of most of the input scenes. The window size is set to 256. We set the learning rate to 2e-5.

We report our result with a single run. However, for each model we run twice; and we found the average development accuracy varies less than 0.5%.

Figure 10: interfaces of human studies.