SlowGuess/NAACL_2022_OCR huggingface dataset:
Correlation between Evidence Types and History Dependence Figure 2 visualizes the flow from evidence types to the dependency of history. Most of them are correlated. Personality and history dependency are most closely related.
Inspired by the successes of pre-trained Transformers on reading comprehension tasks, we bench
Figure 2: Visualization of the flow from the required evidence types to their dependence of history.
marked our TVSHOWGUESS by building baseline solutions on top of these pre-trained models. The key challenge of our task is that the prediction relies on how a character reacts to the scenario with her/his words, therefore the embedding of each utterance should be highly context-aware. This requires handling a whole scene as input, which is usually over the limits of these pre-trained models.
We propose two solutions. The first is to benefit from sparse attention, specifically, Longformer (Beltagy et al., 2020). The second is to organize each utterance with its necessary history context as one row, and have a BERT model to encode each relatively short row independently. For both models, we finally conduct attentive pooling for each character over the contextualized embeddings of all her utterances for prediction.
Our baselines simplified the problem by (1) ignoring the historical scenes in Eq.(1); and (2) making independent prediction of characters within a scene. The former poses the challenge of handling longer contexts and the latter requires specific predictor design. We believe both are important to handle in future work.
5.1 Transformers with Character-Pooling
Our first approach (the top in Figure 3) is denoted as Longformer-Pooling (or Longformer-P).
Scene Encoding The input  $\tilde{S}$  to the model includes the concatenation of all the utterances in an anonymous scene. Each utterance is prefixed by a speaker ID token and suffixed by a separation token, i.e.,
$$
T _ {i} = \left[ \mathrm {P} _ {x _ {i}} \right] \oplus U _ {i} \oplus [ \text {S P L I T} ] \tag {2}
$$
$$
\tilde {S} = T _ {0} \oplus T _ {1} \oplus \dots \oplus T _ {N}, \tag {3}
$$
where  $U_{i}$  is the  $i$ -th utterance and  $[\mathrm{P}_{x_i}]$  is its speaker ID (e.g.,  $[\mathrm{P}_0]$  and  $[\mathrm{P}_1]$ ). [SPLIT] is a special token.  $\oplus$  denotes concatenation. We use a Longformer to encode the whole  $\tilde{S}$ , to make the embedding of each utterance token context-aware, i.e.,  $\mathbf{H} = \operatorname{Longformer}(\tilde{S}) \in \mathbb{R}^{L \times D}$ .
Figure 3: Our two proposed model architectures for the character prediction task.
Character-Specific Attentive Pooling For each character ID  $\mathrm{P}_x$ , we have a token-level mask  $M_x \in \mathbb{R}^{L \times 1}$  such that  $M_x[j] = 1$  if the  $j$ -th word belongs to an utterance of  $\mathrm{P}_x$  and  $M_x[j] = 0$  otherwise. For each  $\mathrm{P}_x$ , we then collect the useful information from all her utterances selected by  $M_x$  as:
$$
A = \operatorname {A t t e n t i o n} (\mathbf {H}) \tag {4}
$$
$$
\alpha_ {x} = \operatorname {S o f t m a x} (A \odot M _ {x}),
$$
where  $\mathrm{Attention}(\cdot)$  is a one-layer feedforward network to compute the token-level attention weights. The character-specific attention  $\alpha_{x}$  is then used to pool the hidden states to summarize a character representation in the input scene  $\tilde{S}$  and make the prediction:  $P(\mathbf{P}_x = c|\tilde{S}) = f_k(\mathbf{H}^T\alpha_x)$ . Here  $f_{k}:\mathbb{R}^{d\times 1}\to \mathbb{R}^{C\times 1}$  is the character classifier for the  $k$ -th TV show.
5.2 Multi-Row BERT
The second approach (the bottom in Figure 3) is denoted as the multi-row BERT (MR. BERT). We split the long scene  $\tilde{S}$  into multiple segments  $\{\tilde{s}_i\}$ . Encoding the segments reduces the overall complexity from  $O(L^2)$  to  $O(RL_s^2)$ , where  $R$  is the number of segments and  $L_{s} \ll L$  is the maximum segment length. To construct each segment  $E_{i}$ , we take an utterance  $T_{i}$  as in Eq. 2 and concatenate it with the nearest history utterances  $T_{i'}(i' < i)$  until arriving at the maximum length  $L_{s}$ . The  $R$  segments for each instance yield  $\{\tilde{s}_i\}$  as follows:
$$
E _ {i} = T _ {t _ {i}} \oplus [ \mathrm {S E P} ] \oplus T _ {t _ {i - 1}} \oplus T _ {t _ {i - 2}} \dots \tag {5}
$$
$$
\left\{\tilde {s} _ {i} \right\} = \left[ E _ {1}; E _ {2}; \dots ; E _ {R} \right].
$$
Then we encode the  $\{\tilde{s}_i\}$  with a BERT encoder:
$$
\mathbf {H} = \operatorname {B E R T} \left(\left\{\tilde {s} _ {i} \right\}\right) \in \mathbb {R} ^ {R \times L _ {s} \times D}. \tag {6}
$$
Different from Longformer-P, we have a segment-level mask  $M_x \in \mathbb{R}^R$  for each character ID such that  $M_x[j] = 1$  if the first utterance in the  $j$ -th row (i.e.,  $T_{t_j}$  in Eq. 5) is said by  $\mathrm{P}_x$ . Applying
the same attentive pooling technique to each segment following Eq. 4, we obtain  $R$  segment embeddings  $\{\mathbf{E}_i\}_{1}^{R}$ . We take the concatenation of these embeddings as the new input to the show-specific predictor and calculate the probability distribution of  $\mathrm{P}_x$  being each character, i.e.,
$$
P \left(\mathrm {P} _ {x} = c \mid \tilde {S}\right) = f _ {k} \left(\left[ \mathbf {E} _ {1}; \mathbf {E} _ {2}; \dots ; \mathbf {E} _ {R} \right]\right). \tag {7}
$$
Compared to Longformer-P, the MR. BERT model takes a smaller number of  $R$  utterances and benefits from their concatenated contextual utterances. To make the selection of  $R$  utterances representative, we applies two tricks: (1) fill-empty, which makes sure each  $\mathrm{P}_x$  has at least one segment selected; (2) the reverse trick, which selects the utterances starting from the end of scene to the start - as the utterances at the end have more histories, they cover more contents from the scene if selected.
We evaluate the instance-level accuracy. An instance refers to a masked speaker in a scene.
6.1 Baselines and Implementation Details
We compare with the vanilla pre-trained Transformer baseline, Vanilla Longformer Classifier. The model conducts direct classification over the concatenation of a character's utterances in the scene. It can be viewed as a discriminative language model of the characters' lines.
We include the implementation details of the baseline and our models in Appendix G.
Overall Results Table 5 compares different models on our TVSHOWGUESS. The proposed architectures beat the vanilla character classifier with large margins (4-5%). However, human performance is
System	FRIENDS	TBBT	Frasier	Gilmore_Girls	The_Office	Overall
dev	test	dev	test	dev	test	dev	test	dev	test	dev	test
Random	35.23	31.59	33.08	37.79	34.74	31.61	36.43	38.90	44.30	46.71	36.79	36.59
Vanilla Longformer	67.79	60.63	61.58	63.95	85.11	82.06	79.84	74.52	70.92	71.60	72.55	69.72
repl with BERT	65.60	59.58	61.58	58.43	85.11	84.30	81.91	70.41	67.56	68.54	71.65	67.76
Our MR. BERT	77.01	73.20	62.60	62.50	90.07	82.51	83.98	78.63	70.92	74.41	76.82	74.52
- context	62.92	57.19	59.54	63.95	81.64	76.23	74.42	67.12	66.00	67.37	68.33	65.54
- reverse trick	70.81	68.71	52.42	59.01	79.40	81.39	78.04	73.97	66.22	68.31	69.45	70.52
- fill-empty trick	74.33	68.56	58.27	63.37	86.10	78.48	72.87	69.86	68.90	73.71	72.28	70.92
Our Longformer-P	77.01	69.91	63.87	66.57	90.32	87.67	82.17	75.07	71.81	76.29	76.95	74.97
maxlen=1000	74.16	66.77	63.36	64.24	86.10	85.65	79.33	72.05	73.83	76.06	75.25	72.74
repl with BERT	68.12	58.83	61.32	63.95	82.63	76.91	68.48	65.75	72.48	71.83	70.49	66.79
Human*	98.68	-	89.82	-	-	-	-	-	-	-
Figure 4: Learning curves of Friends and The Big Bang Theories with increasing training data from other shows.
significantly (21-26%) better than the best models. This shows that models are still far from reaching human level of character understanding.
Among all the shows, TBBT is the most challenging one, while Frasier and Gilmore Girls are relatively simpler. Given that there is no correlation between performance and scene lengths (Table 2), this shows that the difficulty of the task mainly comes from the persona modeling, inference and reasoning. Specifically, the Inside-Scene evidence requires less persona understanding. Therefore, the relatively smaller amount of Inside-Scene cases makes TBBT more difficult. Also the existing models are not good at resolving the related memory or facts from the history, thus the high ratio of history dependent cases in TBBT also leads to lower performance.
Learning Curves We plots the learning curves of Friends and TBBT, with increasing number of shows used as training data (Figure 4). The curves become flat with all shows added, showing that our task has sufficiently data for training.
Scene-Level Performance Besides the instance-level accuracy, we further investigated the scene-
Table 5: Overall performance (%) on our TVSHOWGUESS task. (*) Human evaluation was conducted on a subset of the dataset.
#Speakers Contained
Overall	2	3	4	5	6
FRIENDS	80.6	86.5	80.8	66.3	75.0	56.7
TBBT	67.7	77.0	66.7	57.0	55.0	47.9
Table 6: Scene-Level accuracy decomposition.
FRIENDS	TBBT
#Utterance	Acc	#Utterance	Acc
rachel	7,542	88.3	sheldon	8,131	87.0
joey	6,550	84.5	penny	5,314	75.8
phoebe	5,964	83.1	leonard	7,105	75.3
chandler	6,804	71.2	raj	3,033	52.5
ross	7,259	70.8	amy	1,699	42.1
monica	6,752	64.2	howard	4,013	36.4
Table 7: Accuracy decomposed across characters. We also provide the number of training utterances for each character.
level performance, i.e., the macro-average of instance-level accuracy in all the scenes. Table 6 shows the results, together with the decomposed results on scenes containing different numbers of speakers. The results show the more characters involved, the lower the accuracy is, even though our model is making independent predictions of each speaker. One possibility is that there is fewer available utterances per speaker. In addition, a larger set of speakers may make the logical structure of the conversation more complex.
Character-Level Performance Next, we examined whether our task is uniformly challenging for different characters, or whether there were certain characters that were more difficult to guess. Table 7 shows the results, where the characters are ranked by the accuracy of their guesses. There are clear discrepancies in accuracy by character.
Impact of the Dependence on History The bar charts in Figure 5 show the performance on different history dependence types. The performance of cases that require history supports is in general
Figure 5: Performance breakdown of Friends (the left column) and The Big Bang Theory (the right column).
harder for most of the models ( $\sim 20\%$  lower compared to the cases without dependency of history).
The results suggest that a more thorough extraction of historical events associated with each character is needed to make the model an improvement. This notion aligns with the theories that past experience is an key aspect of building characters' ToM, showing that our TVSHOWGUESS may serve as a good benchmark for the in-depth study of character comprehension from stories.
Another finding is that the cases requiring indirect history dependence (usually Personality and Facts) are even more challenging. The human mind develops detailed profiles of characters when reading stories. The neural models represent each character as a single vector (i.e., the weight vector in the output layer). This indicates a promising future direction of constructing structured persona representations (e.g., based on our schema of evidence) for more accurate character modeling.
Breakdown to Evidence Types The wind-rose charts (bottom) in Figure 5 provide performance breakdown onto our evidence categories. We omit the type of Linguistic style because there are only two cases in Friends so the results are not stable.
As expected, the cases that can be resolved locally without character understanding (Inside-Mention) are relatively easier. All of Personality, Fact and Memory cases have much lower performance as they correspond to heavy dependency on the modeling of history.
The type Exclusion gives the worst overall performance on the two shows. However, this does not indicate difficulty of character understanding - According to the definition, these cases cannot be directly resolved with the scene inputs, but require
the model to have specific strategy to exclude some incorrect answers first.
It is surprising that the Inside-Background type poses difficulties to models, because to human annotators it is similar to standard textual inference. We have identified two possible reasons: (1) As discussed in the introduction, some cases require pragmatic understanding from the surface form to intention, only after which textual inference can be performed; (2) The number of instances of this type is relatively small so the model may fail to recognize the required textual inference skills.
Effect of Scene Contexts Finally, the vanilla character classifier behaves differently compared to the other models. Because it cannot make use of contexts within scenes, there is a performance gap on the Inside-Mention type (hence the drop on the No Dep type). However, it does not suffer from significant differential on the other types. The gap appears because Longformer-P and/or MR. BERT perform considerably better on this type.
Challenges of History Retrieval Our experiments show that the history dependency presents serious challenges for existing models. Finding the evidence from history scenes is a retrieval task (but without groundtruth). We applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results. Our study shows that on our identified cases with Direct Dependency, the top-3 results (from in total 20 candidates) of a state-of-the-art semantic search model only give a recall of  $35.5\%$ . The result confirms that our task requires further advances on semantic retrieval. The detailed task setting and our discussions can be found in Appendix F.
We present the first task and dataset for evaluating machine reading comprehension models for understanding characters in narratives. Based on linguistic, educational, and psychological theories, we proposed a new schema and conducted human studies to analyze the types of evidence and reasoning required in understanding characters. We design a new model architecture and conduct comprehensive experiments as a basis for future studies.
This research was supported, in part, by the NSF (USA) under Grant Numbers CNS-1948457.
Nicholas Andrews and Marcus Bishop. 2019. Learning invariant representations of social media users. arXiv preprint arXiv:1910.04979.
Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361.
Simon Baron-Cohen. 1997. Mindblindness: An essay on autism and theory of mind. MIT press.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198.
Janek Bevendorff, Bilal Ghanem, Anastasia Giachanou, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Rangel, Paolo Rosso, et al. 2020. Overview of pan 2020: Authorship verification, celebrity profiling, profiling fake news spreaders on twitter, and style change detection. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 372-383. Springer.
Gordon H Bower and Daniel G Morrow. 1990. Mental models in narrative comprehension. Science, 247(4938):44-48.
Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. " let your characters tell their story": A dataset for character-centric narrative understanding. arXiv preprint arXiv:2109.05438.
Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a. Reading wikipedia to answer open-domain questions. In Proceedings of ACL 2017, pages 1870-1879.
Henry Y Chen, Ethan Zhou, and Jinho D Choi. 2017b. Robust coreference resolution and entity linking on dialogues: Character identification on tv show transcripts. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 216-225.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021. Summscreen: A dataset for obstructive screenplay summarization. arXiv preprint arXiv:2104.07091.
Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 90-100.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Gregory Currie. 2009. Narrative and the psychology of character. The journal of aesthetics and art criticism, 67(1):61-71.
Pieter De Haan. 1996. More on the language of dialogue in fiction. ICAME JOURNAL, 20:23-40.
Micha Elsner. 2012. Character-based kernels for novelistic plot structure. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 634-644.
Mark Mark Alan Finlayson. 2012. Learning narrative structure from annotated folktales. Ph.D. thesis, Massachusetts Institute of Technology.
Susan T Fiske, Shelley E Taylor, Nancy L Etcoff, and Jessica K Laufer. 1979. Imaging, empathy, and causal attribution. Journal of Experimental Social Psychology, 15(4):356-377.
Lucie Flekova and Iryna Gurevych. 2015. Personality profiling of fictional characters using sense-level links between lexical resources. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1805-1816.
Kenneth J Gergen. 1972. Multiple identity: The healthy, happy human being wears many masks. *Psychology today*, 5(12):31-35.
Morton Ann Gernsbacher, Brenda M Hallada, and Rachel RW Robertson. 1998. How automatically do readers infer fictional characters' emotional states? Scientific studies of reading, 2(3):271-300.
Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge.
Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang. 2020. Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02386.
Rujun Han, I Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, Nanyun Pen, et al. 2021. Ester: A machine reading comprehension dataset for event semantic relation reasoning. arXiv preprint arXiv:2104.08350.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2019. Interactive fiction games: A colossal adventure. arXiv preprint arXiv:1909.05398.
Thomas Holtgraves. 2010. Social psychology and language: Words, utterances, and conversations.
Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1534-1544.
XJ Kennedy, Dana Gioia, and Dan Stone. 2013. Literature: An introduction to fiction, poetry, drama, and writing. Pearson.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Wojciech Krysciński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209.
Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043-5054.
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. Tellmewhy: A dataset for answering why-questions in narratives. arXiv preprint arXiv:2106.06132.
Aaron W Li, Veronica Jiang, Steven Y Feng, Julia Sprague, Wei Zhou, and Jesse Hoey. 2020. Aloha: Artificial learning of human attributes for dialogue agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8155-8163.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155.
Kaixin Ma, Tomasz Jurczyk, and Jinho D Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multiparty dialog. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2039-2048.
François Mairesse and Marilyn Walker. 2007. Personage: Personality generation for dialogue. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 496-503.
Gerald Matthews, Ian J Deary, and Martha C Whiteman. 2003. *Personality traits*. Cambridge University Press.
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. arXiv preprint arXiv:1809.01984.
Brian McHale. 2004. Free indirect discourse. Narrative Theory: Major issues in narrative theory, 1:187.
Robert McKee. 1997. Story: style, structure, substance, and the principles of screenwriting. Harper Collins.
Gerald Mead. 1990. The representation of fictional character. Style, pages 440-452.
Christopher Moore, Kim Barbour, and Katja Lee. 2017. Five dimensions of online persona.
Daniel G Morrow. 1985. Prominent characters and events organize narrative understanding. Journal of memory and language, 24(3):304-319.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain qa techniques: A comprehensive study. arXiv preprint arXiv:2106.03826.
Isabel Briggs Myers and Mary H McCauley. 1988. Myers-Briggs type indicator: MBTI. Consulting Psychologists Press Palo Alto.
Walter Nash. 1990. Language in popular fiction. Routledge.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 188-197.
Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36-76.
James W Pennebaker and Laura A King. 1999. Linguistic styles: language use as an individual difference. Journal of personality and social psychology, 77(6):1296.
Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Assigning personality/profile to a chatting machine for coherent conversation generation. In *Ijcai*, pages 4279–4285.