SlowGuess/ACL_2020_OCR huggingface dataset:
Crime Scene Investigation Dataset We performed experiments on an extension of the CSI dataset<sup>6</sup> introduced by Frermann et al. (2018). It consists of 39 CSI episodes, each annotated with word-level labels denoting whether the perpetrator is mentioned in the utterances characters speak. We further collected scene-level binary labels indicating whether episode scenes are important and should be included in a summary. Three human judges performed the annotation task after watching the CSI episodes scene-by-scene. To facilitate the annotation, judges were asked to indicate why they thought a scene was important, citing the following reasons: it revealed (i) the victim, (ii) the cause of death, (iii) an autopsy report, (iv) crucial evidence, (v) the perpetrator, and (vi) the motive or the relation between perpetrator and victim. Annotators were free to select more than one or none of the listed reasons where appropriate. We can think of these reasons as high-level aspects a good summary should cover (for CSI and related crime series). Annotators were not given any information about TPs or narrative structure; the annotation was not guided by theoretical considerations, rather our aim was to produce useful CSI summaries. Table 2 presents the dataset statistics (see also Appendix B for more detail).
Implementation Details In order to set the hyperparameters of all proposed networks, we used a small development set of four episodes from the CSI dataset (see Appendix B for details). After experimentation, we set the temperature  $\tau$  of the softmax layers for the TP-specific attentions (Equation (5)) to 0.01. Since the binary labels in the
$^{6}$ https://github.com/EdinburghNLP/csi-corpus
overall
episodes	39
scenes	1544
summary scenes	454
per episode
scenes	39.58 (6.52)
crime-specific aspects	5.62 (0.24)
summary scenes	11.64 (2.98)
summary scenes (%)	29.75 (7.35)
sentences	822.56 (936.23)
tokens	13.27k (14.67k)
per episode scene
sentences	20.78 (35.61)
tokens	335.19 (547.61)
tokens per sentence	16.13 (16.32)
supervised setting are imbalanced, we apply class weights to the binary cross-entropy loss of the respective models. We weight each class by its inverse frequency in the training set. Finally, in supervised SUMMER, where we also identify the narrative structure of the screenplays, we consider as key events per TP the scenes that correspond to an attention score higher than 0.05. More implementation details can be found in Appendix C.
As shown in Table 2, the gold-standard summaries in our dataset have a compression rate of approximately  $30\%$ . During inference, we select the top  $M$  scenes as the summary, such that they correspond to  $30\%$  of the length of the episode.
Is Narrative Structure Helpful? We perform 10-fold cross-validation and evaluate model performance in terms of F1 score. Table 3 summarizes the results of unsupervised models. We present the following baselines: Lead  $30\%$  selects the first  $30\%$  of an episode as the summary, Last  $30\%$  selects the last  $30\%$ , and Mixed  $30\%$ , randomly selects  $15\%$  of the summary from the first  $30\%$  of an episode and  $15\%$  from the last  $30\%$ . We also compare SUMMER against TEXTRANK based on tf*idf (Mihalcea and Taurau, 2004), the directed neural variant described in Section 3.1 without any TP information, a variant where TPs are approximated by their expected position as postulated in screenwriting theory, and a variant that incorporates information about characters (Gorinski and Lapata, 2015) instead of narrative structure. For the character-based TEXTRANK, called SCENESUM, we substitute the  $f_{i}, f_{j}$  scores in Equation (6) with character-related importance scores  $c_{i}$  similar to the defini
Table 2: CSI dataset statistics; means and (std).
Model	F1
Lead 30%	30.66
Last 30%	39.85
Mixed 30%	34.32
TEXTRANK, undirected, tf*idf	32.11
TEXTRANK, directed, neural	41.75
TEXTRANK, directed, expected TP positions	41.05
SCENESUM, directed, character-based weights	42.02
SUMMER	44.70
Table 3: Unsupervised screenplay summarization.
F1	Coverage of aspects	# scenes per TP
Lead 30%	30.66	-	-
Last 30%	39.85	-	-
Mixed 30%	34.32	-	-
SUMMARUNNER*	48.56	-	-
SCENESUM	47.71	-	-
SUMMER, fixed one-hot TPs	46.92	63.11	1.00
SUMMER, fixed distributions	47.64	67.01	1.05
SUMMER, -P, -R	51.93	44.48	1.19
SUMMER, -P, +R	49.98	51.96	1.14
SUMMER, +P, -R	50.56	62.35	3.07
SUMMER, +P, +R	52.00	70.25	1.20
Table 4: Supervised screenplay summarization; for in SUMMER variants, we also report the percentage of aspect labels covered by latent TP predictions.
tion in Gorinski and Lapata (2015):
$$
c _ {i} = \frac {\sum_ {c \in C} [ c \in S \cup \operatorname {m a i n} (C) ]}{\sum_ {c \in C} [ c \in S ]} \tag {9}
$$
where  $S$  is the set of all characters participating in scene  $s_i$ ,  $C$  is the set of all characters participating in the screenplay and  $main(C)$  are all the main characters of the screenplay. We retrieve the set of main characters from the IMDb page of the respective episode. We also note that human agreement for our task is 79.26 F1 score, as measured on a small subset of the corpus.
As shown in Table 3, SUMMER achieves the best performance (44.70 F1 score) among all models and is superior to an equivalent model which uses expected TP positions or a character-based representation. This indicates that the pre-trained network provides better predictions for key events than position and character heuristics, even though there is a domain shift from Hollywood movies in the TRIPOD corpus to episodes of a crime series in the CSI corpus. Moreover, we find that the directed versions of TEXTRANK are better at identifying important scenes than the undirected version. We found that performance peaks with  $\lambda_{1} = 0.7$  (see Equation (6)), indicating that higher importance is given to scenes as the story progresses (see Appendix D for experiments with different  $\lambda$  values).
In Table 4, we report results for supervised models. Aside from the various baselines in the first block of the table, we compare the neural extractive model SUMMARUNNER*7 (Nallapati et al., 2017) presented in Section 3.2 with several variants of our model SUMMER. We experimented with randomly initializing the network for TP identification  $(-P)$  and with using a pretrained network  $(+P)$ . We also experimented with removing the regularization terms,  $O$  and  $F$  (Equations (7) and (8)) from the loss  $(-R)$ . We assess the performance of SUMMER when we follow a two-step approach where we first predict TPs via the pre-trained network and then train a network on screenplay summarization based on fixed TP representations (fixed one-hot TPs), or alternatively use expected TP position distributions as postulated in screenwriting theory (fixed distributions). Finally, we incorporate character-based information into our baseline and create a supervised version of ScENESUM. We now utilize the character importance scores per scene (Equation (9)) as attention scores – instead of using a trainable attention mechanism – when computing the global screenplay representation  $d$  (Section 3.2).
Table 4 shows that all end-to-end SUMMER variants outperform SUMMARUNNER*. The best result (52.00 F1 Score) is achieved by pretrained SUMMER with regularization, outperforming SUMMARUNNER* by an absolute difference of 3.44. The randomly initialized version with no regularization achieves similar performance (51.93 F1 score). For summarizing screenplays, explicitly encoding narrative structure seems to be more beneficial than general representations of scene importance. Finally, two-step versions of SUMMER perform poorly, which indicates that end-to-end training and fine-tuning of the TP identification network on the target dataset is crucial.
What Does the Model Learn? Apart from performance on summarization, we would also like to examine the quality of the TPs inferred by SUMMER (supervised variant). Problematically, we do not have any gold-standard TP annotation in the CSI corpus. Nevertheless, we can implicitly assess whether they are meaningful by measuring how well they correlate with the reasons annotators cite to justify their decision to include a scene in the summary (e.g., because it reveals cause of death
or provides important evidence). Specifically, we compute the extent to which these aspects overlap with the TPs predicted by SUMMER as:
$$
C = \frac {\sum_ {A _ {i} \in A} \sum_ {T P _ {j} \in T P} [ \operatorname {d i s t} (T P _ {j} , A _ {i}) \leq 1 ]}{| A |} \tag {10}
$$
where  $A$  is the set of all aspect scenes,  $|A|$  is the number of aspects,  $TP$  is the set of scenes inferred as TPs by the model,  $A_{i}$  and  $TP_{j}$  are the subsets of scenes corresponding to the  $i^{th}$  aspect and  $j^{th}$  TP, respectively, and  $\text{dist}(TP_{j}, A_{i})$  is the minimum distance between  $TP_{j}$  and  $A_{i}$  in number of scenes.
The proportion of aspects covered is given in Table 4, middle column. We find that coverage is relatively low  $(44.48\%)$  for the randomly initialized SUMMER with no regularization. There is a slight improvement of  $7.48\%$  when we force the TP-specific attention distributions to be orthogonal and close to expected positions. Pre-training and regularization provide a significant boost, increasing coverage to  $70.25\%$ , while pre-trained SUMMER without regularization infers on average more scenes representative of each TP. This shows that the orthogonal constraint also encourages sparse attention distributions for TPs.
Table 5 shows the degree of association between individual TPs and summary aspects (see Appendix D for illustrated examples). We observe that Opportunity and Change of Plans are mostly associated with information about the crime scene and the victim, Climax is focused on the revelation of the motive, while information relating to cause of death, perpetrator, and evidence is captured by both Point of no Return and Major Setback. Overall, the generic Hollywood-inspired TP labels are adjusted to our genre and describe crime-related key events, even though no aspect labels were provided to our model during training.
Do Humans Like the Summaries? We also conducted a human evaluation experiment using the summaries created for 10 CSI episodes. We produced summaries based on the gold-standard annotations (Gold), SUMMARUNNER*, and the supervised version of SUMMER. Since  $30\%$  of an episode results in lengthy summaries (15 minutes on average), we further increased the compression rate for this experiment by limiting each summary to six scenes. For the gold standard condition, we randomly selected exactly one scene
Our adaptation of SUMMARUNNER that considers content and salience vectors for scene selection.
8https://github.com/ppapalampidi/SUMMER/tree/master/video_summaries
Turning Point	Crime scene	Victim	Death Cause	Perpetrator	Evidence	Motive
Opportunity	56.76	52.63	15.63	15.38	2.56	0.00
Change of Plans	27.03	42.11	21.88	15.38	5.13	0.00
Point of no Return	8.11	13.16	9.38	25.64	48.72	5.88
Major Setback	0.00	0.00	6.25	10.25	48.72	35.29
Climax	2.70	0.00	6.25	2.56	23.08	55.88
Table 5: Percentage of aspect labels covered per TP for SUMMER, +P, +R.
System	Crime scene	Victim	Death Cause	Perpetrator	Evidence	Motive	Overall	Rank
SUMMARUNNER*	85.71	93.88	75.51	81.63	59.18	38.78	72.45	2.18
SUMMER	89.80	87.76	83.67	81.63	77.55	57.14	79.59	2.00
Gold	89.80	91.84	71.43	83.67	65.31	57.14	76.53	1.82
Table 6: Human evaluation: percentage of yes answers by AMT workers regarding each aspect in a summary. All differences in (average) Rank are significant ( $p < 0.05$ , using a  $\chi^2$  test).
per aspect. For SUMMARUNNER* and SUMMER we selected the top six predicted scenes based on their posterior probabilities. We then created video summaries by isolating and merging the selected scenes in the raw video.
We asked Amazon Mechanical Turk (AMT) workers to watch the video summaries for all systems and rank them from most to least informative. They were also presented with six questions relating to the aspects the summary was supposed to cover (e.g., Was the victim revealed in the summary? Do you know who the perpetrator was?). They could answer Yes, No, or Unsure. Five workers evaluated each summary.
Table 6 shows the proportion of times participants responded Yes for each aspect across the three systems. Although SUMMER does not improve over SUMMARUNNER* in identifying basic information (i.e., about the victim and perpetrator), it creates better summaries overall with more diverse content (i.e., it more frequently includes information about cause of death, evidence, and motive). This observation validates our assumption that identifying scenes that are semantically close to the key events of a screenplay leads to more complete and detailed summaries. Finally, Table 6 also lists the average rank per system (lower is better), which shows that crowdworkers like gold summaries best, SUMMER is often ranked second, followed by SUMMARUNNER* in third place.
In this paper we argued that the underlying structure of narratives is beneficial for long-form summarization. We adapted a scheme for identifying narrative structure (i.e., turning points) in Hollywood movies and showed how this information
can be integrated with supervised and unsupervised extractive summarization algorithms. Experiments on the CSI corpus showed that this scheme transfers well to a different genre (crime investigation) and that utilizing narrative structure boosts summarization performance, leading to more complete and diverse summaries. Analysis of model output further revealed that latent events encapsulated by turning points correlate with important aspects of a CSI summary.
Although currently our approach relies solely on textual information, it would be interesting to incorporate additional modalities such as video or audio. Audiovisual information could facilitate the identification of key events and scenes. Besides narrative structure, we would also like to examine the role of emotional arcs (Vonnegut, 1981; Reagan et al., 2016) in a screenplay. An often integral part of a compelling story is the emotional experience that is evoked in the reader or viewer (e.g., somebody gets into trouble and then out of it, somebody finds something wonderful, loses it, and then finds it again). Understanding emotional arcs may be useful to revealing a story's shape, highlighting important scenes, and tracking how the story develops for different characters over time.
We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the European Research Council (Lapata; award 681760, "Translating Multiple Modalities into Text") and of the Leverhulme Trust (Keller; award IAF-2017-019).
Apoory Agarwal, Sriramkumar Balasubramanian, Jiehan Zheng, and Sarthak Dash. 2014. Parsing
Screenplays for Extracting Social Networks from Movies. In Proceedings of the 3rd Workshop on Computational Linguistics for Literature, pages 50-58, Gothenburg, Sweden.
David Bamman, Brendan O'Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361, Sofia, Bulgaria.
David Bamman, Ted Underwood, and Noah A. Smith. 2014. A Bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370-379, Baltimore, Maryland.
John B Black and Robert Wilensky. 1979. An evaluation of story grammars. Cognitive science, 3(3):213-229.
Charles Oscar Brink. 2011. Horace on Poetry: The 'Arts Poetica'. Cambridge University Press.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602-610, Suntec, Singapore.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484-494.
James E Cutting. 2016. Narrative theory and the dynamics of popular movies. Psychonomic bulletin & review, 23(6):1713-1743.
Yue Dong. 2018. A survey on neural network-based summarization methods. ArXiv, abs/1804.04589.
David K. Elson, Nicholas Dames, and Kathleen R. Mckeown. 2010. Extracting social networks from literary fiction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 138-147, Uppsala, Sweden.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Syd Field. 2005. Screenplay: Foundations of Screen-writing. Dell Publishing Company.
Mark Alan Finlayson. 2012. Learning Narrative Structure from Annotated Folktales. Ph.D. thesis, Massachusetts Institute of Technology.
Lea Frermann, Shay B Cohen, and Mirella Lapata. 2018. Whodunnit? crime drama as a case for natural language understanding. Transactions of the Association of Computational Linguistics, 6:1-15.
Gustav Freytag. 1896. Freytag's technique of the drama: an exposition of dramatic composition and art. Scholarly Press.
Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066-1076, Denver, Colorado. Association for Computational Linguistics.
Amit Goyal, Ellen Riloff, and Hal Daumé III. 2010. Automatically producing plot unit representations for narrative text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 77-86, Cambridge, MA.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. NEWSROOM: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 708-719, New Orleans, USA.
Michael Hauge. 2017. Storytelling Made Easy: Persuade and Transform Your Audiences, Buyers, and Clients - Simply, Quickly, and Profitably. Indie Books International.
Marti A Hearst. 1997. Texttiling: Segmenting text into multi-paragraph subtopic passages. Computational linguistics, 23(1):33-64.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693-1701. Morgan, Kaufmann.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Anna Kazantseva and Stan Szpakowicz. 2010. Summarizing short stories. Computational Linguistics, 36(1):71-109.
Joy Kim and Andres Monroy-Hernandez. 2015. Storia: Summarizing social media content based on narrative theory using crowdsourcing. CoRR, abs/1509.03026.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Wendy G. Lehnert. 1981. Plot units and narrative summarization. Cognitive Science, 5(4):293-331.
Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
Interjeet Mani. 2012. Computational Modeling of Narative. Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers.
Neil McIntyre and Mirella Lapata. 2010. Plot induction and evolutionary search for story generation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1562-1572, Uppsala, Sweden.
Rada Mihalcea and Hakan Ceylan. 2007. Explorations in automatic book summarization. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 380-389.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359.
Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1707-1717.