SlowGuess/ACL_2020_OCR huggingface dataset:
Pinelopi Papalampidi $^{1}$  Frank Keller $^{1}$  Lea Frermann $^{2}$  Mirella Lapata $^{1}$
$^{1}$ Institute for Language, Cognition and Computation
$^{2}$ School of Computing and Information Systems
p.papalampidi@sms.ed.ac.uk, keller@inf.ed.ac.uk,
lea.frermann@unimelb.edu.au, mlap@inf.ed.ac.uk
Most general-purpose extractive summarization models are trained on news articles, which are short and present all important information upfront. As a result, such models are biased by position and often perform a smart selection of sentences from the beginning of the document. When summarizing long narratives, which have complex structure and present information piecemeal, simple position heuristics are not sufficient. In this paper, we propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models. We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenings (i.e., extract an optimal sequence of scenes). Experimental results on the CSI corpus of TV screenings, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode and improve summarization performance over general extractive algorithms, leading to more complete and diverse summaries.
Automatic summarization has enjoyed renewed interest in recent years thanks to the popularity of modern neural network-based approaches (Cheng and Lapata, 2016; Nallapati et al., 2016, 2017; Zheng and Lapata, 2019) and the availability of large-scale datasets containing hundreds of thousands of document-summary pairs (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Narayan et al., 2018; Fabbri et al., 2019; Liu and Lapata, 2019). Most efforts to date have concentrated on the summarization of news articles which tend to be relatively short and formulaic following an "inverted pyramid" structure which places the most essential, novel and interesting el
Figure 1: Example of narrative structure for episode "Burden of Proof" from TV series Crime Scene Investigation (CSI); turning points are highlighted in color.
ements of a story in the beginning and supporting material and secondary details afterwards. The rigid structure of news articles is expedient since important passages can be identified in predictable locations (e.g., by performing a “smart selection” of sentences from the beginning of the document) and the structure itself can be explicitly taken into account in model design (e.g., by encoding the relative and absolute position of each sentence).
In this paper we are interested in summarizing longer narratives, i.e., screenplays, whose form and structure is far removed from newspaper articles. Screenplays are typically between 110 and 120 pages long (20k words), their content is broken down into scenes, which contain mostly dialogue (lines the actors speak) as well as descriptions explaining what the camera sees. Moreover, screenplays are characterized by an underlying narrative structure, a sequence of events by which
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1920-1933
Figure 2: We first identify scenes that act as turning points (i.e., key events that segment the story into sections). We next create a summary by selecting informative scenes, i.e., semantically related to turning points.
a story is defined (Cutting, 2016), and by the story's characters and their roles (Propp, 1968). Contrary to news articles, the gist of the story in a screenplay is not disclosed at the start, information is often revealed piecemeal; characters evolve and their actions might seem more or less important over the course of the narrative. From a modeling perspective, obtaining training data is particularly problematic: even if one could assemble screenplays and corresponding summaries (e.g., by mining IMDb or Wikipedia), the size of such a corpus would be at best in the range of a few hundred examples not hundreds of thousands. Also note that genre differences might render transfer learning (Pan and Yang, 2010) difficult, e.g., a model trained on movie screenings might not generalize to sitcoms or soap operas.
Given the above challenges, we introduce a number of assumptions to make the task feasible. Firstly, our goal is to produce informative summaries, which serve as a surrogate to reading the full script or watching the entire film. Secondly, we follow Gorinski and Lapata (2015) in conceptualizing screenplay summarization as the task of identifying a sequence of informative scenes. Thirdly, we focus on summarizing television programs such as CSI: Crime Scene Investigation (Fr
ermann et al., 2018) which revolves around a team of forensic investigators solving criminal cases. Such programs have a complex but well-defined structure: they open with a crime, the crime scene is examined, the victim is identified, suspects are introduced, forensic clues are gathered, suspects are investigated, and finally the case is solved.
In this work, we adapt general-purpose extractive summarization algorithms (Nallapati et al., 2017; Zheng and Lapata, 2019) to identify informative scenes in screenplays and instill in them knowledge about narrative film structure (Hauge, 2017; Cutting, 2016; Freytag, 1896). Specifically, we adopt a scheme commonly used by screenwriters as a practical guide for producing successful screenplays. According to this scheme, well-structured stories consist of six basic stages which are defined by five turning points (TPs), i.e., events which change the direction of the narrative, and determine the story's progression and basic thematic units. In Figure 1, TPs are highlighted for a CSI episode. Although the link between turning points and summarization has not been previously made, earlier work has emphasized the importance of narrative structure for summarizing books (Mihalcea and Ceylan, 2007) and social media content (Kim and Monroy-Hernandez, 2015). More recently, Papalampidi et al. (2019) have shown how to identify turning points in feature-length screenplays by projecting synopsis-level annotations.
Crucially, our method does not involve manually annotating turning points in CSI episodes. Instead, we approximate narrative structure automatically by pretraining on the annotations of the TRIPOD dataset of Papalampidi et al. (2019) and employing a variant of their model. We find that narrative structure representations learned on their dataset (which was created for feature-length films), transfer well across cinematic genres and computational tasks. We propose a framework for end-to-end training in which narrative structure is treated as a latent variable for summarization. We extend the CSI dataset (Frermann et al., 2018) with binary labels indicating whether a scene should be included in the summary and present experiments with both supervised and unsupervised summarization models. An overview of our approach is shown in Figure 2.
Our contributions can be summarized as follows: (a) we develop methods for instilling knowledge about narrative structure into generic su
pervised and unsupervised summarization algorithms; (b) we provide a new layer of annotations for the CSI corpus, which can be used for research in long-form summarization; and (c) we demonstrate that narrative structure can facilitate screenplay summarization; our analysis shows that key events identified in the latent space correlate with important summary content.
A large body of previous work has focused on the computational analysis of narratives (Mani, 2012; Richards et al., 2009). Attempts to analyze how stories are written have been based on sequences of events (Schank and Abelson, 1975; Chambers and Jurafsky, 2009), plot units (McIntyre and Lapata, 2010; Goyal et al., 2010; Finlayson, 2012) and their structure (Lehnert, 1981; Rumelhart, 1980), as well as on characters or personas in a narrative (Black and Wilensky, 1979; Propp, 1968; Bamman et al., 2014, 2013; Valls-Vargas et al., 2014) and their relationships (Elson et al., 2010; Agarwal et al., 2014; Srivastava et al., 2016).
As mentioned earlier, work on summarization of narratives has had limited appeal, possibly due to the lack of annotated data for modeling and evaluation. Kazantseva and Szpakowicz (2010) summarize short stories based on importance criteria (e.g., whether a segment contains protagonist or location information); they create summaries to help readers decide whether they are interested in reading the whole story, without revealing its plot. Mihalcea and Ceylan (2007) summarize books with an unsupervised graph-based approach operating over segments (i.e., topical units). Their algorithm first generates a summary for each segment and then an overall summary by collecting sentences from the individual segment summaries.
Focusing on screenplays, Gorinski and Lapata (2015) generate a summary by extracting an optimal chain of scenes via a graph-based approach centered around the main characters. In a similar fashion, Tsoneva et al. (2007) create video summaries for TV series episodes; their algorithm ranks sub-scenes in terms of importance using features based on character graphs and textual cues available in the subtitles and movie scripts. Vicol et al. (2018) introduce the MovieGraphs dataset, which also uses character-centered graphs to describe the content of movie video clips.
Our work synthesizes various strands of re-
search on narrative structure analysis (Cutting, 2016; Hauge, 2017), screenplay summarization (Gorinski and Lapata, 2015), and neural network modeling (Dong, 2018). We focus on extractive summarization and our goal is to identify an optimal sequence of key events in a narrative. We aim to create summaries which re-tell the plot of a story in a concise manner. Inspired by recent neural network-based approaches (Cheng and Lapata, 2016; Nallapati et al., 2017; Zhou et al., 2018; Zheng and Lapata, 2019), we develop supervised and unsupervised models for our summarization task based on neural representations of scenes and how these relate to the screenplay's narrative structure. Contrary to most previous work which has focused on characters, we select summary scenes based on events and their importance in the story. Our definition of narrative structure closely follows Papalampidi et al. (2019). However, the model architectures we propose are general and could be adapted to different plot analysis schemes (Field, 2005; Vogler, 2007). To overcome the difficulties in evaluating summaries for longer narratives, we also release a corpus of screenplays with scenes labeled as important (summary worthy). Our annotations augment an existing dataset based on CSI episodes (Frermann et al., 2018), which was originally developed for incremental natural language understanding.
Let  $\mathcal{D}$  denote a screenplay consisting of a sequence of scenes  $\mathcal{D} = \{s_1,s_2,\ldots ,s_n\}$ . Our aim is to select a subset  $\mathcal{D}' = \{s_i,\dots ,s_k\}$  consisting of the most informative scenes (where  $k < n$ ). Note that this definition produces extractive summaries; we further assume that selected scenes are presented according to their order in the screenplay. We next discuss how summaries can be created using both unsupervised and supervised approaches, and then move on to explain how these are adapted to incorporate narrative structure.
3.1 Unsupervised Screenplay Summarization
Our unsupervised model is based on an extension of TEXTRANK (Mihalcea and Tarau, 2004; Zheng and Lapata, 2019), a well-known algorithm for extractive single-document summarization. In our setting, a screenplay is represented as a graph, in which nodes correspond to scenes and edges between scenes  $s_i$  and  $s_j$  are weighted by their simi
larity  $e_{ij}$ . A node's centrality (importance) is measured by computing its degree:
$$
\operatorname {c e n t r a l i t y} \left(s _ {i}\right) = \lambda_ {1} \sum_ {j <   i} e _ {i j} + \lambda_ {2} \sum_ {j > i} e _ {i j} \tag {1}
$$
where  $\lambda_1 + \lambda_2 = 1$ . The modification introduced in Zheng and Lapata (2019) takes directed edges into account, capturing the intuition that the centrality of any two nodes is influenced by their relative position. Also note that the edges of preceding and following scenes are differentially weighted by  $\lambda_1$  and  $\lambda_2$ .
Although earlier implementations of TEXT-RANK (Mihalcea and Tarau, 2004) compute node similarity based on symbolic representations such as tfidf, we adopt a neural approach. Specifically, we obtain sentence representations based on a pretrained encoder. In our experiments, we rely on the Universal Sentence Encoder (USE; Cer et al. 2018), however, other embeddings are possible. We represent a scene by the mean of its sentence representations and measure scene similarity  $e_{ij}$  using cosine. As in the original TEXTRANK algorithm (Mihalcea and Tarau, 2004), scenes are ranked based on their centrality and the  $M$  most central ones are selected to appear in the summary.
3.2 Supervised Screenplay Summarization
Most extractive models frame summarization as a classification problem. Following a recent approach (SUMMARUNNER; Nallapati et al. 2017), we use a neural network-based encoder to build representations for scenes and apply a binary classifier over these to predict whether they should be in the summary. For each scene  $s_i \in \mathcal{D}$ , we predict a label  $y_i \in \{0,1\}$  (where 1 means that  $s_i$  must be in the summary) and assign a score  $p(y_i | s_i, \mathcal{D}, \theta)$  quantifying  $s_i$ 's relevance to the summary ( $\theta$  denotes model parameters). We assemble a summary by selecting  $M$  sentences with the top  $p(1 | s_i, \mathcal{D}, \theta)$ .
We calculate sentence representations via the pre-trained USE encoder (Cer et al., 2018); a scene is represented as the weighted sum of the representations of its sentences, which we obtain from a BiLSTM equipped with an attention mechanism. Next, we compute richer scene representations by modeling surrounding context of a given scene.
We encode the screenplay with a BiLSTM network and obtain contextualized representations  $s_i'$  for scenes  $s_i$  by concatenating the hidden layers of the forward  $\vec{h_i}$  and backward  $\overleftarrow{h_i}$  LSTM, respectively:  $s_i' = [\vec{h_i};\vec{h_i}]$ . The vector  $s_i'$  therefore represents the content of the  $i^{th}$  scene.
We also estimate the salience of scene  $s_i$  by measuring its similarity with a global screenplay content representation  $d$ . The latter is the weighted sum of all scene representations  $s_1, s_2, \ldots, s_n$ . We calculate the semantic similarity between  $s_i'$  and  $d$  by computing the element-wise dot product  $b_i$ , cosine similarity  $c_i$ , and pairwise distance  $u_i$  between their respective vectors:
$$
b _ {i} = s _ {i} ^ {\prime} \odot d \quad c _ {i} = \frac {s _ {i} ^ {\prime} \cdot d}{\left\| s _ {i} ^ {\prime} \right\| \| d \|} \tag {2}
$$
$$
u _ {i} = \frac {s _ {i} ^ {\prime} \cdot d}{\max \left(\left\| s _ {i} ^ {\prime} \right\| _ {2} \cdot \left\| d \right\| _ {2}\right)} \tag {3}
$$
The salience  $\nu_{i}$  of scene  $s_i$  is the concatenation of the similarity metrics:  $\nu_{i} = [b_{i};c_{i};u_{i}]$ . The content vector  $s_i'$  and the salience vector  $\nu_{i}$  are concatenated and fed to a single neuron that outputs the probability of a scene belonging to the summary.
3.3 Narrative Structure
We now explain how to inject knowledge about narrative structure into our summarization models. For both models, such knowledge is transferred via a network pre-trained on the TRIPOD $^4$  dataset introduced by Papalampidi et al. (2019). This dataset contains 99 movies annotated with turning points. TPs are key events in a narrative that define the progression of the plot and occur between consecutive acts (thematic units). It is often assumed (Cutting, 2016) that there are six acts in a film (Figure 1), each delineated by a turning point (arrows in the figure). Each of the five TPs has also a well-defined function in the narrative: we present each TP alongside with its definition as stated in screenwriting theory (Hauge, 2017) and adopted by Papalampidi et al. (2019) in Table 1 (see Appendix A for a more detailed description of narrative structure theory). Papalampidi et al. (2019) identify scenes in movies that correspond to these key events as a means for analyzing the narrative
<sup>3</sup>Aside from salience and content, Nallapati et al. (2017) take into account novelty and position-related features. We ignore these as they are specific to news articles and denote the modified model as SUMMARUNNER*.
4https://github.com/ppapalampidi/TRIPOD
1USE performed better than BERT in our experiments.
2We found cosine to be particularly effective with USE representations; other metrics are also possible.
Turning Point	Definition
TP1: Opportunity	Introductory event that occurs after the presentation of the story setting.
TP2: Change of Plans	Event where the main goal of the story is defined.
TP3: Point of No Return	Event that pushes the main character(s) to fully commit to their goal.
TP4: Major Setback	Event where everything falls apart (temporarily or permanently).
TP5: Climax	Final event of the main story, moment of resolution.
Table 1: Turning points and their definitions as given in Papalampidi et al. (2019)
structure of movies. They collect sentence-level TP annotations for plot synopses and subsequently project them via distant supervision onto screenplays, thereby creating silver-standard labels. We utilize this silver-standard dataset in order to pretrain a network which performs TP identification.
TP Identification Network We first encode screenplay via a BiLSTM equipped with an attention mechanism. We then contextualize them with respect to the whole screenplay via a second BiLSTM. Next, we compute topic-aware scene representations  $t_i$  via a context interaction layer (CIL) as proposed in Papalampidi et al. (2019). CIL is inspired by traditional segmentation approaches (Hearst, 1997) and measures the semantic similarity of the current scene with a preceding and following context window in the screenplay. Hence, the topic-aware scene representations also encode the degree to which each scene acts as a topic boundary in the screenplay.
In the final layer, we employ TP-specific attention mechanisms to compute the probability  $p_{ij}$  that scene  $t_i$  represents the  $j^{th}$  TP in the screenplay. Note that we expect the TP-specific attention distributions to be sparse, as there are only a few scenes which are relevant for a TP (recall that TPs are boundary scenes between sections). To encourage sparsity, we add a low temperature value  $\tau$  (Hinton et al., 2015) to the softmax part of the attention mechanisms:
$$
g _ {i j} = \tanh  \left(W _ {j} t _ {i} + b _ {j}\right), \quad g _ {j} \in [ - 1, 1 ] \tag {4}
$$
$$
p _ {i j} = \frac {\exp \left(g _ {i j} / \tau\right)}{\sum_ {t = 1} ^ {T} \exp \left(g _ {t j} / \tau\right)}, \quad \sum_ {i = 1} ^ {T} p _ {i j} = 1 \tag {5}
$$
where  $W_{j}, b_{j}$  represent the trainable weights of the attention layer of the  $j^{th}$  TP.
Unsupervised SUMMER We now introduce our model, SUMMER (short for Screenplay
Summarization with Narrative Structure). We first present an unsupervised variant which modifies the computation of scene centrality in the directed version of TEXTRANK (Equation (1)).
Specifically, we use the pre-trained network described in Section 3.3 to obtain TP-specific attention distributions. We then select an overall score  $f_{i}$  for each scene (denoting how likely it is to act as a TP). We set  $f_{i} = \max_{j\in [1,5]}p_{ij}$ , i.e., to the  $p_{ij}$  value that is highest across TPs. We incorporate these scores into centrality as follows:
$$
c e n t r a l i t y \left(s _ {i}\right) = \lambda_ {1} \sum_ {j <   i} \left(e _ {i j} + f _ {j}\right) + \lambda_ {2} \sum_ {j > i} \left(e _ {i j} + f _ {i}\right) \tag {6}
$$
Intuitively, we add the  $f_{j}$  term in the forward sum in order to incrementally increase the centrality scores of scenes as the story moves on and we encounter more TP events (i.e., we move to later sections in the narrative). At the same time, we add the  $f_{i}$  term in the backward sum in order to also increase the scores of scenes identified as TPs.
Supervised SUMMER We also propose a supervised variant of SUMMER following the basic model formulation in Section 3.3. We still represent a scene as the concatenation of a content vector  $s'$  and salience vector  $\nu'$ , which serve as input to a binary classifier. However, we now modify how salience is determined; instead of computing a general global content representation  $d$  for the screenplay, we identify a sequence of TPs and measure the semantic similarity of each scene with this sequence. Our model is depicted in Figure 3.
We utilize the pre-trained TP network (Figures 3(a) and (b)) to compute sparse attention scores over scenes. In the supervised setting, where gold-standard binary labels provide a training signal, we fine-tune the network in an end-to-end fashion on summarization (Figure 3(c)). We compute the TP representations via the attention scores; we calculate a vector  $tp_{j}$  as the weighted sum of all topic-aware scene representations  $t$  produced via CIL:  $tp_{j} = \sum_{i\in [1,N]}p_{ij}t_{i}$ , where  $N$  is the number of scenes in a screenplay. In practice, only a few scenes contribute to  $tp_{j}$  due to the  $\tau$  parameter in the softmax function (Equation (5)).
A TP-scene interaction layer measures the semantic similarity between scenes  $t_i$  and latent TP representations  $tp_j$  (Figure 3(c)). Intuitively, a complete summary should contain scenes which
We make our code publicly available at https://github.com/ppapalampidi/SUMMER.
Figure 3: Overview of SUMMER. We use one TP-specific attention mechanism per turning point in order to acquire TP-specific distributions over scenes. We then compute the similarity between TPs and contextualized scene representations. Finally, we perform max pooling over TP-specific similarity vectors and concatenate the final similarity representation with the contextualized scene representation.
are related to at least one of the key events in the screenplay. We calculate the semantic similarity  $\nu_{ij}$  of scene  $t_i$  with TP  $tp_j$  as in Equations (2) and (3). We then perform max pooling over vectors  $\nu_{i1},\ldots ,\nu_{iT}$ , where  $T$  is the number of TPs (i.e., five) and calculate a final similarity vector  $\nu_i^\prime$  for the  $i^{th}$  scene.
The model is trained end-to-end on the summarization task using  $BCE$ , the binary cross-entropy loss function. We add an extra regularization term to this objective to encourage the TP-specific attention distributions to be orthogonal (since we want each attention layer to attend to different parts of the screenplay). We thus maximize the Kullback-Leibler (KL) divergence  $\mathcal{D}_{KL}$  between all pairs of TP attention distributions  $tp_{i}, i \in [1,5]$ :
$$
O = \sum_ {i \in [ 1, 5 ]} \sum_ {j \in [ 1, 5 ], j \neq i} \log \frac {1}{\mathcal {D} _ {K L} \left(t p _ {i} \| t p _ {j}\right) + \varepsilon} \tag {7}
$$
Furthermore, we know from screenwriting theory (Hauge, 2017) that there are rules of thumb as to
when a TP should occur (e.g., the Opportunity occurs after the first  $10\%$  of a screenplay, Change of Plans is approximately  $25\%$  in). It is reasonable to discourage  $tp$  distributions to deviate drastically from these expected positions. Focal regularization  $F$  minimizes the KL divergence  $\mathcal{D}_{KL}$  between each TP attention distribution  $tp_{i}$  and its expected position distribution  $th_{i}$ :
$$
F = \sum_ {i \in [ 1, 5 ]} \mathcal {D} _ {K L} (t p _ {i} \| t h _ {i}) \tag {8}
$$
The final loss  $\mathcal{L}$  is the weighted sum of all three components, where  $a,b$  are fixed during training:  $\mathcal{L} = BCE + aO + bF$ .