Venue: NAACL 2018

# What's this Movie about?

A Joint Neural Network Architecture for Movie Content Analysis

Philip John Gorinski

Mirella Lapata

Institute for Language, Cognition and Computation

School of Informatics, University of Edinburgh

10 Crichton Street, Edinburgh, EH8 9AB

P.J.Gorinski@sms.ed.ac.uk, mlap@inf.edu.ac.uk

###### Abstract

This work takes a first step toward movie content analysis by tackling the novel task of movie overview generation. Overviews are natural language texts that give a first impression of a movie, describing aspects such as its genre, plot, mood, or artistic style. We create a dataset that consists of movie scripts, attribute-value pairs for the movies' aspects, as well as overviews, which we extract from an online database. We present a novel end-to-end model for overview generation, consisting of a multi-label encoder for identifying screenplay attributes, and an LSTM decoder to generate natural language sentences conditioned on the identified attributes. Automatic and human evaluation show that the encoder is able to reliably assign good labels for the movie's attributes, and the overviews provide descriptions of the movie's content which are informative and faithful.

+
Footnote †: The Silence of the Lamb’s in sans serif.

+
Footnote †: The Silence of the Lamb’s in sans serif.

+
Footnote †: The Silence of the Lamb’s in sans serif.

+
Footnote †: The Silence of the Lamb’s in sans serif.

+
Footnote †: The Silence of the Lamb’s in sans serif.

+
Footnote †: The Silence of the Lamb’s in sans serif.

## 1 Introduction

Movie summarization is the task of automatically summarizing a screenplay in order to gain a general impression of its content. This may include describing the movie's main characters and plot, its genre, artistic style, and so on. As more and more movies are being produced every year1, there is an ever growing need to facilitate this task. Potential applications include producing shorter versions of scripts to help with the decision making process in a production company, enhancing movie search by generating descriptions of what the movie is about, and notably, supporting movie recommendation engines by abstracting over specific keywords to more general concepts.

Footnote 1: footnotemark:

Figure 1 gives an example of the type of movie content analysis we would like to obtain automatically. The information is taken from Jinni, a large database (and movie recommendation engine) which indexes movies based on attributes and their values2 (see the top half of Figure 1) and further aggregates these into a comprehensive overview (see the second half of Figure 1). Jinni's movie attributes were created by film professionals based on analysis of user reviews and metadata. There are hundreds, and they aim to describe aspects such as mood, style, plot, and setting for any released movie or TV show. Although some of these attributes could not be possibly ascribed without information from external sources (e.g., _Praise_, or _Based on_), others could be inferred by watching the movie or reading the video

Figure 1: Jinni attributes, their values, and overview for “The Silence of the Lamb’s”. Underlined attribute values appear in the overview.



screenplay (e.g., _Genre, Plot, Flag, Mood, Place_).

This work takes a step toward automatic script summarization by jointly modeling the tasks of movie attribute identification and overview generation. Specifically, we propose a novel neural network architecture which draws insights from encoder-decoder models recently proposed for machine translation Bahdanau et al. (2015) and related sentence generation tasks Wen et al. (2015); Mei et al. (2016); Lebret et al. (2016). Our model takes the screenplay as input and generates an overview for it. Rather than representing the script as a sequence, we employ feed-forward neural networks Zhang and Zhou (2006); Kurata et al. (2016) to encode the screenplay into various attributes (e.g., _Plot_, _Genre_) and their labels (e.g., thriller, romance), viewing movie content analysis as a multi-label classification problem. Our decoder generates movie overviews using a Long Short-Term Memory network LSTM; Hochreiter and Schmidhuber (1997), a type of recurrent neural network with a more complex computational unit which is semantically conditioned Wen et al. (2015, 2016) on this attribute specific representation. Our model is trained end-to-end using screenplays and movie overviews as the supervision signal.

In both automatic and human-based evaluations our neural network architecture outperforms competitive baselines and generates movie overviews which are well-received by human judges. To the best of our knowledge, this is the first work to automatically analyze and summarize the content of screenplays.

## 2 Related Work

Recent years have seen increased interest in the computational analysis of movie screenplays. Ye and Baldwin (2008) create animated storyboards using the action descriptions of movie scripts. Danescu-Niculescu-Mizil and Lee (2011) use screenplays to study the coordination of linguistic styles in dialog. Bamman et al. (2013) induce personas of film characters from movie plot summaries. Agarwal et al. (2014a, 2014b, 2015) extract social networks from scripts, create _xkcd_ movie narrative charts, and automate the Bechdel test which is designed to assess the presence of women in movies. Gorinski and Lapata (2015) summarize screenplays by selecting important scenes. Our work joins this line of research in an attempt to automatically induce information pertaining to a movie's content such as its genre and plot elements.

There has been a surge of interest recently in repurposing sequence transduction neural network architectures for various generation tasks such as machine translation Sutskever et al. (2014), sentence compression Chopra et al. (2016), and simplification Zhang and Lapata (2017). Central to these approaches is an encoder-decoder architecture modeled by recurrent neural networks. The encoder reads the source sequence into a list of continuous-space representations from which the decoder generates the target sequence. Previously proposed architectures are not directly applicable to our task for at least two reasons: (a) the correspondence between screenplays and overviews is very loose, and (b) the screenplay is not strictly speaking a sequence (a screenplay is more like a book consisting of thousands of sentences), and cannot be easily compressed into a vector-based representation from which to generate the overview.

Rather than attempting to decode the overview directly from the screenplay, we encode the latter into attribute-value pairs which we then decode into overviews. We conceptualize the generation task as a joint problem of multi-label categorization, where each screenplay is assigned to one or more categories, and content-sensitive natural language generation. Many machine learning techniques have been proposed for building automatic text categorization systems (see Sebastiani, 2002 and Dalal and Zaveri, 2011 for overviews), including neural networks Belanger and McCallum (2016); Kurata et al. (2016). Our encoder is a feedforward neural network, which, however, is able to capture label interactions which are important for our content analysis task. Our decoder employs an enhanced LSTM architecture which directly maximizes the probability of the overview given the screenplay's attribute values. Conditional LSTMs have been applied to various related tasks, including image description generation Vinyals et al. (2015), the verbalization of database records Mei et al. (2016); Lebret et al. (2016), and the generation of dialogue acts Wen et al. (2015, 2016).

## 3 The Jinni Movie Dataset

Our dataset was built on top of ScriptBase, a collection of 1,276 movie scripts, which Gorniski and 

Lapata (2015) obtained by automatically crawling web-sites such as imsdb.com. We crawled Jinni3 in order to obtain attributes and overviews (see Figure 1) for each movie in ScriptBase. As mentioned earlier, attributes have values which are essentially labels/tags describing the movie's content, whereas overviews are short summaries giving a first impression of the movie. The crawl resulted in 917 movies which Jinni and ScriptBase had in common. We further split these into training, development and test sets, with 617, 200, and 100 instances, respectively. We concentrate on the six types of attributes shown in Table 1 whose values we hypothesize can be inferred from analyzing the movie's screenplay.

Footnote 3: Data set is available from [http://www.jinni.com/](http://www.jinni.com/).

Table 1 provides an overview of the number of labels used in our experiments. Jinni contains a wealth of attribute values varying from nine for _Flag_ to more than 400 for \(Plot\). Additionally, value names for some attributes are synonyms or near-synonyms (e.g., Nudity and Brief Nudity for _Flag_). We reduced the set of attribute values to those that occurred most frequently (column Frequent in the table) and merged synonyms into a common label (column Merged).

## 4 Neural Network Architecture

We could approach the movie overview generation task using an attention-based encoder-decoder model Bahdanau et al. (2015). The encoder would transform the screenplay into a sequence of hidden states with an LSTM Hochreiter and Schmidhuber (1997) or another type of computational unit Cho et al. (2014). The decoder would use another recurrent neural network to generate the overview one word at time, conditioning on all previously generated words and the representation of the input, while an attention mechanism would revisit the input sequence dynamically highlighting pieces of information relevant for the generation task. As mentioned earlier, viewing screensplays as a sequence of sentences is problematic both computationally and conceptually. Even if we used a hierarchical encoder Tang et al. (2015); Yang et al. (2016) by first building representations of sentences and then aggregating those into a representation of a screwalpay, it is doubtful whether a fixed length vector could encode the content of the movie in its entirety or whether the attention mechanism would effectively isolate the parts of the input relevant for generation.

We therefore propose an architecture that consists of two stacked neural network models for the tasks of movie attribute identification and overview generation. Figure 2 illustrates our model. We use simple feed-forward neural networks to impose some structure on the input by identifying the labels that most likely apply to the screenplay. We subsequently employ a semantically conditioned LSTM Wen et al. (2015, 2016)

Figure 2: Neural network architecture: given feature vector \(x\) representing a screenplay, we employ feedforward multi-label classification networks to encode the movie into a content vector \(p_{\theta}\) representing attribute labels; this encoding is fed into an LSTM with a content selection cell.

 

to select the content for which to generate sentences. This architecture is advantageous for a number of reasons. Firstly, by imposing structure on the screenplays, the generation network is faced with a more compact and informative representation. This allows us to make use of a content selection LSTM similar to Wen et al. (2015); Wen et al. (2016), generating fluent and label-specific outputs. Secondly, it enables us to train the screenplay encoder (aka classification network) and the decoder jointly, in an end-to-end fashion.

### Multi-label Encoder

As shown in Figure 1, the overview highlights various aspects of the movie, essentially devoting a sentence to each attribute. This observation motivates us to encode the screenplay as a set of attributes (with their values) and then decode these into a sentence one by one. We treat attribute encoding as a multi-label classification problem: an attribute (e.g., _Genre_ or _Plot_), will typically have multiple values (aka labels) which are suitable for the movie and should occur in the generated sentence. Furthermore, these labels naturally influence each other. For example, a movie whose _Genre_ is Crime is also likely to be a Thriller while it is less likely to be a Parody. In traditional multi-label classification such interactions are either ignored Read et al. (2011); Tsoumakas and Katakis (2006); Godbole and Sarawagi (2004); Zhang and Zhou (2005), or represented by label combinations Tsoumakas and Vlahavas (2007); Read et al. (2008). A few approaches assume or impose an existing structure on the label space Schwing and Urtasun (2015); Chen et al. (2015); Huang et al. (2015); Jaderberg et al. (2014); Stoyanov et al. (2011); Hershey et al. (2014); Zheng et al. (2015).

We employ a neural network approach with the aim of abstracting the screenplay into a set of meaningful labels whose correlations are discovered automatically, during training. As shown on top of Figure 2, our encoder is a feed-forward neural network where individual neurons represent the labels to be classified. The input to the network is a feature vector representing the screenplay (we discuss the specific features we use in more detail shortly):

\[h_{n}=\sigma(W_{n}x_{n}) \tag{1}\]

The input is split into \(k\) segments by feature type, and the feature segments are fed into \(k\) separate fully connected hidden layers. The hidden layer outputs are then combined using simple elementwise addition:

\[f=h_{1}\oplus h_{2}\oplus\cdots\oplus h_{k} \tag{2}\]

The combined feature layer is used to compute an \(l\)-sized output layer, where \(l\) corresponds to the size of the classification label set. The final activation of the output units is obtained by applying the sigmoid function to the output layer:

\[O=\sigma(W_{o}f) \tag{3}\]

In order to better capture label interactions, we adapt a method of network initialization recently introduced in Kurata et al. (2016). In this approach, instead of initializing the model's output weights \(W_{o}\) from a uniform distribution, the first \(p\) rows of the weight matrix are initialized according to \(\lambda\) patterns observed in the data. To this end, we initialize the \(n\)th row of \(W_{o}\) with pattern \(\lambda_{n}\) (equation (4)), which is a vector corresponding to the \(n\)th label-assignment observed in the training data:

\[W^{n}_{o}=i(\lambda_{n}) \tag{4}\]

The initialization weight \(i\) for unit \(l\) of pattern \(\lambda_{n}\) is set to 0 if the corresponding label is not present in the given instance; or to the upper bound \(UB\) of the normalized initialization weights of hidden layer \(h\) and output layer \(o\), scaled by the number of times \(c\) the pattern occurs in the data:

\[i(\lambda^{l}_{n})=\begin{cases}\sqrt{c}\times UB&\text{if $\lambda^{l}_{ n}=1$}\\ 0&\text{otherwise}\end{cases} \tag{5}\]

\[UB=\frac{\sqrt{6}}{\sqrt{|h|+|o|}} \tag{6}\]

We follow Glorot and Bengio (2010) in using \(\sqrt{6}\) as normalization factor for UB, and limit the number of patterns \(\lambda\) to the most frequently observed label assignments.

Our model uses three types of features representing the screenplay's lexical make up, its underlying character relations, and interactions.

Lexical FeaturesAn obvious feature class is the language of the movie. Comedies will be characterized by a different vocabulary compared to thrillers or historical drama. We thus represent each script as a vector of 7,500 dimensions corresponding to the most frequent words in the training corpus. Vector components were set to the 

words' tf-idf values. Words in scripts were further annotated with their sentiment values using the AFINN lexicon (Nielsen, 2011), a list of words scored with sentiment strength within the range \([-5,+5]\). We extracted several features based on these sentiment values such as the sentiment score of the entire movie, the number of scenes with positive/negative sentiment, the ratio of positive to negative scenes, and the minimum and maximum scene sentiment. From scene headings, we were also able to extrapolate the number of internal and external locations per script.

Graph-based FeaturesOur graph-based features are similar to those described in Gorinski and Lapata (2015). Specifically, we view screenplays as weighted, undirected graphs, where vertices correspond to movie characters and edges denote character-to-character interactions (essentially the number of times two characters talk to each other or are involved in a common action). From the graph we extract features corresponding to the number of main and supporting characters, which we identify by measuring their centrality in the movie network (e.g., the number of edges terminating in a given node). We also estimate character polarity by summing the sentiment of each character's utterances as well as the ratio of positive to negative characters in a given script.

Interaction-based FeaturesWe extract features based on how often any two characters interact, i.e., whether they are engaged in a conversation or in the same event (e.g., if a character kills another). We identify interactions as described in Gorinski and Lapata (2015) and measure the number of interactions per scene and movie, the number of positive and negative interactions, and their ratio.

### Movie Overview Decoder

Our decoder generates a movie overview from the multi-label encoding described above. For this, we adapt the LSTM architecture of Wen et al. (2015; 2016) which was originally designed for dialogue act generation (e.g., given input inform(type="hotel", count="182", dogsallowed="dontcare"), the network outputs "there are 182 hotels if you do not care whether dogs are allowed"). The network performs content selection, i.e., decides which attribute labels to talk about, while generating the sentences describing them.

As outlined in the lower part of Figure 2, a sigmoid control gate feeds a content vector, \(p_{0}\), into a traditional LSTM cell to generate a natural language surface form. At each timestep \(t\), the output word \(w_{t}\) is drawn from an output distribution conditioned on the previous hidden layer \(h_{t-1}\)_as well as_ the previous content vector \(p_{t-1}\). The content selection cell effectively acts as a _sentence planner_, retaining or omitting information from the original vector \(p_{0}\) at every time step \(t\) to guide the sentence generating LSTM cell. Our LSTM architecture is defined by the following equations:

\[i_{t} =\sigma(W_{wi}w_{t}+W_{hi}h_{t-1}) \tag{7}\] \[f_{t} =\sigma(W_{wf}w_{t}+W_{hf}h_{t-1})\] (8) \[o_{t} =\sigma(W_{wo}w_{t}+W_{ho}h_{t-1})\] (9) \[\hat{c}_{t} =\tanh(W_{wc}w_{t}+W_{hc}h_{t-1})\] (10) \[r_{t} =\sigma(W_{wr}w_{t}+W_{hr}h_{t-1})\] (11) \[p_{t} =r_{t}\odot p_{t-1}\] (12) \[c_{t} =f_{t}\odot c_{t-1}+i_{t}\odot\hat{c}_{t}+\tanh(W_{pc}p_{t}) \tag{13}\]

where \(\sigma\) is the sigmoid function, \(i_{t},f_{t},o_{t},r_{t}\in[0,1]^{n}\) are input, forget, output, and reading gates respectively, and \(\hat{c}_{t}\) and \(c_{t}\) are proposed cell value and true cell value at time \(t\).

In the original paper, the input \(p_{0}\) to the LSTM is a 1-hot representation of the information that should be included in the natural language output. In our setup, we relax this constraint such that each element of \(p_{0}\in[0,1]\), i.e., we directly use the output of the multi-label encoder just described.

### Training

The proposed architecture is trained jointly in an end-to-end fashion, minimizing the objective:

\[F(\theta)=\sum_{t}y_{t}^{T}\log(\hat{y}_{t})+||p_{T}||+\sum_{t=0}^{T-1}\eta \xi||p_{v+1-P_{t}}|| \tag{14}\]

where \(y_{t}\) and \(\hat{y}_{t}\) are the observed and predicted word distributions over the training data, \(p_{T}\) is the content vector at the final time index \(T\), \(p_{0}\) is the initial content vector as given by the encoder network, and \(\eta,\xi\) are training constants. The second term in the objective penalizes the network for generating output without realizing all required labels, while the third term deters the network from utilizing more than one label at any given time step.

The model is trained on pairs of scripts and sentences extracted from Jinni. To give a concrete example, a training instance for the _Plot_ sentence from Figure 1 would consist of the features representing the movie's screenplay, and the overview's 

_Plot_ sentence "The plot revolves around special agents, mind games, and a psychopath.". The _Plot_ multi-label network encodes the script into content vector \(p_{0}\), and the LSTM learns which "labels" represented in \(p_{0}\) to talk about while its training objective discourages to leave too many labels unmentioned. The observed output error is backpropagated through the LSTM and the embedding network using stochastic gradient descent Bottou (1991) with decaying learning rate.

## 5 Evaluation

In this section we report our evaluation experiments. We begin by assessing how good our encoder is at capturing screenplay content and then proceed to evaluated the generated overviews themselves.

### How Good is the Encoder?

In order to assess encoder's ability to induce structure over screenplays, we focus solely on the top part of the architecture in Figure 2. Specifically, we trained stand-alone models for the six attributes shown in Table 1 on the gold data provided in the Jinni dataset. All networks used the same features introduced earlier and were initialized using the pattern-based method of Kurata et al. (2016). To better capture the fact that we are dealing with multi-label assignments, we used the global error function described in Zhang and Zhou (2006). Given the network output vector \(\hat{y}\) for input \(x\), the true bag of label assignments \(y\) and its complement \(\bar{y}\), the error observed for each instance is computed as:

\[E=\frac{1}{|y||\bar{y}|}\sum_{(k,l)\in y\times j}\exp(-(\hat{y}_{k}-\hat{y}_{ l})) \tag{15}\]

The networks were trained with stochastic gradient descent during back propagation, using the same method as for the full model.

We compared our multi-label encoders (MLE) against several baselines. These include assigning the most frequent attribute labels to each movie based on the attributes' mean distribution (ZeroR), Naive Bayes (NB), Decision Stump (DS), LibLinearLib; Fan et al. (2008) and Support Vector Machines (SVMs; Chang and Lin (2011)). For each comparison system, we trained a binary classifier per attribute label using features identical to the ones used for the MLE.

Table 2 shows F1 performance on the training data for MLE and comparison systems, averaged over 10 folds. As can be seen, MLE performs best, followed by LibLinear. Table 3 compares MLE, ZeroR, and Lib, the strongest baseline, on the test set using F1 and the best parameters found for each system during cross-validation. As can be seen, MLE outperforms Lib across attributes, and is superior to ZeroR by a large margin. F1 differences between MLE and LibLinear are significant (\(p<0.01\)), using approximate randomization testing Noreen (1989).

Overall, the results in Tables 2 and 3 indicate that the classification task is hard. This is especially true for _Plot_ which has the largest number of labels. Nevertheless, the multi-label encoders introduced here achieve good performance on their own, indicating that they are able to capture the content of the screenplay, albeit approximately.

### How Good is the Decoder?

We next evaluate the performance of the jointly trained system which we call Morgan as a shorthand for **Movie OveRv**iw **G**ener**Atio**N** model. Morgan is trained on pairs of screensplays and their corresponding verbalizations in the Jinni dataset. Unfortunately, our dataset is relatively small for neural network training; it contains 617 movies only, i.e., there are 617 sentences for each attribute. To alleviate this problem, we augmented the data as follows. We extracted sentence templates from the training set (209 in total), ex

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline Attributes & ZeroR & NB & DS & SVM & Lib & MLE \\ \hline \hline Mood & 43.6 & 51.2 & 47.1 & 45.3 & 50.7 & **58.4** \\ Plot & 31.3 & 36.7 & 35.4 & 31.5 & 39.6 & **43.9** \\ Genre & 37.1 & 52.4 & 45.8 & 40.6 & 54.9 & **55.3** \\ Attitude & 63.0 & 68.5 & 67.3 & 64.0 & 71.6 & **76.5** \\ Place & 51.3 & 49.7 & 54.9 & 51.4 & 54.2 & **58.6** \\ Flag & 51.7 & 49.3 & 54.7 & 51.4 & 50.9 & **57.0** \\ \hline \hline All & 46.3 & 51.3 & 50.9 & 47.4 & 53.7 & **58.3** \\ \hline \end{tabular}
\end{table}
Table 2: Attribute identification (average %F1 across 10 folds). Best system in bold.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Attributes & ZeroR & Lib & MLE \\ \hline \hline Mood & 43.4 & 48.0 & **61.6** \\ Plot & 32.8 & 38.9 & **42.9** \\ Genre & 37.8 & 54.6 & **58.5** \\ Attitude & 61.2 & 69.0 & **73.1** \\ Place & 48.2 & 46.1 & **54.4** \\ Flag & 48.8 & 46.4 & **54.0** \\ \hline \hline All & 45.4 & 50.5 & **57.4** \\ \hline \end{tabular}
\end{table}
Table 3: Attribute identification (%F1; test set).

 

amples of which are shown in Table 5. We replaced the title and attribute values with variables (shown as capital letters in the table). We then used the templates to generate additional data for each movie by substituting attribute variables in template sentences with permutations of the movie's gold-standard labels. We thereby obtained a total of 31,000 training instances.

The model was trained with a learning rate of 0.5, using a decay of 0.01 over 50 epochs, fixing it for subsequent epochs. Constants \(\eta\) and \(\xi\) in equation (14) were set to \(10^{-4}\) and 100, respectively. At test time, we used screenplay features as input and generated one sentence per attribute. We arranged these into an overview following the ordering _Mood_\(\gg\)_Plot_\(\gg\)_Genre_\(\gg\)_Attitude_\(\gg\)_Place_\(\gg\)_Flag_ which is fixed and attested in all overviews in our dataset.

We compared Morgan against several systems: (1) a random baseline, selecting for each movie and attribute type a random sentence from the training set; (2) a nearest-neighbor baseline (NN) which uses the same screenplay features as Morgan (and cosine similarity) to identify the closest matching script in the training data, and rehashes its overview as output; (3) an attention-based LSTM Bahdanau et al. (2015) trained on script sentence pairs (31,000 in total); and (4) six attention-based LSTMs, one per attribute type, trained on script sentence pairs (on average 5,200 per LSTM). The attention LSTMs were trained on the same screenplay features as Morgan, with the attention mechanism at each timestep \(t\) focusing on parts of the input. Example overviews generated by Morgan, the attention LSTMs, and the nearest neighbor system are shown in Table 6.

We evaluated system output with multi-reference BLEU4 Papineni et al. (2002), using sentences from the extended gold-standard as references. Table 5 (first column) summarizes our results. As can be seen, Morgan outperforms the attention based models, the nearest neighbor system, and the random baseline. The attention-based models cannot succinctly capture the movie's content in order to render it into meaningful sentences. Although the generated sentences are more or less grammatical on their own (see Table 6), the generated overview lacks coherence, and is fairly repetitive. The model does not reliably learn what type of information to focus on for the generation task. For Morgan this problem is alleviated during the encoding step, which performs content distillation prior to generating overview sentences.

Footnote 4: We use NLTK’s ([http://www.nltk.org/](http://www.nltk.org/)) implementation of BLEU, and report the interpolation of BLEU 1 through 4.

### How are System Overviews Perceived by Humans?

In addition to evaluating system output automatically, we are also interested in how it is perceived by humans. To this end, we ran two judgment elicitation studies on Amazon Mechanical Turk. Both experiments were conducted on 12 movies. In a pre-test we asked 20 workers whether they had seen the movies in our test set and chose the three most popular ones from each of the genres Action, Comedy, Drama, and Romance.

In our first experiment Turkers were presented with an overview taken from the Jinni gold standard, Morgan or one of the comparison systems and asked to rate its coherence (i.e., whether it was readily comprehensible or difficult to follow) on a scale from 1 (incoherent) to 5 (coherent). Subsequently, they had to rate the grammaticality of

\begin{table}
\begin{tabular}{|l l|} \hline Mood & \(T\) can be described as \(M_{1}\) and \(M_{2}\) \\  & The mood of \(T\) is \(M_{1}\) \\ \hline \hline Plot & The plot centres around a \(P_{1}\), \(P_{2}\), and \(P_{3}\) \\  & The plot revolves around \(P_{1}\), \(P_{2}\), and \(P_{3}\) \\ \hline \hline Genre & The main genres are \(G_{1}\), \(G_{2}\), and \(G_{2}\) \\  & \(T\) is \(M_{1}\) and \(M_{2}\) move \\ \hline \hline Attitude & In approach, \(T\) is \(A_{1}\) \\  & The pacing is \(A_{1}\) \\ \hline \hline Place & The setting is \(L_{1}\) \\  & It is located in \(L_{1}\) \\ \hline \hline Flag & Note that the movie involves \(F_{1}\) and \(F_{2}\) \\  & Note that it includes \(F_{1}\), \(F_{2}\), and \(F_{3}\) \\ \hline \end{tabular}
\end{table}
Table 4: Template sentences extracted from Jinni overviews. Variable \(T\) is filled by the movie’s title, whereas \(M\), \(G\), \(P\), \(A\), \(L\), \(F\) correspond to values for attributes _Mood_, _Genre_, _Plot_, _Attitude_, _Place_, and _Flag_, respectively.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Models & BLUE & Coherence & Grammaticality \\ \hline \hline Random & 38.0 & 2.42\({}^{*}\) & 3.83 \\ NN & 40.4 & 3.45 & 3.93 \\ Attn & 23.0 & 2.93\({}^{*}\) & 3.91 \\ typAttn & 37.9 & 3.20 & 3.80 \\ Morgan & **42.0** & **3.72** & **4.08** \\ \hline \hline Jinni & — & 4.27 & 4.22 \\ \hline \end{tabular}
\end{table}
Table 5: BLEU scores and mean coherence and grammaticality ratings for movie overviews. \({}^{*}\) significantly different from Morgan (\(p<0.05\)). Best performing system shown in bold.

 

each overview sentence, again on a scale from 1 (ungrammatical) to 5 (grammatical) and decide whether it appropriately described aspects of the movie's content ("Yes", "No", "Unsure"). We elicited five responses for each overview across six systems (Jinni, typAttn, Attn, Random, NN, and Morgan) and 12 movies. Finally, participants had to answer a question relating to the movie's content, to make sure that they had actually seen the movie. We discarded responses with wrong answers to the content question. Examples of the overview s participants judged are given in Table 6.

Table 5 (columns 2 and 3) summarizes the results of our first judgment elicitation study. All systems perform well with regards to grammaticality. This is not surprising for Random and NN which do not perform any generation. Attn and typAttn also perform well with Morgan achiev

\begin{table}
\begin{tabular}{|l|} \hline Nearest Neighbor \\ \hline The mood of Burn after Reading is stylized. The plot revolves around love and romance, motives, buddies, and friends. Its main genres are romance, crime, and comedy. In approach, Burn after Reading is semi serious. It takes place, at least partly, in the USA. Note that Burn after Reading involves violent content, profanity, and sexual content. \\ \hline attnLSTM \\ \hline The setting is Europe. It is a sci-fi, adventure, and adventure movie. Note that Burn after Reading includes violent content. Burn after Reading is set in the USA. It is located in the USA. It is set, at least in part, in the USA. \\ \hline typed attnLSTM \\ \hline The mood of Burn after Reading is bleak, touching, and feel good. The plot revolves around ambition motives, love and romance, and fall in love. It is an adventure and fantasy, and features thriller themes. In approach, Burn after Reading is serious. Burn after Reading is set, at least in part, in the USA. Note that Burn after Reading involves violent content. \\ \hline Morgan \\ \hline The mood of Burn after Reading is feel good, exciting, and clever. The plot centers around ambition, special agents, and partners. Burn after Reading is a comedy movie. In approach, it is realistic and semi serious. It is set, at least in part, in the USA. Note that it involves profanity. \\ \hline Jinni \\ \hline Burn after Reading can be described as cynical, clever, and exciting. The plot revolves around espionage , disorder, and escapades. Burn after Reading’s main genres are comedy and crime. In approach, it is not serious and realistic. Burn after reading is set, at least in part, in an urban environment. It is located in Virginia. Note that it involves sexual content, profanity, and violent content. \\ \hline Nearest Neighbor \\ \hline Lara Croft: Tomb Raider can be described as touching, offbeat, and humorous. The plot revolves around a psychopath, a chase, and couples motives. Its main genres are drama and comedy. In approach, Lara Croft: Tomb Raider is semi serious. Lara Croft: Tomb Raider is set in the USA. \\ \hline latra Croft: Tomb Raider is set, at least in part, in a dangerous and in a haunted house. The plot centers around buddies, disorder, and criminal heroes motives. The plot centers around escapedas, supernatural, and deadly. The setting is a school or college environment. Note that Lara Croft: Tomb raider includes drugs/alcohol and sexual content. The mood of Lara Croft: Tomb Raider is rough, being, and contemplative. \\ \hline typed attnLSTM \\ \hline The mood of Lara Croft: Tomb Raider is suspenseful, exciting, and stylized. The plot revolves around a police investigation, cons and scams, and couple relations motives. Lara Croft: Tomb Raider’s main genres are fantasy, action, and comedy. In approach, it is serious and fantastic. Lara Croft: Tomb Raider is set, at least in part, in the USA. Note that Lara Croft: Tomb Raider involves profanity and violent content. \\ \hline Morgan \\ \hline Lara Croft: Tomb Raider can be described as suspensionful and rough. The plot centers around danger, a master villain, and deadly. Lara Croft: Tomb raider is an adventure, thriller, and action movie. In approach, it is fantastic. Lara Croft: Tomb Raider takes place, at least partly in Europe. Note that it involves profanity. \\ \hline lanni \\ \hline Lara Croft: Tomb Raider can be described as stylized, exciting, and suspenseful. The plot revolves around time travel, a treasure hunt, and a master villain. The main genres are action, adventure, and fantasy. In approach, it is fantastic and serious. Note that it involves violent content. \\ \hline  & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 6: Overviews generated by Morgan and comparison systems for “Burn after Reading” (top) and “Lara Croft; Tomb Raider” (bottom).

\begin{table}
\begin{tabular}{|p{48.4pt}|p{27.0pt}|p{27.0pt}|p{27.0pt}|p{27.0pt}|p{27.0pt}|p{27.0pt}|} \hline Model & Mood & Plot & Genre & Attitude & Place & Flag & All \\ \hline \hline Random & 37.7 & 39.6 & 34.0 & 43.4 & 35.8 & 50.9 & 19.0* \\ NN & 78.6 & 67.9 & 71.4 & 66.1 & 58.9 & 91.1 & 58.9* \\ Attn & 38.2 & 38.2 & 38.2 & 41.8 & 51.0 & 34.5 & 40.0* \\ UpAtn & 60.0 & 60.0 & 53.3 & 57.8 & **66.7** & 64.4 & 40.0* \\ Morgan & **89.5** & **73.7** & **80.7** & **71.9** & 63.2 & **89.5** & 82.5 \\ \hline Jinni & 91.1 & 89.3 & 92.9 & 82.1 & 67.9 & 75.0 & 91.1 \\ \hline \end{tabular}
\end{table}
Table 7: Proportion of sentences and overviews (All) which describe the movie accurately. \({}^{*}\) significantly different from Morgan (\(p<0.05\)). Best performing system per attribute is in bold.

 

ing highest scores for grammaticality amongst automatic systems. Grammaticality differences between the various systems in Table 5 and the Jinni gold standard are not statistically significant (using a one-way ANOVA with post-hoc Tukey HSD tests). Overviews generated by Morgan are perceived as more coherent in relation to those generated by comparison systems, even though the model does not explicitly take coherence into account. Morgan overviews are not significantly different in terms of coherence from Jinni, typAttn, and NN, but are significantly better than Random and Attn.

Table 7 shows the percentage of sentences (per attribute and overall) which participants think describe the movie's content felicitously. Morgan identifies most aspects of the movie successfully, in some cases close to (_Mood, Place_) or even better (_Flag_) than the original Jinni overview. Morgan is significantly better compared to all other models but not significantly worse than Jinni (using a \(\chi^{2}\) test; see last column in Table 7).

In a second experiment, participants were presented with six overviews for a movie (from Jinni, Attn, typAttn, Random, NN, and Morgan) and asked to rank them (equal ranks were not allowed) in order of relevance (i.e., whether they express content relevant to the movie). Again, we obtained five responses for each movie. As can be seen in Table 8, while Jinni is ranked first most of the time, Morgan is ranked second followed by the NN system. We further converted the ranks to ratings on a scale of 1 to 6 (assigning ratings 6...1 to rank placements 1...6) and performed and ANOVA which showed that all systems are significantly (\(p<0.05\)) worse than Jinni but Morgan is significantly better than the comparison systems.

## 6 Conclusions

In this work we have presented a novel approach to automatic movie content analysis. We have assembled a new dataset which combines ScriptBase Gorinski and Lapata [3], a corpus of movie scripts, with information gathered from Jinni, a large movie database. We proposed an end-to-end model for movie overview generation via _multi-attribute_ encoders and a _semantically conditioned_ LSTM decoder. Experimental results show that our encoders are capable of distilling meaningful structures from the screenplay. When applied to the overview generation task, our end-to-end model outperforms a standard attention-based LSTM. Human evaluation also indicates the overviews generated by our model are felicitous, informative, and rated favorably by humans.

In the future, we would like to investigate how attribute-specific features can improve performance compared to our more general feature set which is invariant for each sentence type. It would also be possible to equip the model with a hierarchical decoder which generates a document instead of individual sentences. Although currently our model relies solely on textual information, it would be interesting to incorporate additional modalities such as video Zhou et al. [9] or audio (e.g., we expect comedies to be _visually_ very different from thrillers, or romantic movies to have a different _score_ from superhero movies). Finally, we would like to examine whether the content analysis presented here can extend to different types of fiction such as novels or short stories.

AcknowledgmentsWe thank the NAACL reviewers for their constructive feedback. We gratefully acknowledge the financial support of the European Research Council (award number 681760).

## References

* Agarwal et al. (2014a) Apoorv Agarwal, Sriramkumar Balasubramanian, Jiehan Zheng, and Sarthak Dash. 2014a. Parsing Screenplays for Extracting Social Networks from Movics. In _Proceedings of the 3rd Workshop on Computational Linguistics for Literature_. Gothenburg, Sweden, pages 50-58.
* Agarwal et al. (2014b) Apoorv Agarwal, Sarthak Dash, Sriramkumar Balasubramanian, and Jiehan Zheng. 2014b. Using Determinantal Point Processes for Clustering with Application to Automatically Generating and Drawing xked Movie Narrative Charts. In _Proceedings of the 2nd Academy of Science and Engineering International Conference on Big Data Science and Computing_. Stan

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline Model & 1st & 2nd & 3rd & 4th & 5th & 6th & AvgRank \\ \hline \hline Random & 1.0 & 5.8 & 16.3 & 22.1 & 19.2 & **35.6** & 4.59 \\ NN & 5.8 & 19.2 & **24.0** & 23.1 & 15.4 & 12.5 & 3.60 \\ Attn & 3.8 & 13.5 & 20.2 & **28.8** & 16.3 & 17.3 & 3.92 \\ UpAttn & 1.9 & 7.7 & 15.4 & 10.6 & **33.6** & 30.8 & 4.58 \\ Morgan & 8.7 & **42.3** & 22.1 & 12.5 & 12.5 & 1.9 & 2.71 \\ \hline \hline Jinni & **78.8** & 11.5 & 1.9 & 2.9 & 2.9 & 1.9 & 1.45 \\ \hline \end{tabular}
\end{table}
Table 8: Relevance rankings (shown as proportions) given to overviews by human subjects. Most frequent rank per system and Jinni is in bold.

 

ford, California.
* Agarwal et al. (2015) Apoorv Agarwal, Jiehan Zheng, Shruti Kamath, Sriramkumar Balasubramanian, and Shirin Ann Dey. 2015. Key Female Characters in Film Have More to Talk About Besides Men: Automating the Bechdel Test. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Denver, Colorado, pages 830-840.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proceedings of ICLR_. San Diego, CA.
* Bamman et al. (2013) David Bamman, Brendan O'Connor, and Noah A. Smith. 2013. Learning latent persona s of film characters. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Sofia, Bulgaria, pages 352-361.
* Belanger and McCallum (2016) David Belanger and Andrew McCallum. 2016. Structured prediction energy networks. In _Proceedings of the 33rd International Conference on Machine Learning_. New York, pages 983-992.
* Bottou (1991) Leon Bottou. 1991. Stochastic gradient learning in neural networks. In _Proceedings of the 7th Neuro-Nimes International Conference_. EC2, Nimes, France.
* Chang and Lin (2011) Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: a library for support vector machines. _ACM Transactions on Intelligent Systems and Technology_ 2(3):27.
* Chen et al. (2015) Liang-Chieh Chen, Alexander G Schwing, Alan L Yuille, and Raquel Urtasun. 2015. Learning deep structured models. In _Proceedings of the 32nd International Comference on Machine Learning_. Lilfe, France, pages 1785-1794.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_. Doha, Qatar, pages 103-111.
* Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _Proceedings of NAACL: HLT_. San Diego, CA, pages 93-98.
* Dalal and Zaveri (2011) Mita K. Dalal and Mukesh A. Zaveri. 2011. Automatic Text Classification: A Technical Review. _International Journal of Computer Applications_ 28(2):37-40.
* Danescu-Niculescu-Mizil and Lee (2011) Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs. In _Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics_. Portland, Oregon, pages 76-87.
* Fan et al. (2008) Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A Library for Large Linear Classification. _Journal of Machine Learning Research_ 9 (Aug):1871-1974.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the 13th International Conference on Artificial Intelligence and Statistics_. Chia Laguna Resort, Sardinia, Italy, pages 249-256.
* Godbole and Sarawagi (2004) Shatanu Godbole and Sunita Sarawagi. 2004. Discriminative methods for multi-labeled classification. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, pages 22-30.
* Gorinski and Lapata (2015) Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Denver, Colorado, pages 1066-1076.
* Hershey et al. (2014) John R. Hershey, Jonathan Le Roux, and Felix Weninger. 2014. Deep unfolding: Model-based inspiration of novel deep architectures. _CoRR_ abs/1409.2574.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_ 9(8):1735-1780.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. _CoRR_ abs/1508.01991.
* Jaderberg et al. (2014) Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep structured output learning for unconstrained text recognition. _CoRR_ abs/1412.5903.
* Kurata et al. (2016) Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence. In _Proceedings of the 2016 Conference of the North Amer 

ican Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California, pages 521-526.
* Lebret et al. (2016) Remi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas, pages 1203-1213.
* Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California, pages 720-730.
* Nielsen (2011) Finn A. Nielsen. 2011. A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In _Proceedings of the ESWC2011 Workshop on Making Sense of Microposts: Big things come in small packages_. Heraklion, Crete, pages 93-98.
* Noreen (1989) Eric Noreen. 1989. _Computer Intensive Methods for Testing Hypotheses: An Introduction_. Wiley, Hoboken, New Jesey.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of 40th Annual Meeting of the Association for Computational Linguistics_. Philadelphia, Pennsylvania, USA, pages 311-318.
* Read et al. (2008) Jesse Read, Bernhard Pfahringer, and Geoff Holmes. 2008. Multi-label classification using ensembles of pruned sets. In _Proceedings of the 8th International Conference on Data Mining_. Pisa, Italy, pages 995-1000.
* Read et al. (2011) Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier chains for multi-label classification. _Machine Learning_ 85(3):333-359.
* Schwing and Urtasun (2015) Alexander G. Schwing and Raquel Urtasun. 2015. Fully connected deep structured networks. _CoRR_ abs/1503.02351.
* Sebastiani (2002) Fabrizio Sebastiani. 2002. Machine Learning in Automated Text Categorization. _Association for Computing Machinery: Computing Surveys_ 34(1):1-47.
* Stoyanov et al. (2011) Veselin Stoyanov, Alexander Robson, and Jason Eisner. 2011. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In _AISTATS_. pages 725-733.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems_. pages 3104-3112.
* Tang et al. (2015) Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. pages 1422-1432.
* Tsoumakas and Katakis (2006) Grigorios Tsoumakas and Ioannis Katakis. 2006. Multi-label classification: An overview. _International Journal of Data Warehousing and Mining_ 3(3).
* Tsoumakas and Vlahavas (2007) Grigorios Tsoumakas and Ioannis Vlahavas. 2007. Random k-labelsets: An ensemble method for multilabel classification. In _European Conference on Machine Learning_. Springer, pages 406-417.
* Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. pages 3156-3164.
* Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal, pages 1711-1721.
* Wen et al. (2016) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California, pages 120-129.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. pages 1480-1489.
* Ye and Baldwin (2008) Patrick Ye and Timothy Baldwin. 2008. Towards Automatic Animated Storyboarding. In 

Proceedings of the 23rd Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence_. Chicago, Illinois, pages 578-583.
* Zhang and Zhou (2005) Min-Ling Zhang and Zhi-Hua Zhou. 2005. A k-nearest neighbor based algorithm for multi-label classification. In _Proceedings of the IEEE International Conference on Granular Computing_. Beijing, China, volume 2, pages 718-721.
* Zhang and Zhou (2006) Min-Ling Zhang and Zhi-Hua Zhou. 2006. Multi-label neural networks with applications to functional genomics and text categorization. _IEEE Transactions on Knowledge and Data Engineering_ 18(10):1338-1351.
* Zhang and Lapata (2017) Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark, pages 595-605.
* Zheng et al. (2015) Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. 2015. Conditional random fields as recurrent neural networks. In _Proceedings of the IEEE International Conference on Computer Vision_. Santiago, Chile, pages 1529-1537.
* Zhou et al. (2010) Howard Zhou, Tucker Hermans, Asmita V. Karandikar, and James M. Rehg. 2010. Movie genre classification via scene categorization. In _Proceedings of the 18th ACM International Conference on Multimedia_. New York, NY, pages 747-750.