MovieSum: An Abstractive Summarization Dataset for Movie Screenplays (2024)

Abstract: Movie screenplay summarization is challenging, as it requires an understanding of long input contexts and various elements unique to movies. Large language models have shown significant advancements in document summarization, but they often struggle with processing long input contexts. Furthermore, while television transcripts have received attention in recent studies, movie screenplay summarization remains underexplored. To stimulate research in this area, we present a new dataset, MovieSum,1 for abstractive summarization of movie screenplays. This dataset comprises 2200 movie screenplays accompanied by their Wikipedia plot summaries. We manually formatted the movie screenplays to represent their structural elements. Compared to existing datasets, MovieSum possesses several distinctive features: (1) It includes movie screenplays, which are longer than scripts of TV episodes. (2) It is twice the size of previous movie screenplay datasets. (3) It provides metadata with IMDb IDs to facilitate access to additional external knowledge. We also show the results of recently released large language models applied to summarization on our dataset to provide a detailed baseline.

heading: 1 Introduction
text: Large language models have shown significant improvements in abstractive summarization in recent years (Zhong et al., 2022; Zhu et al., 2021; Zhong et al., 2021; Zhang et al., 2022), aiming to produce a concise and coherent summary of the input document. However, these models often struggle when the input context is long, particularly when the relevant information is distributed across the document (Liu et al., 2023). To better understand this phenomenon and to advance research, datasets
1Our dataset and code is available at https://github. com/saxenarohit/MovieSum.
are needed that not only contain long-form documents but also have the property that important information is dispersed throughout the document. Movie screenplays have these characteristics: to generate a faithful summary, an understanding of characters and events across the entire length of the screenplay is required.
More recently, narrative summarization research has focused on TV shows and books (Kryściński et al., 2021; Moskvichev and Mai, 2023), with less attention given to movie screenplays (Gorinski and Lapata, 2015; Papalampidi et al., 2020). Notably, Chen et al. (2022) introduced a dataset of TV show transcripts which has gained considerable interest and was included in a long document summarization benchmark (Shaham et al., 2022). But unlike movie screenplays, TV episode transcripts tend to be relatively short and predominantly comprise spoken dialogue with minimal scene or character descriptions. Additionally, they are not selfcontained, as the events or characters from previous episodes can be referred to. In contrast, movie screenplays are structured documents with various screenplay elements such as scene headings, locations, character names, dialogues and detailed scene descriptions. These are written by screenwriters and are characteristically formatted to denote each element.
The largest current movie screenplay dataset (Gorinski and Lapata, 2015, 2018) comprises 917 automatically formatted screenplays (ScriptBase-j), with the most recent movie from 2013. We built MovieSum, a new movie screenplay dataset for abstractive summarization, which consists of 2200 movies, more than twice the size ScriptBase-j. Importantly, our new dataset has been formatted using a professional script writing tool and paired with Wikipedia plot summaries. Each movie is also tagged with its IMDB IDs to facilitate the collection of other external knowledge in rhw future. The dataset consists of movies spanning a wide range
4043
of genres from 1930 to 2023. We provide a detailed description of MovieSum, including the steps for collecting and filtering screenplays and statistics and comparison with other narrative datasets. We conduct extensive experiments to evaluate the performance of stateof-the-art summarization models on MovieSum, demonstrating its utility as a benchmark dataset for narrative summarization research. Experiments suggest that recent models struggle with long abstractive summarization, and we hope that our efforts will inspire further research in this area. Furthermore, we provide qualitative analyses of how the structure of the screenplay can be utilized in generating summaries.

heading: 2 The MovieSum Dataset
text: We present MovieSum, a movie screenplay abstractive summarization dataset that consists of 2200 movie screenplay-summary pairs. All movie screenplays in the dataset are in English.

heading: 2.1 Collection of Movie Screenplays
text: We collected movie screenplays from a range of movie screenplay websites.2 In total, we assembled 5,639 movie screenplays documents in various text format along with metadata of movie name, IMDB identifier, and release year. If the IMDB identifier was missing, we extracted it using the IMDB database. We then manually removed movies based on two criteria. Firstly, we removed any duplicate movie screenplays by using the movie names and release years to identify the duplicates. Secondly, we filtered out screenplays which did not have text or were incomplete.

heading: 2.2 Screenplay Formatting
text: Movie screenplays are structured documents with various script elements such as scene headings (also known as slug lines), characters’ names, dialogues, and scene descriptions (or actions). These elements have specific markers based on spacing. Most of this formatting is lost when extracting text from these movie screenplay documents, making it challenging to retrieve the elements using regular expressions. To ensure the quality of the dataset, after filtering, we manually corrected the movie screenplay and formatted each movie screenplay using Celtx,3 a professional screenplay writing tool. This
2https://www.scriptslug.com/, https://imsdb. com/, https://www.dailyscript.com/
3https://www.celtx.com/
process preserved the format of movie screenplay elements. We further filtered out screenplays with encoding or optical character recognition errors.

heading: 2.3 Collection of Wikipedia Plot Summaries
text: To build a robust summarization dataset, it is necessary to collect high-quality human-written summaries. Similar to previous work (Kočiský et al., 2018), we collected Wikipedia plot summaries, which we found to be of high quality, helped by the fact that Wikipedia summaries follow a consistent set of guidelines for movie plot summaries.4
To collect the Wikipedia plot summary, we first extracted the Wikipedia page of the movie using the movie name and year, then collected text under the Plot section. We filtered out movies where the Wikipedia page or the plot section was unavailable.
This process resulted in 2200 manually formatted movie screenplays with corresponding Wikipedia summaries.

heading: 3 Dataset Analysis
text: This results in a dataset consisting of 2200 manually formatted movie screenplays along with their corresponding summaries. The average length of the screenplays is 29k words, with an average summary length of 717 words. Importantly, this dataset is twice the size of the previously available movie screenplays dataset with formatted movie screenplays (Gorinski and Lapata, 2015). Figure 1a. illustrates the genre distribution of the movies within the dataset and showcases the broad range of genres. In Figure 1b, the distribution of release years is depicted, revealing that the movies span a wide range of years, with a substantial number of them originating in recent years.
To study the abstractiveness of the summary, we report the percentage of novel n-gram in Table 1 as reported by Fabbri et al. (2021); Zhao et al. (2022). It shows that a high number of 3-gram and 4-gram are novel in summary and not present in the movie screenplays implying high abstractiveness of the
4https://en.wikipedia.org/wiki/Wikipedia: How_to_write_a_plot_summary
summary. See Appendix B for further analysis on the abstractiveness of the summary.

heading: 3.1 Comparison with Existing Datasets
text: We compare our dataset with various datasets in the narrative domain, and the statistics are reported in Table 2. These datasets include ScriptBase-j (Gorinski and Lapata, 2015), ScriptBase-alpha (Gorinski and Lapata, 2015), SummScreenFD (Chen et al., 2022), NarraSum (Zhao et al., 2022), NarrativeXL (Moskvichev and Mai, 2023), NarrativeQA (Kočiský et al., 2018), and BookSum (Kryściński et al., 2021). Notably, BookSum and NarrativeXL have a longer average document length but comprise books, not screenplays. SummScreenFD consists of TV show episode transcripts, which are much shorter in both document and summary length. Importantly, SummScreen consists of community-contributed transcripts and primarily comprises dialogues, unlike screenplays, which include detailed scene descriptions. Also, TV show episodes are not self-contained, as events or characters from previous episodes can be referenced. NarraSum contains plot summaries as documents rather than actual screenplays, and therefore has
the lowest average document length among the datasets we compare. NarrativeQA includes both books and movie screenplays, with books being notably lengthy, making the average document length comparable to book datasets. However, it consists of only 789 unformatted movie screenplays.
Both ScriptBase-j and ScriptBase-alpha datasets are close to our screenplay dataset. ScriptBase-j contains formatted screenplays, whereas ScriptBase-alpha comprises the unformatted raw text of screenplays. It is important to note that ScriptBase-j is a subset of ScriptBase-alpha, which consists of 917 formatted screenplays. On the other hand, ScriptBase-alpha includes an additional 359 movies. Our work can be considered as the extension of ScriptBase-j as it also consists of formatted screenplays. At the same time, we overcome two critical limitations of SciptBase-j:
(1) The formatting of the movie screenplay was performed automatically. Although it is easy to detect the scene heading based on rules and string matching, it is challenging to distinguish dialogues, character names, and scene descriptions. The work does not provide any details regarding the auto-
matic formatting strategy. MovieSum, on the other hand, includes all the movies from ScriptBase-j formatted using professional screenplay tools.
(2) Both subsets of ScriptBase consist of movies released until 2013. In contrast, MovieSum also includes recently released movies. This is crucial for ensuring that summarization models remain robust to new movie narrative conventions. The movie release years are graphed in Figure 1b.

heading: 4 Experiments
text: We evaluate the MovieSum dataset using several baselines and state-of-the-art neural abstractive summarization models. We first report the Lead-N baseline, which simply outputs the first N tokens of the movie script as the movie summary. We varied the value N to understand the impact of summary length on performance and report results for Lead-512, Lead-768, and Lead-1024. For the extractive baseline, we used TextRank (Mihalcea and Tarau, 2004), a graph-based unsupervised extractive summarization method. For instructiontuned large language models, we used Vicuna 1.5 13B 16K (Zheng et al., 2023), built on Llama-2 (Touvron et al., 2023), and FLAN-UL2 (Tay et al., 2023; Wei et al., 2022) in a zero-shot setting. For fine-tuned models with long inputs, we utilized LongT5 (Guo et al., 2022), PEGASUS-X (Phang et al., 2023), and the Longformer Encoder-Decoder (LED) model (Beltagy et al., 2020). We fully finetuned these models and report results on the test set. The implementation details of the models are
mentioned in Appendix A.

heading: 4.1 Results
text: Table 3 shows the summarization evaluation results using ROUGE F1 (1/2/L) scores (Lin, 2004) and BERTScore (Zhang et al., 2019) on MovieSum. The Lead baseline performs better with a higher number of words, achieving the best result with 1024 words. This is not surprising, as the ROUGE metric is known to give higher scores for longer summaries (Schluter, 2017). In the case of zeroshot, FLAN-UL2 8K performed substantially better than Vicuna 13B 16K. This confirms that a longer context does not necessarily lead to attention to the full context length. We also tested reducing the input context using TextRank, which only marginally improves the zero-shot performance for Vicuna 13 model. To further utilize the full context, we also report results for moving window chunk-based zeroshot summarization and concatenating the generated summaries. This performs better compared to using only the 16K context length. The best performance was achieved with fine-tuned models with longer context lengths. Pegasus-X, LongT5, and LED perform similarly well, with LED demonstrating superior performance.

heading: 5 Analysis of Screenplay Structure
text: We analyze the importance of screenplay elements, dialogue and scene description, and their impact on summarization performance. We selected the best model from the full fine-tuned experiment and stud-
ied the effects of fine-tuning solely on dialogues and descriptions. Additionally, we investigated the effects of selectively using both dialogue and scene description based on heuristics similar to Pu et al. (2022). Results in Table 3 show a marginal impact on summarization when dialogues or scene descriptions are removed. We also found the heuristic based removal of input context method results is comparable to the full-text results. This suggests that the current model does not fully utilize the inherent structure of the document, and new methods for content selection and summarization should consider both screenplay elements for movie script summarization.

heading: 6 Discussion and Conclusion
text: We introduce the MovieSum dataset, comprising formatted and recent movie screenplays paired with Wikipedia summaries. Our experiments demonstrate that it is a challenging dataset even for a large language model with a long input length. We hope that MovieSum will enable future research in the area of movie screenplay understanding and abstractive summarization.
Limitations
Limitations of the work include that the dataset consists of movie screenplays and their corresponding summaries only in English. Models trained on MovieSum may not generalize well to multilingual summarization tasks or applications requiring cross-lingual understanding.
Ethics Statement
Large Language Models: This paper uses pretrained large language models, which have been shown to be subject to a variety of biases, to occasionally generate toxic language, and to hallucinate content. Therefore, the summaries generated using our dataset should not be released without automatic filtering or manual checking.
Bias: Despite efforts to include a wide range of movies, the dataset may not fully represent the diversity of cinematic styles, languages, or cultural contexts. Models trained on MovieSum may therefore exhibit biases towards the types of movies included.

heading: Acknowledgements
text: This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by UK Research and Innovation (grant EP/S022481/1), Huawei, and the School of Informatics at the University of Edinburgh. We would like to thank the anonymous reviewers for their helpful feedback.

heading: B Additional Statistics of Dataset
text: To further understand the abstractiveness of the summaries we computed the coverage and density of the summaries as discussed by Fabbri et al. (2021). The low density in Figure 2, indicates low overlap between the summary and the screenplays.

heading: C Prompt Template
text: For the zero-shot experiments in Section 4, we used the following prompt template:
Prompt: Summarize the following movie script. Movie Script: {movie script text} Summary:

heading: 10000 20000 30000 40000 50000 60000 70000

heading: D Length Distribution
text: Figure 3 and 4 show the length distribution for movie scripts and their summaries across the training set. The mean length of movie scripts is 29K words, and the average length of summaries is 714 words.

heading: E Example of a Movie Screenplay
text: Figure 5 shows an example of a cleanly formatted screenplay with distinct elements such as scene heading, characters, and dialogues. All the files are converted into XML using Celtx tool.

heading: F Sample of Movie Summary
text: Table 4 shows sample of generated summary of a movie using fine-tuned LED model (full-text).
Movie: Oppenheimer

heading: Future Work
text: We hope that MovieSum will enable future research in the area of movie screenplay understanding and abstractive summarization. Limitations
Limitations of the work include that the dataset consists of movie screenplays and their corresponding summaries only in English. Models trained on MovieSum may not generalize well to multilingual summarization tasks or applications requiring cross-lingual understanding