A Modular Approach for Multimodal Summarization of TV Shows (2024)

Abstract: In this paper we address the task of summarizing television shows, which touches key areas in AI research: complex reasoning, multiple modalities, and long narratives. We present a modular approach where separate components perform specialized sub-tasks which we argue affords greater flexibility compared to end-toend methods. Our modules involve detecting scene boundaries, reordering scenes so as to minimize the number of cuts between different events, converting visual information to text, summarizing the dialogue in each scene, and fusing the scene summaries into a final summary for the entire episode. We also present a new metric, PRISMA (Precision and Recall Evaluation of Summary Facts), to measure both precision and recall of generated summaries, which we decompose into atomic facts. Tested on the recently released SummScreen3D dataset (Papalampidi and Lapata, 2023), our method produces higher quality summaries than comparison models, as measured with ROUGE and our new fact-based metric.

heading: 1 Introduction
text: In this paper, we address the challenging task of summarizing television shows which has practical utility in allowing viewers to quickly recall plot points, characters, and events without the need to re-watch entire episodes or seasons. From a computational standpoint, the task serves as a testbed for complex reasoning over long narratives, involving multiple modalities, non-trivial temporal dependencies, inferences over events, and multi-party dialogue with different styles. An added difficulty concerns assessing the quality of generated summaries for long narratives, whether evaluations are conducted by humans or via automatic metrics.
Most prior work on creative summarization does not consider the above challenges all at once, focusing either on the text modality and full-length narratives with complex semantics (Gorinski and
Lapata, 2015; Chen et al., 2022; Agarwal et al., 2022) or on short video clips which last only a couple of minutes (Tapaswi et al., 2016; Lei et al., 2018; Liu et al., 2020). A notable exception are Papalampidi and Lapata (2023), who incorporate multimodal information into a pre-trained textual summarizer by adding (and tuning) adapter layers (Rebuffi et al., 2017; Houlsby et al., 2019). On the evaluation front, there is no single agreed-upon metric for measuring summary quality automatically, although there is mounting evidence that ROUGE (Lin, 2004) does not discriminate between different types of errors, in particular those relating to factuality (Min et al., 2023; Clark et al., 2023).
While end-to-end models are a popular choice for summarization tasks (Chen et al., 2022; Zhang et al., 2020), more modular approaches have been gaining ground recently (Guan and Padmakumar, 2023; Gupta and Kembhavi, 2022; Sun et al., 2023) for several reasons. Modules can be developed independently, and exchanged for better versions if available, new modules can be added to create new solutions or repurposed for different tasks, and dependencies between modules can be rearranged. Aside from greater controllability, modular approaches are by design more interpretable, since errors can be inspected and attributed to specific components. In this paper we delegate the end-toend task of summarizing from multiple modalities (i.e., TV show video and its transcript) to more specialized modules, each responsible for handling different subtasks and their interactions. Our approach is depicted graphically in Figure 1.
As scene breaks are not always given explicitly, we devise an algorithm to identify them from the order of the speaker names (row 1, Figure 1). Additionally, we select the optimal order in which to re-arrange scenes (row 2, Figure 1), as these often appear non-linearly (e.g., cuts between different subplots or flashbacks). Next, we produce summaries in a two-layer process. A vision-processing
8272
module (Lei et al., 2020; Lin et al., 2022) converts the video to text using visual captioning (row 3, Figure 1), which leverages the strong specialized ability of vision-to-text models and allows us to treat the problem as one of text-to-text summarization. We also summarize each transcript scene independently with a module specialized for dialogue summarization (row 4, Figure 1). Finally, a highlevel summarization module specialized for narrative summarization fuses the sequence of scene summaries into a final summary (row 5, Figure 1).
We also propose a new metric for assessing the factuality of generated summaries by adapting FActScore (Min et al., 2023), a recently introduced metric for detecting hallucination in text generation. We break the generated summary into atomic facts, and check what fraction of them are supported by the reference. This we term fact-precision. We also do the same in reverse, breaking the reference into facts and measuring what fraction are supported by the generated summary, which we term fact-recall. Our metric, PRISMA (Precision and Recall Evaluation of Summary Facts), is the harmonic average of these two scores. Our contributions are:
• We present a novel modular approach to multimodal summarization, where separate subtasks are performed by separate modules;
• Our modules involve detecting scene breaks,
reordering each scene, summarizing the dialogue therein, converting the visual information to text, and fusing the scene-summaries into a final summary (see Figure 1);
• We present two novel algorithms, for determining the optimal order in which to place each scene, and for identifying where the scene breaks are located in the transcript;
• We devise a new metric for summarization, based on splitting text into atomic facts, which captures both precision and recall and correlates significantly with human judgments.

heading: 2 Related Work
text: In the area of long-form summarization, various methods have been proposed to deal with inputs that exceed the context size of a large language model (LLM). Memwalker (Chen et al., 2023) forms a tree of hierarchical summary nodes and traverses it during inference to find the most relevant parts of the input text. Pang et al. (2023) propose a two-layer method where the top layer captures coarse long-range information and produces topdown corrections to a local attention mechanism in the lower layer. Chang et al. (2023) describe two settings for long-form summarization with LLMs: hierarchical merging summarizes chunks of the input sequentially, whereas iterative updating con-
tinually updates a single summary for each of the chunks. We also adopt a two-layer approach but differ in that we divide the input into semantically meaningful chunks, i.e., scenes, rather than uniform chunks, and we summarize each independently before fusing them together.
The problem of generating descriptions for videos has also received much attention, largely independently from long-document summarization. It is common practice (Zhang et al., 2021; Pan et al., 2020; Ye et al., 2022) to extract features from each frame individually, then fuse them into a single feature vector and decode with a language model. Swinbert (Lin et al., 2022) instead uses an end-toend video network, dispensing with image-based encoders, and samples frames densely. Lei et al. (2020) generate descriptions for short videos with a memory-augmented vision transformer. Popular video captioning datasets (Chen and Dolan, 2011; Xu et al., 2016) are only ∼10s in length. YouCook (Zhou et al., 2018) is a recent dataset with longer videos (5min on average), but still far shorter than the TV shows we focus on here.
Multimodal summarization is an extension of video captioning in which the input contains text as well as video. Pan et al. (2023) tackle the analogous problem for still-images with a single model whose architecture allows image and text input and can produce a text description as output. Bhattacharyya et al. (2023) convert the video to text and feed it to a visual storytelling model. Tsimpoukelli et al. (2021) train a vision encoder to produce a sequence of vectors which, when fed to an LLM, produce a textual description of the image contents. Papalampidi and Lapata (2023) apply a similar idea to multimodal summarization, extracting a feature vector from the visual input which is fed, along with the token embeddings from the transcript, to a summarization network. Our method does not try to extract a visual feature vector that functions like a token embedding, but rather extracts actual tokens, i.e., a textual description, from the video.

heading: 3 Decomposition into Modules
text: Our decomposition of the summarization task into modules is motivated by three assumptions specific to TV shows: (1) each scene is somewhat self-contained and, on a coarse level, the events it depicts can be understood independently of other scenes (2) the order in which scenes appear is not necessarily the optimal order to facilitate under-
standing, often shows cut back and forth between different plotlines, and sometimes they are presented non-linearly (3) an effective way to capture visual information in a multimodal text summary is to translate it into natural language. These assumptions motivate our modular approach, which is depicted graphically in Figure 1. Five separate subtasks are performed by separate components: scene-break detection (top row), scene reordering (second row), converting visual information to text (third row), dialogue summarization (fourth row), and high-level summarization (fifth row).
Assumption (1) motivates our choice to detect scene breaks and summarize each independently with a module specialized for dialogue summarization and then later fuse these with a high-level summarization module to produce a final output summary. Assumption (2) motivates our scenereordering algorithm: we do not simply concatenate scene-level summaries in the order in which they appear, but rather we compute an optimal order designed to minimize the number of transitions between different plotlines. Assumption (3) motivates our choice of how to capture the visual information in our summaries. A visual processing module produces a textual description of the video for each scene, which is fed, alongside the dialogue summaries to the high-level summarization module. As a result, the high-level and dialogue summarization modules only need to focus on the single modality of text.

heading: 3.1 The Multimodal Summarization Task
text: Before discussing the details of the various modules, we provide information on our specific task and the dataset we are working with. We develop and evaluate our approach on SummScreen3D1 (Papalampidi and Lapata, 2023), which to our knowledge is the only existing dataset for long-form video summarization. It consists of 5,421 videos of TV episodes (mostly soap operas) varying in length from 30–60min, with accompanying transcripts (on average 6K tokens long) and summaries that were written by fans and scraped from public websites (average length is 200 tokens). These are partitioned into 296 each for validation and testing, and the remaining 4,829 for training. Videos can have multiple summaries from different fansites (the average number of summaries per episode
1https://github.com/ppapalampidi/video_ abstractive_summarization
is 1.53), giving a total of 8,880 training pairs. In SummScreen3D, each data point contains a video, a transcript that includes character names and, sometimes, marked scene breaks, and a closed captions file, which is used to display subtitles and has timestamps but not speaker names. We align the lines in the transcript with those from the closed captions. These do not match perfectly, because of slight errors in the automatic transcription. For each line t in the transcript and utterance c in the caption, we estimate a similarity score as
|f(t,c)| min (|t|,|c|) , where f computes the longest common subsequence. Then we use dynamic time warping to align both sequences (Myers and Rabiner, 1981; Papalampidi et al., 2021). As a result, we obtain alignments of transcript utterances to video segments, which allows us to use the scene-breaks from the former to segment the latter.

heading: 3.2 Scene Detection
text: This module (Figure 1, row 1) partitions the transcript into contiguous chunks. Each line in a transcript begins with the name of the character speaking, and our algorithm seeks a partition where each chunk contains only a small number of characters.
We define a cost for a given partition, by invoking the minimum description length (MDL) principle (Grünwald, 2007), in which the optimal representation of a piece of data is that which contains the fewest of bits. MDL has been shown to have many nice theoretical properties (Grünwald, 2007) and has been used in recent machine learning applications from interpreting hidden representations (Voita and Titov, 2020) to measuring noisefree information content in images (Mahon and Lukasiewicz, 2024). We use it here to formalize the desirable properties of a good segmentation.
Thus, the cost of a partition is the number of bits needed to specify it. We assume the total set of N character names for the entire transcript is given. Then, for each scene, we make a scenespecific codebook for the n characters that appear there, which assigns each character a code as an index from 0, . . . , n − 1, which all have length ≤ ⌈log n⌉. Using exactly ⌈log n⌉ guarantees a prefix-free code (Grunwald, 1998)2. The number of possible codebooks with n out of N characters is ( N n ) . Imposing some order, e.g., lexicographic,
2Because we do not need to convert to concrete codes, we omit the ceiling operator in our optimization objective.
Algorithm 1 Compute the optimal partition of transcript lines into scenes. m← # lines in the transcript P ← an m×m matrix, to store the cost of spans Q ← an m ×m matrix, to store the optimal partition of spans for i = 2, . . . ,m do
for j = 1, . . . ,m− 1 do n← # of unique characters in lines j, . . . , j + i P [j, j + i]← C(N,n) + i logn Q[j, j + i]← ∅ for k = j, . . . , j + i do
if P [j, k] + P [k, j + i] < P [j, j + i] then P [j, j + i]← P [j, k] + P [k, j + i] Q[j, j + i]← Q[j, k] ∪ {k} ∪Q[k, j + i]
on these, the number of bits to specify one is
C(N,n) := log
( N
n
) = log
N !
n!(N − n)!
and the total cost of a given scene is then
C(N,n) + l log n , (1)
where N is the total number of speakers as before, and n and l are, respectively, the number of distinct speakers and the number of transcript lines in the scene. See Appendix D for details and examples. Intuitively, the two terms in (1) impose opposite pressures on the solution. Term 1, the bitcost of specifying a new code book, imposes a cost on each scene break, and means the algorithm only inserts a scene break if it leads to a sufficient reduction in term two to outweigh this cost. Term 2, the bitcost of using each code book to specify the characters in the corresponding scene, penalizes solutions where scenes contain lots of different characters, and encourages instead segmentations where the number of speaking characters in each scene is minimized.
Let m be the number of lines in the transcript. Then, for any 1 ≤ i < j ≤ m, the cost of all scenes from line i to line j under the optimal partition, written as S(i, j), can be expressed recursively as
S(i, j) = min 2≤k≤i S(i, k) + S(k, j) ,
where S(i, j) is defined as zero when i = j. This motivates a dynamic programming solution, similar to the CYK algorithm for context-free parsing (Kasami, 1966), in which we compute, in order, the optimal solution of spans of lengths 2, . . .m, and reuse solutions of smaller spans when computing those of larger spans. Our algorithm runs in O(N3), with only O(N2) calls to compute the
scene cost as per Equation (1). The scene breaks can be transferred to the video because of the dynamic time-warping alignment between the transcript and the timestamps in the closed captions, as described in Section 3.1. Note, that our algorithm determines automatically the number of scenes.

heading: 3.3 Scene Reordering
text: We now discuss how we compute the optimal order of scenes (Figure 1, row 2), prior to summarization. We first define a cost for a given order as
C(s1, . . . , sn) = n−1∑
i=1
1− IOU(si, si+1) , (2)
where IOU is the intersection over union of character names. For example, if the characters in scene s1 are Alice and Bob, and the characters in scene s2 are Bob and Charlie, then IOU(s1, s2) = 1 3 . Additionally, we introduce a constraint that if the same character appears in two different scenes, si and sj , then we should never swap the order of si and sj , as that may violate causality between the two scenes.
We approximately solve this optimization problem by passing from s2 to sn and moving each scene as far to the front as possible without violating the causality constraint, if this leads to an improved total order cost. We continue passing from s2 to sn until no changes are made. Our algorithm runs in O(N2), where N is the number of scenes, typically 30. The change in cost when moving a scene depends only on the IOU cost of that scene and its current and future neighbours, and the IOU cost of all pairs of scenes can be cached.

heading: 3.4 Vision Processing
text: We explore two methods for the vision-processing module which differ in architecture and scope (Figure 1, row 3). SwinBERT (Lin et al., 2022) is a dedicated video captioning model; it operates directly on a sequence of video frames for which it generates a description. SwinBERT is end-to-end trained with a masked language modeling objective, and a sparse attention mask regularizer for improving long-range video sequence modeling. We apply SwinBERT to the video for each scene, sampled at 3 frames-per-second, to obtain a video caption, which we use as a description of video contents.
Kosmos-2 (Peng et al., 2023) is pretrained on several multimodal corpora as well as grounded image-text pairs (spans from the text are associated
Algorithm 2 Minimize the number of transitions between scenes with different speakers.
function IOU(x,y) return #characters in both x and y#characters in x+#characters in y function COMPUTE OPTIMAL ORDER(S) s1, . . . , sn ← scenes in order of appearance in S; C ← n× n matrix; ▷ cache for IOUs for i = 1, . . . , n do
for j=i,. . . ,n do C[i, j]← 1− IOU(si, sj); if (si, sj) == 0 then
C[j, i] = 1 else
C[j, i] =∞ changed← True; while changed do
for i = 2, . . . , 2 do n = min{j|C[j, i] < 1}+ 1 ▷ new idx for i cost1← C[i−1, i+1]−C[i−1, i]−C[i, i+1]; cost2← C[n− 1, i] + C[i, n]− C[n− 1, n]; if cost1 + cost2 < 0 then
move si to position new in S; changed← True break
with image regions) and instruction-tuned on various vision-language instruction datasets. Unlike SwinBERT, it operates on individual images, so we first extract three equally spaced I-frames from the h264 video encoding (Wiegand et al., 2003), and take the captions from each. We prompt Kosmos-2 with “A scene from a TV show in which”.
Both SwinBERT and Kosmos-2 can generate uninformative textual descriptions, e.g., “a man is talking to another man” which we discard. We also modify them to replace unnamed entities such as ‘the man’ with character names where these can be easily inferred. See Appendix B for details of our filtering and renaming procedures.

heading: 3.5 Summary Generation
text: Because of our two-layer summarization approach, we can use a relatively small backbone model, and are still able to encode very long input. In our experiments, we use variants of BART-large (Lewis et al., 2020) for both the dialogue summarization and high-level summarization modules, but any other similar model could be used instead. For the dialogue summarization module (Figure 1, row 4), we use the public Huggingface checkpoint for BART which has been fine-tuned on the SamSum dataset (Gliwa et al., 2019), to output multi-sentence summaries of dialogue. For the high-level summarization module (Figure 1, row 5), we use BART, fine-tuned first for document summarization on the CNN/Daily Mail dataset (Hermann et al., 2015),
and then on SummScreen3D. The input for the latter fine-tuning is our re-ordered scene summaries and visual text descriptions, i.e., the output of the dialogue summarization, vision-processing and reordering modules. The output is the gold summaries from the SummScreen3D (Papalampidi and Lapata, 2023) training set.

heading: 4 Fact Precision, Fact Recall and PRISMA
text: Hallucinations are a widely known issue with abstractive summarization (Song et al., 2018; Maynez et al., 2020; Kryscinski et al., 2020; Gabriel et al., 2021), especially when the output is long (Min et al., 2023) and information is being consolidated from multiple sources (Lebanoff et al., 2019). Automated metrics are crucial for our task and related creative summarization applications where human evaluation is extremely labor-intensive (e.g., watching long videos or reading book-length transcripts), costly, and difficult to design (Krishna et al., 2023).
Our proposal is based on a recent metric, FActScore (Min et al., 2023), which aims to detect hallucination in text generation. FActScore first parses the generated text into atomic facts (i.e., short sentences conveying one piece of information), and then determines whether these are supported by an external knowledge source, such as Wikipedia. The FActScore for some model output is the fraction of the extracted facts that are judged to be supported. Min et al. (2023) recommend using InstructGPT for the first stage of converting the text into facts, and Llama-7B (Touvron et al., 2023) for checking whether these facts are supported (i.e., by zero-shot prompting Llama to estimate whether a generated fact is True or False).3
We adapt this for summary evaluation by replacing the external knowledge source with the gold summaries. Note that this incorporates relevance as well as accuracy. There are many true facts about the TV show being summarized, and only some of them appear in the gold summaries. We assume that facts that do not appear in the gold summary are irrelevant and so should not be marked as correct if they appear in our model summaries. This is our fact-precision metric. For fact-recall, we do the same in reverse: we convert the gold summary into atomic facts, and then check whether each of these is supported by the generated summary. Again, the score is the fraction of such facts that are supported. A summary will get a high fact-precision score if
3https://github.com/shmsw25/FActScore
every atomic piece of information is both true and relevant enough to appear in the gold summary. It will get a high fact-recall score if it contains every atomic piece of information that was contained in the gold summary. The final score for our metric, which we term PRISMA (Precision and Recall Evaluation of Summary Facts), is the harmonic mean of fact-precision FP and fact-recall FR:
PRISMA = 2
1 FP + 1 FR
.
In our implementation, we used GPT4-Turbo for the extraction of atomic facts and the estimation of whether they are supported, as we found that Llama-7B overestimates the number of supported facts. We use the same prompts as Min et al. (2023) and make a separate query, with in-context learning examples, for each fact. In order to penalize repetitive/redundant information, repeated facts are regarded as unsupported. In the case that GPT indicates that the sentence is not properly formed, we convert it to a single fact which is scored as unsupported (see Appendix C for an example).
We examined whether PRISMA correlates with human judgments of factuality. Two annotators, both English native speakers, were asked to watch four randomly selected shows from the SummScreen3D development set. They were then shown facts generated by GPT4 corresponding to summaries produced by several automatic systems (those described in Section 5.1). For each fact they were asked to decide whether it was supported by the episode (1/True or 0/False). Human ratings significantly correlate with GPT4’s estimate on whether a fact is supported (Pearson’s r = 0.5, N = 520, p < 0.01), with an inter-annotator agreement of r = 0.57 (p < 0.01) as upper bound.

heading: 5 Experimental Evaluation
text: Our full model was composed of the modules described in Section 3. For those SummScreen3D transcripts without explicitly marked breaks (approximately 20%), we split into scenes using Algorithm 1. The high-level summarization module was trained for a max of 10 epochs, with early stopping using ROUGE on the validation set, with a patience of 2. The optimizer was AdamW with learning rate 1e-6, and a linear scheduler with 0 warmup steps. Training and inference took place on a single NVIDIA A100-SXM-80GB GPU, taking seven and one hour(s), respectively. The other models in our framework are not fine-tuned.

heading: 5.1 Comparison Models
text: We compare our approach to the end-to-end model of Papalampidi and Lapata (2023), which represents the state of the art on our task, and various text-only models operating on the transcript:
Unilimiformer (Bertsch et al., 2023) is a retrieval-based method that is particularly suited to processing long inputs. It wraps around encoderdecoder transformers, with context size k, and can extend this size arbitrarily by storing an index of all input tokens and replacing the cross-attention query mechanism with the k-nearest neighbours from this index.
LLama-7B, Mistral-7B As our backbone BART models are relatively small, ∼400M parameters, we may be able to obtain better summaries, simply using a larger model; we test this hypothesis with LLama-7B and Mistral-7B, both fine-tuned for three epochs on our training set.
startend We implemented a simple baseline which uses BART fine-tuned for dialogue summarization and takes scenes from the start and end of the transcript up to the maximum that can fit in the context size (1,024). This is inspired by recent work showing that even long-context transformers take information mostly from the beginning and end of the input (Liu et al., 2023).
central is inspired by the method of Papalampidi et al. (2021). Again it uses BART and selects a subset of the input to fit in the context size. It computes a weighted graph whose nodes are scenes and edge-weights are tf-idf similarity scores, then
uses the page rank algorithm to rank scenes in order of importance, and selects the topmost important scenes that can fit in the context window.
We also compare to an upper-bound which tests the gold summaries against each other. Recall that SummScreen3D often contains multiple summaries for each episode. We select one from the three websites with the most uniformly-sized summaries and treat it as if it was the predicted summary, then test it against the remaining summaries.

heading: 5.2 Results
text: Table 1 summarizes our results, as measured by ROUGE-1 (r1), ROUGE-2 (r2), ROUGE-Lsum (rlsum) (computed using the python-rouge package) and our new PRISMA metric. We present results with BERTScore (Zhang et al., 2019) in Appendix E, for the sake of brevity. Results for Papalampidi and Lapata (2023), which we denote as ‘adapter-e2e’, were reproduced with their code.4 For PRISMA, we use a separate query to GPT for each fact to check whether it is supported. There are roughly 70 facts per generated summary, which can lead to significant financial cost if used excessively. Therefore, while we report five random seeds for ROUGE, we select a single seed (randomly) for fact-precision and fact-recall.
Our modular approach outperforms comparison models across all metrics. As shown in Table 1, our method, which we denote as ‘modular’ significantly outperforms the previous end-toend multimodal system of Papalampidi and Lap-
4https://github.com/ppapalampidi/video_ abstractive_summarization
ata (2023), and all comparison text-only models (upper block). A two-sampled t-test shows that improvement over the comparison models is significant at α = 0.01 (calculation in Appendix F). We observe that billion-parameter models like Mistral and Llama struggle with this task (although Mistral is superior), despite being fined-tuned on SummScreen3D. Amongst text-only models, Unlimiformer performs best which suggests that the ability to selectively process long context has a greater impact on the summarization task. As far as our model is concerned, we find that the type of visual processing module has an effect, allbeit small, on the quality of the output sumaries. Overall, Kosmos-2 (Peng et al., 2023) has a slight advantage over SwinBERT (Lin et al., 2022) which we attribute to it having been trained on various image understanding tasks, besides captioning.
PRISMA better reflects summary quality than ROUGE. Interestingly, upper-bound ROUGE scores are lower than those for several models. We regard this as a shortcoming of ROUGE as a metric. Qualitatively, reading the different gold summaries shows that they are more similar to and accurate with respect to each other than any of the predicted summaries, including ours. Although they differ in phrasing and length, they all describe the same key events. PRISMA better reflects this similarity and gives a much higher score to ‘upper-bound’ than to any of the models. The large gap to get to the level of ‘gold upper-bound’ reflects the difficulty of the task. The comparison between ROUGE and PRISMA suggests that the former is useful for detecting which summaries are of very low quality, (e.g., if ROUGE-2 is < 5). However, for distinguishing between multiple relatively decent summaries, PRISMA is more useful.
All modules are important, but the transcript is the most important of all. Table 2 shows the effect of removing four of the five modules. In ‘w/o
scene-detection’, we remove the scene-detection module and just split the input into equally-sized chunks equal to the context size (1,024). In ‘w/o transcript’ we do not include summaries of the dialogue from the transcript, and the only input to the high-level summarization module is the output of the visual processing module. In ‘w/o video’ we do the reverse: use only the dialogue summaries without the output of the vision-processing module. In ‘w/o reordering’, we remove the scene-reordering module and present the scene summaries to the high-level summarization module in the order in which they appear. All these experiments are performed using the Kosmos-2 captions (except ‘w/o video’ which has no captions).
Aside from ROUGE-1 and ROUGE-Lsum, which are roughly the same for ‘w/o reordering’ and ‘w/o scene-detection’, all ablation settings lead to a drop in all metrics. Unsurprisingly, the largest drop is in ‘w/o transcript’ as most information is in the dialogue — a TV show without sound or subtitles is difficult to follow. Interestingly, however, this setting still gets some n-grams (ROUGE) and facts (PRISMA) correct, showing that our model is extracting useful information from the video. This is also clear from the ‘w/o video’ setting, which shows a drop in all metrics. Qualitatively, we observe that the vision-processing module is most useful for identifying locations, e.g., ‘at the hospital’ which is generally not mentioned in the dialogue. For ‘w/o scene-detection’ and ‘w/o reordering’, the difference is moderate for ROUGE. For PRISMA, it is more substantial and highly significant when taken over all facts in the dataset.
Our scene-detection algorithm is more accurate than uniform baselines. Here, in Table 3, we report an empirical test of the accuracy of our scenedetection algorithm (Section 3.2, Algorithm 1). These figures are produced using an episode which has explicitly marked scene breaks. We assign each transcript line a label based on the scene it is in with
respect to these explicit scene breaks, e.g., all lines in the first scene get the label ‘0’. Then we do the same for the scene breaks produced by our algorithm, and compare the two sets of labels using unsupervised label comparison metrics, as commonly used in clustering problems: accuracy (ACC), normalized mutual information (NMI), and adjusted Rand index (ARI), defined as, e.g., in Mahon and Lukasiewicz (2023). We compare to two baselines. The first, denoted ‘uniform’, divides each episode into n equally-sized scenes, where n is the average number of scenes per episode. The second, denoted ‘uniform oracle’, does the same except sets n to the true number of scenes in that episode.
Our method produces more accurate scene splits than both baselines, despite taking no information from the ground-truth labels. Moreover, many of the occasions on which it differs from the groundtruth scene splits appear at least as reasonable as the ground-truth splits when we observe only the character names in the transcript. This suggests that errors in the splits are not due to the algorithm itself but to the fact that, at present, it only uses character names, ignoring the speech itself. A future extension is to use named entities or all nouns from the speech as well as character names.
Humans prefer summaries generated by our modular approach We selected 8 episodes from SummScreen3D, and asked human participants (recruited through the Prolific platform) to watch the episode, then rank summaries in order of perceived quality. The summaries we included in this study were ours, the gold summary, unlimiformer (Bertsch et al., 2023), ‘adapter e2e‘ (Papalampidi and Lapata, 2023), and finetuned Mistral. We instructed participants to priortize informativeness and factual correctness, with a secondary emphasis on fluent well-written text. We included an attention-check question that participants would only be able to answer if they had watched the full
episode, and asked for a short justification for the chosen ranking. The full instructions can be found in Appendix H. We recruited five annotators for each of the eight episodes.
Table 4 presents the ranking given to the systems by our participants. Specifically, we calculate Elo (1967) scores for each of the summary methods we perform evaluation on. We adjust scores once after all comparisons are made so as to be independent of the order of episodes. We are interested only in relative scores and correlations with other metrics, which are independent of the adjustment factor K and the starting score. So, we set K = 1 and the starting score to 12 , and the score for each method reduces to the fraction of pairwise comparisons they won. We can also then observe that our PRISMA metric is a much better predictor of human preference (Pearson’s r = 0.792) than ROUGE-2 (r = 0.594) and ROUGE-Lsum (r = 0.512).

heading: 6 Conclusion
text: We addressed the task of summarizing TV shows from videos and dialogue transcripts. We proposed a modular approach where different specialized components perform separate sub-tasks. A scenedetection module splits the TV show into scenes, and a scene-reordering module places these scenes in an optimal order for summarization. A dialogue summarization module condenses the dialogue for each scene and a visual processing module produces a textual description of the video contents. Finally, a high-level summarization module fuses these into an output summary for the entire episode. We also introduced PRISMA, a new metric for longform summarization, based on splitting predicted and reference summaries into atomic facts. It captures both precision and recall, and correlates significantly with human judgments. In the future, we plan to test our method on even longer inputs, and explore settings where transcripts are not available.
Limitations
While the modular approach we propose has advantages, such as allowing specialization of individual modules, and the ability to replace one module without affecting the others, it also has the disadvantage that it is difficult to fine-tune all modules. We fine-tune only the high-level summarization module, whereas for a monolithic end-to-end model, all parameters can be trained.
Our proposed PRISMA metric requires multiple calls to GPT which incurs a financial cost. We estimate that all the results presented in this paper cost about $1300. This is still many times cheaper than hiring human evaluators to extract and score facts manually, which we estimate would take 400 person-hours and cost about $15,000.
There is still a significant gap, in terms of PRISMA, between our summaries and the upper bound of comparing the gold summaries to each other. This shows that the task is challenging and requires further advances to reach human-level.

heading: Acknowledgments
text: We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of Amazon and the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1), and the feedback and advice from Liz Norred, Zhu Liu, Vimal Bhat and Rohith Mysore Vijaya Kumar.

heading: A Example Summaries
text: Table 5 shows the gold summary for an episode from the SummScreen3D testset. Tables 6 and 7–9 show the summary of our model, and those of comparison models, respectively, for the same episode.

heading: B Caption Filtering and Inference of Names
text: The output from the visual processing module for a given scene is a textual description of the visual information for that scene. Sometimes, this description is vague and merely adds noise to the high-level summarization module input. This is also clear from manually viewing the TV show episodes: many scenes are just headshots of characters talking, and do not convey plot information. For both SwinBERT and Kosmos-2, we filter out any captions that contain the following phrases ’a commercial’, ‘talking’, ‘is shown’, ‘sitting on a chair’, ‘sitting on a couch’, ‘sitting in a chair’, ‘walking around’. Addtionally, we replace occurrences of the phrase “is/are seen” with “is/are”. This is because the captioning datasets these models are trained on often use the phrase “is/are seen”, e.g., “a person is seen riding a bicycle” instead of “a person is riding a bicycle”, but we do not want such passive voice constructions in our summaries.
The captions do not contain character names, as these names cannot be inferred from the video alone. Therefore, for each sentence in the output of the visual processing module, we employ the following method to insert names, where they can be easily inferred. We categorize each name appearing in the transcript for the scene as male, female or neutral, using the vocabulary list for English names from Python’s NLTK. Similarly, we assume noun-phrases “he”, “a man”, or “a boy” are male and noun-phrases “she”, “a woman”, “a girl” are female. If there is only one male name, then we replace all male noun phrases with that name, similarly for female names. For example, from the caption for Scene 2 in One Life to Live, (aired 10- 18-10), as shown in Figure 2, the output of the visual processing module contains the sentence “a man is kissing a woman”, and the only names in the transcript for that scene are “Brody”, which is listed as male, and “Jessica”, which is listed as female. Our method to insert names then transforms this caption to “Brody is kissing Jessica”.

heading: C PRISMA Metric
text: Often, facts extracted by GPT4 are too vague and uninformative to be useful for assessing summary quality. Therefore, we perform a simple filtering to remove such facts. Specifically, we eliminate out any facts containing the following words for phrases: “someone”, “somebody”, “something”,
“is a person”, “are people”, “is a character”, “are characters”. Additionally, we eliminate all facts containing only two words, as we observe they are almost always uninformative and often are an ungrammatical use of a transitive verb in an intransitive context. Table 10 shows the full list of facts extracted from our summary for The Bold and the Beautiful, (aired 05-05-06), showing which are eliminated, which are judged supported and which are judged unsupported.

heading: D Scene Detection Details and Examples
text: Here, we provide further details on the scene detection algorithm from Section 3.2 and report a measure of its accuracy.
We calculate the cost for a given partition by assuming that the scene breaks under this partition and the placeholders where the speakers would go are given, and then asking how many bits are needed to fill in the speaker names. Then the receiver can infer the length of the scene-specific codebook as 2m, where m is the length of each code in the scene. Note, we still need prefixfreeness, as just knowing the number of speaker lines in a scene doesn’t allow us to distinguish between 10 followed by 11 and 101 followed by 1.
For example, consider the following sequence of character names from As the World Turns (aired 01-09-07): ’Elwood’, ‘Casey’, ‘Elwood’, ‘Casey’, ‘Elwood’, ‘Casey’, ‘Elwood’, ‘Casey’, ‘Elwood’, ‘Meg’, ‘Paul’, ‘Meg’, ‘Paul’, ‘Meg’, ‘Paul’, ‘Luke’, ‘Meg’, ‘Paul’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Paul’, ‘Meg’,
‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Meg’, ‘Luke’, ‘Adam’, ‘Gwen’, ‘Adam’, ‘Gwen’, ‘Adam’, ‘Gwen’, ‘Adam’, ‘Gwen’, ‘Adam’.
Abbreviating the character names as their first letters, the correct partition is
1. ECECECECE,

heading: 2. MPMPMPLMPLMLMLMLMPMPMLM-
text: LMLML,
3. AGAGAGAGA
Now consider the cost, as defined in Section 3.2 of this partition. The total vocabulary size is
N = |{E,C,M,P, L,A,G}| = 7 .
For the first scene, n1 = 2, n2 = 3, n3 = 2, l1 = 9, l2 = 28, l3 = 7. Letting ci denote the cost for the
ith scene, we then have similarly,
c1 = log
( 7
2
) + 9 log 2 ≈ 13.392
c2 = log
( 7
3
) + 28 log 3 ≈ 49.508
c3 = log
( 7
2
) + 9 log 2 ≈ 13.392 ,
giving a total cost of approximately 13.392 + 49.508 + 13.392 = 76.292.
Now compare this to the cost of a different partition of this sequence, say into scenes of uniform length:
1. ECECECECEMPMPMP,
2. LMPLMLMLMLMPMPM,
3. LMLMLMLAGAGAGAGA
In that case
c1 = log
( 7
4
) + 15 log 4 ≈ 34.392
c2 = log
( 7
3
) + 15 log 3 ≈ 28.167
c3 = log
( 7
4
) + 16 log 4 ≈ 36.392 ,
giving a total cost of approximately 34.392 + 28.167 + 36.392 = 98.951. Therefore, because our algorithm computes the global minimum to this cost function, it is guaranteed to choose the correct partition over this uniform one. In fact, it selects the correct partition.

heading: E Additional Results
text: E.1 BERTScore Tables 11 and 12 report results on summary quality according to BERTScore (Zhang et al., 2019). Tables 11 shows BERTScore for our model and comparison models, the analogue of Table 1 in the main paper, and Table 12 shows BERTScore for the ablation settings, the analogue of Table 2 in the main paper.
We find this metric does not distinguish well between the different settings, and scores all models very similarly. Even the “w/o transcript” setting, which qualitatively misses most of the important information in the episode, and scores poorly on ROUGE and PRISMA, gets a high BERTScore. Moreover, even within each setting, the scores are very similar across different episodes, again in contrast to ROUGE, PRISMA and qualitative evaluation (some episodes appear much easier to summarize than others). The average standard deviation across episodes, within each setting, is 2.2, 1.39, and 1.41 for BERTScore-precision, BERTScorerecall and BERTScore-f1 respectively. In contrast, the same standard deviation for ROUGE-1, ROUGE-2, ROUGE-Lsum are 6.66, 3.32, 6.52, while for factscore-precision, factscore-recall and PRISMA, they are 14.92, 15.14 14.94, respectively.
This suggests that BERTScore always returns a similar score, regardless of the input. This inability of BERTScore to adequately distinguish different settings was also reported by Papalampidi and Lapata (2023).

heading: F Significance Calculations
text: The improvements in our model over comparison models, as shown in Table 1 are statistically sig-
nificant at α < 0.01. We now calculate an upper bound on all metrics to establish this, for an unpaired unequal variances T-test, i.e.. a Welch test. The t-value for such a test is calculated as
tw = µ1 − µ2√ σ21 + σ 2 2 , (3)
where µi and σ2i are the mean and variance, respectively, for the ith sample. Let µ1, σ21 denote the mean and variance for our model, and µ2, σ22 those for some comparison model. Excluding Llama7B as a special case, which we consider below, then, for ROUGE1, µ10.6, and σ2 ≤ 0.42. Therefore, the denominator in (3) is ≤≈ 0.328. Also, µ = 44.86 µ2 ≤ 42.24. Thus,
tw ≥ 44.86− 42.24√
0.62 5 + 0.422 5
≈ 4.47 .
Similarly, for ROUGE2
tw ≥ 11.83− 10.32√
0.152 5 + 0.342 5
≈ 4.78 .
and for ROUGE-Lsum
tw ≥ 42.49− 40.40√
0.562 5 + 0.342 5
≈ 4.72 .
Using the Welch-Satterthwaite equation, the degrees of freedom ν can also be lower-bounded at
5, giving a critical value, for α = 0.1 of 3.37, which is less than our t-value for each for ROUGE1, ROUGE2 and ROUGE-Lsum.
For FActScore, we do not run multiple random seeds because of the financial cost, but if we compare distributions over individual facts, we see all comparisons are very highly significant. Each setting involves ∼ 15, 000 facts, with the ‘no transcript” ablation having fewer, ∼ 4, 000. Expressing fact-precision and fact-recall as fractions rather than percentages, and recalling that Bernoulli distribution with mean p has variance p(1−p), we see that the denominator in (3) is upperbounded by 24000 , so the t-value is lower-bounded by 2000(µ1 − µ2). This is in the hundreds for all µ1, µ2 from the factscore-precision and factscorerecall columns in Table 1, which is far above the critical value of 2.33.

heading: G Correlation Between Metrics
text: Figure 3 shows the pairwise correlation (Pearson) between all metrics that we report in Section 5. This is taken across all data points in all settings. All metrics are at least weakly correlated with each other, with the strongest correlations between the different varieties of ROUGE. This suggests that our PRISMA metric captures information outside of what is captured by ROUGE.

heading: H Human Evaluation Setup
text: We request that each participant read the following guidelines before ranking the summaries.
When comparing summaries, you should take into account the following three points, in order of importance. (1) Firstly, most importantly, does it give plenty of detail about the events in the episode, rather than repeating the same information multiple times? (2) Does it report events and characters that really are in the episode, rather than irrelevant information, e.g. about when and where the episode was aired or reviews it received? (3) Is the text well-written, containing full properly-formed sentences devoid of spelling, grammar, and punctuation errors?
The following is an example of a good summary:
Clint goes to Robert’s home and has two guys assault him for what he did to Jessica. But he tells Robert he still gets to keep the job that Clint has enabled him to keep at LU and James can keep his grant. Meanwhile, Jessica and Brody are happily planning their future together with their new baby but agree to get the paternity test so that they know. In the place where Eli has kept her, Dani manages to untie herself from the chair and rushes to the other room in an attempt to save Starr and Hope, assuming they are on the other side. But when she gets through, she is shocked and elated to see her mom. They cry and rejoice but are horrified to find out that Eli, whom they’ve both seen as a trusted family member and upstanding citizen, is a serial killer and psycho. She admits she does not remember much but is going to get better and be with her daughter and husband. Dani unties her mom and they are ready to go home while Eli is gone. Eli goes to find unconscious Greg in his hospital room and is ready to inject a lethal dose and kill him knowing what Greg was planning, until Shaun catches him and he runs. Greg is still unconscious however, and Eli escapes. And he comes back in time to prevent Tea and Dani from escaping. Meanwhile, Hannah holds Starr and baby Hope captive and is determined to keep Starr away from Cole for as long as it takes for him to forget all about Starr and fall in love with Hannah. But when she returns to Marty’s, James notices that she is carrying his bullet on the chain in her purse. He finds that very odd since it was last in Starr’s possession. He also finds it very odd when Nate informs him and Robert that Eli called and let him, Todd and Blair all talk to Dani but made no mention of Starr and nobody knows where Starr and Hope are. He senses they are in danger and is determined to save them.
The following is an example of a bad summary, which repeats the same point again and again instead of covering all the information in the episode:
Ford is beaten up by two men that Clint hires, but Clint does not fire him from his job at NU, even though Ford expects that he will be fired. Clint is angry because Ford took advantage of his daughter, Jessica, while she was vulnerable. Jessica and Brody are worried about who the father of their baby is, so they are doing a paternity test. They are doing the paternity test so they can find out who the father is.They are worried about who the father is, but they are going to do a paternity test so they can find out who the father is.They are worried about who the father is, but they are going to do a paternity test so they can find out who the father is.The paternity test will tell them who the father is, and they are worried about it because then they will find out who the father is. They are doing the paternity test to find this out, and find out who the father is with the paternity test. They are very worried about it, but are doing a paternity test to find out for sure. This will tell them who the father is.
The following is an example of a bad summary that is poorly written, discusses irrelevant material and contains garbled sentences:
Clint pays two large men to come with him to Ford’s apartment and beat him up for what Ford@; Ford did, the apartment downstairs, Ford did to his daughter. Ford thinks he will be fired from NU but Clint says he is not. Ford: I guess I’m gonna be out of a job, huh? At The The.. the hospital, Brody meets Jessica to do a paternity test. .. Brody and.. The hospital. The test is ready to go. The episode airs at 9 p.m. ET/PT on Thursday, December 14. The series
finale airs on December 15 at 8 p.M. ET on ABC. Jessica: whatever happens, we’ll raise this baby together,. Brody: I’m glad you think so honey. Jessica: let’s just do the test to be sure. Dani: Mom? I never thought I’d find you here, I thought you were dead. Tea: I’m so happy that you’re here. Now let’s get out of here before Eli comes back. The season finale airs at 10 p. M.E. on December 16 at 8:30 p.S. ET.M ET on E.C. ET at the Palace Hotel in New York City, New York, and on December 18 at 9 a.C.. The finale will air at 9 P.M.. The season. The finale airs. The premie.. The premieThe finale he series.. The season finale. The.. season finale..The. season finale aired.. The. finale airs.. September 14 at 9:00 p. m. ET, September 14, 2013,
The ‘good summary‘ example is a gold summary from the train set. The ‘bad summary‘ examples are composed by us.
We also ask each participant a question about the contents the episode they are assigned, to check that they have watched and understood the full episode. The list of episodes, and corresponding questions and answers are shown in Table 13.

heading: Future Work
text: Meanwhile, Jessica and Brody are happily planning their future together with their new baby but agree to get the paternity test so that they know. In the place where Eli has kept her, Dani manages to untie herself from the chair and rushes to the other room in an attempt to save Starr and Hope, assuming they are on the other side. But when she gets through, she is shocked and elated to see her mom. They cry and rejoice but are horrified to find out that Eli, whom they’ve both seen as a trusted family member and upstanding citizen, is a serial killer and psycho. She admits she does not remember much but is going to get better and be with her daughter and husband. Dani unties her mom and they are ready to go home while Eli is gone. Eli goes to find unconscious Greg in his hospital room and is ready to inject a lethal dose and kill him knowing what Greg was planning, until Shaun catches him and he runs. Greg is still unconscious however, and Eli escapes. And he comes back in time to prevent Tea and Dani from escaping. Meanwhile, Hannah holds Starr and baby Hope captive and is determined to keep Starr away from Cole for as long as it takes for him to forget all about Starr and fall in love with Hannah. But when she returns to Marty’s, James notices that she is carrying his bullet on the chain in her purse. He finds that very odd since it was last in Starr’s possession. He also finds it very odd when Nate informs him and Robert that Eli called and let him, Todd and Blair all talk to Dani but made no mention of Starr and nobody knows where Starr and Hope are. He senses they are in danger and is determined to save them. The following is an example of a bad summary, which repeats the same point again and again instead of covering all the information in the episode:
Ford is beaten up by two men that Clint hires, but Clint does not fire him from his job at NU, even though Ford expects that he will be fired. Clint is angry because Ford took advantage of his daughter, Jessica, while she was vulnerable. Jessica and Brody are worried about who the father of their baby is, so they are doing a paternity test. They are doing the paternity test so they can find out who the father is. They are worried about who the father is, but they are going to do a paternity test so they can find out who the father is. They are worried about who the father is, but they are going to do a paternity test so they can find out who the father is. The paternity test will tell them who the father is, and they are worried about it because then they will find out who the father is. They are doing the paternity test to find this out, and find out who the father is with the paternity test. They are very worried about it, but are doing a paternity test to find out for sure. This will tell them who the father is. The following is an example of a bad summary that is poorly written, discusses irrelevant material and contains garbled sentences:
Clint pays two large men to come with him to Ford’s apartment and beat him up for what Ford@; Ford did, the apartment downstairs, Ford did to his daughter. Ford thinks he will be fired from NU but Clint says he is not. Ford: I guess I’m gonna be out of a job, huh? At The The. the hospital, Brody meets Jessica to do a paternity test. Brody and. The hospital. The test is ready to go. The episode airs at 9 p. m. ET/PT on Thursday, December 14. The series
finale airs on December 15 at 8 p. M. ET on ABC. Jessica: whatever happens, we’ll raise this baby together,. Brody: I’m glad you think so honey. Jessica: let’s just do the test to be sure. Dani: Mom? I never thought I’d find you here, I thought you were dead. Tea: I’m so happy that you’re here. Now let’s get out of here before Eli comes back. The season finale airs at 10 p. M. E. on December 16 at 8:30 p. S. ET. M ET on E. C. ET at the Palace Hotel in New York City, New York, and on December 18 at 9 a. C. The finale will air at 9 P. M. The season. The finale airs. The premie. The premieThe finale he series. The season finale. The. season finale. The. season finale aired. The. finale airs. September 14 at 9:00 p. m. ET, September 14, 2013,
The ‘good summary‘ example is a gold summary from the train set. The ‘bad summary‘ examples are composed by us. We also ask each participant a question about the contents the episode they are assigned, to check that they have watched and understood the full episode. The list of episodes, and corresponding questions and answers are shown in Table 13