TeaserGen: Generating Teasers for Long Documentaries
Authors: Weihan Xu
Paul Pu Liang
Haven Kim
Julian McAuley
Taylor Berg-Kirkpatrick
Hao-Wen Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Teaser Generation
Multimodal Learning
Vision-Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Teasers are an effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling capability for the input videos, while necessitating maintaining audiovisual alignments, managing scene transitions and preserving factual accuracy for the output teasers. Due to the lack of a publicly-available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, a collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose a new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration from the documentary using a pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-vision models. For narration-video matching, we explore two approaches: a pretraining-based model using pretrained contrastive language-vision models and a deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining-based approach is more effective at identifying relevant visual content than directly trained deep autoregressive models.

Published as a conference paper at ICLR 2025
TEASERGEN: GENERATING TEASERS FOR LONG DOCUMENTARIES
Weihan Xu!
Paul Pu Liang²
Haven Kim
Julian McAuley²
Taylor berg-Kirkpatrick'
Hao-Wen Dong
Duke University Massachusettts Institute of Technology
University of California San Diego *University of Michigan
bechannxx@dukkeee. edu, ppliangemit.edu (hak@04,jmcaule) tberg)@ucsd. edu, hudong@umi ch. edu
ABSTRACT
Teasers are effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling the input videos, while necessitating maintaining audiovisual alignments, managing scene changes and preserving factual accuracy for the output teasers. Due to the lack of publicly- available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration of the documentary using pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-viste vision models. For narration-video matching, we explore two approaches: a pretraining- baased model using pretrained contrastive language-vision models and deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining- based approach more effective at identifying relevant visual content than directly trained deep autoregressive models.
INTRODUCTION
Teasers are effective tool for promoting video contents such documentaries, movies, vlogs, commercials and educational videos. However, creating an effective teaser for long videos possesses unique challenges: first, it requires modeling and understanding long-range multimodal data streams of video, audio and narrations; second, necessitates maintaining the text-visual correspondence between the teaser narrations and visuals; third, it needs managing smooth scene transitions beyond ctume-by-frame audiovisual matching; finally, it entails preserving the factual accuracy in the gener- ated teaser narrations and the accompanying visuals. These challenges together create an exciting yet underexplored ground toward long-range multimodal modeling.
Progress in teaser and trailer generation has been hindered due the lack of a publicly-available dataset. Existing work on movie trailer generation (Huang et al., 2020; Soldan et al., 2021; Chi al., 2024) relies on either private datasets or datasets without paired data, creating a barrier for the community to reproduce and follow up their research. In this paper, we present new documentary dataset with 1,269 high-quality documentaries paired with their teasers. The proposed DocumentaryNer dataset features various modalities such video, speech, music, sound effects, narrations and tags. With the proposed dataset, we explore generating teasers for long documentaries.
In this work, we adopt narration-centered approach for documentary teaser generation. Given long documentary, we first generate the teaser narration from the transcribed narration, and then select the most relevant visual content from the documentary to accompany the generated teaser narration. We leverage a pretrained large language model (LLM) with specially-designe prompts to generate teaser narration that has attracting story-like narratives and phoookng ending question. For

---

Published as a conference paper at ICLR 2025
Teaser
Body Content
sccne change
scene change
0:51
0.52
But will the Benobo's shart seeres with us hamans?
Zachogists Burbarn Fruth the university Musich Kinshssa [truneated]]
Title Maficine from jungle- Rainforest pharnacy| DW Documentary
User Ammolnted Taga: Education, minforest, modicine, psychology, spes, zsture Africa, monkey, Barbarn Fruth, Democratic Republic of the Congo, natural medicine, bosobos, tropical minforest, endangered specits, animals, phantacy phommscuticcal research, DW, Deutsche Welle, documentary
Narration with Timestamgs (in saz)
[2.6 15.92) Benobo Apes among closest living rolatives and Benobo's here in Affia''s 's Congo basin have valuable treasure. They know which plants bealing properties. Maybee
them.
(17.99 26.59] It's always interesting when one specifie individual within a group cats something differest. That could tum out to be some kind of medicinal plant.
[28855 - 44,91] Seientists in the Democratic Republic of Congo and Gemmany are hapeful the Apes have somethhing to teach us. And now by anbusadar.. It woakl could j'mj for joy. going put my lag.
be this
natura substance (6633-99.31 But will the Benobo's shore thoir sccress with humans?
Figure 1: In this paper, we aim to generate teasers, including both the visual and narration, from a long documentary. This figure shows sample from our DocumentaryNet dataset, consisting of 1,269 high-quality documentaries paired with their teasers.
narration-videe matching, we first explore framing the task as a constrained optimization problem by applying thresholding mechanism to maximize the total text-visual matching scores produced by pretrained language-vision model, while keeping the total length of the selected video clips within a desired range. Further, explore training deep sequential model that directly learns the mapping between the narration and the visual content, and we propose various decoding strategies to maximize the narration- virraaioo-visual alignment while maintaining cohesive output teaser. We compare propose models with four baseline models (Lin et al., 2023b; Narasimhan et al., 2021b; Son et al., 2024; He et al., 2023a) using objective evaluation metrics and a subjective survey. The experimental results show that the text-visual matching score-based approach is more effective for narration-video matching compared the directly trained sequential model. Due the complexity of the task of teaser generation, we focus on narration and visual generation in this paper as an initial step toward the ultimate goal of generating teasers with risually-aligned sound effects and background music. Our contributions can be summarized as follows:
We propose DocumentaryNet, a publicly-availabl dataset consisting of 1,269 high-quality documentaries paired with their teasers.
We propose TeaserGen, a narration-centerae teaser generation system that can effectively compress >30-min documentaries into <3-min teasers.
The DocumentaryNet dataset, along with all source code and demos, can be found on our website.
2 RELATED WORK
Teaser and trailer generation Prior work has studied the task of teaser or trailer generation. Chen et al. (2004) proposed to identify and extract semantically important story units and segments of an action movie through movie tempo analysis. Liu & Jiang (2015) learned to select key frames in semi-supervised manner with support vector machine. Smeaton et al. (2006) learned to locate the shots that could be contributors to the trailer of an action movie with a support vector machine. However, those three methods do not account for temporal consistency throughout those frames. Argaw et al. (2024a) proposed a unimodal bequence-to-sequene
model that selects the key shots from movies without audio outputs. CCANet (Wang al., 2020) presented a ranking network that uses a co-attention mechanism between movies and trailers to guide the creation of training pairs, with moments strongly correlated with the trailers expected receive higher scores than those that are less correlated. However, they do not attempt to generate audio narration or text captions to accompany the trailer with temporal correspondence, which is essential in documentary settings. Wang et al. (2024) proposed a human Al co-creation system to assist video podcasters in crafting
httpps:/wxx3. .i gitb. Se/TeassrrGe_Officiialll
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of related methods in trailer generation and multimodal summarization
Input modality
Outpat modality
Temporal multimodal
Type'
Frames
Video
Article
Narration
Query
Text
Video
Frames
information fusion
MMS (2017)
E
CLIP-IT (202
x
A2Summ (20236)
x
x
UniVTG (2023h)
x
x
<
x
LIVS (2024b)
X
x
TGT (2024a)
X
x
x
v
MSMO (2018)
V
X
V
MM-AVS (2021)
X
VideoXum (2023a)
x
X
Ours
x
<
-E-EExtractive; A-Abstractive
Achieved by dense video captioning
leasers. In this work, we propose a two-stage system that can output narration and visual contents with temporal correspondence.
Multimodal summarization Multimodal summarization aims to create concise summaries by integrating information from multiple modalities. It can be categorized into several types based on how content is extracted and summarized. The first category involves directly extracting text and visual content from the main material. For instance, Argaw al. (2024b) generates video summaries by leveraging both text and video inputs. They utilize large language models (LLMs) to extract key sentences from transcribed texts, which are then paired with time-aligned video segments to create pseudo-ground truth summaries. A2Summ (He et al., 2023b) produces extractive summaries with a unified multimodal transformer- based model to predict key sentences and its time aligned video segments. Similarly, MMS (Li et al., 2017) generates textual summaries from set of documents, images, audios, and videos related to a specific topic. Another category of multimodal summarization focuses on obtaining textual information through dense video captioning. For example, Lin al. (2023a) constructs the VideoXum dataset based on the video captioning dataset ActivityNet (Krishna al., 2017) and introduces an end-to-end cross-modal video summarization model, VTSUM-BLIP to achieve video-to-video summarization, video-to-text summarization, and combined video-to-video and text summarization. However, they only input visual contents and generate text summary based on the encoded visual features. A third category involves converting abstractive summaries into extractive labels before identifying key sentences and frames. For instance, MM-AVS (Fu et al., 2021) introduces Jump- Atte-nttentoon mechanism align features between text and video by first converting abstractive summaries into extractive labels, then extracting key sentences and frames.
our work, generate teasers from long documentaries using both narration and visual content. Unlike traditional multimodal summarization, teaser generation requires handling longer input sequences, higher compression rates, and ensuring the narration remains engaging and story-driven
Cross-modal alignment Recent work in multimodal alignment includes discrete alignment and continuous alignment (Liang et al., 2023). Contrastive learning (Cao et al., 2017; Huang et al., 2017; Grave et al., 2018) is a common approach in discrete local alignment. Optimal transport-based approaches (Villani, 2009) support global alignment between discrete elements across modalities. For continuous alignment, dynamic time warping (Kruskal, 1983) can be used to segment and align multi-view time series data. this work, we explore using a pretrained contrastive vision-language model as well as directly learning the cross-modal mapping between the narrations and visuals.
3 DATASET
We present DocumentaryNet, collection of 1, 1,269 high-quality documentaries from three reputable sources: DW Documentary, Public Broadcasting Service (PBS) and National Geographic. We download the videos and metadata from YouTube, where the metadata include video title, duration, and user-ane annetaandt tags. Each documentary in our dataset includes short teaser at the beginning. which allows to acquire teaser documentaryy pairs through splitting the documentary into its teaser and main content, as shown in Figure 1. We will refer to the main documentary section as the body content. To find the boundary between each teaser and its body content, we recruit three annotators
3

---

Published as a conference paper at ICLR 2025
Table 2:
Comparison of relevant datasets for video summarization and movie trailer generation. Our dataset is the first publicly available dataset with music and sound effects modality.
Modality
Mean duration (min)
Compression rate (%)
Publicly
Samples
Visual
Text
Speech
Music&SFX
Main
Teaser
available
SumMe (2014)
2.4
0.73
30.4
TVSum (2015)
50
3.9
1.17
27.9
CNN (2021)
X
203
2,1
0.12
5.7
Daily Mail (2021)
<
v
1,970
1.4
0.05
3.6
BLISS (2023b)
v
v
X
13,303
0.17
(2024)
LIVS-T
v
X
1,200
12.2
VideoXum (2023a)
v
X
14,001
13.6
MovieNet (2020)
<
1,100
116
1.9
(2024g)
23,604
DecumentaryNot (ours)
1,269
31.3
1.3
to mark the start of the body content in each documentary. The teasers and the body contents have an average length of 79 seconds and 31.3 minutes, respectively, resulting in a 4. 15% compression rate.
To extract narrations from the documentaries, we first separate audio track to three tracks: dialogue, music and sound effects and then use speech transcription model to transcribe narration from the dialogue track. We adopt a pretrained model (AudioSep. 2023) for sound separation. Since the main content audio is extensive, we split it into 60-second intervals before separating it into the three tracks. To ensure smooth transitions between segments, apply windowing function with window size of 5% based on the standard audio sampling rate 44, 100 Hz. To acquire frame-sentend pairs 2023) to transcribe the narration from the
for training, we adopt Whisper WWhhsppr-Timestamppee dialogues track and estimate the timestamps for each sentence. We also include silence detection information in this dataset for future research. More details of the dataset can be found in Appendix A.
4 METHOD
Documentaries often rely on the narration convey information, while the visual plays supplemen- tary role in strengthening the narrative. In this paper, we adopt narration-centered approach by first generating the teaser narration from the transeribed narration of the documentary (Section 4.1) and selecting the most relevant visual content to accompany the teaser narration (Sections 4.2 and 4.2.2).
4.1 GENERATING TEASER NARRATIONS BY PROMPTING LARGE LANGUAGE MODEL
We leverage a pretrained large language model to generate the teaser narration from the transcribed narration of the documentary. We then adopt text-to-speech model (TTS, 2023) to synthesize the generated script into audio narration. We note that extractive models often fail to construct a coherent narration due the frequent inserted interviews in documentaries, as exemplified in Appendix B.
To accommodate longer documentaries, we divide the transcribed narration of each main documentary into 10 segments, as the average transcript contains around 3,900 words. For each segment, we prompt GPT-4o (GPT-4o, 2024) to generate a one-sentence summary, resulting in a total of ten single-sentence summaries per documentary. Moreover, since documentary teasers often resemble a story and conclude with chouught-provokinn the ten summarized sentences into a story-like narration based on the ten summarized sentences' and, query the
question, we instruct GPT-4o (GPT-4o, 2024) rewrite
further, propose an ending question to end the teaser narration." For TeaserGen-Pa model separately, we retain the names characters rather than replacing them with pronouns the story-like narration. We will examine the effectiveness of this approach in Section 5.6.
We the following prompt: "Rewrite the paragraph into engaging story opening in 10 sentences less, keeping names and avoiding heing replaced by pronouns." We use the following prompt: "Given the title and the provided summary, formulate one thought-provoking and concise question that relate directly the summary."
4

---

Published as a conference paper at ICLR 2025
Optinaal chreshold
Pretrained vision-aaguagee model
Body content
Optnnnaall chrechodd
The buzzed with searly 10-Scoc-tall abandoned <plins satuee at one Egypt's largest trchzeological stoe.
Synthesized
Selected video
mirration
Destred leagth
Destred length
Figure 2: TeaserGen-PT selects video clips from the body content to accompany the generated narration by measuring the matching score between the input sentence and each video frame of the body content and applying thresholding mechanism to find the optimal video clips. We apply binary search algorithm to find the optimal threshold for each sentence in the narration scripts of the documentaries. We provide the details of the search algorithm in Appendix O.
4.2 SELECTING ACCOMPANYING VIDEO CLIPS FOR THE TEASER NARRATIONS
To accompany the GPT-summarized narration track with corresponding visual content, we formulate this task sentence-by by-nnceeeb---eeecee optimization problem that aims find the most relevant and important video clips in the body contents to pair with each sentence in the generated narration. Let V be an input video represented as a sequence of frames sampled a certain frame (1 frame per second in this work), i.e V = {x1, X2, X3...., xx, ,,, where is the total number of frames. Let S be the input narration represented the total number of sentences. We first synthesize each sentence 8, into waveform, and let т; be the duration of the synthesized speech of sentence 8, Our goal to find a number of video clips from the body content that can accompany the generated waveform of sentence s; and have a total duration of Tú. In the following two sections, we will introduce two approaches we propose to tackle this problem.
sequence of sentences, i.e., S {$1, ($1,S2... ,mm) where m is
4.2.1 INTERVAL-BASER APPROACH USING PRETRAINED LANGUAGE- VISSIIN MODELS
In the first approach, we frame this task as a constrained optimization problem, where we want to maximize the total text-visual correspondence between each sentence and the selected video clips while making the total length the selected video clips be the length the synthesized waveform of the sentence. To achieve this, video temporal grounding model (Lin et al., 2023b), which measures the importance of the video clips (as defined by whether to be included the text-visual correspondence. As shown in Figure 2, we first generate VTGHLS curve for each sentence throughout the main documentary, using a pretrained video temporal grounding model (Lin et al., 2023b). We then adopt binary search to find the threshold value that leads to the desired length. We will refer to this model as TeaserGen-PPT
the VTGHL score (VTGHLS) produced by pretrained
highlight in video highlight detection) well as
However, we notice this naive implementation leads to changes and repeatedly selected video clips across sentences. To alleviate the werly-ffragmented video issue, we only consider clips that are longer than 3 seconds to be selected in our algorithm. Further, to discourage the algorithm from selecting the same video clips across sentences, we only allow one second overlap with each of the video clips selected for the previous note that this issue is prominent in documentaries as a documentary focuses on the same topic and subject throughout the videos.
videos with frequent scene
sentences, We
4.2.2 LEARNING-BASHR CONTEXT- APPROACH USING DEEP SEQUENTIAL MODELS
In the second approach, we adopt a train sequential model to learn the mapping between the narration and the visual content. We frame this leqquence-to-sequeee learning where the input is sequence
5

---

Published as a conference paper at ICLR 2025
Transfoener
Postieal
Seonthhig mechasion
peftution prler
Dr &
DC
DP
De +
DP @
neightee hased Ipooil deoolaag boan saurcl
CLIP
CLIP
Bedly oxstest
Figure 3: An overview of the proposed TeaserGen-L.R model-((lef) deep sequential model that learns the direct mapping between narration and visual content, and (right) the proposed decoding strategy using beam search and a smoothing mechanism
of sentence embeddings at each frame: Exext = (clext exxt texs) img ..))). In this work, we adopt (e)mk, e2
), and the output is sequence
of the corresponding image embeddings: Eing (e) transformer model (Vaswani et al., 2023) to learn the mapping time, we find the frame from the body content that has the closet embedding to the generated image embedding the selected frame, as shown in Figure 3. We will refer to this model as TeaserGen-LR.
Feext - Fiig At inference
In practice, as the transformer operates at the frame level, multiple frames can share the same input sentence embeddings, and tend to generate similar output image embeddings within a sentence, which leads to overly-repeated scenes. To alleviate this issue, we leverage pretrained diffusion prior model (Ramesh et al., 2022; Prior, 2022) that models the mapping from the CLIP-text embedding space to the CLIIP-image embedding space. The diffusion prior model allows the model to generate more diverse output image embeddings, and, further, it helps reduce the modality gap between the CLIP image and text embedding spaces (Liang et al., 2022).
At inference time, we first explore decoding the output image embeddings by finding their nearest neighbors in the body content in the CLIP-image embedding space. In practice, however, we find that this naive approach leads to highly repetitive outputs. This is partly due the nature of documentaries, which focus on the same topic and subject throughout the video, and thus frames at different parts of the documentaries may be semantically similar and have close CLIP -mage embeddings.
To alleviate the overly-repetitive output issue across different sentences, we propose to add an additional regularization term to the decoding algorithm and adopt beam search to find the optimal (imi)) be the generated image embeddings from the model. Our goal is to find a sequence of frames in the body content that have the image embeddings E+ing- = (e]ma ....,
sequence of frames. Let Kiin = (e]me,
....,,eii (imng) that maximize the following score function:
((EimR, EAiMR) = Ei d (o!mes ,c, -.iim))
- jij different sentences (c +img simg
Eis
(1)
where d(.,,) is the distance metric (cosine similarity in this work) and À E R adjusts the strength of the regularization term. The regularization term in Equation (1) encourage the algorithm find diverse frames from the body content as the match to the raw output image embeddings. We apply beam search with a beam size Kj to find the optimal decoded sequence. For computational efficiency, we consider only the top K2 frames in the body content that are closest to the output embedding as candidates for selection. from selecting the same frames within one sentence, we also apply smoothing mechanism that replaces & repeatedly selected frames (xi,... } with nearby consecutive frames centered around x;: [ ( X 1B 11
this work, we use K₁ = 5, K₂ = 10. Finally, to prevent the model
5
EXPERIMENTS AND RESULTS
5.1 IMPLEMENTATION DETAILS
Following Argaw et al. (2024a;b), we extract frames from the videos at a rate of one frame per second (1 fps). For TeaserGen-PT we use -VTP-/VTBB/3 (Radford et al., 2021) for extracting visual and sentence embeddings, chosen to match the embedding size of the pretrained video temporal grounding model. For TeaserGen-L.R we use VIT-L//1 (Radford al., 2021) for embedding extraction,

---

Published as a conference paper at ICLR 2025
aligning with the embedding size of the pretrained diffusion prior. The backbone transformer of the diffusion prior consists of 12 blocks with 12 attention heads. For TeaserGen-LLR we utilize the L2 distance between the ground truth image embeddings and the generated ones as the loss function. The batch size is set to 16, and optimize using the Adam optimizer (Kingma & Ba, 2017) with learning rate of 1e-4. We train the proposed models for 15 epochs and select the best model on the validation set. During evaluation, we track the decoded frame numbers in the body content and use 5 512-dimensional CLIP embeddings for fair comparison. We evaluate our model on test set of 49 documentaries. We include dataset split details in Appendix K. All experiments are conducted on an NVIDIA RTX A6000 GPU.
transformer layers with a hidden dimension of 768, and
5.2 BASELINE MODELS
We consider six baseline models:* (Results for the two video summarization models (He et al., 2023a; Son et al., 2024) can be found in Appendix L..
This baseline model selects clips based on the duration of the generated speech. The model randomly selects M clips from the main content, each corresponding to the duration of the synthesized speech for sentences in S {s1, $2. ... ,Sm)
Random selection:
CLIP-NN: This baseline CLIP nearest neighbor (CLIP-NN) model selects key frames by finding frames in main documentary that are closet the input sentences in CLIP embedding space.
CLIP-IT (Narasimhan et al., 2021b): This baseline model generates a teaser by selecting key frames based on any input query. Since CLIP-IT cannot process long video effectively, divide the original video into 10 sub-clips and pair each sub-clip with a sentence from the GPT- generated script. For each paired sub-clip and sentence, suppose the duration of the synthesized speech of sentence is Tr, we extract top т; key frames that align with the assigned sentence. Finally, we concatenate all the selected frames in sequence construct a teaser.
UniVTG (Lin et al., 2023b): This baseline model selects key shots with user-defined keywords or queries to construct teaser. Since UniVTG cannot take input videos over 10 minutes, we adopt the same approach as in CLIP-IT to construct a teaser.
5.3 OBJECTIVE EVALUATION METRICS
To evaluate the performance of our proposed methods, we adopt the following evaluation metrics:".
Retrieval-based metrics Following Hesham et al. (2018); Argaw et al. (2024a), we calculate the F1 score by comparing the frames selected the generated teaser to those of the ground truth. While the Fl score provides us an estimate on the overlap between generated teaser and the ground truth, they should not be treated as the only standards as teaser generation is inherently a generative task.5
Repetitiveness To measure the repetitiveness of generated teaser, we propose a new metric:
REP = Number of unique frames
Number of frames
The repetitiveness score computed on the ground truth is 7.86%. suggesting that there are some repeated frames in the ground truth teasers."
Scene change rate (SCR) To measure the degree of fragmentation, we propose the SCR metric:
SCR Number consecutive frames within same scene
(3)
Number of frames
SCR measures the frequency of scene changes of the teasers. The estimated SCR of the ground truth teasers is 27.6% by manual inspections on 10 video samples.
CLIPScore To evaluate the alignment between the narration and the visual content, we compute the CLIPScore (Hessel al., 2021). ). Let ex img and ex be the text and image embeddings at frame and cos oos_sim(-) be the cosine similarity. CLIPScore is defined as
CLIPScore 2.5 1 Σx max (cos_sim sim ( img eeext) ..0)
(ex
(4)
"We intended compare model Argaw al. (2024a). However, we cannot repraduce their results as trained on a private dataset, and their code and pretrained models are not available. 'Due to space concern, we provide details of the calculation in Appendix C. We also include objective evaluation generated narration scripts in Appendix I
7

---

Published as a conference paper at ICLR 2025
Table 3: Objective evaluation results with L.MM-generated narrations.
Query
Decoding
(R)
(9)
(%)
CLIPScore
VTGHLS
Baseline models
Random
Random
1.67
0.56
0.75
CLIP-NN (2023b)
Narration
Greedy
x
0.11
92.73
8.29
0.69
0.79
Rank
CLIP-it (2021b)
Narration
Rank
1.24
99.39
0.56
0.61
Pretrahning-based models
TeaserGen-PI
Title
Thresbolding
1,85
0
13.16
0.56
1,02
TeaserGen
Thresholding
1.07
PPucr.GenPP..CIII
Narration
Threshold
1,31
27.23
24.10
0.58
0.74
Learning-baid models TeaserGen-1.R
Narration
Greedy
x
1.56
31.97
27.18
0.58
0.74
TeaserGen-LR
Narration
Greedy
1,38
26.83
35.48
0.62
0.78
TeaserGen-L.R TeaserGen-LR
Narration
Ream seurch
x
1.88
24.16
41.97
0.58
0,74
Narration
Beam Search
19.39
0.77
Ground truth
100
77.86
27.6
0.58
0.64
vTGHLS Proposed by Lin al. (2023b), VTGHLS measures the importance of the frame (as defined by whether to be included as a highlight in video highlight detection) in addition to the text-visual correspondence. We estimate a VTGHLS 0.64 for the ground truth teasers.
5.4 OBJECTIVE EVALUATION RESULTS
As shown in Table 3, the proposed TeaserGen-PT model that uses the title the query outperforms both baseline models as indicated by the higher Fl score and a closer scene change rate that the ground truth. Similarly, Lea,,rGen-LR, when using beam search, achieves higher Fl score than than the baseline models and a closer scene change rate to that of the ground truth. In contrast, the two baseline models, i.e., UniVTG (Lin et al., 2023b) and CLIP-it (Narasimhan et al., 2021b), both lead to a scene change rate that is three to four times greater than that of the ground truth. Further, while the CLIP-NN baseline achieves the highest CLIPScore, it results in high repetitiveness, partly because the nearest neighbor search can easily lead to repeatedly selected video clips. Moreover, TeaserGen-Le equipped with the diffusion prior model achieves us the second highest CLIPScore. demonstrating the effectiveness of diffusion prior in bridging the gap between the CLIP image and text embedding spaces. We also observe that transformer-batt models (TeaserGien-L.R) tends to result in higher scene change rate than the pretraining-bart models (TeaserGGen-PTT)
For TeaserGen-PP models, while it achieves a higher CLIPScore when using narration as the query, it achieves higher Fl score when using title as the query. This is possibly because a documentary often revolves around specific topic or character, which usually captured by the title, whereas narrations provide more detailed but denser descriptions. For TeaserGen-LR models, in terms of F1 score, Lea,erGen-LR, with beam search and diffusion prior, outperforms other transformer based models. We find that decoding with the beam search method proposed in Section 4.2.2 can effectively reduce repetitiveness by discouraging the selection of clips from different scenes.
5.5 SUBJECTIVE EVALUATION
To further measure the quality of our generated scripts and teasers, we conduct subjective test. We randomly select 10 documentaries from the test dataset and split them into 2 versions, with each version containing demos for documentaries. We then assess the coherence, alignment, engagingness, and realness (see Appendix D for the questions we ask in the survey) of the generated teasers on a Likert scale from to Additionally, we ask participants to compare the effectiveness of two approaches: summarizing each chunk of content directly versus using carefully designed GPT prompts, also on a Likert scale from 5. This comparison focuses on how both methods perform in terms of consistency, informativeness, and engagingness. We recruit 20 participants for the subjective test, with 11 working on version A and 9 working on version B. Among all participants, 14 out of 20 have video editing experience.
We report in Table 4, the mean values from subjective tests, along with the 95% confidence intervals. Human evaluations further confirm that TeaserGen-PT surpasses baseline models with higher scores
8

---

Published as a conference paper at ICLR 2025
Table 4: Subjective evaluation result with L.MM-generated narrations.
Model
Query
Decoding
Coherencet
Alignment
Engagingnesrt
Realnesst
UniVTG (2023h)
Title
Rank Rank
261 # 0.50
2.62 + 0.47
2.67 # 0.57
2.66 = 0.54
CLIP-it (2021b)
Narration
2.61 + 0:46
- 0.4
2.57 + 0:46
2.51 0.46
TeaserGen-P7 PT -enLLL
Threshold
3.14 +:
2.84 0.57
281 + 0.49
2.94 0.50
Narration
Greedy
2.90 + 0:45
2,88
+ 0.48
2.71 0.42
2.71 0.44
TeaserGen-IR
Narration
Beam search
2.84 + 0.46
2.69 + 0.51
2.71 4 0.42
2.64 = 0.41
Table 5: Subjective evaluation results of the LLM- narrations
Narration
Organizationt
Infonmativenesst
Engagingnesst
Naive summarization
3.58 + 0.57
3.72 + 0.47
3.60 + 0.56
Finely-tuned scripts
3.88 0.44
3.82 0.54
3,70 = 0.46
in terms of coherence, alignment, engagingness, and realness. The low coherence of baseline model in human study further proves that the baseline model is overly-fragmented, While TeaserGen-LA with beam search decoding has higher Fl score and CLIPScore compared to TeaserGen-PT using the title as the query in objective metrics, subjective evaluations reveal that it performs lower in terms of coherence, alignment, engagingness, and realness. This discrepancy is likely due to its higher scene change rate, which reflects more fragmented sequences and quicker transitions between clips. We encourage the readers to watch the video samples on our demo website."
5.6 ABLATION STUDY
Experiment changing the matching score function In this experiment, compare using CLIPScore (Radford et al., 2021) versus VTGHLS (Lin et al., 2023b) as the matching metric in Section 4.2.1 to examine the effects of changing the matching score function. Unlike CLIPSSore,, VTGHLS consider both the importance of a frame and the text-visual correspondence. The goal of this experiment is examine whether incorporating the importance of a frame is beneficial for video-narration matching. As shown in Table 3, we find that the model that uses VTGHLS as the matching score function outperforms that uses CLIPScore instead, resulting in higher F1 score, lower repetitiveness and lower scene change rate, closer to that of the ground truth. This highlights the benefits of using matching score that takes into account the importance of a frame
Effectiveness of the proposed prompting approach for teaser narration generation In this experiment, we examine the teaser generation approach proposed in Section 4.1. We conduct a subjective listening test to compare the teaser narrations generated by naive summarization prompt and those generated by our proposed prompting approach in terms of organization, informativeness and engagingness (see Appendix E for the questions we ask in the survey). We report in Table 5, the mean values from subjective tests, along with the 95% confidence intervals. The results indicate that finely-tuned prompts outperform naive summarization methods across all three dimensions. This indicates story-like conversion and ending questions can make narrations scripts closer to human expectations. We provide in Appendix B examples of the extractive summary, ground truth narration, LLM with non-finctuned prompts and LLM with finetuned prompts.
Effects of the smoothing mechanism and the diffusion prior model In this experiment, we ex- amine the effectiveness of the smoothing mechanism and the diffusion prior on the teaser generation approach proposed in Section 4.2.2. Table shows that applying smoothing mechanism can signifi- cantly decrease the repetitiveness as well as increase the Fl score, demonstrating the effectiveness of the smoothing mechanism proposed in Section 4.2.2. Further, Table 6 shows that applying diffusion prior leads to lower repetitiveness, higher language-vision correspondence (as indicated by higher CLIP and VTGHL.S). The decreased repetitiveness possibly result from the sampling process in diffusion prior, which is a generative model that generate different image embeddings for the same input text embeddings. The increased CLIPScore possibly come from the fact that diffusion prior can bridge the gap between textual embeddings and visual embeddings. Comparing the two results of TeaserGen-LP using beam search decoding, we also find applying diffusion prior can result in a higher scene change rate, partially because diffusion prior would further encourage scene changes.
In addition to the above ablation study, we present additional experimental results exploring the impact of X, inference using ground truth narration, and the effects of minimum clip length and

---

Published as a conference paper at ICLR 2025
Table 6: Effects of the smoothing mechanism and diffusion prior for TeaserGen-L.
Decoding
Smoothing
(Syt
REP (%)
SCR (%)
Cl.IPScore
VTGHLS
Greedy
0.53
81.32
27.38
0.59
0.74
Groedy
1.56
31.97
27.18
0.58
0.74
Greedy
1.38
26.83
35.47
0.62
0.78
Beam search
24.16
41.97
0.58
Beam Search
1,88
19.39
46.56
0.63
0.77
overlapping tolerance in TeaserGen-PT, as detailed in Appendices F, H and N. Qualitative examples are also available on our demo page.
DISCUSSION AND LIMITATIONS
Alignment between objective and subjective evaluation results From Tables 3 and 4, find that the scene change rate metric aligns well with the coherence and engagingness scores in our subjective evaluation, where lower scene change rate is preferable. Several survey participants also pointed out that too frequent scene changes negatively affect the viewer experience. Further, we notice that higher CLIPScore does not always lead to higher alignment score in human evaluation. This is possibly because CLIPScore is calculated on the frame-level, while humans perceive narration-video correspondence the clip-level. This necessitates clip-level narration-video correspondence metric for objective evaluation in future work, which is not trivial as it inherently involves scene change detection and clip-level video understanding.
Limitations and future work Finally, we want to point several notable limitations of our work. First, our proposed model leverage several assumptions that might not hold for other media. For example, we assume that scene change always happens when we move from one sentence to next in the narration. addition, the our proposed two-stage approach inherently assume that narration plays more significant role than visual content, which is a reasonable assumption for many documentaries but might not work for more visual-centered media such as vlogs and silent movies. Second, the proposed narration-vides matching approach cannot accurately match interview scenes commonly seen documentaries as the pretrained language-vision model cannot associate names with their corresponding faces, which would require separate video understanding module. Third, teaser narration generation a creative process. While we have proposed several prompting strategies for generating better teaser narrations, the proposed method still falls short in terms of artistic quality and creativeness compared to scripts created by professionals. Last but not least, we only consider documentary teaser generation in this paper due to the scarcity of public datasets for other media. However, to examine the generalizability of our proposed models, we apply our proposed models to other media in a zero-shot setting, including old movies and educational videos. We provide some qualitative examples of the generated teasers on the demo website and objective evaluations on the teasers created in zero-shot scenario in Appendix M. For future work, we would like to further scale up the dataset and further explore end-to-end approaches for teaser generation. Moreover, would also like to explore generating music and sound effects to accompany the generated teasers.
7 CONCLUSION
We have presented new documentary dataset, DocumentaryNet, that consists of 1,269 documentary- teaser pairs with multimodal data streams of video, speech, music, sound effects, narrations and tags. With DocumentaryNet, we have proposed the TeaserGen system for generating teasers for long documentaries. We adopt a narration-centered approach and approach teaser generation by first generating the narration using pretrained LLM and then selecting accompanying video clips from the main content. We have explored two approaches for narration-video matching: TeaserGen-Pi is based on pretrained contrastive language-vision model and a thresholding mechanism, whereas TeaserGen-LR a learning baanin model that directly models the mapping between the narrations and visuals. Through objective and subjective evaluation, the proposed system against several baseline models. We hope our work pave pathway towards long-range multimodal modeling by exploring this new task of documentary teaser generation.
have demonstrated the effectiveness of
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our proposed models do not create new narrations and visuals from scratch; instead, they focus on summarizing, reorganizing, and rewriting existing content. the narration generation phase, approach does not involve training the model with copyrighted materials. We utilize GPT-4 to generate the teaser narration by summarizing and rewriting the original narration. the visual matching phase, for TeaserGen-PP, we employ a contrastive language-visiee model to select video clips from the main documentary, rather than deploying a generative model. For TeaserGen-LLA we have developed generative model that generates CLIP-image embeddings from the narrations. We note that TeaserGen-LR inherently learn the underlying distributions of DocumentaryNet in the CLIP embedding space, but these CLIP-image embeddings need be further decoded into video frames. In this work, we use nearest neighbor based algorithm to decode the generated CLIP-image embeddings by finding the closest video frames in the original documentary, which does not involve learning the underlying distribution of DecumentaryNet.
Moreover, we note that our system may produce inaccurate teasers where faces are paired with the wrong names or where the generated narrations are paired with irrelevant visuals. Further investigation is needed to avoid unintended misinformation or hallucination of the proposed system. Despite these challenges, generating educational content previews and enhancing video accessibility. We hope our work contribute to the long-term goal of enriching human experience and augmenting human creativity through thoughtful use of Al technology. For copyright concerns, we will not distribute the raw videos from our datasets, but will provide access publicly available YouTube links and data preprocessing scripts.
envision practical and significant applications of our model such as
REPRODUCIBILITY STATEMENT
For reproducibility, we will release all the source code. Due to copyright concerns, we will release only the YouTube links to the videos, along with the metadata and annotations.
ACKNOWLEDGEMENT
We would like to thank Ruoyu Wang and Hongyi Yin for helping with the data annotation process. This work was partially supported by the NSF under grants 2146151 and 2200333.
REFERENCES
Dawit Mureja Argaw, Mattia Soldan, Alejandro Pardo, Chen Zhao, Fabian Caba Heilbron, Joon Son Chung, and Bernard Ghanem. Towards automated movie trailer generation, 2024a. URL https: wtt.rii..or/abs//24///. 03477.
Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanich Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. Scaling up video summarization pretraining with large language models, 2024b. URL
03398.
AudioSep. Cdx submission, 2023.
URL https://ggttaabaaccoo...ooom
cdx-submi dox-submissions.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Transitive hashing network for heterogeneous multimedia retrieval. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, pp. 81-87. AAAI Press, 2017.
Hsuan-Wal Chen, Jin-Hau Kuo, Wei-Ta Chu, and Ja-Ling Wu. Action movies segmentation and summarization based on tempo analysis. In Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval, MIR '04, pp. 251-258, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581139403, doi: 10. 011144/126711.1266111 URL https: /tte::///oooieeeeee org/10 httgg/...11/111111 1145/1926711, 1026752.

---

Published as a conference paper at ICLR 2025
Xiaowei Chi, Yatian Wang, Aosong Cheng, Pengjun Fang, Zeyue Tian, Yingqing He, Zhaoyang Liu, Xingqun Qi, Jiahao Pan, Rongyu Zhang, Mengfei Li, Ruibin Yuan, Yanbing Jiang, Wei Xue, Wenhan Luo, Qifeng Chen, Shanghang Zhang, Qifeng Liu, and Yike Guo. Mmtrail: A multimodal trailer video dataset with language and music descriptions, 2024. URL https:
//arxiv.org/aaaeaee ab2//407.242..... 20962.
Xiyan Fu, Jun Wang. and Zhenglu Yang. MM-AVS: A full-scale dataset for multi-modal sum- marization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 5922-5926, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v w .naacl-main.477 https://aclanthoologyy 2at.aannnoooogyoor/222222 org/ mmainn
URL
GPT-4o. Gpt-4o, 2024. URL
Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised alignment of embeddings with wasserstein procrustes, 2018. URL URL.ittps://rrxv.orr///
11222.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In European Conference Computer Vision, 2014. URL https://api.
semanticscholar org /rgg/CorpsII:::2 2111099.
Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivastava, and Zhaowen Wang. Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023a.
He, Jun Wang. Jielin Qiu, Trung Bui, Abhinav Shrivastava, and Zhaowen Wang. Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Mohammad Hesham, Bishoy Hani, Nour Fouad, and Eslam Amer. Smart trailer: Automatic gen- eration of movie trailer using only subtitles. In 2018 First International Workshop on Deep and Representation Learning (IWDRL), pp. 26-30, 2018. doi: 10. 10.1 1099/WDRL.22188835522
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. URL https://arxii org/abs/27 04. 08718.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. CoRR, abs/2007.10937,, 2020. URL https://arvv...rr//aba.a 10937.
Xin Huang, Yuxin Peng, and Mingkuan Yuan. Cross-modal common representation learning by hybrid transfer network, 2017. URL https:///rxivv http:///wwwvo.oo///////ei/e aa0/a///.... 0ttps/1766.00153.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv httg/bbs/142....
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos, 2017. URL
Joseph B. Kruskal. An overview of sequence comparison: Time warps, string edits, and macro- molecules. SIAM Rev. 25(2):201-237, April 1983. ISSN 0036-1445, doi: 13.1137/1020045 URL https://doi. orgo 10. 1137/1025045.
Haoran Li. Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. Multi-modal summarization for asynchronous collection of text, image, audio and video. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1092-1102. Copenhagen. Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653//./D11-1111 URL htttp:://aclanthologg.oooo org// D17-1114.
12

---

Published as a conference paper at ICLR 2025
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. 2023. URL https://arxxv.orrt abs/2209. 034330.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. In NeurlPS, 2022.
Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. Videoxum: Cross-modal visual and textural summarization of videos. IEEE Transactions on Multimedia, 2023a.
Kevin Qinghong Lin, Pengchuan Zhang. Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding, 2023b.
Xingchen Liu and Jianming Jiang. Semi-supervised learning towards computerized generation of movie trailers. 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 2990-2995. IEEE Press, 2015, doi: 10.1109/5MC2201555500 URL htLLttts://ooiirrg/// 1109/ 2015.520
Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language-guided video sum- marization. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 13988-14000 Curran Associates, Inc., 2021a. URL file/7503cfacd 1051000006060000111111111111-000
neurips. httpppscccppp..pppppppppppppp wwwes//appe///0///// pdf.
Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language- video summa- rization. CORR, abs/2107.0065 2021b. URL https: httpaaaaiiv..ggb//////aea 00650..
Diffusion Prior. diffusion prior, 2022.
URL
Jielin Qiu, Jiacheng Zhu William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Ding Zhao, Bo Li, and Lijuan Wang. Mmsum: A dataset for multimodal summarization and thumbnail generation of videos. 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https: //arxiv. http//ab//21100.999 00020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv preprint arXhv:2204.06115, 2022.
Audio Slicer. Audio slicer, 2022. URL /ttptt//////gtttoo..c/o////////p httpc//openomp//aaiooiieeeeee
com/
Alan F. Smeaton, Bart Lehane, Noel E. O'Connor, Conor Brady, and Gary Craig. Automatically selecting shots for action movie trailers. In Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, MIR '06, pp. 23 231-238, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595934952. doi: 10. 1181144/1777777 URL https:
//doi. www.orgg//// / 1111/1/1/////1.... / 117877799
Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba Heilbron, Chen Zhao, Silvio Giancola, and Bernard Ghanem. MAD: A scalable dataset for language grounding in videos from movie audio descriptions. CoRR, abs/21 2021. URL httpp::/aaaxi.ooo//aaaaaaa/// 00431.
Jaewon Son, Jachun Park, and Kwangsu Kim. Csta: Can-based spatiotemporal attention for video summarization, 2024. URL
http://aaxxxvv..gaa//aa/ /0000 905.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5179-5177, 2015. URL htl.httpss//aae http://ppp..smmnnncccc..lo. htt.://aw.rrrggoooosso///////11. 767566335
13

---

Published as a conference paper at ICLR 2025
TTS. Text-to-sper speech guide, 2023. URL https://platform.. http://aaaaoooooopaa...ooom ctt.:aaa...mmm/ccsssuiuieeeeeee
text-to- to- speesh.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need, 2023. URL https//////xiv..oaaaaabaa
03762.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Lezi Wang, Dong Liu, Rohit Puri, and Dimitris N. Metaxas. Learning trailer moments in full-length movies, 2020. URL https://arxiv. www.:orr/bbbb/......
Sitong Wang, Zheng Ning, Anh Truong, Mira Dontcheva, Dingzeyu Li, and Lydia Chilton. Podreels: Human-ai co-creation of video podcast teasers, 2024. URL httpppss//axxvvoog/////// 2311. 2311.05867. 05867.
Whisper-Timeetamped whisper timestamped, 2023, URL https://gittubb.omm/iiii
whisper-timestampee timestamped.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. MSMO: Multimodal summarization with multimodal output. In Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 4154-4164, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D188 1/488-4488. // //aclanthology.. wwwpthooogy.org//D.0 www.oo/-o 1448.
URL https:
A DETAILS OF THE DOCUMENTARYNET DATASET
Three-stream Sound Separation We extract the audio track of the documentary using the ffmpeg library. We then use a pretrained three-stream audio separation model (AudioSep, 2023) to separate each audio track into three stems: dialogue, music and sound effects. Due the length of the audio track of the documentary, we split each documentary into 60-second chunks and then run the separation model chunk by chunk. Finally, we smooth the transitions with window size equal 5% of the audio sampling rate (44,100 Hz in this work). While we do not use the music and sound effect tracks in this work, we include them in the DocumentaryNet dataset we believe they would be helpful for future research teaser generation.
Silence Detection we use a silence detector (Slicer, 2022) to detect periods of silence within each separated audio track. We randomly select 20 samples to select the optimal threshold value for classifying segments as noise or sound. binary label assigned to each segment, with 'I' indicating sound and 'O" indicating silence. The final thresholds used are -40 dB for the music track, -25 dB for the dialogue track, and -40 dB for the sound effects track.
B COMPARISON OF THE GENERATED NARRATIONS
We compare original narration, LLM-generated teaser narrations, LLM-generated teaser narrations with finetuned prompts in Tables 11 and 12. We leverage T5 (Text-to-Text Transfer Transformer) model to generate extractive summary on transcribed narration.
C DETAILS OF THE EVALUATION METRICS
Ground truth matching In this work, we only consider those non-fully black video clips that can be extracted from body contents. Therefore, we remove those without match. In order to find those without match, randomly pick 15 documentaries with teaser and body contents frame pairs. Considering the low FPS, there might be some shift when we extract frames. Therefore, due to frame shift, allow higher tolerance in distance. We calculate the lowest 20 % L, distances between each teaser frame and each body contents frame. The selected lowest distance threshold is 88.92. To find the matched interval, first extract CLIP feature with CLIP((L-14). We select top 20
14

---

Published as a conference paper at ICLR 2025
images with the highest cosine similarity to find those close in semantic meanings. Then calculate pixel-by-pixel L2 distance between teaser frame and those 20 images. We pick the one with the lowest distance and smaller than the selected threshold. In order to remove dark frames, randomly select frames from the body content of 10 documentaries and calculate the average brightness of their raw pixels. We define the darkest frames those in the lowest 5% of brightness.
Repetitiveness Since we are not able to find some of the frame in body contents due to frame shift, we caleulate the lower bound of repetitiveness by considering all non matched frames as different frames. The lower bound is 0.0786.
The upper bound is calculated by considering all consecutive non matched frames as new clip as those consecutive non matched frames might contain more than one clip in practice. We also randomly pick 10 documentaries teaser and calculate number of video clips per teaser. The average number of clips per teaser is around 27.6 %.
Scene Change Rate
VTGHLS For each sentence in the ground truth teaser, we first generate a score function using Lin et al. (2023b). This function measures frame importance (defined by its likelihood of being included as highlight video highlight detection) along with text-visual correspondence. Next, we calculate the VTGHLS for each matched frame. By averaging VTGHLS values over all matched frames of all documentaries in DocumentaryNet, we obtain an estimated VTGHIS of 0.64.
The estimated VTGHLS value of 0.64 for the ground truth, as reported in Table 3, is included solely for comparison purposes. It is not used at any stage in our training or inference pipeline.
D SUBJECTIVE EVALUATION QUESTIONS FOR TEASER GENERATION
We ask the following questions our subjective evaluation survey to evaluate our proposed models for teaser generation, described in Section 5.5 and reported in Table 4.
Coherence: "To what extent do you feel that the sample maintains coherence and a smooth flow, ensuring that each segment transitions logically and the overall experience feels seamless?"
"To what extend do you feel that the narration and a video match and work well together, making the overall presentation clear and easy to follow?"
Correspondence:
Engagingness: "To what extent do you feel that the sample captures your interest and keep you engaged throughout?"
Realness: "How well do you feel this sample meets your expectations as a teaser for a doeu- mentary in general?"
E SUBJECTIVE EVALUATION QUESTIONS FOR TEASER NARRATION GENERATION
We ask the following questions in our subjective evaluation survey to evaluate our proposed approach for generating teaser narration, as described in Section 4.1 and reported in Table
Organization: To what extent do you feel that the narration present a well well-organized and consistent storyline?
To what extent do you feel that the narration introduces the main characters and conflicts clearly and concisely?
Informativeness:
Engagingness: To what extend do you feel the narration keeps you engaged and builds curiosity about what happens next?
F EXPERIMENT ON INFERENCE WITH GROUND TRUTH NARRATIONS
Although we cannot have ground truth narration when making teasers reality, we conduct an ablation study to compare model behavior real narrations and machine generated narrations. We did not apply the diffusion prior model for TeaserGen-LP in this experiment. We report objective evaluation results in Table and subjective evaluation results in Table 8. As shown in Table
15

---

Published as a conference paper at ICLR 2025
Table 7: Objective evaluation results for the experiment inference with ground truth narrations
Backbone
Query
Decoding
Smoothing
(%)
REP (9)
SCR (98)
CLIPScore
VTGHLS
Baseline models
Randem
Random
165
0.54
0.64
UniVTG (2023b)
Tille
Rank
1.67
0
88.50
0.55
0.79
CLIP-it (2021b)
Narration
Rank
1.20
99.24
0.54
0.44
Pretraining-bant models
TeserGGenFF
Title
Threshold
1.46
1.03
TeaserGin-PP PT
Narration
Thresheld Threshold
1.20
6.12
24.42
0.55
1.40
TeaserGen
Narration
Learning-basn based models
TeaserGen LR
Narration
Greedy
0.88
68.87
34.19
0.57
0.71
TeaserGen-L.R
Narration
Greedy
1.76
11.91
34.07
0.56
0.71
TeaverGen
Narration
search
2.20
Ground truth
100
77.86
27.6
0.58
0.64
Table 8: Subjective evaluation results for the experiment inference with ground truth narrations
Model
Query
Decoding
Coherencef
Correspondencef
Engagingnesst
Realnesst
UniVTG (2023b)
Title
Rank
2.64 = 0.53
2.72 + 0.60
2.76 + 0.53
2.64 1: 0.57
TeaserGen-PT
Title
Threshold
3.45 1 0.47
2.96 1 0.66
2.91 1 0.58
2.97.1 0.63
TeaserGen-LA
Narration
NN
2.93 0.58
2.69 + 0.62
2.74 = 0.62
2.61 0.67
TeaserGen- PT achieves the highest score in terms of coherence, correspondence and engagingness in human evaluation.
G EXPERIMENT ON FINETUNING THE PRETRAINED VIDEO TEMPORAL GROUNDING MODEL
We prepare highlight detection dataset by constructing query and highlight frame pairs with times- tamped narration. We have around 10,333 samples to finetune the pretrained UniVTG. However, Table 9 shows the finetuned model result in worse performance due to the relatively small size of the finetuning dataset comparing with the dataset in pretraining.
H EFFECTS OF THE REGULARIZATION TERM
We show in Table 10 the results when using different values of À in Equation (1). As shown in Table 10, we find that a higher À can reduce the repetitiveness and lead to a higher F1 score but the difference is not significant.
Table 9: Objective evaluation results for the finetuning experiment with TeaserGen-PT
Query
Decoding
Finetune
Fl (%)
REP (%)
SCR (%)
CLIPScore
VTGHLS
Title
Thresholding
1.85
0
13.16
0.56
1.02
Narration
Thresholding
1.07
21.38
22.58
0.58
1.45
Narration
Thresholding
1.23
41.39
24.01
0.56
1.05
16

---

Published as a conference paper at ICLR 2025
Table 10: Effect of the regularization term À in Equation (1)
Decoding
X
F1 (%6y
REP
SCR
CLIPScore
VTGHLS
Beam search
1.88
24.16
41.97
0.58
0.74
Beam search
10
1.91
24.01
41.78
0.58
0.74
Beam search
100
1.91
24.06
41.73
0.58
0.74
I OBJECTIVE EVALUATION ON LLM-GENERATED NARRATION
To assess the quality of generated narration scripts, we report common text summarization evalu- ation metries: FI of ROUGE in Table 13 by comparing LLM-Generated narration with audio transcription of ground truth teaser. The referred text from the audio transcription of ground truth teaser where dialogues are also contained. We also compare the TeaserGen performance with fully extraction from body contents narration and naive summarization. Note that our generated narrations intentionally exclude dialogue and include an engaging ending question, which inherently reduces overlap and leads to lower ROUGE scores. To address this limitation, we also conduct subjective evaluation metries better assess the quality of our generated narrations, as ROUGE alone does not fully capture the nuances of narration quality in this context.
Table 13: Objective evaluation results of the LLM-generated narrations
Evaluation
Naive Summarization
Fully Extraction
Ours
ROUGE-1
0.14
0.15
ROUGE-2
0.02
0.02
0.02
ROUGE-I 0.12
0.14
0.12
J COMPARISON WITH EXISTING TEASER/TRAILER-BON CONTENTS DATASET
In Table 14, compare DocumentaryNet with two existing teaser/trailer-bout body contents dataset. DocumentaryNet the first publicly available teaser/trailer-body contents dataset.
Table 14: Comparison with Existing Teaser/Trailer-oot Body Contents Dataset
Dataset
Number of Samples
Total Lengths(hours)
Public Available
MovieNet (2020)
1100
2126.7
TGT (2024a)
22,604
X
Ours
1,269
655.7
K DATASET SPLIT DETAILS
Table 15, we show the details of dataset splitting.
Table 15: Dataset Split Details
Split
Number of Samples
Total Hours
Train
1026
514
Validation
57
32
Test
17

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of LLMM-generated teaser narrations Exammle I
Model
Narration
Extractive
archneologists searching for clues reveal more about the pharaoh queen. hetshepsood's 's temple the key the of her dynasty, says di zbignicw szybranski. a doctor of ranski team, the temple was buried and badly damaged and little was known about owner. ram ram-headed sphinx has been excavated at one of the largest anchaeological sites ancient Tggpt. it's thought that leddge Sheppset started the greatest display of Sphinces known la ancient Egypliians the necropolis around 100 ancient tombs 120 miles from Laxor Aswan, cnn's richard quest excavating what could he one the sphinxes of Harshopsut. quest finds a small cryo-like body in the quarry gopel. in southem egypt, near the tomb of king henry ihn al al-queda he was huried there in 1586 he. doctors of ransky is investigating the paintings she left on the walls head chef's 's at temple. the imagery holds clues her life and reveals family power struggle with her nephew, who also her stepson. Tutmost III. despite being female. Hadshepsaat is often depicted as man. but hidden in the hicrogly phic lext, Colleen linds evidence of her real gender. at the Necrogolis cartonage. it was skillfully how they made the eyes of an adult mummy buried outside the tomh. the mask was meant to help the afterlife from gold, this one is more beautiful than come once again. american archacologist finds dramatic evidence of one individual's life and death. the vagina the whole here, still very distended, so we know within 24 hours of giving birth, that she died. but the female pharaoh had chefsuit wanted to be more than just equal. doctors of Ransky are working to restore temple of Hachepsut, ancient cartiquakes and landslides have damaged the temple's 's upper terrace, which is feet higher than the valley of the kings. the bones and delicate cartonage have been moved into lab in aswan, savan. hundreds of sphinxes remain at carmax temple site. ten foot high ahandaned stalue is the largest they' seen the quarry. jožn and maria Ward suy. the couple are looking for black inclusions within the stone itself. there are no records whatsoever of any unfinished statue that intact like this, sphinxes from sosilla were shipped 100 miles down the nile to camse temple. the evidence suggests they came from the city. not so far from where they now -ohnn w. medonaid jr.
Aswan, team linds an ancient face musk made
LLM-
Zbigniew Sxy Stybranski. are excavating and restoring the unique temple of the revolutionary female pharach Hetshepsood in I Laxor, Egypt, to uncover more about her enigmatic reign and contributions, while other teams investigate her quarries to understand her extensive bailding projects John and Maria are excavating an abandoned statue one of Egypt's largest archacological sites, while Martina Bartanova's 'ss team in Aswan discovers ancient buman remains that could halt their exploration of an unopened tomb. Martimu and her leam are excavating a site w where they discover a child's remains and miniature sphinx, while dealing with the challenges of preserving delicate artifacts and documenting their findings for the Egyptian government. The paragraph describes various archncological efforts in Egypt, including the discovery of ministure model likely carved for conffict with her stepson, the challenges faced by team uncovering artifacts near her temple, and the work of Yale professors using digital technology to study ancicnt inscriptions. The couple explores Karnak Temple to investigate inscriptions revealing Hadshepsut's s portrayal al as male pharaoh, while archacologists at Dra' abu Ei Naga study the roles of women ancient Egyplian society through the excavation of a tomb filled with fragmented human remains. The team is meticulously organizing human remains, including mummified rrgans and detached body parts, while making significant discoveries such as young woman who likely died in childbirth and an ancient face mask, all under the pressure of impending strong winds. roles mortal dangers faced by women in ancient socicty.
Archacologists, led
generated
the investigation of Queen Hatshepsut's
power and her
American archacologist Susanne Onstein is uncovering while also exploring the monumental building campnigns of female pharaoh Hatshepsut, who used colossal obelisks as both religious devotion and propaganda to immortalize her reign. The excavation team has uncarthed ancient pottery fragments and evidence of temples buill Hatshepsul and her stepion Tutmose III, revealing historical power play and ongoing restoration elforts, while another team in Aswan uncovers Middle Kingdom burials, all amidst the personal reflections and quarry
challenges faced by the archacologists. John and Maria uncarthed nearly 10-foot-tall abandoned sphinx and are now investigating at the Camax temple to find matching sphinx, focusing on identifying black inclusions in the sandstone to determine its origin. John and Maria's discovery sphinxes from Sosilla, transported Camac Temple. alongside Dr. Ransky's 's excavation and restoration efforts, has unearthed significant artifacts and revived the legacy of the remarkable Pharach Queen Hanshepsur.
LLM- generated (luned)
Under the scorching Egyptian sun, Dr. Zbigniew Szybranski led a team of archncologists in Luxor, meticulously excavating the unique temple of revolutionary female pharach Hetshepsood. The buzzed with excitement John Maria unearthed a nearly 10 fuot tall abundoned sphins statue al one of Egype's 's largest archaeological sites. Meanwhile, in Aswan, could
Martina Bartanova's 's team stumbled upon ancient human remains The discovery of a child's remains and miniature sphinx added layers of mystery to their quest. At Kamak Temple. John and Maria delved into inscriptions revealing Hetshopsood's as portrayal as male pharaoh. while Yale professors used digital technology study ancient texts. In Dra abu El Naga, another team meticulously organized fragmented human remains, uncovering stories of women in ancicnt Egyptian society. American archacologist Susanne Onstein explored the monumental building campaigns of Hetshepsood, whose colossal obelisks stoed as testaments and evidence of iemples huilt by Hetshepsood and her stepson Tutmose III, the archaeologists faced personal reflections and challenges. The winds of history whispered through the sands, as each discovery brought them closer to unraveling the enigmatic legacy of Pharaoh Queen Hetshepsood. How do the discoverics and restoration efforts at Hatshopsut's 's temple and other archaeological sites in Egypt contribute to our understanding of her reign and the breader role of women in ancient Egypiian society?
their exploration an unopened tomb.
reign. Amidst pottery fragments
Ground trath
Fgypt, richest source of archaeological trea sures on the planet, beneath this desert landscape, why secrets this ancient civilization? Wow! You can see why Pharaoh's 's chosen place, now, for full scason of excavations, our cameras have unprecedented sccess, follow teams on the front line of archacology. I'm writing so fast because I'm subsided! It's an entrance, we can see entrance. Revealing varied secrets. I have just been told that they have found something Making discnveries, they could rewrite ancient history. This time, new secrets about one of Egynt's greatest rulers, the Pharnoh Queen, Hatshepout, doctors of Ranski discovers very treasures that her magnificent temple had reached to be remembered for millions of years. The Darnels uncovered how sbe formed a mysterious double identity to scize power. For my beloved daughter, not son, and John and Maria, honor rare and intriguing statue.
18

---

Published as a conference paper at ICLR 2025
Table 12: Comparison of L.LM-generated teaser
Example II
Model
Narration
Extractive
the mountains here were a bartleground when US and NATO troops were still in the country. outside bombs and airstrikes have left their mark. damage lines the road her large garden. Rochanak Wardak takes us down memary lane. gynecologist dr. rochanak's 's patients are still suffering the deprivation of that war. her shoulders aren't I neck hart. makes no difference if cat or not, with the Taliban, things are good now, it's peaceful. there was a lot of war and fighting. every pregnant woman wants know the sex of her baby. yes, boy power of the family. the drive through Kabul will reduce more drug users and street children than we' 'ee seen hefore, more than half of the population is hungry; food prices are soaring. women sit in front of hakeries, begging for leaf bread. but it's enough fill our stomachs. frida ghitis: the hunger crisis is on the government of the Taliban. she says they want gender segregated society. but girls dare to dream of being Afghunistan. she says she has lost since the Taliban returned to power. aisha khan: if were president, I would form an all-femall government. she says women would be allowed to choese any job they wanted, study and live freely. her country is tird. tired of betrayal. you are songless and voiceless. can's 's aisha nusri meets with teen who wants join the Taliban Afghans fear Taliban nale, but he and his friends brush aside such concerns, she adds. they say talk of explosions, targeted killings and arbitrary arrests nothing more than Western propaganda, the traditional pottery of Istarlif darling when u.s. troops were still in the country. the pressure on visitors has affected the potters of the mountain village. do
to the same. aisha and sister are trying find their place
spirit
the Islamic Emirate.. she says many
so foreign
which now closed tourists and monopoly by taliban. no one buys their wares, they may not be they have always done. "we'r afraid of new wars and more hardship," " says Abdul manan. he says the traditional life has survived much, but now feels under threat again.
LLM- generated
We traveled west from Kabul to the war-torn province of Shidan-Wardal to meet Dr. Rochanak Wardak, a former politician and Taliban supporter, who reminisces about efforts the country, halling its development and education, leading Dr. Rochamak, gynecologist who now runs small home clinic, witness parients still suffering from the war's deprivation. With the cessation fighting, travel has hecome safer, and while the region now peaceful under the Taliban, there remains cultural emphasis on the importance of having boys for family power and security. with some boys she helped deliver now becoming fighters. Dr. Rosha Nakh, who works for the Taliban oversccing female and maternal health, hopes for a change their stance women's activities, but faces the harsh reality of a male-dominated regime and widesprcad poverty in Kabul. The Taliban blame the hunger crisis on international sanctions and aid cuts, asserting their right to enforce their visson of Islamic rule and demanding autonomy. while policies focus more on controlling women's is lives than addressing ecomomic issues. The Taliban's 's ban on secondary education for girls has left young women like Aisha, who once aspired to be doctors and pursued careers. foeling they struggle to fiod their place in the new Afghanistan. Aisha foels like prisener and blames the West for abandoning Afghanistan to the Taliban. which has led her to withdraw from public life and dream of an all- all-female government to ensure women's 's freedom and opportunitics. The men would stay home experience the restrictive life of Afghan women, while Isatullah, a former farmer turned Taliban member, dismisses fears of Taliban rule as Western propaganda and expreases his support for the Sharia based Islamic Emiratte. impacted the raaditioaal pottery husiness in the mountain village of Istarlif, leaving craftsmen like Abdul Manan and his son Ramish struggling to sustain their family craft. Abdul Manan, a traditional artist, expresses fear for his livelihood and safety amidkt oogoing conflicts and deserted marketplaces. highlighting the persistent threat their peaceful existence despite not taking sides in wars.
a new Afghanistan governed by parliament. War devastated
and hopeless
Talibun's restrictions on visitors, especially womeen. have severely
LLM- generated
The journey west from Kabul to the war-torn province of Maidan-Wardak was fraught with memories of conflict and hope. Dr. Rochanak Wardak, former politician and Taliban supporter, welcomed us into her small home clinic. Once a gynesologist with dreams of parliamentary Afghanistan, Dr. Rochanak now witnesses the lingering scars of war in her patients' eyes. The cessation of lighting has brought a fragile peace, but the cultural emphasis on having boys for family power remains. Some of the hoys Dr. Rochamak helped deliver Nakh, working for the Talihan in female and maternal health, dreams of a shift the regime's stance on women's 's activities. However, the malc-dominated reality and widespread poverty Kabul blame international sanctions for the hunger crisis, focusing more on controllin women's 's lives than addressing economic issues. Young women like Aisha, once aspiring doctors, now feel lost and hopeless, their dreams crushed by the Taliban's ' ban secondary education for girls. In the mountain village of Istarlif, traditional potter Abdul Manan and his Ramish struggle to sustain their family craft amidsi severe restrictions on visitors. Abdul Manan's 's fear for his livelihood and safety undersoores the persistent threat their peaceful existence, poignant reminder of the ongoing conflicts that continue shape Afghanistan's 's future. How does Dr. Rochanak Wardak reconcile her support for the Taliban with her aspirations for a progrossive Afghanistan, espocially in light of the Taliban's restrictive policies on women's 's education and activitics?
(tuned)
fighters, stark reminder of the cycle of coaflict. Dr. Rosha
long shadow over her hopes. The Taliban
Ground truth
Taliban fighters rale streets, armed with weapons once used by their enemies. We see them over Kabul. The Taliban's s return to power has changed millions of lives, espccially the lives of Afghan women. I feel like prisoner. nothing is in my own hands anymore, before I could do whatever I wanted. Of course, I'm very much happy. You know, the head is no war. is their idcology. but what's it like living today's
We are living like other people the world. The Taliban are back Afghanistan?
19

---

Published as a conference paper at ICLR 2025
L RESULTS FOR THE BASELINE VIDEO SUMMARIZATION MODELS
L.1 BASELINE MODELS
We include the following two video summarization models as our baseline models and report results in Table 16. We observe that TeaserGen achieves relatively higher FI score and also provides narrations with higher audio-visual correspondence that are effective in engaging the audience.
CSTA (Son al., 2024): This baseline model proposes CNN-based Spatio Temporal Attention (CSTA) to summarize videos. Note that CSTA cannot process videos with more than 5,000 frames, so we chunked the long videos and recombined them after processing.
A2Summ(He et al., 2023a):This baseline model proposes a unified multimodal transformer- based model to extract the most important information from different modalities. A2Summ cannot handle the long documentaries in our dataset, divide them into 10 chunks and recombine them afterward.
Table 16: Objective evaluation results with LLM-generated narrations.
Model
Query
Decoding
DP
FI (%)
REP (%)
SCR (%)
CLIPScore
VTGHLS
Baseline models
Random
Random
1.67
4.05
7.81
0.56
0.75
CIIPPNNN
Narration
Greedy
0.11
92.73
8.29
0.69
0.79
UniVTG (2023b)
Title
Rank
1.82
89.68
0.58
1.01
CLIP-it (2021b)
Narration
Rank
1.24
0.61
CSTA(2024) A2Summ(2023g)
X
1.78
5.00
0.77
95.23
0.54
0.61
evetraining-based models
TeaserGeu-PTT
Title
Thresholding Thresbolding
1.85
0.56
1,02
TeaserÜenn-PT
Narration
1.07
21.38
22.58
0.58
1.45
TeaserGen.
Narration
Threshold
Learning-base based models
TeaserGien L.R
Narration
Greedy
1,56
31,97
27.18
0.58
0.74
TenserGen-I R
Narration
Greedy
1.38
26.83
35.48
0.62
0.78
TeaseGGen-LR
Narration
Beam scarch
1.88
0.74
TeaserGen-I.R
Narration
Ream Search
1.88
19.39
46.56
0.63
0.77
Ground truth
100
77.86
27.6
0.58
0.64
M ZERO-SHOT EVALUATION
We present the objective evaluation results in Table 17. We observe that TeaserGen achieves relatively higher Fl score and also provides narrations with audio-visual correspondence that are effective in engaging the audience. Zero-shot samples can be found on our demo page
Table 17: Objective evaluation on Zero-Shot Examples
Model
(E)
SCR (8)
CLIPScore
VTGHLS
CSTA(2024)
4.90
A2Summ(20233)
0
96.25
0.54
0.57
TeaserGen.
TeaserGen-LR
20.3
48.69
0.62
0.87
N ABLATION STUDY ON MINIMUM CLIP LENGTH AND OVERLAPPING TOLERANCE
During our experiments, we aimed avoid having clip lengths that were too short, as this would make the output fragmented. However, setting the minimum clip length too high would exclude too many clips, thereby sacrificing audio-visual correspondence. We conducted an ablation study on different combinations, as shown in the Table 18 for Pe with narrttion being the query. We observe that selecting as our minimum clip length results in a scene change rate that is mostly close to the ground truth and provides higher audio-visual correspondence.
20

---

Published as a conference paper at ICLR 2025
Table 18: Experiments Minimum Clip Length and Overlapping Tolerance
Clip Length
Overlapping
FI(S)
(9)
(S)
CLIPScore
VTGHLS
0.97
20.19
48. 12
0.57
1.55
0.98
3
U
0.96
19.96
22.49
0.57
1.45
5
1.14
21.01
15.33
0.57
1.44
5
1.08
26.32
13.44
0.57
1.44
10
1.16
26.82
8.38
0.57
1.41
O DETAILS OF SEARCH ALGORITHM
The goal the proposed searching algorithm to maximize total video-narration correspondence while maintaining the desired length. The desired length is defined as the duration of the synthesized speech for each sentence. To facilitate smooth transitions between concatenated video clips, we add a one-second buffer time between sentences ensure that the length of the audio track length is greater than or equal to that of the visual content. For each sentence in the documentary, we generate a score curve (with the x-axis representing video time and the y-axis representing the score). This curve reflects frame importance (defined by its likelihood of being included as highlight in video highlight detection) and text-visual correspondence the sentence. During the search process, the algorithm selects threshold such that the intersections of the score curve and the threshold determine the start and end times of video clips. These clips are chosen based on the condition that their scores exceed the threshold. Multiple clips can satisfy this condition. The search begins at the top of the score curve, lowering the threshold incrementally (by 0.001 steps) until the total length of the selected clips matches the duration of the generated audio waveform. This final threshold is considered optimal, as illustrated Figure 2. For each sentence in given documentary, we associate with a score curve. The sentence-level VTGHLS is calculated by averaging the VTGHLS values of all frames fall in that sentence whose VTGHLS exceeds the threshold in the corresponding score curve. Finally, compute the overall average VTGHLS across all sentences in all documentaries within the test set. To improve computational efficiency, we take advantage of the monotonic increase in the total length of selected video clips the threshold decreases. Instead of linearly decreasing the threshold, a binary search algorithm is applied to quickly identify the optimal threshold.
The estimated VTGHLS value of 0.64 for ground truth is used in Table is only for comparison purpose and it is not used anywhere in our training or inference pipeline.
21


Reviews:
[{"review_id": "Ey3Q6gteW4", "reviewer": "Reviewer_uVty", "summary": "This paper introduces TeaserGen, a method for generating teasers for long videos. To address the lack of suitable datasets, it presents the DocumentaryNet dataset, which contains 1,269 documentaries paired with their teasers. The dataset includes streams for video, speech, music, sound effects, and narrations. The proposed method is a two-stage system: first, it generates the teaser narration using a large language model (LLM), and then it uses vision-language models to select the most relevant visual content.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The newly introduced dataset could be valuable for the research community.\n- The task is interesting and meaningful.\n- The experiments are thorough, and the performance appears good.", "weaknesses": "- The system heavily relies on LLMs and vision-language models, which may lead to error accumulation. How can we evaluate whether the teaser narration generated by GPT is effective?\n- Some video summarization and highlight detection methods could also be applied to generate teasers, but the paper lacks a comparison with these approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jkQmn7Fzg", "reviewer": "Reviewer_339o", "summary": "The paper addresses the challenge of creating effective teasers for long videos, the authors introduce DocumentaryNet, a dataset comprising 1,269 documentaries paired with their teasers. The proposed system operates in two stages: \n\nTeaser Narration Generation: A pretrained large language model (LLM), is prompted to create engaging, story-like narratives with a thought-provoking ending question from the transcription.\nVisual Content Selection: For relevant video segments to be selected to match the narration using two methods: a pretrained contrastive language-vision model (TeaserGen-PT) and a deep learning-based sequential model (TeaserGen-LR) that aligns video frames to narration.\nTeaserGen-LR frames the narration-video matching as a sequence-to-sequence learning task, where it learns a direct mapping between the sequence of sentences in the teaser narration and the video frames. It uses transformer-based architecture with a diffusion prior to embed narration and visual sequences into a shared embedding space, enabling more nuanced and context-aware matching.\n\nThe authors used both objective metrics (like F1 score, CLIPScore, and scene change rate) and subjective user surveys that evaluate in terms of consistency, informativeness, and engagingness, to assess the quality of the teasers. They found that TeaserGen-PT with threshold-based selection often provided better coherence and alignment between video and narration, while TeaserGen-LR benefited from enhanced narration-video correspondence.", "questions": "1. The paper claims generalizability of proposed models, however the work is only shown qualitatively through some examples. It would be interesting to get some quantitative results for the same.\n\n2. Can we see the framework's results on how it adapts to other videos with complex visual elements that don’t directly correspond to narration, such as scenes with symbolic or artistic visuals or videos with limited narrative.\n\n3. Comparison with other existing video summarization models, on how the framework is against other models on this task.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes frameworks to generate teasers from documentary using audiovisual alignments and scene-changes.\n2. The paper demonstrates robust experiments and comparisons to baseline models.\n3. The authors use thorough evaluation on their dataset using both objective metrics (like F1 score and scene change rate) and subjective evaluations (coherence, engagingness) to validate their results.\n4. The paper has shown extensive ablation studies.", "weaknesses": "1. Limited dataset scale, both for training and testing. Though the dataset is domain specific, still the scale is limited with just 1.2k documentaries.\n2. Reliance on pretrained LLM, for teaser narration generation without any check for hallucinations or error compounding due to this step.\n3. The work is very domain specific, the framework’s reliance on pretrained language-vision models for narration-video alignment, while effective for documentaries, may struggle with complex visual elements that don’t directly correspond to narration, such as scenes with symbolic or artistic visuals or videos with limited narrative.\n4. Comparison with other existing video summarization models is not shown in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZRA3bg93yW", "reviewer": "Reviewer_VbpS", "summary": "- The paper presents TeaserGen, a two-stage system for creating teasers from long documentaries, addressing challenges like audiovisual alignment, smooth transitions, and factual accuracy. To support this, the authors developed DocumentaryNet, a dataset of 1,269 documentaries with teasers, including multimodal elements like video, narration, and sound effects. \n\n- TeaserGen first generates teaser narration from the documentary’s transcript using a large language model, creating an engaging summary. It then pairs visuals with narration through either a pre-trained contrastive language-vision model or a deep sequential model to match visuals accurately.\n\n- Results show that TeaserGen outperforms baseline models in maintaining coherence and alignment, offering a streamlined approach to automated teaser generation. DocumentaryNet and TeaserGen together provide valuable tools for advancing multimodal content modeling in documentary summarization.", "questions": "- Could you clarify the notation used in rows 210 and 211? Specifically, is  $S$  intended to represent a sequence of language tokens or audio waveforms? Consistent notation throughout the paper would enhance understanding and prevent confusion.\n\n- Have you considered evaluating alternative architectures or conducting ablation studies to assess the robustness of relying solely on the pretrained VTGHL model? Exploring different models could strengthen the validation of your approach.\n\n- What is the rationale behind setting the minimum clip length to three seconds and the overlap between clips to one second? Providing theoretical or empirical justification for these specific constraints would help in understanding their impact on the results.\n\n- Given that a frame rate of 1 FPS may be insufficient for capturing dynamic video content, have you experimented with higher frame rates? How does increasing the frame rate affect the model's ability to capture motion and improve prediction accuracy?\n\n- How does your model address the issue of repetitive scenes arising from multiple frames sharing identical sentence embeddings? Have you explored methods to incorporate fine-grained temporal information or annotations to enhance diversity and temporal coherence?\n\n- Could you elaborate on how the VTGHLS threshold of 0.64 was determined? Additionally, have you investigated how varying this threshold influences the results, perhaps through an ablation study?\n\n-  Given that the TeaserGen-LR model uses only three transformer layers, have you tested deeper architectures to see if they capture complex patterns more effectively? Would increasing the number of layers improve performance?\n\n- Have you considered using standardized quantitative evaluation metrics like ROUGE, BLEU, BERTScore, or perplexity for assessing the generated narration text? Including these metrics could enhance the reproducibility and comparability of your study.\n\n- Relying solely on L2 distance as the loss function might not fully capture perceptual similarities. Have you experimented with alternative loss functions, such as perceptual loss, SSIM loss and so on to potentially achieve more nuanced image generation?\n\n- Considering that the models were trained for only 15 epochs on a relatively small test set of 49 documentaries, have you explored training for more epochs or using a larger dataset? How might this affect model performance and generalizability?  To test the generalizability of your approach, have you considered evaluating the model on additional datasets beyond the 49 documentaries? How does the model perform on different genres or types of video content?\n\n- Could you provide details on the computational resources required by the diffusion prior model? Understanding the computational overhead would help assess the practicality and scalability of your approach in real-world applications.\n\n- Higher scene change rates may enhance visual diversity, have you evaluated their impact on the narrative coherence of the teasers? How do you balance diversity with maintaining a coherent and engaging storyline?\n\n- You employ different CLIP models (CLIP-ViT-B/32 and CLIP-ViT-L/14) for different components of your system. Have you considered the potential inconsistencies this might introduce? Would using the same CLIP model throughout improve the alignment between textual and visual data?\n\n-How does the frame extraction rate influence the model's performance in terms of capturing essential visual information? Have you analyzed the trade-offs between computational efficiency and the richness of visual features at different frame rates?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Tackles a unique problem in automated teaser generation for documentaries with TeaserGen, a creative, narration-centered two-stage approach that combines large language models with language-vision models for cohesive narration and visual alignment, showing effective and innovative use of existing technologies.\n\n- Provides solid empirical support with comparisons to baseline models across objective (e.g., F1 score, CLIPScore) and subjective metrics, as well as the introduction of DocumentaryNet, a multimodal dataset with documentary-teaser pairs that enriches resources available for this research area.\n\n- The work has significant potential impact by addressing a real-world gap in video summarization for documentary-style content, with applications in multimedia and educational fields, and establishes a foundation for further multimodal research, likely to stimulate new directions in long-form video modeling.", "weaknesses": "- The paper presents an innovative approach to video teaser generation using pretrained language-vision models, but several issues need to be addressed to enhance its clarity and robustness. In rows 210 and 211, there is a notation inconsistency where $S$ is defined as a sequence of language tokens, yet later each $S_i$ is referred to as a waveform (audio signal). This inconsistency creates confusion, and it's crucial for the notation to consistently represent either language tokens or audio waveforms throughout the paper to avoid misunderstandings.\n\n- In Section 4.2.1, the method relies heavily on a single pretrained VTGHS model without sufficient ablation studies or comparisons with alternative architectures, which weakens the validation of the approach. The constraints imposed such as a minimum clip length of three seconds and a one-second overlap between clips, appear arbitrary and lack theoretical or empirical justification, raising questions about their effectiveness and impact on the results. Additionally, using a frame rate of only one frame per second (1 FPS), as mentioned in rows 453 to 455, is inadequate for videos that change dynamically. This low frame rate hinders the model's ability to effectively capture motion and make accurate predictions.\n\n- In Section 4.2.2, the model extracts features at a low frame rate of 1 FPS, causing multiple frames to share identical sentence embeddings. This coarse temporal resolution fails to capture dynamic changes within the video, leading to overly repetitive scenes. The absence of fine-grained temporal annotations in the dataset prevents the model from effectively distinguishing and assigning unique embeddings to semantically similar frames occurring at different times. As a result, the approach struggles to maintain diversity and temporal coherence in the generated visual content.\n\n- Regarding threshold selection, in row 320, the paper states, `We estimate a VTGHLS of 0.64 for the ground truth teasers`, but there is insufficient explanation about the criteria used to select this threshold value. The paper does not analyze how varying this threshold affects the results. Since there are ablation studies on changing the matching score function, as mentioned in Section 5.6, it is necessary to explore this threshold value in more detail to understand its impact on the overall pipeline.\n\n- Table 4 may present an unfair comparison because the TeaserGen models utilize advanced decoding techniques like beam search, which can enhance performance, while the baseline models do not use these techniques. For a fair comparison that accurately reflects each model's true capabilities, all models should employ similar decoding methods.\n\n- A significant limitation in the methodology is the heavy reliance on subjective metrics without incorporating standardized quantitative measures, as mentioned in rows 468 to 470. This over-reliance weakens the study's reproducibility and generalizability. While subjective listening tests provide valuable insights, the absence of automatic evaluation metrics commonly used in natural language processing and computer vision (such as ROUGE, BLEU, BERTScore, or perplexity for the generated narration tex) makes it difficult to objectively compare the results with other approaches or validate the findings across different contexts.\n\n- The TeaserGen-LR model uses a limited architecture with only three transformer layers, which may not capture complex patterns as effectively as the diffusion prior's more extensive 12-block backbone. Relying solely on L2 distance as the loss function might not fully capture perceptual similarities, leading to less nuanced image generation. Additionally, training the model for only 15 epochs on a small test set of 49 documentaries raises concerns about underfitting and limits the generalizability of the results.\n\n- The study does not discuss the potential computational overhead introduced by incorporating the diffusion prior, which could affect the practicality and scalability of the approach. While higher scene change rates may increase visual diversity, they might also compromise the narrative coherence of the generated teasers. Addressing these issues would enhance the study's robustness and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyc3QcZCqg", "reviewer": "Reviewer_Hqts", "summary": "This paper presents TeaserGen, a two-stage system for generating promotional teasers for long documentaries. Leveraging a their proposed dataset, DocumentaryNet, the authors aim to generate teasers by first synthesizing teaser narrations from documentary transcripts using a large language model. They then use a language-vision model to select relevant visual content that aligns with the narration. In that process, to avoid repeat frames, they proposed some methods to alleviate. The study compares a pretraining-based model (TeaserGen-PT) and a deep sequential learning model (TeaserGen-LR) for narration-video alignment. Experimental results show some advantages of the pretraining-based approach over directly trained autoregressive models.", "questions": "1)Line229-230: You should detail this part about how to find the threshold and does it sensitive and what is the pretrained model here. especially in the supplementary material.\n\n2)Line267-268: Please illustate well about the diffusion prior and how it helps and also do you conduct the ablation study about this.\n\n3)Line693-706: You collect these media from three main sources, but in the appendix A that you explain about the dataset, I didn't see any discussion about the copyright for these videos.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1)The DocumentaryNet dataset of 1,269 documentary-teaser pairs fills a gap in the community by providing a publicly available resource for multimodal summarization and teaser generation, and the system shows potential for applications in media, advertising, and education for automated content promotion.\n\n2)The inclusion of various evaluation metrics, including scene change rate and repetitiveness, provides a more nuanced assessment of the teaser’s quality.", "weaknesses": "1)The approach predominantly relies on pretrained language-vision models, limiting novelty. This reliance raises questions about the model’s true capacity to generate creative outputs, as it functions more as an information retrieval system instead of a real generative system.\n\n2)The proposed model does not consider the alignment of audio cues like music or sound effects with visual elements, which limits its ability to produce emotionally engaging teasers, I've checked in your demo, there also exists this issue, and none of above baseline methods consider this aspect, and if you could consider this, that would be a great contribution in this domain. In this aspect, you get all frames by your frame-matching system driven by the text narration you got from llm, so the audio should also be different correspond to each frame in this proposed hypothesis. And by the way, this multi-stage method, means multi-stage information loss and also your pretrained model from general domain also have information loss, so I don't think this method work well for the task that you work with in this paper.\n\n3)The sentence-by-sentence matching approach does not effectively capture scene continuity, leading to potentially fragmented visual sequences that lack coherence even though you utilized the smoothing and regularisation.\n\n4)The experiments are mainly based on one proprietary dataset (other methods also didn't train on your dataset) and lack extensive ablation studies for elements such as the diffusion prior model and threshold sensitivity, which are critical to understanding the model’s flexibility and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)