Movie101: A New Movie Understanding Benchmark
Year: 2023

Abstract:
To help the visually impaired enjoy movies, automatic movie narrating systems are expected to narrate accurate, coherent, and role-aware plots when there are no speaking lines of actors. Existing works benchmark this challenge as a normal video captioning task via some simplifications, such as removing role names and evaluating narrations with ngram-based metrics, which makes it difficult for automatic systems to meet the needs of real application scenarios. To narrow this gap, we construct a large-scale Chinese movie benchmark, named Movie101. Closer to real scenarios, the Movie Clip Narrating (MCN) task in our benchmark asks models to generate role-aware narration paragraphs for complete movie clips where no actors are speaking. External knowledge, such as role information and movie genres, is also provided for better movie understanding. Besides, we propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation, which achieves the best correlation with human evaluation. Our benchmark also supports the Temporal Narration Grounding (TNG) task to investigate clip localization given text descriptions. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines. The dataset and codes are released at https://github.com/yuezih/Movie101.

heading: 1 Introduction
text: The estimated number of visually impaired people worldwide was about 285 million by 2020, according to reports (He et al., 2020). While regulations are in place to ensure increased access for these audiences to experience the culturally dominant movies and TV shows on popular media platforms, technologies that provide them with genuine experience are becoming increasingly important. Audio description (AD, also known as video description)
∗*Corresponding Author.
is a form of such technology intended for visually impaired audiences to experience the movie or TV show by hearing what is happening on-screen. However, producing movie narration scripts is not trivial, often requiring a professional writer to oversee the original movie. The high cost of narration generation (Lakritz and Salway, 2006) greatly hinders the production of movies with AD and thus limits the opportunities for visually impaired users to experience movies.
To address this issue, attempts have been carried out to automate AD production. Datasets of movies with ADs are constructed to support the research on automatic AD generation, including the MPII-MD dataset (Rohrbach et al., 2015) and M-VAD dataset (Torabi et al., 2015), with shotlevel ADs or scripts aligned to the visual contents of movie. Consequently, different solutions for automatic movie narrating have been proposed based on these datasets (Rohrbach et al., 2017).
However, existing benchmarks suffer from several limitations. Firstly, there is a gap between the designed tasks and the actual movie narration scenario. These tasks mainly focus on generating single-sentence narrations for shots of a few seconds. They can not support the generation of coherent narrations for longer plots, which is critical for the visually impaired to better understand the movie, and the timestamps of these shots are carefully annotated, which are difficult to obtain for new movies in real application. Meanwhile, these tasks treat the very distinctive movie narrating task as a normal video captioning task through some simplifications such as replacing role names with SOMEONE, resulting in the inability to connect roles to plots. Secondly, these benchmarks evaluate the generated narrations with ngram-based metrics, which can over-penalize a semantically correct but textually inconsistent narration, especially when there is only one reference available. In addition, these existing datasets are all in English. However,
4669
about one-fifth of the world’s population speaks Chinese as their mother tongue, of whom more than 17 million are visually impaired (Yu and Bu, 2021). Therefore, building a Chinese movie narration benchmark is necessary.
Intending to address the limitations of the existing narrating benchmarks, in this work, we propose a new benchmark with 101 Chinese movies for movie understanding, named Movie101. We collect the movies from the barrier-free channel on Xigua Video platform1, where normal movies are remastered with ADs. Through automatic process and manual correction, we obtain the ADs and actor lines from the raw videos. We crawl rich metainformation relevant to the movies as well. Finally, Movie101 contains 30,174 narration clips totaling 92 hours, with data samples as shown in Fig. 1. As our investigation shows that narrations mostly occur at those times when no actors are speaking (see Appendix A), to achieve realistic movie narrating, we propose the Movie Clip Narrating (MCN) task that requires a model to narrate where there are no lines. It brings a potential benefit for identifying where to narrate in an unlabeled new movie, since the timestamps of the actor lines are easily accessible2. Meanwhile, in order for the audience to accurately comprehend the role-related plots, concrete role names should be contained in the generated narration. For the MCN task, we reorganize the Movie101 dataset, merging the narration clips between two actor dialogues into a longer clip, to simulate real-scenario movie narrating. We thus obtain 14,109 long clips of variable length for narration generation. Moreover, to better evaluate the quality of model-generated narrations, we conduct
1https://www.ixigua.com/channel/barrier_free 2The timestamps of the lines can be obtained from the movie script or by automatic methods such as OCR and ASR.
human evaluations and design a new metric specific to movie narrating, namely Movie Narration Score (MNScore), which well aligns with human evaluation. In addition to the MCN task, our dataset also supports the Temporal Narration Grounding (TNG) task, which asks a model to locate target clips in the movie according to some text descriptions. For both tasks, we benchmark the performance of existing methods, and further propose our improved models by incorporating auxiliary external knowledge. In addition to MCN and TNG tasks, Movie101 can also potentially support other movie understanding tasks such as visual question answering and action recognition, etc.
The main contributions of this paper are as follows: 1) We propose a new benchmark for movie understanding, Movie101, with a large number of video-aligned text descriptions in Chinese. 2) We propose two primary tasks, MCN and TNG, and a new narrating evaluation metric MNScore, where MCN is more in line with the needs of actual movie narrating, while MNScore is more consistent with human evaluation. 3) We benchmark state-of-theart models and propose improved models enhanced by external knowledge for MCN and TNG, respectively. We expect our proposed Movie101 benchmark can inspire more explorations on narrating and understanding a whole movie.

heading: 2 Related Works
text: Datasets. Existing datasets to support the automatic narration generation task include MVAD (Torabi et al., 2015) and MPII-MD (Rohrbach et al., 2015), which are merged into LSMDC (Rohrbach et al., 2017). M-VAD, which is collected based on an automatic AD segmentation and alignment method, contains 47K videos from 92 DVDs,
with an average length of 6.2s, each with an aligned narration. MPII-MD contains 68K videos from 94 movies with an average duration of 3.9s, about half of which come with paired scripts and the other half with paired ADs. In addition to movies, TV shows are also good data sources for automatic narration generation. Lei et al. (2020) propose TV Show Caption (TVC), a variant of TV Show Retrieval (TVR). It contains 11K short videos averaging 9.1s in length, and 26K captions describing the visual content, dialogues, and subtitles. All the existing datasets are in English.
Video Captioning. As a classic vision and language task, the video captioning task requires a model to generate natural language descriptions for given videos. Solutions for normal video captioning go through stages from pre-designed templates (Kojima et al., 2002; Guadarrama et al., 2013) to sequence-to-sequence generation with deep neural networks (Pasunuru and Bansal, 2017). A challenging variant for this task is dense video captioning (Krishna et al., 2017), which requires the generation of multi-sentence descriptions for long multievent videos. The two-stage generation approach, which firstly performs proposal detection on the video and then generates descriptions for each proposal separately, has been the dominant approach (Krishna et al., 2017; Park et al., 2019; Rohrbach et al., 2014; Xiong et al., 2018). Recently, some works avoid event detection and generate paragraph descriptions directly based on the video, such as the one-stage paragraphing model (OVP) (Song et al., 2021), obtaining competitive performance compared to previous works, inspired by which we propose our knowledge-enhanced movie narrating model. Identity-aware video description that distinguishes different persons is more practical in real applications. Park et al. (2020) attempt to achieve role-aware movie narrating by distinguishing different people using labels such as PERSON1, PERSON2, etc. However, it fails to generate concrete role names and falls short in terms of practicality.
Temporal Sentence Grounding. The temporal sentence grounding (TSG) task aims to localize the moment in a video based on a natural language query (Gao et al., 2017). A two-step pipeline has been the mainstream approach, which first produces a large number of moment candidates via sliding windows, then ranks them with their similarity to the query sentence. The following works try to improve the grounding performance
by enhancing interaction between video and query modalities (Liu et al., 2021; Li et al., 2022) or introducing novel detection heads (Lei et al., 2021; Zhang et al., 2020a). Specifically, for interaction methods, Liu et al. (2021) adopt an Iterative Alignment Network (IA-Net) to iteratively interact interand intra-modal features within multiple steps. Li et al. (2022) explicitly decompose video and query into multiple structured hierarchies and learn finegrained semantic alignment among them. In this work, we propose to incorporate external knowledge based on the IA-Net model structure.

heading: 3 Dataset

heading: 3.1 Data Collection
text: Movie Acquisition. To the best of our knowledge, there are only a handful of platforms that provide accessible movies in Chinese. The barrierfree channel of Xigua Video is one such platform that provides over 100 accessible movies online, and new movies are still being released that can support further expansion of our dataset. From Xigua Video, we collect all 101 movies available to date and crawl as much meta information as possible for each movie, including title, introduction, genres, directors, actors, etc. We emphasize actors in particular, including actor names, role names, actor portraits, role rankings, and other information about important roles. We expect such information can benefit the movie narrating task and general movie understanding tasks. Narrations and Lines Extraction. As the movie lines and narrations are only available in the subtitle and audio format respectively from the platform, we therefore leverage OCR and automatic speech recognition (ASR) tools for transcription. For lines, we extract text from subtitles by open-source OCR toolkit PaddleOCR3 at 2.4 FPS, and manually remove the irrelevant subtitles from the beginning and the end of each movie. For narrations, we extract the audio track from the movie and utilize the ASR service provided by iFlyTek4, which detects the speech in the audio and transcribes it into text. In addition, the service supports identifying different speakers, which helps discriminate the narrator from the actors. However, the ASR service is not perfect, and its outputs contain errors such as wrong characters, unreasonable sentence breaking, and misidentification of narrations as movie
3https://github.com/PaddlePaddle/PaddleOCR 4https://www.xfyun.cn/doc/asr/lfasr/API.html
Table 1: Movie101 and other Movie Narrating and Temporal Sentence Grounding datasets. (* indicates statistics based on Chinese characters.)
Task Dataset Video num. Text num. Avg. video len. Avg. text len. Avg. actions Avg. role names
Narrating
M-VAD 47K 47K 6.2 sec. 10.8 - - MPII-MD 68K 68K 3.9 sec. 9.6 1.4 0.37
TVC 109K 262K 9.1 sec. 13.4 1.9 0.75
Movie101-N 14K 14K 20.4 sec. 80.7* 12.3 2.0
Grounding
Charades-STA 10K 16K 31 sec. 7.2 1.1 0 ActivityNet 20K 72K 118 sec. 13.5 2.1 0.02
TVR 22K 109K 76 sec. 13.4 1.9 0.75
Movie101-G 101 30K 6,144 sec. 47.3* 6.9 1.1
Swordsmen Buddy Young
Fantasy Costume
Detective Crime Action
Romance Comedy
dialogues, etc. Therefore, we recruit human annotators to further correct the ASR transcription errors and remove non-narration texts manually to improve the data quality. We also delete the irrelevant fragments at the beginning (e.g., movie synopsis, cast introductions) and the summary narration at the end. For coherency, we further organize the narration fragments at the clip level. We merge every two fragments if their temporal gap is less than 1 second. we also apply a paragraph-length threshold of 100 characters to limit over-merging to avoid excessively long clips. We take punctuation into account as well, for example, a period in Chinese is likely to mean the end of a narrative paragraph. Further detailed descriptions of data quality can be found in Appendix B. Movie101-N and Movie101-G. For real-life movie narrating, models are expected to narrate in the breaks between different actor dialogues.
0% 5% 10% 15% 20% 25%

heading: 4 12 20 28 36 44 52 60 68 76 84 92 100

heading: Clip Duration (seconds)
text: Thus, we reorganize Movie101 to fit this task format. Concretely, we first merge the independent lines in Movie101 into dialogues, where two lines with a temporal gap shorter than 5 seconds are considered to belong to one dialogue. Then, we merge all the narration clips between two adjacent dialogues into a long paragraph. In this way, we obtain Movie101-N with narration paragraphs separated by dialogues, which well simulates the practical narrating challenge. Meanwhile, with rich videotext pairs in Movie101, we create another variant dataset to support the temporal grounding tasks, named Movie101-G, where narrations are taken as queries and aligned videos serve as targets. For validation and testing, we carefully select 10 movies of different genres for each respectively.

heading: 3.2 Dataset Statistics
text: Movie Properties. Movie101 contains 101 movies, involving 41 genres (a movie can belong to up to 4 genres) and 645 roles in total. Fig. 2 shows the numbers of movies in the top 10 most popular genres, with comedy, romance, and action in the top 3. Clip Properties. Movie101 contains a total of
30,174 short narrated clips with an average duration of 11.0 seconds and an average length of 47.3 Chinese characters. From the narrations, we locate role names and action words with the metadata and the Chinese Part-Of-Speech (POS) tagging toolbox HanLP5, to detail role and action content in narrations. Fig. 3 shows the distribution of the number of role names and actions per single clip. The narrating variant dataset Movie101-N contains 14,109 long narration clips of an average length of 20.4 seconds and 80.7 characters. The comparison in Table 1 shows that Movie101-N contains much longer video clips and text descriptions than existing movie narrating datasets, while the length distribution in Fig. 4 indicates that the clip length varies a lot. Movie101-G contains 30,174 clips to be located from 101 movies. The average video length of 6,144 seconds also greatly exceeds existing TSG datasets.

heading: 4 Movie Clip Narrating

heading: 4.1 Task Description
text: In order to help the visually impaired keep up with the plot in the movie, we first propose a Movie Clip Narrating (MCN) task, which aims to generate a plot-related paragraph description given a clip in Movie101-N. Besides, the narration styles may vary across different genres of movies. The role portraits are important external knowledge for a model to accurately describe the subject of actions. Thus, we also provide this information in Movie101-N to support the MCN task.
5https://github.com/hankcs/HanLP

heading: 4.2 Proposed Method
text: For the MCN task, with multimodal inputs including video, movie genres, role names, and actor portraits, we propose a Transformer-based (Vaswani et al., 2017) model with an encoderdecoder framework, namely Role-pointed Movie Narrator (RMN), where the encoder mainly encodes video clips and the decoder generates narrations, as shown in Fig. 5 (a).
On the encoder side, taking into account the frame-level visual information, the video clip is embedded into a sequence of frame-level features. To emphasize the roles, we extract face features from each frame and concatenate them to the corresponding frame feature sequentially based on the confidence scores of face detection. With learnable genre embeddings, genres are also represented as a sequence of genre features. After video and genre representation, we apply a Transformer encoder to perform cross-encoding. Then, we follow the One-stage Video Paragraphing model (OVP) (Song et al., 2021) to use a dynamic memory bank to refine the video-part representations, which updates at each decoding step.
On the decoder side, in addition to the Transformer decoder, we enable the model to directly choose a complete role name from the movie cast according to context during token-by-token generation via a pointer network (Gu et al., 2016). At the decoding step t, with the decoder hidden state ht, we first calculate the token scores yvoct among normal vocabulary. Then we design a Role Selector module to get the name scores among external
role vocabulary. Concretely, with the decoder’s video-part attention distribution αt, we perform a weighted summation among video representations to get a context-filtered video feature. Then the role scores yrolet are computed with the contextfiltered video feature as query and portrait features as key. Finally, the prediction distribution at step t is calculated as follows:
yt = f([y voc t ;λy role t ]) (1)
where [; ] means concatenation, λ is a gate computed from ht, f() is the softmax function.

heading: 4.3 Evaluation
text: Existing movie narration benchmarks directly adopt ngram-based metrics including CIDEr, BLEU, and METEOR as in normal video captioning. However, there are pitfalls for these metrics, such as underestimating semantically correct but textually inconsistent phrases, which have been widely reported (Zhang et al., 2020b; Shi et al., 2022). For movie narrating, a movie clip can be narrated in multiple expressions, while there is only one reference. Thus, text matching is inadequate to measure the quality of a narration paragraph.
To better evaluate the generated narrations in the MCN task, we conduct a manual evaluation to investigate how humans assess different narrations. We randomly select 30 movie clips, each with 5 candidate narrations, of which 3 are derived from the predictions of different models and 2 are obtained by disturbing the ground truth narrations. Next, we recruit 10 annotators to individually rank the candidates for each video in terms of accuracy, informativeness, and textual quality. Accuracy defines how the narration accurately describes the
video, especially roles, actions, and objects; informativeness defines how richly the narration reveals the video content; textual quality is determined by the narration fluency and grammatical correctness.
With the human evaluation results, we investigate a wide range of objective metrics as follows: (1) State-of-the-art video captioning metrics based on deep neural networks including CLIPScore (Hessel et al., 2021), BERTScore (Zhang et al., 2020b) and EMScore (Shi et al., 2022), which are reported outperforming ngram-based metrics in video captioning evaluation; (2) Textual quality metrics including n-grams diversity(Shetty et al., 2017) (DIV) and causal language model perplexity (PPL); (3) F1 score of role name generation (RoleF1). For every two candidate narrations of a video, we use human ranking as a reference to determine whether these metrics correctly judge which of the two candidates is better or worse, and the accuracy is used for evaluating metrics’ correlation with human judgment. Finally, we settle on a new metric Movie Narration Score (MNScore) as follows:
mns = 1 · ems+ 4 · berts+ 1 · rf1
6 × 100 (2)
where mns, ems, berts and rf1 refer to MNScore, EMScore, BERTScore and RoleF1, respectively. As shown in Table 2, BERTScore outperforms ngram-based metrics in narration evaluation accuracy, while our new proposed MNScore achieves the best alignment with human evaluation. More details about the implementation of the candidate narrations and the above metrics are presented in Appendix C.

heading: 4.4 Experiments
text: Implementation Details. In our proposed method, models are trained with next-token language modeling by the maximum likelihood estimation (MLE) objective. For videos, we use CLIP (Radford et al., 2021) pre-trained on large-scale image-text pairs and MIL-NCE (Miech et al., 2020) pre-trained on HowTo100M videos (Miech et al., 2019) to extract frame-level CLIP and S3D features with dimensions of 512 and 1024, respectively, at 1 FPS, and further concatenate them. For faces in video frames and portraits, we use the Arcface model (Deng et al., 2019) pre-trained on MS1M (Guo et al., 2016) to extract face features. When there are insufficient faces detected within a frame, the
face feature extractor compensates by substituting the extracted features with zero-vectors. Results & Analysis. We choose Vanilla Transformer (Zhou et al., 2018) and state-of-the-art video paragraphing model OVP (Song et al., 2021) as the MCN baselines.
As shown in Table 3, RMN outperforms the baselines by a large margin, especially on RoleF1. This indicates that our model learns to generate role names from external knowledge with the help of the pointer network. To verify the contribution of the genre and face representations in our RMN model, we also perform an ablation study by progressively adding these representations as input. From the results, face features extracted from video frames bring significant gains in role awareness, which shows that using face features to bridge the video content and external actor portraits is beneficial for generating role-related narrations. Qualitative results can be found in Appendix D.

heading: 5 Temporal Narration Grounding

heading: 5.1 Task Description
text: To help people locate clips of interest during movie entertainment, an AI agent should be able to understand users’ intentions and locate the target clips. To achieve this goal, we propose the Temporal Narration Grounding (TNG) task. Given a clip narration as the query, TNG aims to predict the starting and ending time of the clip in the whole movie.

heading: 5.2 Proposed method
text: Existing temporal sentence grounding models can hardly handle an entire movie input with limited computational resources. Thus, we propose a twostage framework for the TNG task, with global shot retrieval to coarsely locate the target clip in the first
stage and local temporal grounding to finalize the precise timestamp of the target clip in the second stage, as shown in Fig. 5 (b). Global Shot Retrieval. To find the approximate location of the target, we treat it as a text-video retrieval subtask. We divide a movie into 20s-long shots, and the shot with the highest similarity to the text query will be used as the anchor for further grounding in the second stage. For training such a retrieval system, we construct a temporary dataset Movie101-GSR(temp). Concretely, after cutting the movie into shots, each shot and each annotated narration in Movie101 are judged with the temporal overlap whether they can be considered as an aligned video-text pair.6
We build the retrieval model by transferring a Chinese Vision-Language Pre-training (VLP) model ChineseCLIP (Yang et al., 2022) (CNCLIP) from image-text to video-text. Specifically, the shot frames are separately encoded as image features by the visual encoder of CNCLIP, and the final video feature is obtained by performing mean pooling over the CLS tokens of all frames. We then perform contrastive learning between the video and text features on Movie101-GSR(temp) to fine-tune the modified CNCLIP. Local Temporal Grounding. After obtaining the anchor shot in the first stage, we further lo-
6A shot and a narration with a temporal overlap larger than half of the duration of either the shot or the narration are regarded as aligned.
calize the target clip within a 200-second window around the anchor shot. This requires the temporal sentence grounding in a 200s-long movie clip, where comprehending the actions of different roles is critical. Therefore, based on the state-of-the-art TSG model IA-Net (Liu et al., 2021), we propose Role-aware Narration Locator (RNL). With a bidirectional GRU (Chung et al., 2014) visual encoder, we encode the input frame features to get temporal context-aware frame representations V . We in addition extract face features from the frames and encode them with a fully connected (FC) layer to filter key face information F . Then we finalize the visual representation by summing V and F . For text encoding, to relate role names in the text query with roles in the video, we extract face features from the portraits that correspond to the role names and encode them as visual token representations with a FC layer, which are then concatenated to the query’s textual token representation sequence. During training, for each target, we randomly select a 200s-long clip window that covers the target in each training epoch. We also construct a temporary dataset Movie101-LTG(temp) with fixed window to separately evaluate the second-stage model performance.

heading: 5.3 Experiments
text: Implementation Details. For Global Shot Retrieval, we use average Recall@n (n ∈ 1, 5, 10) to evaluate the retrieval performance on all movies. For Local Temporal Grounding, following previous works (Zhang et al., 2020a), we use “R@n, IoU@m” as metrics, which are defined as the percentage of at least one of top-n proposals having a larger temporal IoU than m with the ground truth. We fine-tune CNCLIP-huge on our Movie101-GSR(temp) for Global Shot Retrieval, and benchmark two code-released state-of-the-art temporal grounding models 2D-TAN (Zhang et al., 2020a) and IA-Net(Liu et al., 2021) on Movie101LTG(temp) for Local Temporal Grounding. In our RNL model, the video frame, face, and text feature extractors are pre-trained MIL-NCE, Arcface
(same as in the MCN task) and BERT-base-Chinese (Devlin et al., 2019), respectively. Results & Analysis. Table 4 and Table 5 show the performance of models on Global Shot Retrieval and Local Temporal Grounding, respectively. Our RNL outperforms baselines by introducing roleaware video and text encoding, indicating that distinguishing actions of different roles is critical for grounding movie narration. Furthermore, we perform an ablation study to verify the effectiveness of role-aware encoding. As shown in Table 5, adding face features to either video or text representations outperforms our base method IA-Net. RNL with both role-aware video and text encoding achieves the best performance. Table 6 shows the performance of combined inference by Global Shot Retrieval and Local Temporal Grounding. We in addition show the performance of k-way re-ranking, where the top-k shots retrieved in the first stage are respectively used as the anchors in the second stage, and all predictions obtained are re-ranked with their confidence scores. The experimental results show that k-way re-ranking improves Rank@5 performance but harms Rank@1 performance. Qualitative results can be found in Appendix D.

heading: 6 Conclusion
text: In this work, we propose Movie101, a Chinese large-scale video benchmark for movie understanding. To assist visually impaired people in enjoying movies, we propose a more realistic Movie Clip Narrating task to address the automatic movie description issue and design a human-preferencecompatible metric MNScore for narrating evaluation. Movie101 also supports the Temporal Narration Grounding task, which is more challenging than the previous TSG benchmarks. Furthermore, our experiments validate the importance of external knowledge including genres and roles for movie understanding. However, there is still a significant gap between our models and expert annotations. This reveals that further research endeavors are still needed to help visually impaired people enjoy movies by AI.

heading: Limitations
text: Keeping narration coherent within a movie is crucial for visually impaired people to enjoy the movie. In this work, we move a step forward for this target by setting the ground-truth texts in the Movie Clip Narrating task as narration paragraphs and providing longer video clips as inputs. However, how to ensure description coherence across different clips within a movie has not been studied in this work. This requires a higher-level comprehending ability of models to process the whole movie and connect different plots. We leave this to our future investigation.

heading: Ethics Statement
text: We propose Movie101, a new benchmark to support exploring technologies to benefit the accessibility of the visually impaired. There are two potential ethical issues with our work, regarding data source and crowdsourcing services, respectively. We state each of them as follows: Data Source. The collected movies are publicly available from Xigua Video, and are allowed to be crawled according to the service contract of the website7. Considering the copyright issue, we will only release the url list of movies. Besides, our data source does not contain any information that names or uniquely identifiable individuals or offensive content. Crowdsourcing Services. We recruited 20 Chinese college students (12 females and 8 males) via social media. For ASR outputs cleaning, workers were required to correct errors in the narration text while watching the movie. For each movie, it took about 2 hours with a payment of 50 RMB ($7.40 USD). To review corrections, for each movie, it took about 30 minutes with a payment of 25 RMB ($3.70 USD). Our payment is fair and reasonable in China, especially since the work is easy and fun. Before the annotation works began, we introduced the future use of the data in the task document to ensure that everyone was informed.

heading: Acknowledgements
text: This work was partially supported by the National Key R&D Program of China (No.2020AAA0108600) and the National Natural Science Foundation of China (No. 62072462).
7https://www.ixigua.com/robots.txt

heading: A Narration Distribution
text: Clips where ‘no actors are speaking’ refer to ANY scene wherein no verbal dialogue is being employed by the actors, regardless of whether they are visually present or absent. This definition encompasses, for example, a scene focused solely on a depiction of the sky. We detail the dialogues and narrations in the 101 collected movies. By merging the actor lines, we obtain a total of 15,307 dialogues, constituting 15,206 dialogue gaps with a total duration of 99.4 hours. The 30,174 narration clips we collect fill in 95.3% of the dialogue gaps in terms of quantity and cover 92.9% in terms of duration. Therefore, it is reasonable to assume that where there are no lines, there is a need for narration.

heading: B Dataset Quality Description
text: We adopt a two-stage annotation process to ensure the quality of the narrations. In the first stage, a group of workers is recruited to clean the data according to our guidelines. In the second stage, another group of workers further checks and corrects the annotation data. Our heuristics used to divide the paragraphs are designed based on our observation experience. We further conduct a manual evaluation of the narration quality. Of the randomly sampled 300 paragraphs, (1) in terms of narration recognition, 96.7% are textually consistent with original ADs; (2) as for the paragraph coherence, 90% maintain complete and coherent semantics, 7.7% should be merged with contexts, and 2.3% should be divided into multiple paragraphs. Thus, the narration is of good quality to support downstream tasks.
C Implementation Details
Candidate Narrations. In Section 4.3, We provide 5 different candidate narrations for each sampled movie clip for human evaluators to rank. These candidates are created as follows:
1. generated by the Vanilla Transformer (Zhou et al., 2018);
Table 7: Key hyperparameters and computational burden for models training.
Model Batch size Learning rate Training epochs GPU hours / epoch
VT 150 1e−4 ≤ 100 ∼3min on single RTX 2080ti OVP 56 1e−4 ≤ 100 ∼40min on single RTX 3090 RMN 56 1e−4 ≤ 100 ∼1h on single RTX 3090 CNCLIP 16 2e−6 ≤ 1 ∼1h on 4 RTX A6000 nodes 2D-TAN 64 1e−4 ≤ 30 ∼40min on single RTX 3090 IA-Net 64 8e−4 ≤ 15 ∼20min on single RTX 3090 RNL 64 8e−4 ≤ 15 ∼20min on single RTX 3090
2. generated by the OVP model (Song et al., 2021); 3. generated by our proposed RMN model; 4. generated by disturbing the ground truth with
role name removal and replacement; 5. generated by disturbing the ground truth with
nouns and verbs replacement.
Metrics Implementation. For CLIP-based metrics including CLIPScore and EMScore, we finetune ChineseCLIP-huge (Yang et al., 2022) on our dataset in the same way as in Section 5.2. For each movie clip and generated narration, CLIPScore is calculated with the mean pooled feature of 10 uniformly selected frames and the overall text feature, while EMScore is calculated with all selected frame features and textual token features. For BERTScore, we use the BERT-base-Chinese (Devlin et al., 2019) model checkpoint to calculate, and rescale the raw BERTScore with baseline8. For DIV, we calculate 1-gram diversity and 2-gram diversity following Shetty et al. (2017), and average them. For PPL, we obtain the perplexity of each narration with the causal Ernie 3.0 model (Sun et al., 2021) following the calculation of HuggingFace9. For RoleF1, we extract role names from the ground truth and the generated narration. We measure how the generated narration covers the roles appearing in the movie clip by Recall; given that these generated role names may also come from the model’s hallucination, for example from a wrong movie, we also take Precision into account. Finally, we calculate the F1 score with Precision and Recall. Hyperparameters and Computation. We detail the key hyperparameters and computational burden for the models training in Table 7. For each model, the results are derived from a single run.
8https://github.com/Tiiiger/bert_score/blob/ master/journal/rescale_baseline.md
9https://huggingface.co/spaces/ evaluate-metric/perplexity

heading: D Qualitative Result

heading: D.1 Movie Clip Narrating
text: Fig. 6 shows the qualitative results of the MCN task, including the generation results of baselines and our proposed RMN model, and the evaluation results of previous metrics and our proposed MNScore. Vanilla Transformer and OVP can correctly mention some actions but fail to generate correct role names because these roles never appear during training. However, with the help of the Role Selector module, our RMN could well relate roles in video clips with their role names. In addition, these cases demonstrate that our newly proposed MNScore evaluates more consistently with humans.

heading: D.2 Temporal Narration Grounding
text: Fig. 7 shows the qualitative results of our proposed two-stage method. Through Global Shot Retrieval, we obtain an anchor shot near the target clip from the whole movie, which further helps Local Temporal Grounding to locate the final target.
GT: 他凑近⼩星，得意地捶捶⾃⼰胸⼝，林冲⼜皱着眉撑起下巴。(He comes close to Xiaoxing, proudly pounding his chest. LinChong then frowns and props up his chin.） VT:林佳看着他们，点头，点头。(LinJia looks at them, nods her head, nods her head. ) OVP: 林佳看着丽丽，⼜看向⾃⼰说道。(LinJia looks at Lili, and then looks at herself and says. ) RMN: 陈姗姗抬起头，看着林冲的背影有些害怕，(ChenShanshan lifts her head and looks at LinChong’s back with some fear. )
GT: 说完想说的话叶薰才离开，⼜不⽢⼼地回头看了眼屋内。⻩达躲在墙后⾯听到了叶 薰讲的所有话，叶薰离开后他才从墙后⾛出来。(After saying what she want to say YeXun leaves only to look back at the house again reluctantly. HuangDa hides behind the wall and hears all that YeXun speaks, and he comes out from behind the wall only after YeXun leaves. ) VT: 他⼀边回头⼀边往外⾛，⼀边⼀边回头看了看，⼜看了看⾃⼰⼀眼，⼜看了看⾃⼰ 的背影，⼜低头没有说话，(As he turns back and walks out the door, he looks back, looks at himself again, looks at his back again, and looks down again without speaking, ) OVP: 他站在⻔⼝的毒贩已经停下了脚步，看着这⾥，他转过身来看着脚下的两个⼈， (The drug dealer he is standing in the doorway stop and look at the place, and he turn to look at the two men at his feet, ) RMN: ⻩达还是⼀副模样，他稍微落寞的样⼦，⼀时间还算回到，⻩达跟在⻩达身后， 他先是⼀屁股坐在椅⼦上，(HuangDa is still the same, slightly melancholy, for a while … come back, HuangDa follows behind HuangDa, he first buttocks in the chair. )

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓
text: VT 14.30 6.24 11.75 21.65 3.87
OVP 0.01 0 6.70 17.46 4.37 RMN 3.67 0 7.43 27.22 3.63
GT:在⼤家的嘲笑声中夏洛也尴尬的笑了⼀下并坐在席位上，(In the sound of everyone’s ridicule, XiaLuo smiles awkwardly and sits in the seat, ) VT:他看着⾯前的菜⼑，脸上满是泪⽔，他⾯⾯带笑意，(He looks at the chopper in front of him, his face is full of tears, and he has a smile on his face, ) OVP: 李⾏表情极其尴尬，他看着场上的⼈若有所思。(LiXing’s expression is extremely embarrassed, and he looks at the people as if in thought. ) RMN:夏洛看着⼿机，似乎陷⼊了沉思。⻢⼩丽缓缓开⼝，(XiaLuo looks at the phone and seems to be in deep thought. MaXiaoli slowly opens her mouth, )
GT: 现场⼤屏幕上的数字转了起来，⻩达和主持⼈转身看向⼤屏幕，数字转动了⼀会⼉ 之后停了下来，(The numbers on the big screen turn up, and HuangDa and the host turn to look at the big screen, the numbers turns for a while and then stops. ) VT: 他们在台上观察着，时间间的位置上台下的观⾳室内，(They watch from the stage, the position between time on the stage in the chamber of the observer, ) OVP: 第⼆天，三⼈来到现场，孟云和余⻜⼀起看着屏幕上的选择题，三⼈离开了。(The next day, the three come to the scene, MengYun and YuFei look at the multiple choice questions on the screen together, the three leave. ) RMN: ⻩达看着台下的电脑，这时余⻜和⻩达拉着⼿来到台球厅，他们相互打量着这⼀ 切，(HuangDa looks at the computer under the stage, at this time, YuFei and HuangDa come to the billiard room with hands holding, they survey all this each other )
GT: 王多⻥站在保险公司⼤厦最顶层，穿着红⾊⾐服绿裤衩，⼀只⼿背在后⾯，另⼀只 ⼿扶着巨⼤的“瘦”字，双腿交叉带着⼀脸享受闭上双眼，倚靠在“瘦”字上⾯。镜头缓 缓上升拉伸，王多⻥变得越来越渺⼩，最后完全看不⻅了。(WangDuoyu stands at the top of the insurance company building, wearing red clothes and green pants, one hand behind the back, the other hand holding the huge “thin” character, legs crossing, with a face of enjoyment, closing eyes, leaning on the “thin” character. The camera slowly rises, WangDuoyu becomes smaller and smaller, and finally completely invisible. ) VT: 在众⼈的⾼举⾏下，张彪也在这⼀场，下⾯的⾼楼下，阿俊也摔倒在地上。(In the crowd hold under the high, ZhangBiao is also in this scene, below the high floor, Arjun also falls to the ground. ) OVP: 随着⻜机的轰鸣声，继续朝下抓捕，⽽在楼梯上的临时，他选择了⼀个⾼挑的身 影，这时，焦急的他选择了⼀个按钮，⼤厦向下⻜去。(With the roar of the plane, continues capture downward, while on the stairs of the temporary, he chooses a tall figure, at this point, anxious he chooses a button, the building flies downward. ) RMN: 随后，王多⻥在天台上朗着升机，踏上了⾏程，登基本的装饰演员，王多⻥独⾃ 在空中，美丽在云⼤楼⾥摆着各种姿势，王多⻥顺着绳索向上攀爬 (Subsequently, WangDuoyu boards on the rooftop … , embarking on a trip, … decorative actors, WangDuoyu is in the air alone, posing in a variety of positions beautifully in the cloud building; WangDuoyu climbs upward along the rope )
GT: 桃⼦边说话边拦下⼀辆出租⻋，然后坐上⻋快速离去了。⻩达⼀个⼈愣在原地。 (Taozi stops a cab as she talks, then gets in and quickly leaves. HuangDa freezes alone. ) VT: 卢⼩⻥看到了他的眼神，他低头看着他，然后低下头，(LuXiaoyu sees the look in his eyes, and he looks down at him, then lowers his head, ) OVP:江丰回过头来看着他，然后叹了⼝⽓，(JiangFeng looks back at him, then sighs, ) RMN: ⻩达听到这话愣住了，他回头⼀看，(HuangDa freezes when he hears this, and he looks back, )

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓
text: VT 0.67 0 9.64 19.67 4.83
OVP 5.57 0 11.62 26.83 3.73 RMN 11.89 0 9.80 29.77 3.37

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓
text: VT 0 0 0 7.90 4.73
OVP 0 0 4.02 9.31 3.90 RMN 4.97 7.06 11.84 29.83 3.20

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓
text: VT 0.01 0 4.98 9.47 4.33 OVP 31.28 7.95 16.65 24.65 3.87 RMN 13.12 0 10.75 34.76 3.53

heading: Candidate CIDEr BLEU@4 METEOR MNScore Manual Ranking↓
text: Query: 影⽚开始挂满鲜花的欧式⼤铁⻔缓缓打开，铁⻔内是⼀座欧式建筑， ⼀辆⼩汽⻋从⻔外⾏驶⽽进，它穿过摆满花束的院⼦中，⼀名保安在礼堂前 ⼀⼿拿起路障，⼀⼿指挥着汽⻋向前，这辆⻋没有停下。 (At the beginning of the film, a large European-style iron gate full of flowers slowly opens. Inside the iron gate is a European-style building. A car drives in through the gate, and crosses the courtyard full of flowers. A security guard holds a barricade in front of the auditorium with one hand and directs the car forward with the other; this car does not stop. )
Query: 时间⼜过了⼀天，⼀只纸⻜机穿过学校的楼顶，只⻅秋雅⼀个⼈孤零 零的站在楼顶，这时袁华慢慢⾛了过来，秋雅看了⼀眼袁华便皱起眉头，⼀ 边摸着⾃⼰的⼩辫⼦，⼀边低下头，袁华吐了嘟嘴，含着眼泪询问： (Another day, a paper airplane flies through the roof of the school building. QiuYa is standing alone on the roof, when YuanHua slowly walks over. QiuYa takes a look at YuanHua and then frowns, while touching her pigtails, while lowering her head, YuanHua spits out his mouth and asks with tears. )
GT182s 200s anchor shot177s 197s
prediction183s 205s
GT2095s 2121s anchor shot2097s 2117s
prediction2102s 2122s
Query: 不知过了多久夏洛醒了过来，他发现洗⼿间的⽔⻰头关不上了，镜⼦ 也碎了，他看着镜⼦中的⾃⼰说道，他打开卫⽣间的⻔，⼀束耀眼的光芒刺 向他，他⽤⼿臂挡住眼睛，缓缓抬起头，他的瞳孔⾥看到的是教室的场景。 (A long time later, XiaLuo wakes up. He finds the bathroom faucet can not be turned off, and the mirror is also broken. He says while looking at himself in the mirror. He opens the bathroom door, and a dazzling light shines towards him. He blocks his eyes with his arm, and slowly raises his head. In his pupils is the scene of the classroom. )
Query: ⻢冬梅拉着⾏李⾛向⻄虹市的⻋站，⻋站上⽅的⼤屏上播放着夏洛的 相约98，画⾯⼀转时间来到了⾼考后，袁华穿着格⼦衫，脖⼦上挂着围⼱耐 着寒冷，在⼀个顶部积雪的公共电话亭⾥拨打着电话，(MaDongmei walks towards the Station of Xihong City with her luggage. The large screen above the station is playing XiaLuo’s meeting 1998. The scene switches to the time after the college entrance exams, YuanHua is wearing a plaid shirt with a scarf around his neck to withstand the cold, dialing the phone in a public phone booth with snow on top. )
GT3526s 3548s anchor shot3557s 3577s
prediction3528s 3548s
GT725s 748s anchor shot677s 697s
prediction724s 746s
Query: ⽕苗瞬间点燃了教室的窗帘，同学们乱作⼀团，纷纷拿出书本灭⽕， 校⻓还拿出⼀瓶墨⽔泼向燃烧的窗帘，这时夏洛的妈妈跑进教室，夏洛缓缓 转过头，看到是⺟亲来了，缓缓⾛向他，突然间就跪倒在地上，⼀把抱住他 的⼤腿。(The fire instantly ignits the curtains in the classroom, and the students are in a mess, and all taking out books to put out the fire. The school principal also takes out a bottle of ink throwing to the burning curtains. At this time, XiaLuo’s mother runs into the classroom, XiaLuo slowly turns his head, seeing his mother coming, slowly walking towards her, and suddenly falling to her knees with a hug on her thighs. )
Query: 夏洛的记者发布会开始了，⼀群记者围着夏洛，夏洛的两边站着秋雅 和张扬，⼤家都喜笑颜开，只有夏洛，⾯对记者⼀⾔不发。曾经他弹着吉他， 冬梅在⼀旁举着灯牌静静聆听，在家弹吉他时，冬梅在旁边拖着地，还有他 最爱吃的茴⾹打⻧⾯，这时夏洛突然举起左⼿示意⼤家安静，(XiaLuo’s press conference begins. A group of reporters surround XiaLuo. On either side of XiaLuo stand QiuYa and ZhangYang. Everyone is happy and smiling, only Xia Luo faces the reporters without saying a word. Once when he plays the guitar, Dongmei in the side holding a light sign quietly listening. When playing the guitar at home, Dongmei is mopping the floor next to him, and there is also his favorate Dalu noodles. At this time, XiaLuo suddenly raises his left hand to signal everyone to be quiet. )
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section: Limitation
3 A2. Did you discuss any potential risks of your work? Section: Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section: Abstract, Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section: Introduction, Dataset, Movie Clip Narrating, Temporal Narration Grounding, Appendix
3 B1. Did you cite the creators of artifacts you used? Section: Introduction, Dataset, Movie Clip Narrating, Temporal Narration Grounding, Appendix
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section: Dataset, Ethics Statement
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section: Dataset, Ethics Statement
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section: Ethics Statement
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section: Introduction, Dataset, Temporal Narration Grounding, Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section: Introduction, Dataset
C 3 Did you run computational experiments? Section: Appendix
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section: Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section: Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section: Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section: Dataset, Movie Clip Narrating, Temporal Narration Grounding, Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Introduction, Dataset, Movie Clip Narrating, Ethics Statement
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We report key information about the requirements for human annotators. (Section: Dataset, Movie Clip Narrating)
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section: Ethics Statement
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section: Ethics Statement
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section: Ethics Statement

heading: Future Work
text: Before the annotation works began, we introduced the future use of the data in the task document to ensure that everyone was informed