Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024)

Abstract: Video storytelling is engaging multimedia content that utilizes video and its accompanying narration to attract the audience, where a key challenge is creating narrations for recorded visual scenes. Previous studies on dense video captioning and video story generation have made some progress. However, in practical applications, we typically require synchronized narrations for ongoing visual scenes. In this work, we introduce a new task of Synchronized Video Storytelling, which aims to generate synchronous and informative narrations for videos. These narrations, associated with each video clip, should relate to the visual content, integrate relevant knowledge, and have an appropriate word count corresponding to the clip’s duration. Specifically, a structured storyline is beneficial to guide the generation process, ensuring coherence and integrity. To support the exploration of this task, we introduce a new benchmark dataset E-SyncVidStory with rich annotations. Since existing Multimodal LLMs are not effective in addressing this task in oneshot or few-shot settings, we propose a framework named VideoNarrator that can generate a storyline for input videos and simultaneously generate narrations with the guidance of the generated or predefined storyline. We further introduce a set of evaluation metrics to thoroughly assess the generation. Both automatic and human evaluations validate the effectiveness of our approach. Our dataset, codes, and evaluations will be released.

heading: 1 Introduction
text: Video storytelling (Li et al., 2019; Bhattacharya et al., 2023) is a tactic that utilizes the naturally engaging video format to share stories and has been widely used as a powerful tool for marketing, educational, or entertainment purposes. The video and its accompanying narration are both critical to successful video storytelling. Therefore, two aspects
*Corresponding Author.
of automatic video storytelling have been explored: generating or retrieving videos to illustrate written stories (He et al., 2023; Lu et al., 2023); and generating narrations based on visual scenes (Li et al., 2019). Nowadays, people frequently use convenient devices to make visual recordings. These initial recordings can be transformed into compelling video stories by integrating synchronized narrations. There is a strong need for automatically generating narrations for a given video.
Existing research on video-to-text generation mostly focuses on creating a single sentence to summarize the video (Wang et al., 2019) or generating fine-grained captions for video clips (Krishna et al., 2017). Gella et al. (2018) and Li et al. (2019) take a step further by generating sequential visual sum-
9479
maries for video scenes, creating a coherent video story. However, in real-life applications, when narrating an ongoing video, each narration should not only reflect the current visual scene, but also have an appropriate word sequence length to fit the duration of the video clip. Including relevant external knowledge is also important, as it can attract and convince the audience, particularly for marketing purposes. Additionally, we believe that a logical storyline helps maintain the coherence and integrity of the narratives.
In this paper, we introduce the task of Synchronized Video Storytelling, which aims to generate synchronized narrations for videos. As illustrated in Figure 1, the sequential visual scenes in the video are accompanied by corresponding narrations. Each narration has a suitable word sequence length and incorporates relevant knowledge to effectively engage the audience. They can be directly utilized as voiceovers for video recordings. This distinguishes our task from existing ones, as illustrated in Figure 2. Furthermore, our generation follows a structured storyline, similar to the human writing process, beginning with a brief outline and then drafting the complete story (Yang et al., 2022b). This proposed task is complex and cannot be effectively solved by traditional frameworks or evaluated by existing benchmarks. A possible approach is to utilize powerful Multi-modal Large Language Models (MLLMs) (Han et al., 2023; Sun et al., 2023) for zero-shot or few-shot generation. However, most MLLMs face challenges with time alignment and sequential video comprehension (Huang et al., 2023). Moreover, generating narrations for longer videos becomes difficult for MLLMs due to the requirement of excessively long visual tokens. Another approach is to generate cap-
tions for video shots and use them as input for the LLMs (Bhattacharya et al., 2023; Lin et al., 2023b). However, this relies heavily on the performance of the captioning model. Furthermore, current models are ineffective in generating coherent, appealing, and length-appropriate narrations. Additional training on specific instruction datasets is necessary.
To support the exploration of the proposed task, we create a benchmark dataset named ESyncVidStory (E-commerce Synchronized Video Storytelling). Given the rapid growth of ecommerce platforms, many sellers have product information and visual materials, but may lack the time and resources to convert these into engaging advertisement videos. Therefore, they seek an automatic and efficient solution. Our dataset can provide value for both research and practical applications in this area. E-SyncVidStory contains 6.0K videos with 41.3K video clips in total. Each video is segmented as sequential visual scenes with synchronized Chinese narrations. As the primary objective of Ad stories is to engage with the audience and showcase products (Bhattacharya et al., 2023), they might not always present a sequence of events as a classic story. Still, like classic ones, advertising stories should be coherent and follow a logical storyline, which is annotated in our dataset1. All annotations are obtained through automatic preprocessing, GPT-based refinement, and manual checks.
We further propose an effective framework integrating vision models and LLMs to address the task of synchronized video storytelling. Our specifically designed instruction structure allows simultaneous
1The storyline is expressed by categorized script labels. These labels are divided into 12 types, with their definitions provided in Appendix A.4.
generation of the storyline and the synchronized story. To enhance the effectiveness of visual embedding, we compress the long visual tokens into shorter visual information, which includes the longterm memory of previous clips and compressed information of the current clip. To evaluate the generated stories, we propose a set of evaluation metrics to comprehensively measure the generated results. Extensive experiments on both specific domain (E-SyncVidStory) and general domain (Li et al., 2019) Video Storytelling verify the effectiveness of our proposed framework.
The main contributions of our work include: 1) We introduce the new task of synchronized video storytelling, which is more challenging as it requires generating a synchronous, informative, and coherent story for a given video. 2) To support the exploration of this task, we collect a benchmark dataset in the advertising domain, namely E-SyncVidStory, which contains rich annotations, and also can support more research beyond video storytelling; 3) We propose an effective framework called VideoNarrator, which takes relevant knowledge and sequential video scenes as inputs. It simultaneously supports storyline generation and controllable video story generation; 4) We introduce a set of systematic evaluation metrics to comprehensively measure the story generation results. Automatic and human evaluation results verify the effectiveness of our method.

heading: 2 Related Works
text: Video Narration Generation. Bridging vision and natural language is a longstanding goal in computer vision and multimedia research (Li et al., 2019). Previous research on video-to-text generation primarily focuses on video captioning tasks, generating single-sentence factual descriptions for the whole video (Wang et al., 2019; Xu et al., 2016). Some works aim to provide more detailed and comprehensive descriptions by generating multisentence paragraphs (Krishna et al., 2017; Zhou et al., 2018a). However, most of these works ignore the overall coherence of the video descriptions. Gella et al. (2018) and Li et al. (2019) step further to generate a coherent and succinct story from abundant and complex video data. However, these datasets only generate short summaries for long videos and do not handle synchronized video storytelling. Additionally, all these works only focus on the visual content and do not take into account the
need to incorporate external knowledge.
Multi-modal Large Language Models. With the success of Large Language Models (LLMs), many works have tried to build Multi-modal LLMs by combining models from other modalities. In the video field, VideoChat (Li et al., 2023b) integrates video foundation models and LLMs via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. VideoChat Longvideo (OpenGVLab) incorporates LangChain into VideoChat to support videos that are longer than one minute. VideoChatGPT (Maaz et al., 2023a) design a 100k video instruction dataset, successfully empowering LLMs to comprehend videos. Video-LLaVA (Lin et al., 2023a) learns from a mixed dataset of images and videos, and performs joint visual reasoning on images and videos. However, all of these methods are not effective in comprehending sequential video clips. VtimeLLM (Huang et al., 2023) design a boundary-aware three-stage training strategy, enhancing results for dense video comprehension. However, it struggles with longer video inputs since the maximum training length is 2048. Like other MLLMs, it also lacks efficiency in generating video descriptions that are constrained by appropriate text length and specific storylines.

heading: 3 Synchronized Video Storytelling

heading: 3.1 Task Definition
text: Given a video V composed of sequential video clips {v1, v2, ..., vn} and the related knowledge composed of several knowledge items K = {k1, k2, ..., km}, synchronized video storytelling aims to incorporate the knowledge to generate a narration si for each video clip vi, and all narrations form a coherent story S = {s1, s2, ..., sn} for the video. The word count 2 of each si should be within the appropriate range ri. To enhance the coherence of the overall story, the generation is guided by a predefined or automatically predicted storyline, which is expressed using a sequence of script labels {l1, l2, ..., ln}. Each script label li, belonging to one of the 12 predefined types, summarizes the key point of the corresponding narration. More details about the script label are presented in the Appendix A.4.
2If the time length of video clip vi is t seconds, considering the normal speaking speed of 5 words per second, the appropriate word length range can be [5t− 3, 5t+ 3] with an error margin of 3 words in either direction.
Table 1: Comparison between E-SyncVidStory and existing dense video captioning (top half) and video story generation (lower half) datasets.
Input Modality Storyline Num. videos Avg. clips(per video) Avg. text len. (per video) Avg. text len. (per second)
YouCook2 (Zhou et al., 2018b) Video - 2K 7.7 67.7 (en) 0.21 ActivityNet Captions (Krishna et al., 2017) Video - 20K 3.7 47.6 (en) 0.40 Charades-STA (Gao et al., 2017) Video - 10K 1.8 11.0 (en) 0.36 ViTT (Huang et al., 2020) Video - 8K 5 110.5 (en) 0.44
VideoStory (unreleased) (Gella et al., 2018) Video - 20K 6.1 - - Video Storytelling (Li et al., 2019) Video - 105 13.5 162.6 (en) 0.22 E-SyncVidStory Video & Knowledge yes 6K 6.9 194.1 (zh) 5.21
Figure 3: The distribution of the number of 12 types of script labels in the benchmark.

heading: 3.2 E-SyncVidStory Dataset
text: Data Collection. We select high-quality advertisement videos with high click-through rates from the web and collect these videos along with their related information which will be utilized as knowledge points. As shown in Figure 1, knowledge points are some short expressions about different attributes of the product. We automatically divide each video into sequential video clips (Yue Zhao, 2019) and obtain their synchronized narrations through the automatic speech recognition (ASR) tool (Zhang et al., 2022). However, these preprocessed narrations still contain errors. To minimize the labor costs to correct them, we utilize the powerful tool GPT-4 (OpenAI, 2023) to correct the errors and predict script labels for each narration. The prompts for ASR refinement and script label classification are displayed in our Appendix A.6. With the aforementioned automatic pre-processing process, the results are already quite satisfactory. We then recruit crowd workers to review the narrations, check script label classification, and correct any remaining errors. This way, we build our proposed dataset, named E-SyncVidStory.
Statistics and analysis. E-SyncVidStory contains 6,032 videos, with a total of 41,292 video clips. The advertising videos cover various industries including personal care, makeup, clothing, household supplies, maternal parenting, and electronics, etc. The video length ranges from 2 to
220 seconds with an average of 39 seconds, and the average story length is 194 words. As shown in Figure 4 (a), the clip numbers within each video vary greatly. Thus we apply relative clip position embedding as described in Section 4.1. The duration of video clips is displayed in Figure 4 (b). Since a single clip within a video can have a maximum duration of 30 seconds, visual compression is needed when comprehending the video. The distributions of script labels are shown in Figure 3. Most narrations focus on the product effect, demonstrating the product’s benefits. They also prefer social proof, using positive experiences to persuade the audience. While this work focuses on creating a story from a structured video, this dataset could also aid investigation in generating a well-structured video from multiple unsequential clips, as discussed in Appendix A.5.
Dataset Comparison. To the best of our knowledge, E-SyncVidStory is the first synchronized video storytelling benchmark. As shown in Table 1, the average text length per second is much longer than the previous datasets because it allows for synchronous narration. We also annotate the structured storyline and corresponding knowledge of videos, which can be used to generate more informative and coherent narrations.

heading: 3.3 Automatic Evaluation Metrics
text: Following Li et al. (2019), we evaluate the generated narrations with NLP metrics such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and CIDEr (Vedantam et al., 2015). However, it is not reasonable to only consider the scores based on Ground Truth, as the generated story may include different knowledge even from the same aspect. For example, while the visual scene displays the list of ingredients, the narration might select a portion of them to describe. Thus we propose reference-free metrics to evaluate the results, in the aspect of visual relevance, knowledge relevance, controllable accuracy, and fluency.
Visual Relevance. Following Shi et al. (2022), we use EMScore and EMScoreref to evaluate the visual relevance between the video and generated narrations. The textual and visual embeddings for evaluation are encoded using Chinese-Clip (Yang et al., 2022a). Unlike the traditional visual relevance evaluation, we only consider the similarity between visually relevant words3 and the video clips. Because connecting words and knowledge within a sentence which enriches the video narra-
3We tokenize si using jieba (Sun, 2019) and retain only the nouns, verbs, and adjectives as visually relevant words.
tions should not hurt the visual relevance even if they have a low similarity score.
Knowledge Relevance. A compelling advertising story should incorporate relevant knowledge to effectively attract customers. Firstly, each narration should incorporate relevant knowledge points while avoiding unrelated messages, denoted as information similarity (InfoSim). Additionally, the knowledge points within a story should be diverse, avoiding the emphasis on only a small amount of information. This is referred to as information diversity (InfoDiverse). The detailed evaluation process is displayed in our Appendix A.2.
Controllable Accuracy. We evaluate the accuracy of the word sequence length by checking if the word count of sub-story si falls within the appropriate range as described in Section 3.1. When generating stories controlled by a predefined storyline, we utilize GPT-4 to assess the controllable accuracy of the script labels. The prompt for this evaluation can be found in Appendix A.6.
Fluency. We apply intra-story repetition (Yao et al., 2019) which measures sentence repetition within a story through word overlaps.

heading: 4 Method
text: As shown in Figure 5, our proposed VideoNarrator consists of the visual feature extractor, the memory consolidation component, the video projection layer, and the Large Language Model (LLM). Inspired by Maaz et al. (2023b), we utilize the powerful CLIP-L/14 visual encoder (Radford et al., 2021) and the Q-Former module from BLIP-2 (Li et al., 2023a) to obtain frame-level visual features. These features already achieve a good alignment with the textual modality. To capture temporal information, we further feed them into a linear video projector. However, inputting all video frames would result in significant computational complexity and memory usage (Song et al., 2023). Moreover, it would make it challenging to extract prior knowledge or maintain coherence with the previously generated story. To address these difficulties, we consolidate the visual information from previous clips into fixedlength memory tokens. Additionally, we compress the visual frames of the current clip, retaining only the key information.

heading: 4.1 Visual Embedding
text: Visual Compression. Considering the visual information redundancy (Tong et al., 2022), we first compress the original frames in each video clip by removing a frame from any adjacent pair frames if their similarity is above a threshold value τ . This removal process will be continued until the similarity between each pair of adjacent frames falls below τ .
Memory Consolidation. For long videos consisting of multiple video clips, when generating a narration for the current clip vi, we only need a more concise message of previous video clips. Inspired by Song et al. (2023), we perform memory consolidation by merging the most similar visual tokens. Specifically, we maintain fixed-length tokens as visual memory. We iteratively repeat the merging process 4 until the memory tokens of previous video clips reach the fixed length.
Video Clip Positional Encoding. The relative position of each video clip is important in video story generation. For example, when approaching
4Assume {fi}ki=1 are the visual features of k frames. In each merging process, we : (1) Evaluate the cosine similarity of each adjacent pair. (2) Find the pair (fm, fm+1) with the highest similarity. (3) Merge fm and fm+1 by averaging their features and remove fm+1.
the end of a video, it is more likely to evoke a response from the audience rather than introduce the product. However, the model could only acknowledge the absolute position of a video clip. Thus, we propose a relative clip position embedding and add it to the video embeddings. If a clip is located at p% of the whole video, then p is its relative position. This value p is passed through a positional embedding layer, whose parameters are updated during training. The encoded position feature is then added to the visual embeddings.

heading: 4.2 Prompting
text: The input prompt for VideoNarrator is displayed in Figure 6. For each video clip, the controllable signal is provided just before the narration, which can ensure controllable accuracy. In our instruction, we concisely explain the task requirements. As shown in Figure 6, these red words can make it
easier for the LLM to target relevant information. In the underlined instruction, we emphasize that the model should primarily use the information provided in the prompt and avoid using unrelated knowledge stored in the LLM.

heading: 4.3 Training
text: As shown in Figure 6, we construct our dataset as multimodal instruction training samples. For a video consisting of n clips, we maximize the probability of generating each script label li and narration si as follows:
n∑
i=1
P (li, si|XInstrution,K, ri, vj≤i)
We update the parameters of the video projector and the video clip positional embedding layer. Additionally, we apply LoRa (Hu et al., 2022) adapter to fine-tune the LLM, enabling it to possess the ability of controllable generation and relevant knowledge combination.

heading: 5 Experiments

heading: 5.1 Experiment Settings
text: Implementation Details. In our experiments, we use pre-trained models as visual feature extractors
as described in the above Section. In experiments on E-SyncVidStory (advertising domain), we employ the Baichuan-7B model (Yang et al., 2023) as the LLM model, known for its excellent performance in Chinese-related tasks. We construct our dataset as illustrated in Figure 6, resulting in 41k instruction samples. We apply 90% of them as the training set and the left as the testing set. Our proposed framework is trained for 15 epochs, using a learning rate of 1e−4 and a batch size of 32. The LoRA parameters are set to r = 64 and alpha = 16. The training of our 7B model took around 48 hours on 4 A6000 GPUs.
In experiments on Video Storytelling dataset (general domain) (Li et al., 2019), the LLM model is replaced by LLaMA-2 (Touvron et al., 2023) for
English generation. We make minor adjustments to fit the task setting of Synchronized Video Storytelling. Specifically, we apply the text length of the ground truth as the length requirement, and use the video topic as its relevant knowledge. The training prompt is similar to that presented in Figure 6, while the system role is changed to “You are an AI assistant that can understand videos”, the product information is replaced with the event topic, and the requirement for script labels has been removed.
Compared Baselines We compare our model with the following baselines. (1) Multi-model Pipieline. Specifically, we apply LLaVA (Liu et al., 2023) to get textual summaries for video clips and prompting GPT-3.5 for zero-shot and fewshot generation. (2) End2end MLLMs, including zero-shot performance of Video-ChatGPT (Maaz et al., 2023a), Video-LLaVA (Lin et al., 2023a), and VTimeLLM (Huang et al., 2023) with 7B LLMs. These models take the video as inputs and generate narrations in an end2end manner. (3) Fine-tuned End2end MLLMs. Specifically, we finetune the VTimeLLM model, which is effective for dense video comprehension and supports Chinese generation. More details about these baselines can be found in our Appendix A.1.
As for experiments on the Video Storytelling dataset (Li et al., 2019), we compare our model with state-of-the-art methods: the retrieval-based GVMF (Lu and Wu, 2022) and the generativebased VerbalizeVideos (Bhattacharya et al., 2023).
Ablation Study Setting. We carry out ablation studies to validate the contribution of different components in our proposed framework VideoNarrator. a. w/o LLM LoRa Finetune Keep the LLM frozen during training in order to verify whether the LLM is effective for controllable generation. b. w/o Storyline To verify whether the model can generate a nicely structured video story without provided explicit script labels. c. w/o Video Clip Position To verify the effectiveness of relative clip position embedding. d. w/o Visual Compression Retain all frames to verify the effectiveness of visual compression.
e. w/o Visual Memory Retain all video features of previous clips to verify the effectiveness of memory consolidation.

heading: 5.2 Automatic Evaluation Results
text: Advertising Video Storytelling. VideoNarrator can either generate a storyline to guide the generation, or use a predefined storyline to generate a story based on user preference. We conduct experiments for both situations on E-SyncVidStory. With Generated Storyline. As shown in Table 2, our model outperforms the baseline models, even when fine-tuned on our dataset. For detailed evaluations shown in Table 3, existing MLLMs demonstrate their ability to generate visually relevant and informative stories. However, they tend to focus on a limited amount of prior knowledge and may repeat it, resulting in a lower InfoDiverse score. Furthermore, their fluency and ability to control the length of word sequences are poor. Based on the ablation studies, we find that by fine-tuning the LLM with our instruction data, our model can achieve the ability to control the word sequence length and comprehend video features, showing an increase in controllable accuracy and visual relevance. Applying visual compression and visual memory consolidation can improve the model’s ability to extract relevant knowledge and retain story coherence, resulting in better performance in knowledge relevance and fluency. Explicit storylines and relative positional embedding can encourage narrations to follow a more logical outline of persuasion, improving the performance in knowledge relevance. With Predefined Storyline. Our model is also capable of generating stories with predefined storylines. Users can modify the generated storyline or offer a specific storyline to control the generation process, generating customized narrations. Table 4 shows the results generated with the predefined storyline, demonstrating ideal performance in controllable accuracy and other aspects.
General Domain Video Storytelling To validate the effectiveness of our proposed VideoNarrator, we conduct experiments on the Video Storytelling
dataset (Li et al., 2019), which covers a more general domain. As shown in Table 5, our model surpasses the state-of-the-art results. This showcases its effectiveness in synchronized video storytelling across a broader domain.

heading: 5.3 Human Evaluation and Qualitative Cases
text: We conduct human evaluations as well on three metrics: (1) Visual Relevance measures the relationship between the generated narrations and the video shots. (2) Attractiveness measures how well the stories can invoke the users’ interests. (3) Coherence measures the inter-sentence coherence and the completeness of the whole story. All metrics are rated from 0 to 3. We compare our models with the few-shot generation results of existing powerful LLMs. Specifically, we apply LLaVA1.5 or GPT4V to generate descriptions for each visual scene. The captions of LLaVA1.5 are used as inputs for GPT3.5 to generate stories, and the ones generated by GPT-4V are provided for GPT4. The prompt is shown in Appendix A.7. We randomly sample 30 stories and ask 13 advertisers to conduct the evaluation. As shown in Table 6, our model outperforms
GPT-3.5. We find that GPT3.5 often generates redundant statements and sometimes describes the visual scene instead of generating attractive narration. Regarding GPT-4V+GPT4, its performance is significantly improved compared to GPT3.5. However, our model still achieves better results in terms of story coherence and attractiveness, as well as very similar visual relevance scores. Additionally, it lacks length controllability, with an accuracy of 55.0%, which impacts its practical use.
Figure 7 displays some example stories generated by our proposed framework.

heading: 6 Conclusion
text: This paper introduces the new task of synchronized video storytelling, which aims to generate synchronous narrations for sequential video scenes. These narrations are guided by a structured storyline and can incorporate relevant knowledge, resulting in a coherent and informative story. The new task is more practical than existing tasks of dense video captioning and visual storytelling because the generated narrations can be directly combined with the visual scenes, creating an engaging videoformat story. We collect a benchmark dataset called E-SyncVidStory and introduce an effective framework named VideoNarrator. Both automatic and human evaluations verify the effectiveness of our proposed framework.

heading: Limitations
text: In this work, our main focus is on generating synchronous narrations based on given videos. However, in real life, it is also important to transform unstructured video clips into well-structured ones. This aspect can be further explored in future research with the support of our proposed ESyncVidStory. Additionally, our VideoNarrator model still faces the challenge of hallucination. For instance, most training samples show the application of face cream to the face after rubbing it on the hand. As a result, the model sometimes assumes that people will do the same when viewing someone rubbing the product on their hands. To address this issue, more constraints will be incorporated in future work.

heading: Ethics Statement
text: We acknowledge the Code of Ethics and Professional Conduct and strictly adhere to the rules throughout this research. There are two potential ethical issues with our work. The first pertains to the data source, while the second relates to the use of crowdsourcing services.
Data Source. The datasets are collected from public advertisement videos on the e-commerce website 5. Given the copyright considerations, we will only release the URLs and features of the videos. Furthermore, our data source does not contain any information that identifies individuals by name or any offensive content.
Crowdsourcing Services. After refining the data using the powerful GPT-4, we only need to conduct further checking of the annotations. We have hired 5 workers to review and correct any remaining errors. Each video took approximately 2 minutes to complete, and the workers received reasonable payment for the local area.

heading: Acknowledgements
text: We thank all reviewers for their insightful comments and suggestions. This work was partially supported by the National Natural Science Foundation of China (No. 62072462) and the Beijing Natural Science Foundation (No. L233008).
5https://www.taobao.com/

heading: A Appendix

heading: A.1 Details of Compared Baselines
text: Multi-model Pipelines Following Zhang et al. (2023), we convert the visual information of sequential video clips into textual summaries. This information, along with the product details and length requirements, is fed into a strong generative model GPT-3.5 to produce coherent narrations. Specifically, we apply key frames from each video clip (as described in Section 4.1), generate English captions, and translate them into Chinese. We combine these captions to create the visual summaries.
Figure 13 shows our generation prompt. For fewshot generation, we select three example video stories with the highest click-through rate from the same domain and apply them in the prompt.
End2end MLLMs Most existing MLLMs, although promising, struggle to directly generate captions aligned correctly with sequential video clips. When an entire video is inputted, they often fail to properly ground the correct time ranges for each clip (Huang et al., 2023). For such models, we generate narrations sequentially. Specifically, we input the current video clip along with the previously generated narrations to create the current narration. We make experiments with Video-LLaVA (Lin et al., 2023a) and Video-ChatGPT (Maaz et al., 2023a), which show SOTA results on video understanding tasks. Since these models are pre-trained in English only, we apply translations using the GPT-4 model. We then manually create well-designed instructions: “You are a creative advertiser who is very familiar with the advertisement video.\n Product Information: {knowledge points}\n Previous advertisement copy: {generated narrations} \n Please proceed with the previous advertisement copy and create a new advertisement copy inspired by the video, using between {x1} to {x2} words. Ensure that the advertisement copy closely relates to the visual scene.\n Advertisement copy: ”
We also conduct experiments using VTimeLLM (Huang et al., 2023), a model pre-trained for time-
aligned video understanding and is effective for dense video captioning. It supports Chinese generation with the ChatGLM3 (Du et al., 2022) structure. The generation prompt is shown in Figure 8.
Finegrained End2end Large Multimodal Models To the best of our knowledge, VTimeLLM is the most effective Chinese MLLM for sequential video comprehension. We fine-tune VTimeLLM on our proposed dataset, which is transformed into its QA format, as illustrated in Figure 8. As illustrated in Table 2 and Table 3, our model surpasses the fine-tuned results in all metrics.

heading: A.2 Evaluation Metric for Knowledge Relevance
text: As described in Section 3.3, we evaluate two aspects of knowledge relevance: information similarity and information diversity. To evaluate InfoSim, we calculate the similarity of each knowledge point with the entire sentence si, selecting the maximum score as the coarse knowledge similarity We also calculate the maximum similarity of each word in si, and the average value is considered as the fine-grained knowledge similarity. InfoSim is calculated as the average of coarse and fine-grained similarity. To evaluate InfoDiverse, we check if a knowledge point ki has a similarity higher than 0.9 with a sentence or its segmented words. If this condition is met, we consider that the sentence includes knowledge point ki. The InfoDiverse score is calculated as the number of covered knowledge points divided by the video time duration. Given a story S = {si}ni=1 and its prior knowledge K, the knowledge relevance scores are evaluated by:
InfoSim = 1 2|si| ∑
si
( max k∈K fTk fsi+
1 |W (si)| ∑
w∈W (si) max k∈K
fTk fw
)
InfoDiverse = 1
T ∣∣∣∣∪si { kt ∈ K ∣∣ max w∈W (si)∪si fTktfw > 0.9 }∣∣∣∣,
where W (si) represents all the words in sentence si, T refers to the duration of the whole video. fsi , fk, and fw refer to the normalized embeddings of sentence si, knowledge point k, and segmented word w, respectively.

heading: A.3 Human Evaluation Annotators
text: All the annotators are employees from the advertising company and have a thorough understanding of advertisement videos. The team of 13 annotators includes 5 women and 8 men, ranging in age from 22 to 35. The Pearson Correlation score of their evaluation results is 0.55.

heading: A.4 Script Label Definition
text: Script Labels are predefined structures to control the generated story. In this work, we focus on advertising stories in the e-commercial era. We analyze the collected advertising stories and categorize the script labels into 12 types: social proof/influence, sore point, call to action, design of appearance, ingredient/material/texture statement, product trial, product effect, product security, specific characteristics, authoritative certificate, production process, and others. Each of them corresponds to one of the advertising persuasion strategies (Kumar et al.,
2023). The definition of the 12 script labels is displayed in Figure 9.

heading: A.5 Discussions of Further Research
text: In our paper, the visual scenes are logically sorted. But at times, a user may wish to input unordered visual recordings and obtain sorted clips with a coherent narrative. We also hope to invoke additional research on the reordering task using the annotation provided in our proposed dataset. In Figure 11, we present the quantitative results sorted by the robust GPT-4V model. These results are still imperfect. Considering its reasoning process, the usage process should be positioned between the product’s pre-use and post-use stages. Starting the advertisement with the pain point could make it
more appealing, capturing the users’ needs right from the beginning. This highlights the need for further research.

heading: A.6 Prompt for Data Processing
text: The prompt for ASR refinement is displayed in Figure 10. The prompt for script label classification is displayed in Figure 12.

heading: A.7 Prompt for Few-Shot Story Generation
text: The prompt for few-shot story generation with GPT3.5 and GPT4 is displayed in Figure 13.

heading: Future Work
text: This aspect can be further explored in future research with the support of our proposed ESyncVidStory. Additionally, our VideoNarrator model still faces the challenge of hallucination. For instance, most training samples show the application of face cream to the face after rubbing it on the hand. As a result, the model sometimes assumes that people will do the same when viewing someone rubbing the product on their hands. To address this issue, more constraints will be incorporated in future work