A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them In Zero Shot
Authors: Aanisha Bhattacharyya
Yaman K Singla
Balaji Krishnamurthy
Rajiv Ratn Shah
Changyou Chen
Venue: EMNLP 2023 Main (2023)
Keywords: video understanding
large language models
persuasion strategies
zero-shot
long video understanding
TL;DR: Video understanding lags far behind NLP; LLMs excel in zero-shot. Our approach utilizes LLMs to verbalize videos, creating stories for zero-shot video understanding. This yields state-of-the-art results across five datasets, covering fifteen tasks.

Abstract: Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey
meaning. There is a dearth of large annotated training datasets in the multimedia domain hindering the development of supervised learn-
ing models with satisfactory performance for real-world applications. On the other hand, the rise of large language models (LLMs) has
witnessed remarkable zero-shot performance in various natural language processing (NLP) tasks, such as emotion classification, question-
answering, and topic classification. To leverage such advanced techniques to bridge this performance gap in multimedia understanding, we
propose verbalizing long videos to generate their descriptions in natural language, followed by performing video-understanding tasks on
the generated story as opposed to the original video. Through extensive experiments on fifteen video-understanding tasks, we demonstrate that our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. Furthermore, to alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.

A Video Is Worth 4096 Tokens:
Verbalize Videos To Understand Them In Zero Shot
Aanisha Bhattacharyya
A
Yaman K Singla
Balaji Krishnamurthy M
Rajiv Ratn Shah
Changyou Chen
A
Adohe Media and Data Science Research (MDSR).
HrT-Dethi,
State University of New York Buffalo
depict some form of story. Despite this, most work in the multimedia understanding domain has been about simple vidcos containing a single ac- tion or photo streams (Li et al., 2020). Beyond understanding objects, actions, and scenes lies in- terpreting causal structure, making sense of visual, textual, and audio input to tie disparate moments together they give rise cohesive narrative of events through time. This requires moving from reasoning about single activity and static moments to sequences of images and audio that depict events as they occur and change. Progressing from single- action videos to story videos allows us to begin to reason about complex cognitive tasks like emotions depicted and persuasion strategies used.
Abstract
Multimedia content, such as advertisements and story videos, exhibit a rich blend of cre- ativity and multiple modalities. They incor- porate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. There is a dearth of large annotated training datasets in the multimedia domain hin- dering the development of supervised learn- ing models with satisfactory performance for world applications. On the other hand, the rise of large language models (LLMs) has witnessed remarkable zero-shot performance various natural language processing (NLP) lasks, such emotion classification, question- answering, and topic classification. To leverage such advanced techniques to bridge this perfor- mance gap in multimedia understanding, propose verbalizing long videos to generate their descriptions natural language, followed by performing video-understanding tasks the generated story as opposed to the origi- nal video. Through extensive experiments on fifteen video-understanding tasks, we demon- strate that our method, despite being zero-shot, achieves significantly better results than super- vised baselines for video understanding, Fur- thermore, alleviate lack story understand- ing benchmarks, publicly release the first dataset on crucial task in computational social science on persuasion strategy identification.
Recently, large video pre-trained models (LVMs) like VideoMAE (Tong et al., 2022), Intern Video (Wang et al., 2022), and VideoCLIP (Xu et al., 2021) have proved to be powerful in enhancing reasoning skills on video data. For e.g., Intern- Video showed a performance increase in action classification and question answering tasks. Never- theless, these models have few shortcomings that impair their performance on video understanding tasks: 1) LVMs are mostly trained on short videos (<10s) consisting of majorly motion-centric actions, such as those present in Kinetics (Kay et al., 2017) and Something Something v2 (Goyal et al., 2017); and 2) they often require significant amount of task-specific finetuning data to perform on video- understanding tasks like summarization, question- answering, and emotion classification. On the other hand, stories are often longer than 10 seconds and are typically much more complex than motion- centric videos. For example, we test our mod- cls on five datasets with average vidco lengths of 12.4 minutes (video story) and 3.5 minutes (other tasks). Our videos contain dialogues, text overlaid on frames, fast-moving scenes, graphics like sym- bols and cartoon characters, and rhetoric elements like emotions and taglines, other than motions and
1 Introduction
"We are, as a species, addicted stories. Even when the body goes to sleep, the mind stays up all night, telling itself stories." Jonathan Gottschall
Most videos we encounter in the day-to-day, like movies, documentaries, advertisements, and user- generated content like Tiktok and Youtube shorts,
"Fqual Contribution, Contact Sxumar@@adhh.coom for questions and suggestions.

---

Transoript: well hey presont Thank You Marshall... ... are you ready blow out the candle.... Marshall let's push chase the pool... Wow the surprises we got... Go firet okay put on the blindfold ready ..... for more fun click the red subcoribe button
wrong pineta how you ... bitthday cake is its ... paw patrol got as
Generated Story A fumily had the perfect idea for Paw Patrai-hemmed birthday party. The kids had blast blowing bubbles in the beckyard and
play with pinsta the backyerd and pool. birthdyy party ended bitildyy girl huggedd badge on masoot game. Everyone a great time. and birthday cake was Pawv Patrol thomed delight. This vlog shows family fun can be had at any birthday party with a littl imagination.
the pool with
played Paw Petrol costumn.. The bithday Patrol playhouse,
not
They also set up LEGO train
Comice maseat
game catis and pin
Topic Celebratione
Emotion Cheerfu
Persuasion Strategy: Social Identity Social Proof
Action Consider using your imaginetion create a fun and memorablee birnday party for your family or friendss.
Reason: The reason create uniquely fun-filed and momorabic bithday party for your famil or Iriends
Figure I: An example of story generated by the proposed pipeline along with the predicted outputs of the video- understanding tasks on the generated story. The generated story captures information across scenes, characters, event sequences, dialogues, emotions, and the environment. This helps the downstream models to get adequate information about the video to reason about it correctly. The original video can be watched at pettps//youtte be/ _ar__amwjjjp jmwoPAAcCC..
actions. Further, the video domain, there are not large enough datasets for finetuning LLMs on the downstream tasks.
ple sources and modalities such as dialogue, text, characters, and scenes. All of them provide both intersecting and mutually exclusive information helpful to understand and reason about the video. As a first work, we propose to address these chal- lenges by exploiting the power of LLMs in the video domain. The LLMs are used decompose long videos into stories and then reason about videos using the generated stories instead of the raw information. In this way, we can retain the rele- vant evidence and exclude the remaining irrelevant evidence from interfering with the decision. The main contributions of our paper are:
Recently, zero-shot performance in the natural language processing (NLP) domain has increased substantially owing to the recent growth of gener- ative large language models (LLMs). Instead of fine-tuning L.Mss, in-context learning has recently gained noticeable attention to exploring the rea- soning ability of LLM, where several onput-output exemplars are provided for prompting (Wei et al., 2022; Kojima et al., 2022; Min et al., 2022). For ex- ample, the chain of thought prompting (Wei et al.. 2022) has been discovered empower LLMs to perform complex reasoning by generating interme- diate reasoning steps.
1) We propose converting long videos from the mul- timodal domain to "small" coherent textual stories by verbalizing keyframes, audio, and text-overlaid scenes with the help of a powerful LLM and in- structions (Figs. 5). We experimentally test the story generation capability of our method and note that our method outperforms mate-of-theeart story generation methods.
Owing to in-context learning, tasks like emotion recognition, named entity recognition (Wang et al., 2023), and even higher-order composite tasks like table understanding (Ye al., 2023) have shown performance improvements. However, the capa- bility of LLMs on video reasoning tasks is still unexplored. There are several technical challenges preventing leveraging LLMs for video-based rea- soning tasks. First, video in the raw form can be quite long, spanning minutes and thus contain- ing many frames. Directly encoding all frames via pre-trained models could be computationally intractable and interfere with huge amounts of ir- relevant information. Second, for effective video understanding, we need information about multi-
2) Next, we test the utility of generated stories by conducting extensive experiments on five bench- mark datasets covering fifteen video understand- ing tasks. Experimental results demonstrate that our methods achieve better results than both fine- tuned and zero-shot video understanding baseline models without using any human-annotater sam- ples. Through this, for the first time in literature, we show that the essence of a mulliolll video can be represented in text while being in-

---

& Viceo kare
'Libel INl Srup Toys"
Dura
Ournetted Story
Chefter
Fitent Per Slver "Roal Boac"
many Histro aberthe
OCR Modkl
Shop feo, The ingin oriees up
door
Pranstcondeiigg
colle beins helli by nard Lafto dlwers oan inepucollec mon ant con get procoeghh amphosiing
"poeamss scmerith!
Capilons
Top Cloaderr
Sueplin
Telko.pe pot ship scr
for gaseration
celection Atngette the stretteerec Slopp key.
Obgaeta::
pervoe your
Samping Age
"gattr:: "I "ooo pot dlue eax beaot" "ocwersnn
Resso
Roffernes ustradde from
you can cilled jum or glrmmet over 199 peryte cant enoug?
Recegn Bon
The overview of our framework to generate a story from a video and perform downstream video- understanding tasks. First, we sample keyframes from the video which are verbalized using BLIP-2. We also extract OCR from all the frames. Next, using the channel name and ID, query Wikidata to get company and product information. Next, we obtain automatically generated captions from Youtube videos using the Youtube API. All of these are concatenated as a single prompt and given as input an LLM and ask to generate the story of the advertisement. Using the generated story, we then perform the downstream tasks of emotion and topic classification and persuasion strategy identification. This video can be watched at https: //yutu. be/ZBLKTAL11CI.
Figure 2:
formed through the different modalities like audio. raw pixels of frames, text overlaid scenes, emo- tions, and product and business information. This text representation can then be used to perform story-understanding tasks instead of the original video. We show that finetuning on video stories rather than videos leads to better performance (Ta- ble 2). Through an ablation study, we also show that none of the individual modalities is able to perform as well as the combined knowledge when crystallized in story using LLMs (Table 8).
grained descriptions by generating multi-sentence paragraphs. However, most work is concentrated cooking videos and action videos. Li et al. (2020) is arguably the first to propose dataset for long videos with complex event dynamics, aim- ing to generate a coherent and succinet story from the abundant and complex visual data. We use their dataset to test our method. Further, recently, two datasets specifically targeting long-form video understanding were released: Long Video Under- standing (LVU) (Wu and Krahenbuhl, 2021) and Holistic Video Understanding (HVU) (Diba et al., 2020). In consonance with the idea of stories ex- pressed in long-form videos, these datasets go be- yond actions and objects and release rich semantic labels like concepts, attributes, and events in HVU, and relationship, scene, way of speaking, etc. in the LVU dataset. We use these datasets to test long- video understanding.
3) Further, given the lack of datasets for complex story ridco-understanding tasks, we release the first dataset for studying persuasion strategies in advertisement videos (Fig. 3)', This enables initial progress on the challenging task of automatically understanding the messaging strategies conveyed through video advertisements.
2 Related Work
Advertisement Story Understanding: Other than user-generated content, the other most impor- tant source of story content is grand-generated con- tent. Through these videos, brands try to commu- nicate with their customers. There has been some work on understanding advertisements (Hussain et al., 2017; Ye and Kovashka, 2018; Zhang et al., 2018; Ye al., 2019; Savchenko et al., 2020; Pilli et al., 2020; Gaikwad et al., 2022; Kumar et al., 2023). Hussain et al. (2017) released dataset containing image and video advertisements and their emotion, topic, action-reason, symbolism, and other labels. We show our performance on all the
Visual Storytelling:
Bridging language and mul- timedia is longstanding goal in multimedia derstanding (Mogadala al., 2021). Earlier works mainly target image and video captioning tasks, where a single sentence factual description is gen- erated for an image or a segment (Xu et al., 2015). Recent works (Krause et al., 2017; Liang et al., 2017; Yu et ul., 2016; Krishna et al., 2017; Li et al., 2020) aim to provide more comprehensive and fine-
'Visit https:/github.co http://gituu..com/miias--cooo htmm://ias-recaach//iddooooooo to access the videos and their annotations.

---

video tasks they released in their work. Subsequent papers improve performance benchmarks on their image ads dataset.
tion tasks in the text domain. This powerful abil- ity has been widely verified on natural language tasks, including text classification, semantic pars- ing, mathematical reasoning, etc. Inspired by these advances of LLMs, we aim to explore whether they could tackle reasoning tasks on multimodal data (i.e. videos). Therefore, we propose a storytelling framework, which leverages the power of LLMs to verbalize videos in terms of text-based story and then performs downstream video understanding tasks on the generated story instead of the original video. Our pipeline can be used to verbalize videos and understand videos to perform complex down- stream tasks such as emotion, topic, and persuasion strategy detection.
Understanding Persuasion In Ad Stories: Fur- ther, we contribute one crucial task on the advertise- ment story understanding task: persuasion strategy identification. The primary purpose of all brand communication is change people's ' beliefs and actions (i.e. to persuade). There has been limited work in computer vision on persuasion. Among the limited prior works, Bai et al. (2021) tried to answer the question of which image is more persua- sive; and Joo et al. (2014) introduced syntactical and intent features such as facial displays, gestures, emotion, and personality, which result in persuasive images. On the other hand, decoding persuasion in textual content has been extensively studied in natural language processing from both extractive and generative contexts (Habernal and Gurevych, 2016; Chen and Yang, 2021; Luu et al., 2019). All of the marketing messages employ one of set of strategies to persuade their target customers. Ku- mar et al. (2023) came up with exhaustive list of strategies used by brands to persuade consumers and also released an image-based ads dataset an- notated with those strategies. We build on that list and annotate video ads for persuasion strategics.
3.1 Video Verbalization
To obtain a verbal representation of a video, we employ a series of modules that extract unimodal information from the multimodal video. This infor- mation is then used to prompt a generative language model (such as GPT-3.5 (Brown et al., 2020) and Flan-t5 (Chung et al., 2022)) to generate coherent narrative from the video. The overall pipeline is depicted in Fig. In the following, we delve into each component of the framework in details.
1. Video Metadata: Understanding the context of story is crucial, and we achieve this by gather- ing information about the communicator (brand). We leverage the publicly available video title and channel name from the web. Additionally, we uti- lize Wikidata (Vrandečić and Krötzsch, 2014), a collaborative knowledge base that provides com- prehensive data for Wikipedia, to obtain further details such as the company name, product line, and description. This information helps us com- prehend the story elements and establish connec- tions with the brand's 's business context. For non- advertisement videos, we skip this step and retrieve only the video title.
Large Language Models on Reasoning: Large language models (LLMs) have been shown to con- fer a range of reasoning abilities, such as arithmetic (Lewkowycz et al., 2022), commonsense (Liu et al., 2022), and symbolic reasoning (Zhou et al., 2022), as the model parameters are scaled up (Brown et al., 2020). Notably, chain-of-thought (CoT) (Wei et al., 2022) leverages a series of intermediate reason- ing steps, achieving better reasoning performance on complex tasks. Building on this, Kojima et al. (2022) improved reasoning performance by simply adding "Let's think step by step" before each an- swer. Fu al. (2022) proposed generating more reasoning steps for the chain to achieve better per- formance. Zhang et al. (2022) proposed selecting examples of in-context automatically by cluster- ing without the need for manual writing. Despite the remarkable performance of LLMs in textual reasoning, their reasoning capabilities video- understanding tasks are still limited.
2. Text Representation of Video Frames: We extract two types of textual information from video frames. Firstly, we capture the literal text present on the frames. Secondly, we analyze the scene de- picted in each frame to gain deeper understanding. In the upcoming sections, we will elaborate on both of these aspects.
3 Method
a. Visual and Scenic Elements in Frames: For videos with a duration shorter than 120 seconds, we employ an optical flow- heuristic using the GMFlow model (Xu et al., 2022) to extract
Large Language Models (LLMs) have been demon- strated to perform well for downstream classifica-

---

Persuasion Strategies: Foot in the door. Guarantoes, Anchoring and Compansoon
WAGRANTY
WOTCH
Fixt tim siore that cums mind
RETM poice Nlu frae, le sote vnlado
Be Snniaed Sarrstii
Pos find faacch
den' walk until toe
Persuasion Strategies: Social Identity, Reciaracity
My Twvnttes
[MUSICI]
Selling Ast Del
dowilai stove yona
3ME porzsetf out there
It's elooy to be loen [MELSSC]]
Figure 3: Examples of videos with their annotated persuasion strategies, Relevant keyframes and ASR captions are shown in the figure, along with the annotated strategies. These two videos can be watched https://bit.ly: 3Ie3JG0, https: hypp:://bit.l//00//...
keyframes. In shorter advertisement videos, scene changes often indicate transitions in the story, re- sulting in keyframes with higher optical flow val- ues. The GMFlow model effectively captures these story transitions. We select frames with an optical flow greater than 50 and prioritize frames with max- imum pixel velocity. However, for longer videos, this approach yields a large number of frames that are difficult to accommodate within a limited con- text. To address this, we sample frames at a uni- form rate based on the native frames-per perss--pe-- (fps) of the video (see Table 11 for a comparison between uniform sampling and Pyscenedetect). Ad- ditionally, we discard frames that are completely dark white, they may have high optical flow but lack informative content.
to PP-OCR (Vrandečić and Krötzsch, 2014). We filter the OCR text and use only the unique words for further processing.
3. Text Representation of Audio: The next modal- ity we utilize from the video the audio content extracted from We employ an Automatic Speech Recognition (ASR) module to extract transcripts from the audio. Since the datasets we worked with involved YouTube videos, we utilized the YouTube API to extract the closed caption transcripts associ- ated with those videos.
Prompting: We employ the aforementioned modules to extract textual representations of var- ious modalities present in a video. This ensures that we capture the audio, visual, text, and outside knowledge aspects of the video. Once the raw text is collected and processed, we utilize it to prompt generative language model in order to generate herent story that represents the video. To optimize the prompting process and enable the generation of more detailed stories, we remove similar frame captions and optical character recognition (OCR) outputs, thereby reducing the overall prompt size.
Using either of these methods, we obtain a set of frames that represent the events in the video. These frames are then processed by pretrained BLIP-2 model (Li et al., 2023a). The BLIP model facili- tates scene understanding and verbalizes the scene by capturing its most salient aspects. We utilize two different prompts to extract salient informa- tion from the frames. The first prompt, "Caption this image," is used to generate a caption that de- scribes what is happening in the image, providing understanding of the scene. The second prompt, " "an you tell the objects that are present in the im- age?," helps identify and gather information about the objects depicted each frame.
The prompt template is given in Appendix A.1.2. Through experimentation, we discovered that using concise, succinct instructions and appending the text input signals (such as frame captions, OCR. and automatic speech recognition) at the end sig- nificantly enhances the quality of video story gen- eration. For shorter videos (up to 120 seconds), we utilize all available information to prompt the LLM for story generation. However, for longer videos. limit the prompts to closed captions and sam- pled frame captions. The entire prompting pipeline zero-shot and relies on pre-trained L.MMs. our story generation experiments, we employ GPT-3.5 (Brown et al., 2020), Flan-t5 (Chung et al., 2022),
b. Textual elements in frames: We also extract the textual information present in the frames, as text often reinforces the message present in scene and can also inform viewers on what expect next (Wang al., 2021). For the OCR module, we sam- ple every 10th frame extracted at the native frames- per-second of the video, and these frames are sent

---

and Vicuna (Chiang al., 2023). A temperature of 0.75 is used for LLM generation. The average length of the generated stories is 231.67 words. Subsequently, these generated stories are utilized for performing video understanding tasks.
Number Videos by Strategy
Number Videos
307
3.2 Downstream Video Understanding Tasks
122
For each downstream video understanding task, we explain the task in detail to the LLMs and provide the options to choose from. We adopt an in-context learning approach by using task descriptions as prompts This method enables the LLMs to ac- quire the skills necessary to solve the understanding task efficctively. The provided context encompasses comprehensive information about the classification task, including details about different classes, along with their corresponding definitions and concepts. By exposing the language model this context, becomes adept at selecting the correct option when presented with a generated text input.
Socol Iccenityy
Connone and compatioon
anceccomnn Reactance
Uncless
Secipecty
yoot in the ttor
Aution
social Imgald
Scaroty
social panl
Persuasion Strategy
Figure 4: Distribution of persuasion strategies in our dataset
attribute prediction. These diverse datasets allow us to evaluate the performance and capabilities our framework thoroughly.
In the story generation pipeline illustrated in Fig. 2, we leverage the stories derived from videos and employ separate prompting systems classify tags for downstream tasks. Each task operates in- dependently, ensuring that the context of one task does not interfere with another. Here too, we utilize three generative language models, namely GPT-3.5, Flan-t5, and Vicuna. In these experiments, we em- ploy a lower temperature setting of 0.3. Our per- formance is evaluated against various fine-tuned models from the literature. Moreover, we utilize our generated stories and ground truth labels to perform fine-tuning on Roberta model for var- ious tasks. Both Vicuna and GPT-3.5 generated stories are used to train the Roberta model. This allows us to compare the performance of models fine-tuned on video data with those trained solely on video-verbalized-rex data.
1. The Video story dataset (Li et al., 2020) con- tains 105 videos, from four types of common and complex events (i.e. birthday, camping, Christmas, and wedding) and corresponding stories written by annotators. has longer videos (average length 12.4 mins) and longer descriptions (162.6 words on average). Moreover, the sentences the dataset are more sparsely distributed across the video (55.77 sec per sentence). Metrics: Following Li et al. (2020). we use several NLP metrics, viz., BLEU- N, ROUGE-L, METEOR and CIDEr to measure the similarity between the story generated by the model and ground truth.
2. The Image and Video Advertisements (Hus- sain et al., 2017) contains 3,477 video advertise- ments and the corresponding annotations for emo- tion and topic tags and action-reason statements for each video. There are a total of 38 topics and 30 unique emotion tags per video. Further, we have 5 action-reason statements for each video for the action-reason generation task. For our experiment, we use 1785 videos, due to other videos being un- available/privated from Youtube.
4 Evaluation and Results
4.1 Datasets
To test the effectiveness of our framework, we con- duct experiments involving fifteen distinct tasks across five datasets. Firstly, we employ a video story dataset to evaluate the story generation task. Secondly, utilize vidco advertisements dataset assess topic and emotion classification, well as action and reason generation. Then, the persuasion strategy dataset to evaluate the task of understand- ing persuasion strategies within stories, and finally, HVU and LVU for concept, user engagement, and
Metrics: Following Hussain et al. (2017), for the topic and emotion classification task, we evaluate our pipeline using top-1 accuracy the evalua- tion metric. Further, since Hussain al. (2017) did not use any fixed set of vocabulary for anno- tations, rather they relied on annotator-provided labels, the labels are often very close (like cheerful, excited, and happy). Therefore, based on near-

---

Mathod
Model Type METEOR
CIDEr
ROUGEE-
BLBHU-1
BLRUU-
BLEL--3
BLEU-4
Random
Random
Retrivval
13.1
50.2
45.1
23.1
Finetumed
Narmttx (Ia al., 20220)
Rarneval
19.6
98.4
29.5
69.1
43.11
25.3
15.0
EMR (1a al., 2020)
Retrivall
19.1
88.8
64.5
BRNN (1. 2020)
Ratrieval
18.1
81.0
28.3
51.4
36.6
20.3
11.3
ResBRNN al., 2020)
Retrieval
19.6
94.3
29.7
66.0
41.7
24.3
Pscudo-GII+
Retrieval
20.1
103.6
29.9
69.1
43.5
26,1
15.6
ResBRNN-ANN
GVMF (Lu and Wa, 2022)
Retrieval
20.7
107.7
30.8
70.5
44.3
26.9
1..9
(Li al., 20236)
Generative
15:49
42.9
Zero-sho
GPT-3.5
Generative
24.8
102.4
24.3
63.8
56.4
11..2
38.6
Our Pmmework
Vicuna
Genentive
17.4
73.9
20.9
70.49
60.00
48.25
33.20
Flant-5-op
Genentive
4.8
34.6
Uniformly Sampled BLIP-2 Captions
Generative
21.7
108.9
24.04
55.19
18.5
10.7
33.76
Table Comparison on story generation task on the video-story dataset. We see that our framework despite being zero-shot outperforms all the fine-tuned generative prior art on all metrics. Further, it also outperforms fine-tuned retrieval models, which choose from a fixed set of frame descriptions on most metrics. Best models are denoted in green and runner-ups blue
Topic
Action
Reuson
Training
Model
All labels
Clubhed
Random
Random
2.63
3,37
14.3
8.37
3,34
3.34
Finetuned
VideoMAE (Tong al., 2022)
24.72
29.72
85.55
11.17
Hussin (2017)
35.1
48.45 45
Intem-Video (Wang et al. 2022)
57.47
36.08
36.39
5.47
6.8
7.1
Zero-shoc
VideoChat ILi
9.07
5.1
Our ramewwrk
GPT33 Generssed Story GPT33 Classiter
51.6
11.88
79.69
35.02
06.27
59.59
Zero-shox
GPT:3.5 Genented Story Flan-1--xx Chassifier
60.5
10.8
79.10
79 22
81.72
GPE33 Generused Story Vicuna Clasaller
22.92
10.8
67.35
29.6
21.39
30.99
Generated Story + GPT-3.5 Classifier
46.7
Vicuna Generated Story + Flan-15-xxl Clissilier
57.38
9.8
76.60
50.11
77. 3M
60.66
Generated Story + Clasifier
20.72
21.00
Finctuned
Geverated Story Roberta Classifer
21.3
33.02
84.20
64.67
42.96'
39.09!
Table 2: Comparison of all the models across topic, emotion, and persuasion strategy detection tasks. We see that our framework, despite being zero-shot, outperforms finetuned video-based models on the topic classification. persuasion strategy detection and action and reason classification tasks and comes close the emotion classification task. Further, the Roberta classifier trained on generated stories outperforms both finetuned and zero-shot models on most tasks. Best models are denoted in green and runner-ups blue.
ness in Plutchik (1980) wheel of emotions, we club nearby emotions and these seven main cate- gories: joy, trust, fear, anger, disgust, anticipation, and unclear. For the action-reason task, following Hussain et al. (2017), we evaluate our accuracy on the action and reason retrieval tasks where 29 ran- dom options along with 1 ground truth are provided to the model to find which one is the ground truth. Further, we also generate action and reason state- ments and evaluate the generation's 's faithfulness with the ground truth using metrics like ROUGE, BLEU, CIDEr, and METEOR.
sion strategy set: Social Identity, Concreteness, An- choring and Comparison, Overcoming Reactance, Reciprocity, Foot-in-the-Door, Authority, Social Im- pact, Anthropomorphismn, Scarcity, Social Proof, and Unclear. In order to make the class labels cas- ier to understand for non-expert human annotators, we make list of 15 yes/no type-questions contain- ing questions like "Was there any expert (person or company) (not celebrity) encouraging to use the product/brand? Was the company showcasing any awards (e.g.. industrial or government)? Did the video show any customer reviews or testimonials?" (complete list in Appendix:Tabl Table 6).
3. Persuasion strategy dataset: Further, we con- tribute an important task on advertisement story un- derstanding task, namely persuasion strategy iden- tification. Fig. shows few examples from the curated dataset. For this task, collected 2203 video advertisements from popular brands avail- able on the web publicly and use the persuasion strategy labels defined by Kumar et al. (2023). We use the following 12 strategies as our target persua-
Each human annotator watches 15 videos such that each video gets viewed by at least two anno- tators and answers these questions for each video. Based all the responses for video, we assign labels to that video. We remove videos with an inter-annotator score of less than 60%. After re- moving those, we get a dataset with 1002 videos, with an average length of 33 secs and a distribution

---

Task
Model
METEOR
CIDEr
ROUGE:L
BLEU-1
BLEU-2
BLEU-3
BLEU-4
Action
GPT-3.5
20.46
41.7
18.7
14.8
9.4
Flan-15-xd
61.5
50.0
21.8
42.6
16.8
13.08
10.08
Reason
GPT-3.5
13.34
16.7
7.8
27.1
20.8
14.7
10.4
Reason
Hlan-t5-xll
8.35
24.9
5.9
39.4
24.7
16.7
12.0
Reuson
Vicuna
15.82
27.9
7.75
24.6
19.3
14.1
10.3
Reason given action
GPT-3.5
13.77
29.4
8.7
33.5
24.9
17.9
13.2
Reason given action
Flan-(5xxl
4.29
19.0
7.6
23.2
15.0
10.2
7.5
Reason given action
Vicuna
13.62
24.4
7.61
22.6
17.7
12.8
9.2
Table Comparison of the different zero-shot models on the action reason generation tasks. Note that there are fine-tuned generative models in the literature for this task and the number of annotated videos is too small to train a generative model. Best models are denoted in green
as shown in Fig. 4. This dataset is then used for the persuasion strategy identification task. Metrics: We evaluate the performance using top-1 accuracy metric. Videos have varied number of strategies, therefore, we consider a response to be correct if the predicted strategy is present among the list of ground-truit strategies.
erative, the ROUGE-L score is lower compared to retrieval-based methods due to less overlap with ground truth reference video stories. However, overall metrics indicate that our generated stories exhibit higher level of similarity to the reference stories and effectively capture the meaning of the source video.
Long-Foree Form Video Understanding (LVU): Wu and Krahenbuhl (2021) released a benchmark comprising of 9 diverse tasks for long video under- standing and consisting of over 1000 hours of video. The various tasks consist of content understand- ing ("relationship'. 'speaking style', "scenc/place" ) user engagement prediction ('YouTube like ratio', "YouTube popularity"). ") and movie metadata predic- tion ('director', 'genre", 'writer', 'movie release year"). Wu and Krahenbuhl (2021) use top-1 clas- sification accuracy for content understanding and metadata prediction tasks and MSE for user en- gagement prediction tasks.
The performance com- parison between our pipeline and other existing methods across six tasks (topic, emotion, and per- suasion strategy classification, as well as action and reason retrieval and generation) is presented in Tables 2 and 3. Notably, our zero-shot model outperforms finetuned video-based baselines in all tasks except emotion classification. Further, our text-based finetuned model outperforms all other baselines on most of the tasks.
Video Understanding:
Unlike the story generation task, there are lim- ited baselines available for video understanding tasks. Moreover, insufficient samples hinder train- ing models from seratch. To address this, we utilize state-of-the-arr vidco understanding models, Vidco- MAE and Intern Video. Intern Video shows strong performance on many downstream tasks. Analyz- ing the results, we observe that while GPT-3.5 and Vicuna perform similarly for story generation (Ta- ble 1), GPT-3.5 and Flan-t5 excel in downstream tasks (Table 2). Interestingly, although GPT-3.5 and Vieuna-generater stories yield comparable re- sults, GPT-3.5 exhibits higher performance across most tasks. Vicuna-generated stories closely follow GPT-3.5 5 in terms of downstream task performance.
5. Holistic Video Understanding (HVU): HVU (Diba et al., 2020) is the largest long video under- standing dataset consisting of 476k, 31 and 65k samples in train, val, and test sets, respectively. A comprehensive spectrum includes the identification various semantic elements within vidcos, con- sisting of classifications scenes, objects, actions, events, attributes, and concepts. To measure perfor- mance on HVU tasks, similar to the original paper, we use the mean average precision (mAP) metric on the validation set.
4.2 Results
Video Storytelling:
The performance comparison between our pipeline and existing methods is pre- sented Table 1. We evaluate multiple generative and retrieval-based approaches and find that our pipeline achieves state-of-te of-te--ffthh---ee the-art art results. It is im- portant to note that as our method is entirely gen-
Next, compare the best models (as in Table 2) on the LVU and HVU benchmarks with respect to the state-of-the-arr models reported in the lit- erature. Tables 4 and report the results for the comparisons. As can be noted, the zero-shot mod- els outperform most other baselines. For LVU,

---

Training
Medel
relationhip
way. .spcaking
secoe
like_rate
view_coont
view.
ccantt
director
genre
writer
yeur
Trainal
R101-slewfist-NG NN and Krahenbuhl, 2021)
52.4
35.8
54.7
0.386
13.77
53.0
Trainal
Vileolen (Son al,, 2019)
37.9
54.9
0.320
51.9
38.5
Trained
Xiso et al. (2022)
50.95
34,07
44,19
0.353
4.3%5
40.19
48.11
31.43
29.65
Trainol
(2021)
$0.95
32.56
0.444
37.76
48.17
27.06
25.31
Trainol
Objes Tranformers (Wo and Krahenbubi, 2023)
53.1
39.4
36.9
0.250
3.55
51.2
54.6
34.5
39.1
shot (Ouns)
gonerated story
39,07
80.2
58.1
52.4
Zero-sbo (Ouns)
GPT-35 generated story GPT-355 classiffer
68.42
32.95
$4.54
0.031
12.09
75.26
50.84
32.16
75.96
(Ours)
GPT-3.5 generated story
38.41
0.054
39.27
35.98
Table 4: Comparison of various models on the LVU benchmark. We see that our framework, despite being zero-shot, outperforms fine-tuned video-based models on 8/9 tasks. Best models are denoted green and runner-ups in
blue
Training
Model
Scene
Object
Action
Event
Attribate
Coucept
Overall
Trained
3D-Resnet
50.6
28.6
48.2
35.9
22.5
35.8
Trained
3D-STCNet
51.9
30.1
50.3
35.8
29.9
22.7
36.7
Trained
HATNet
55.8
34,2
51.8
38.5
33.6
26.1
40
Trained
3D-Resnct (Multitask)
51.7
29.6
48.9
36.6
24.1
37
Trained
HATNet (Multitas))
57.2
35.1
53.5
39.8
34,9
27.3
41.3
Zero-Shar Shot (Ours)
GPT-3.5 generated story + Flan-t5-xxl xxl classifier
59.66
98.89
98.96
38.42
67.76
86,99
75.12
Zero-Shot (Ours)
GPT-3.5 generated story GPT-3.5 classifier
60.2
99.16
98.72
40.79
67.17
88.6
75.77
Table 5: Comparison of various models the HVU benchmark (Diba al., 2020).
The models scores are as reported in Diba et al. (2020). We see that our framework, despite being zero-shot, outperforms fine-tuned video-based models on all the tasks. Best models are denoted green and runner-ups blue.
the zero-shot models work better than the trained Roberta-based classifier model. For HVU, we con- vert the classification task to retrieval task, where in a zero-shot way, we input the verbalization of a video along with 30 randomly chosen tags con- taining an equal number of tags for each category (scene, object, action, event, attribute, and concept). The model is then prompted to pick the top 5 tags that seem most relevant to the video. These tags are mapped back the main category tags, which are treated as the predicted labels.
overcome these challenges, we propose leverag- ing the advancements in large language models (LLMs) within the field of natural language pro- cessing (NLP). LLMs have shown remarkable zero- shot accuracy in various text-understanding tasks. Thus, our approach involves verbalizing videos to generate stories and performing video understand- ing on these generated stories in a zero-shot manner. We utilize signals from different modalities, such automatic speech recognition, visual scene descrip- tions, company names, and scene optical character recognition, to prompt the LLMs and generate co- herent stories. Subsequently, we employ these sto- ries to understand video content by providing task explanations and options. Our proposed method demonstrates its effectiveness across fifteen video- understanding tasks getting state-of-ther the-ttr-rt results across many of them. The entire pipeline operates in a zero-shot manner, eliminating the reliance on dataset size and annotation quality.
Furthermore, as a comparative and ablation study of our approach, we evaluate the performance using only the BLIP-2 captions and audio transerip- tions (Table 8). Our findings highlight that gener- ated stories leveraging both audio and visual sig- nals outperform those using vision or audio inputs alone. This emphasizes the significance of verbal- izing a video in enhancing video understanding.
5 Conclusion
Acknowledgement: This work is partially sup- ported by NSF AI Institute-2229873, NSF RI- 2223292,, an Amazon research award, a Google fellowship, and Adobe gift fund. Any opin- ions, findings, conclusions. recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Na- tional Science Foundation, the Institute of Educa- tion Sciences, or the U.S. Department of Education.
In this study, we delve into the challenge of under- standing long videos. These videos, including ad- vertisements and documentaries, encompass wide range of creative and multimodal elements, such text, music, dialogues, visual scenery, emotions, and symbolism. However, the availability of anno- tated benchmark datasets for training models from scratch is limited, posing significant obstacle. To

---

6 Limitations
Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling cntruuctionfinetune language models.
In this work, we showed that verbalizing videos results in better performance on downstream tasks when compared to finetuned video based computer vision models. We verbalized vidcos using video transcripts, scene captions, optical character recog- nition, and other techniques. The overall perfor- mance of the pipeline intricately linked to the out- put generated to its constituent components. How- ever, it is important to acknowledge the potential for hallucinations arising from each component, which can introduce false information and imag- inative outputs. The final output's hallucinations and human values alignment are dependent on the underlying LLM, which need more research. We have tried give a few qualitative samples of the same in the Appendix.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Luc Van Gool, 2020. Large scale holistic video under- standing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23- 28, 2020, Proceedings, Part V 16, pages 593-610. Springer.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-bane baseed prompt- ing for multi-step reasoning. arXiv preprint arXiv:2210.00720.
Bhagyashree Gaikwad, Bhargav Kurma, Manasi Pat- wardhan, Shirish Karande, and Niranjan Pedanekar. 2022. Can pretrained language model make sense with pretrained neural extractors? an application to multimodal classification. De-Factify- Workshop on Multimodal Fact Checking and Hate Speech Detec- tion, AAAI.
References
Chongyang Bai, Haipeng Chen, Srijan Kumar, Jure Leskovec, and VS Subrahmanian. 2021. M2p2: Mul- timodal persuasion prediction using adaptive fusion.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. 2017. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages
IEEE Transactions on Multimedia.
Breakthrough. 2023. Pyscenedetect: Video scene cut detection and analysis tool.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are shw-shot learners.
Ivan Habernal and Iryna Gurevych. 2016. What makes convincing argument? empirical analysis and de- tecting attributes of convincingness in web argumen- tation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Zaeem Hussain, Mingda Zhang. Xiaozhong Zhang. Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. 2017. Automatic un- derstanding of image and video advertisements.
Jiaao Chen and Diyi Yang. 2021.
hierarchical models for predicting persuasive strate- gies in good-faith textual requests. Proceedings of the AAA/ Conference on Artificial Intelligence, (14).
Jungseock Joo, Weixin Li, Francis F Steen, and Song- Chun Zhu. 2014. Visual persuasion: Inferring com- municative intents of images. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 4 with 90%* chatgpt quality.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang. Chloe Hillier, Sudheendra Vijayanaraiimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint
Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- hert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gauray Mishra, Adams
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.1199

---

Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generat- ing descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 317-325.
Aditya Mogadala. Marimuthu Kalimuthu, and Dietrich Klakow. 2021. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. Journal of Artificial Intelligence Research, 71:1183-1317.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE in- ternational conference on computer vision, pages 706-715.
Stephen Pilli, Manasi Patwardhan, Niranjan Pedanekar, and Shirish Karande. 2020. Predicting sentiments in image advertisements using semantic relations among sentiment labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- tern Recognition Workshops, pages 408-409.
Yaman Kumar, Rajat Jha, Arunim Gupta, Milan Ag- garwal, Aditya Garg, Tushar Malyan, Ayush Bhard- waj. Rajiv Ratn Shah, Balaji Krishnamurthy, and Changyou Chen. 2023. Persuasion strategies in ad- vertisements. Proceedings of the AAAT Conference on Artificial Intelligence.
Robert Plutchik. 1980. Chapter 1 a general psycho- evolutionary theory of emotion. In Robert Plutchik and Henry Kellerman, editors, Theories of Emotion. pages 3-33. Academic Press.
Rui Qian, Tianjian Meng. Boqing Gong, Ming- -suan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. 2021. Spatiotemporal contrastive video representa- tion learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pages 6964-6974.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.1 748588
Andrey Savchenko, Anton Alekseev, Sejeong Kwon, Elena Tutubalina, Evgeny Myasnikov, and Sergey Nikolenko. 2020. Ad lingua: Text classification improves symbolism prediction in image advertise- ments. In Proceedings of the 28th International Con- ference on Computational Linguisties, pages 1886- 1892.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2020. Video storytelling: Textual sum- maries for events. IEEE Transactions on Multimedia, 22(2):554-565.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Mur- phy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learn- ing. In Proceedings of the IEEF/CVP international conference computer vision, pages 7464-7473.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video un- derstanding.
Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training.
Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. 2017. Recurrent tpacc-rransition gan for visual paragraph generation. In Proceedings of the IEFE international conference on computer vision, pages 3362-3371.
Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledgebase. Commum. ACM, 57(10):78-85.
Jiacheng Liu. Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022. Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:2210.03078.
Jing Wang, Jinhui Tang, Mingkun Yang. Xiang Bai, and Jiebo Luo. 2021. Improving ocr-based image captioning by incorporating geometrical relationship. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1306-1315.
Youwei Lu and Xiaoyu Wu. 2022. Video storytelling based on gated video memorability filtering. Elec- tronics Letters, 58(15):576-778
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models.
Kelvin Luu, Chenhao Tan, and Noah A Smith. 2019. Measuring online dehaters' persuasive skill from text over time. Transactions of the Association for Com- putational Linguistics, 7:537-550.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. Sen Xing, Guo Chen, Junting Pan, Ji- ashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022. Internvideo: General video foundation mod- els via generative and discriminative learning. arXiv preprint 22X.v:2212.03191.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in in-context learning work? arXiv preprint arX: 2ati:2222..122..3. 12837.

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans. Olivier Bousquet, Quoe Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint 12X6v:2005.0062..
Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903.
Chao- Yuan Wu and Philipp Krahenbuhl. 2021. Towards long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1884-1894.
Fanyi Xiao, Kaustav Kundu, Joseph Tighe, and Davide Modolo. 2022. Hierarchical self-supervised repre- sentation learning for movie understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9727-9736.
Haofei Xu, Jing Zhang. Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. 2022. Gmflow: Learning optical flow via global matching.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettle- moyer, and Christoph Feichtenhofer. 2021. Video- clip: Contrastive pre- pre-training training for zero-shot video-text understanding. arXiv preprint arXiv:2109 14084.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual atten- tion. In International conference on machine learn- ing, pages 2048-2000 1018-205.1110 PMLR.
Keren Ye and Adriana Kovashka. 2018. Advise: Sym- bolism and external knowledge decoding adver- tisements. In Proceedings of the European Confer- Computer Vision (ECCV), pages 837-855.
Keren Ye, Narges Honarvar Nazari, James Hahn, Za- cem Hussain, Mingda Zhang, and Adriana Kovashka. 2019. Interpreting the rhetoric of visual advertise- ments. IEEE transactions on pattern analysis and machine intelligence, (3(4):3388-122
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yonghin 2023. Large language models are versatile decomposers: Decompose evi- dence and questions for table- based reasoning, arXiv prepriat arXiv:2301.13808.
Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016. Video paragraph captioning using hierarchical recurrent neural networks. Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4593.
Mingda Zhang, Rehecca Hwa, Adriana Kovashka. 2018. Equal but not the same: Understanding the implicit relationship between persuasive images and text. arXiv preprint arXiv: (ot::080020000 08205.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing large language models. arXiv preprint arXiv:2210.03493.

---

A Appendix
exclude any empty or stop words from the final text." "
A.1 Experimentation Details
For downstream tasks, template prompt format with instruction about the specific ths previ- ous generated verbalization and vocabulary for the downstream task is prompted to the LLM. Here is the example for the topic detection lask, for other tasks context and vocab were changed accordingly.
Video and Brands selection We covered 135 For- tune 500 brands, covering I 113 industries, the ads spanned the years 2008-2022. The videos have an avg duration of 33 secs. Nearly 45% of the videos have audio in them. The collected advertisement videos have variety of characteristics, including different scene velocities, human presence and an- imations, visual and audio branding, a variety of emotions, visual and scene complexity, and audio types. The total number of unique annotators is 484. Annotators were university students who were given guidelines to annotate videos and were duly compensated for their work based on local rates. We use the following 12 strategies persuasion strategy set: Social Identity, Concrete- ness, Anchoring and Comparison, Overcoming Re- actance, Reciprocity, thot--n-the-Door, Authority, Social Impact, Anthropomorphismn Scarcity, Social Proof, and Unclear. In order to make the class la- bels easier to understand for non-expert human an- notators, we make a list of 15 yes/no type-questions containing questions that are mentioned below.
"Given [topics) identify the most relevant topic from the dictionary keys from topic_ vocab related to the story of the video advertisement given be- low. Consider the definitions given with topics in the topic_vocal vocab dictionary, to identify which topic is most relevant, don't It add any extra topics that are not given in dictionary keys and answer with just the most relevant topic. Story: : (verbalization]"
target
A.2 A few examples of the stories generated using our method
https: http:::/www..ytuu//e com/watch?v= 1PdD8NwVfw0: Kathy Ames had always wanted to pursue a doctoral degree but was unsure about the time commitment. When she discovered Grand Canyon University, she knew she had found the perfect fit. Grand Canyon University offered a flexible schedule that would allow her to balance her personal and family life with her studies, She along with other students gathered in the classroom, excitedly listening to their coach, Scott Saunders, explain the program. Afterward, Kathy made her way to the library and settled into a chair with her laptop.
1.
A.1.1 Sampling Rate
The frames for longer videos (>2 mins), every 10th frame the native fps of the video is sampled and similar captions are deduplicated to reduce the context size for Prompting. Refer Table 11 for a comparison between uniform sampling and Pyscenedetect.
She studied diligently, surrounded by her peers and classmates. In the evenings, she met with her peers around the table to discuss the topics of the day. Everyone was always eager to help and sup- port each other. After a long day, Kathy made her way back to her living room where she relaxed on the couch with glass of water and lamp provid- ing soothing light.
A.1.2 Prompt format
For verbalization, a template prompt format has been used, including all the data components as objects, captions, ast, oct, meta-data.
"Please write a coherent story based on the fol- lowing video advertisement. Use only the informa- tion provided and make sure the story feels like continuous narrative and at the end include one sentence about what product the advertisement was about. Do not include any details not mentioned in the prompt. Use the elements given below to create a coherent narrative, but don't 1 use them as it iseThe advertisement for the company fcompany | The video titled [title), with captions that include [caption], voice-over: [transcripts), and object recognition descriptions: : [ocr). The following objects are present in the advertisement and should be used to help create the story: (objects] Please
Kathy was grateful for the opportunity to pursue her dream at Grand Canyon University. She was able to learn from experienced faculty and gain real-world experience that would prepare her for success after graduation.
The advertisement for Grand Canyon University was about offering private, Christian education at an affordable price.
2. https: http//wwwottttt.
com/watch?v=f. 6QQ6IVa6E: The woman holding the book stepped onto the patio and looked up to the sky. She was

---

Question
Strategy Authority
Question
Strategy
Was there any expert (person or company) (not celebrity) encour- aging to use the prod- uct/brand?
Did the video show any normal customers (non- expert, non-celebrity) using the product?
Social Identity
Did the video show- case any awards or long usage history of the product/brand?
Authority
Did the video show any customer reviews or testimonials?
Social Proof
Was the product/brand comparing itself with other competitors or existing solutions?
Anchoring and Com- parison
Were any num ber/statisties men-
Concreteness
tioned?
Did the video talk about any specific fea- tures or provide infor- mation about the prod- uct/brand?
Concreteness
Were there any men- tion of any offers on the brand/product?
Reciprocity
Were the offers limited available for a short period of time?
Scarcity
Was the product/brand told to be free or avail- able on a discount?
Foot in the Door, Reci- procity
Was the brand/product described as simple, easy-to-use, start using with minimal re- sistance?
Overcoming tance, Foot in the Door
Reac-
Was the brand/product talking about bigger so- cietal impact?
Social Impact
Did the brand provide any guarantees which might help reduce risk of people to try out the product?
Overcoming
Reac-
Did the video provide any resources, tips, guides, or tools related to the product?
Reciprocity
tance
When brand prod- uet is portrayed as human-like?
Anthropomorphism
Table The questions we asked to non-expert annotators to help them identify persuasion strategy contained the video advertisement.

---

things
tint.
ULIA
ULTA
ULTA
ULTA
[MUSIC]
The woman in the Ulta Beauty advertisement skillfully applied makeup face, creating a beautiful pastel tint. With each brush stroke, her look became brighter and more dreamy. She beamed with confidence and her face glowing with beauly. the advertisement continued, she highlighted her eyes and cheeks with subtle pastel shades, adding to the stunning effect. With each addilion, face up vibrant luminosity. and dreamy.. The Ulta Beauty advertisement showcased the power of perfect pastel tint, and how it can enhance any look. This advertisement was for Ulta Beauty.
was finished, her face was work
perfect combination bright
FROMAACLE 02 CHAMPIONSHI
NO CARDS
NO POINTS NO NONSENSE
MUSIC]
I' profes Cee siooall Pornula Drift driver, 3yan Tuerok
check out. Ch.app wwn.hi..p...............oo...... in Atlanta
Ryan Tuerck is Red Bull Racing F1 driver, and he's lired of the nonsense that comes with earning rewards. That's why he loves Advance Auto Parts' Speed Perks Rewards there's s no points, no cards, just no nonsense. Through Speed Perks, Ryan can get the parts he needs for his race car quickly, easily and without the hassle of Iraditonal rewards programs. With Speed Perks, Ryan can get the parts he needs to keep his car running fast and Auto Parts' Speed Perks Rewards gives Ryan the freedom lo focus on what he does best racing. This advertisement was for Advance Auto Parts' Speed Parks Rewards.
ahead of the competition. With no cards, no points and no nonsense, Advance
Figure 5: A few examples of stories generated with the proposed pipeline. ASR outputs are shown in Red. As can seen, generated stories capture information about scenes, characters, event sequences, dialogues, emotions, and the environment. This helps the downstream models to get adequate information about the video to reason about correctly. These videos can be watched at https: //tpss:///ittt/ http///t...../2///2/..... https://bitt http//.t.......eeeee 1y/3pG.RRr.
ready to take the day. Taking out her phone, she opened the furniture catalog app, scrolling through the various designs. She quickly decided on the perfect pieces to brighten up her home. Next, she headed to The Home Depot for the supplies she needed. As she entered the store, the woman was delighted to find all the tools and materials she needed, from the Ryobi Cordless Vacuum to the Leaf Blower. She was spotted the Splatter an object recognition tool that allowed her easily find the perfect paint color for her project. With her shopping done, the woman made her way to the checkout line with a cup of coffee in hand. She couldn't wait to get to work and make her home more beautiful. She knew that with the help of The Home Depot, Today was the Day for Doing. This advertisement was for The Home Depot- the me--stop-hopp for all your home improvement needs.
up phone with the Sherwin -illiimms logo the screen. A girl appears from behind white sheet, peeking out of a white tent if signify the timelessness and neutrality of this color. The voice-over begins, as the camera pans to a living room with a staircase, and then to a dining room with a white table, chairs, and white vase. The words "Color of the Month: Shell White, Sherwin- Williams" appear the screen, as the camera zooms in the vase. The words are followed by Öur app makes it a snap,teferring to Color Snap, the company's new way of painting home. The advertisement ends with the Sherwin-Williams logo, emphasizing the company's ' commitment to excellence in home painting. This advertisement was promoting the company's 's color of the month, Shell White.
more excited when she
4. https://www/ youtube..uubbe..mmmoooo.mo... com/watch?v= CDjBIt70fp4: The story began with green light glowing in the dark, symbolizing the presence of powerful technology that can change the way we work. This technology was an advanced graphics card, the NVIDIA Quadro FX 1700. It was compared bide-by-side with its successor, the Quadro 2000, and it was clear that the Quadro
3. https: /ww.:////www///om http:////wwwyobubbe.../oo www.tbe...b............... com/watch?y= PJIHiQJBDMW: The advertisement for the company Sherwin-Williams opens on kitchen table strewn with shells and wicker baskets, with two glasses iced water beside them. A vase with blue pattern sits in the foreground, and person holds

---

2000 was far more powerful. The Quadro FX 1700 had a qt x700 vs qt x700 capacity, while the Quadro 2000 had a green light that shone brighter and further than before. As the comparison was being made, the results were clear: the Quadro 2000 was the superior product. This advertisement for the company NVIDIA showcased the power of the Quadro 2000, and the improved performance it could bring organization. The advertisement concluded with a green light, signaling that NVIDIA had the answer to improving workflows. The product the advertisement was about was the NVIDIA Quadro 2000."
differently while constructing the verbalization for the vidcos.For this experiement we use subset of (Hussain al., 2017) dataset, considering videos that have spoken audio present.
We use ROUGE-I to get the longest common sub- sequence (LCS) between the generated verbaliza- tion and the individual components, which captures the overlapping content, providing an indication of their semantic similarity.
As generated verbalizations are abstractive as compared extractive, we also use cosine similar- ity between the Roberta embeddings of the gener- ated verbalization and the individual components.
We find that despite the order of the components in the prompt, the LLMs tend to utilize the audio components in the videos, in an extractive way.
youtube............. www..yyutube.comm/aatta com/watch?v= SXTSJC5NrLs: Mary Kom was training hard to make her mark as boxer, and she had nothing but her focus and determination to help her succeed. She was foree to be reckoned with in the boxing ring and her hard work was soon paying off. But she knew that she had to take her training to the next level if she wanted to win the boxing championship. That's when she decided to take her own advice and step out of her comfort zone. She headed out to the woods, determined to make the most of her time in nature.The woods were perfect place for Mary challenge herself physically. She was soon standing on the rocks a river, pushing her body to its limits with every movement. She was standing tall, her black top glinting in the sunlight, and her energy was palpable. The sun was setting, and it was the perfect time to relax and unwind.Finall she ended her day with a yoga routine, her body twisting and turning in beautiful display of strength and grace. It was day of hard work and fun, and left Mary feeling inspired and ready to take on anything. The video was an advertisement for the movie Mary Kom, released by Viacom18 Studios."
5.
http:///wwwwyuuub.e.mm
A.3 Hallucinations Present In the Automatically Generated Stories
The current large language models. do hallucinate produce imaginary outputs. However, they able to capture the gist of the videos. helping im- prove on downstream tasks. Some examples have been shown in Table-7.
A.4 Ablation
Among the different components of information input present in the prompt, the LLM utilizes them

---

Table 7: Hallucinations Present In the Automatically Generated Stories
TRI.
Story
Hollacinations
(C.o0:7//00tu022 01/ Japna75. 3
The womas the abveriseneut is standig frost of red backgoondd smiliag leightyy Sie and a mas stiig at sade look content, il liky ve just uime from a cnmossation The Neilis lopo is vaille, indasiing that dhe udvertsenens in foe the compparr and is producre The voice-over for the atvommmmmm iomodacee Selrn Gomer as the carr of the miler for poputar Toilight Suge. Her populaiit has made her perfect cloise pronote Netflc's products The voice over thas inforts vizwers that the company's straaiing
The may lalilly mullires "Sc" lesa Gomeer to "Tulligts Saya" The L1Me when contiso he- tween sareed entry lirkages.
usd mosilate. The uan and comus the aevertiacmnnt
atter's emaggies hase with
they hith conmaction, and comfortette The knowledge But they find Nedlis.
inforttan they friends Pnrtope dvee on indrresnding happy exch
od-entieneen ends with Netflic leating
Biy are zoing The mun plase was pointig Skgisd Caniig compterr and the man the beadset peopured sus game The man with lazalas was low wordd of guming. his finen Bying actes the keyboord as he tried achiove dominatins. Masrahill, the mon with givers waished intetty it the IKCi I'V ip the room The womon was playing a game PC garing. while the man with glasses and beodphotes was engrossed is pame of Espors, man a headsct played a game of PSA, while mudaa nas leci up comgater senon A gidl was playag a pane PC sauing while a. mas was looking it cumpater cas wal colorful lights The man weating glaeen and hestphonss was sitting in from of compent. paying gime of IK gaming The woman wearing headphones was plaing gume on conguter and the man with glases was snilling he played a game of PS4. A pasaen's a laad was labding janting unsuae, while lights and Accr Aupine 82 MI3 PC cac purple light
unscontanding
compasy's
catlo "Moer Aspic RT
MK2' "PSi" sae uul rola- mood the peompe and wen hal-
luclnoeed by the LLM
was plating ganz of Espatt Findll, conguter the video pasline The advertiseee for Skytech Gaming, comppny offering giming would Sksteck Gasting acrenaline, gaming pasiio and
with and
playing quitar with lapoope
the leat. Oac sanaariiin leplope beny
gamon, allowing
every
anticipation
1c bel 2bw ishiirs
A pouss gill la waking dhoogh le brakoom when she soices mus maning up stain lis strokers Sle intigued and fulines bit,, curionse
socae with a lal- lusiuated though non mestioond in the peompe
whire is going Amiing the top of stains finds a yourg man daading with backgrak. Acmes room she sees a get in dress andding from of minor The gir emile and lonts cortere At that moment the soice over of the Aoadeny Spotts Qundorss of can leard." "Aadeny Sports and Guidoons is the place to get all the bes, leards is thonsaasia of comfotaions for beck to school Right aufl, los priz, even day." Subkielly u baak cui cas be san lising in the dty and the get in the min rop is now starding in fror the nimor The man is wornng his portt and sesttre, and the shoee, rug and saeakers of the spartner are visBle. The gir stiles and looks coment. The some shifls the bedioon where the bed. draws, and mime procet, aling with sign fo Assalany Spettoseas This ahettianan for Acaukayy Sperts Outdeen was shout company's hack-m-schod collecton protuorn In showraadd how Academy price,
Dtastoons prosidos the nigh uneii
for school
by
Mony
sxinning vicon manty
enternses
lilg2
sunding red moon desent landicape and a live performanze Adan Sander Purple backgrounds with tte word "vortds" councyy
"Mdum Sandler"
is socistual with the unventioonantt casees the genertted wary to have the same association
many positilliss whils a bay plasing wilk a hall font of vosue shanes a ture peasiial uad bosudy this As the might falls mon conting on car and the Ultmmes. Then aloo plimpux of The Soats, Deettmon Oryboby. Sant Selya Knights of the Zedix. Edet Vs. Edes Canée and Tuesay, Aggettoalo, Sets Mazos, Goddilla, Casteronia, The Begaaning Buki. Canoo Buskers Rilskkan and Kasea, Scadan, and Ulmansia: Maysell. We uxe soady bos. leights of the redorr, a cheneter wakeng the water, mas in ennoy stra pf dooes, a mon comming may from the rime the maoe. mon seoring silt with green light eyes being the only thnng faat con be seen tinds flying in the sky, red and block bodkgmund with and light bats and ctales, tie difeerett wor te that posobiities tndicked aniss Sulikaly checal caarel foom tis wild capernss beanty ingondicsble the Afficas savama This adventsenen ultirately serves to promose the Ein, Africa Cuts. which the sway of chessuh fanily as they stive tu protett fhcir soepoctie agardd a variity of olotaclos wild.
bobling
uan (biig Be faze, sid with lon ladi, and onu lands. All of find home" and abvrttement corelulo with the Nattle lugo.
this cutminattos opail This ohreniiment Nertix capornes the itea the
/y V.IE
The
chase peneolos Dequiic preacesz anvvnisement
playing the yous wild" insertilerr to cupture the overall infoema. timn of the videis However, to presonce ASR and work-- knowledge of L.M LLM is abic to gasentt as solaceent story.
cestones, steyy
Dieney
The iereanie Liie Hoaely adveraennccil skillully applaal eroke, her look became brighter mon deasyy. She heamed with contidenz
ker lacs, creaing beastife pasne list With each leads her face Whan fuislaal, las face
The valco content mnses Howower izuiion toykk gl sdvetionaan.
bearty As the the effect. With yourr perfect poorl
veraa-
advertieneat continue dhe highlightdd eyes checks with subile shales,
perlisi combination syy lonk.
beight drnsmy. The
This enteteeeett was
Bearty
The To be/ vintooo
Jacak had bezu warting Juassr koop camings wisi wonnd lis feroes She hoce dicaning pallect pair she creri wear en specid cersems and ther would beng out hos her Uhe day budcis loppered come acroec an atvettemmnt for Coazo, which featred par of Round Brilion Dianced Hoop Eartins ldkx White Gold, She inmreditelly knew this was the
vitkor lse au autier tratacip- and mnmimaal clesal infoma- tinn. However the L.M halls
parles pair los he: The alretisenann leal satue-orer dus sail. "Thos casing quakle with glanner and sopiiitiazian" ladki was mesmeroon by the sporcls of the domonds and she was sae snyute win sew them would ahsa he taken in by there hoasty. lockce quickly decided to perchase the corings. She was are they would make the gerfect accesorry for quecill occadin l'ron dus sisles to famils sutletings. she lace tasse caniaga would make les lock simply staniin. The udventicncat fo the Cusico Rourd Briliint Diamedd Hoop Eariins 1441 White Goid prosed he the perfect choice for Incer. They only zakked boach of glamour and sophitication her wandmbb, but made her feel special when she wome them. The advertsement for the Whise Gold.
cinsto u atoey, uaisttaiing the meeane the adveriiement also atribures hals voice over which nut preseni the video.
compasy Cosco for paic Billiin Diamood Hoop Easings
Topic
Emotion
Persuasion
Action
Reason
Model
All labels
Clubbed
BLIP-2 Captions- + Flant-t5-xxl
32.2
7.4
43.11
32.1
52.98
76.26
BLIP-2 Captions + GPT-3.5
32.7
7.9
76.69
30.1
49.91
58.71
Audio Transcription + Flant-t5-xe
49.37
10.1
63.56
21.9
66.17
79.68
Audio Transcription + GPT-3.5
32.88
6.4
75.97
32.25
64.98
61.78
Table 8: Ablation study of using only visual (caption) or audio (transcripts) and LLMs for downstream tasks. It can be noted that the overall model does not perform as well (compared to Table 2) when using only audio or scene description without generating story.

---

Table 9: Comparison of factors contributing the verbalization
Model
Component
ROUGE-I
Cosine Similarity
GPT-3.5
Visual Captions
18.9
50.40
ASR
26.29
52.60
OCR
02.64
22.52
Vicuna
Visual Captions
15.44
48.23
ASR
21.15
48.75
OCR
02.63
22.52
Model
Top-5 Accuracy
mAP
VideoMAR
25.57
24.79
InternVides
7.477
15.62
GPT-3.5 Generated Story + GPT-3.5
34.2
27.53
Vicuna Generated Story GPT-3.5
31.54
27.24
GPT-3.5 Generated Story Flant5
37
27.96
Vicuna Generated Story Flant55
31.13
27.32
Table 10: Top-5 accuracy, and mAP for persuasion strategy detection task
Method
Frame Extraction
METEOR
CIDEr
Rougle-1
BLEU--
BLEU-2
BLEU--
BLEU4
GPT-3.5
Uniform Sampling
24.8
10244
24.3
63.8
47.2
GPT-3.5
Pyscenedetect
24.17
21.17
54,59
Table 1l:
Comparison Pyscenedetect (Breakthrough, 2023) with uniform sampling choosing video frames. Based on downstream performance, we that uniform sampling works better than Pyscenedetect


Reviews:
[{"review_id": "Y7TvfB0C83", "reviewer": "Reviewer_Jhzm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Keyframe sampling**\n\nWe agree that our current method of uniform sampling might discard a few important keyframes. Experimentally, we observe that:\n- There is a lot of information redundancy in the video: similar information is repeated through audio, video frames, OCR, etc. We find that removing video frames completely did not alter the performance on the downstream benchmarks significantly. The numbers w/o visual information are given in Table 4 (Ablation)\n- Uniform sampling captures most important information. Our sampling rate is 2-3 fps (depending on the native video fps). This might miss a few frames but it captures most important ones.\n- For shorter videos (with length than <120 seconds), we use GMFlow based sampling algorithm (Fig.2, Sec.3.1 Line 292).\n\nHowever, we agree with the reviewer’s new experiment. In the limited rebuttal time, we performed an additional experiment where we compare the verbalisations generated on Video storytelling dataset, we find that using pyscenedetect (https://github.com/Breakthrough/PySceneDetect) the scores were comparable.\n\n|Method|Frame Extraction|METEOR|CIDEr|Rouge-l|Bleu-1|Bleu-2|Bleu-3|Bleu-4|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|GPT-3.5|Uniform Sampling|24.8|102.4|24.3|63.8|56.4|47.2|38.6|\n|GPT-3.5|Pyscenedetect|24.17|67.8|21.17|54.59|49.05|41.54|33.88|\n\nWe will perform more experiments and put them in the camera-ready.\n\n**The reason behind the selection of BLIP-2 model over other VQA models:**\nFor our downstream tasks, we needed two main sources of visual information: captions and the objects present in the video. \nWe pose both the tasks as VQA (open-ended generation), Blip2 with Flant5 decoder beats all models such as OSCAR, VinVL, Flamingo. Ref: Table-3, Table-4 of BLIP-2:Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (https://arxiv.org/abs/2301.12597). \nSeeing the performance reported, we went with Blip2 as our choice of model, for extraction of visual information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper uses an original methodology in video understanding for the first time which pass other state of the art methods.\nThe paper is written in clear language.\nThe author did a through literature study and comparison of results with those studies.", "reasons_to_reject": "My concerns have been addressed in the rebuttal.", "questions_to_authors": "", "paper_topic_and_main_contributions": "They are leveraging the powerful understanding and reasoning capabilities of LLMs to generate a story from a usually long video which will be used to do video understanding tasks like: emotions depicted and persuasion strategies used.\nThere is a modality gap between videos consisting of images, audios, texts and etc... and text that is primary input of an LLM. They address this issue by converting the video to text which will be fed to an LLM to generate a story that is used in video understanding tasks instead of the raw video.\nWhen it comes to contributions they specifically did:\n1) Converting long videos from the multimodal domain to “small” coherent textual stories by verbalizing keyframes, audio, and text-overlaid scenes with the help of a powerful LLM and instructions. In their experiments, the story generation capability of their method outperforms state-of-the-art story generation methods\n2) Test the utility of generated stories by conducting extensive experiments on several benchmark datasets covering five video understanding tasks, namely topic, emotion and persuasion strategy classification, action and reason retrieval and generation. Experimental results demonstrate that their method achieve better results than both finetuned and zero-shot video understanding baseline models without using any human-annotated samples.\n3) For the first time in literature, they show that the essence of a highly-multimodal video can be represented in text while being informed through the different modalities like audio, raw pixels of frames, text overlaid on scenes, emotions, and product and business information.\n4) Release the first dataset for studying persuasion strategies in advertisement videos. This enables initial progress on the challenging task of automatically understanding the messaging strategies conveyed through video advertisements.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5fxuY37277", "reviewer": "Reviewer_5637", "summary": "", "questions": "* For **BLIP2** caption extraction, keyframe selection was used as a pre-filtering step. \nAs an alternative, have the authors explored shot detection followed by BLIP-2-based caption extraction for center frames in shots?\n\n* For the persuasion strategy dataset zero-shot results, it is not clearly mentioned that the evaluation was performed on the complete set of 1002 videos or a separate test split.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Performance on other datasets:**\n- Thanks for pointing us to the LVU and HVU datasets! In the limited rebuttal time, we were able to test out our best models on the following tasks across the two datasets. Similar to the tasks reported in the paper, we obtain significant improvements on them.\nWe observe that on 8/9 tasks in the LVU benchmark and 6/6 tasks in the HVU benchmark, we get state-of-the-art results.\nWe observe that on 8/9 tasks in the LVU benchmark and 6/6 tasks in the HVU benchmark, we get state-of-the-art results.\n\n\n1. LVU Benchmark (https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Towards_Long-Form_Video_Understanding_CVPR_2021_paper.pdf):\n\n| Training         | Model                                            | relationship | way_speaking | scene | like_ratio | view_count | director | genre | writer | year  |\n|------------------|--------------------------------------------------|--------------|--------------|-------|------------|------------|----------|-------|--------|-------|\n| Trained          | R101-slowfast+NL [ICCV,2019]                     | 52.4         | 35.8         | 54.7  | 0.386      | 3.77       | 44.9     | 53    | 36.3   | 52.5  |\n| Trained          | VideoBert, Sun et al, 2019, ICCV                 | 52.8         | 37.9         | 54.9  | 0.32       | 4.46       | 47.3     | 51.9  | 38.5   | 36.1  |\n| Trained          | Xiao et al, 2022, CVPR                           | 50.95        | 34.07        | 44.19 | 0.353      | 4.886      | 40.19    | 48.11 | 31.43  | 29.65 |\n| Trained          | CVRL, Qian et al, 2021, CVPR                     | 50.95        | 32.86        | 32.56 | 0.444      | 4.6        | 37.76    | 48.17 | 27.26  | 25.31 |\n| Trained          | Object Transformer, Wu et al, 2019, CVPT         | 53.1         | **39.4**         | 56.9  | 0.23       | **3.55**       | 51.2     | 54.6  | 34.5   | 39.1  |\n| Zero-shot (Ours) | GPT-3.5 generated story + Flant5-xxl [zero shot] | 64.1         | **39.07**        | 60.2  | 0.061      | 12.84      | 69.9     | **58.1**  | **52.4**   | 75.6  |\n| Zero-shot (Ours) | GPT-3.5 Generated story + GPT-3.5 classifier     | **68.42**        | 32.95        | 54.54 | **0.031**      | 12.69      | **75.26**    | 50.84 | 32.16  | **75.96** |\n| Trained (Ours)   | GPT-3.5 generated story + Roberta                | 62.16        | 38.41        | **68.65** | 0.054      | 11.84      | 45.34    | 39.27 | 35.93  | 7.826 |\n\n(like ratio and view count are MSE scores, (less is better), all others are top-1 accuracies, (more is better))\n\n\n2. HVU benchmark (https://pages.iai.uni-bonn.de/gall_juergen/download/HVU_eccv20.pdf)\n| Training  | Model                             | Scene | Object | Action | Event | Attribute | Concept | Overall |\n|-----------|-----------------------------------|-------|--------|--------|-------|-----------|---------|---------|\n| Trained   | 3D-Resnet, Diba et al, 2020, ECCV | 50.6  | 28.6   | 48.2   | 35.9  | 29        | 22.5    | 35.8    |\n| Trained   | 3D-STCNet, Diba et al, 2020, ECCV | 51.9  | 30.1   | 50.3   | 35.8  | 29.9      | 22.7    | 36.7    |\n| Trained   | HATNet, Diba et al, 2020, ECCV    | 55.8  | 34.2   | 51.8   | 38.5  | 33.6      | 26.1    | 40      |\n| Trained | 3D-Resnet, Diba et al, 2020, ECCV [Multitask] | 51.7 | 29.6 | 48.9 | 36.6 | 31.1 | 24.1 | 37   |\n| Trained | HATNet, Diba et al, 2020, ECCV [Multitask]    | 57.2 | 35.1 | 53.5 | **39.8** | 34.9 | 27.3 | 41.3 |\n| Zero-Shot (Ours)  | GPT-3.5+Flant5 | **59.66** | **98.89** | **98.96** | 38.42 | **67.76** | **86.99** | **75.12** |\n| Zero-Shot (Ours) | GPT-3.5+GPT3.5 | **60.2**  | **99.16** | **98.72** | **40.79** | **67.17** | **88.6**  | **75.77** |\n\n(All the tasks are MAP scores, (more is better))\n\n\n**Details regarding the persuasion strategy dataset:**\n- While we tried to accommodate the most important details in the 8 pages, we apologize for missing out on some key details. We will make sure to add these details in the final version. The details are:\n    - Video and Brands selection - 155 Fortune 500 brands, covering 113 industries, the ads spanned the years 2008-2022. The videos have an avg duration of 33 secs. Nearly 45% of the videos have audio in them. The collected advertisement videos have a variety of characteristics, including different scene velocities, human presence and animations, visual and audio branding, a variety of emotions, visual and scene complexity, and audio types. \n    - Unique annotators - 484\n    - Annotators were university students who were given guidelines to annotate videos and were duly compensated for their work based on local rates. \n    - Top-5 accuracy and MAP - \n| Model                            | Top-5 Accuracy | mAP   |\n|----------------------------------|----------------|-------|\n| VideoMAE                         | 25.57          | 24.79 |\n| InternVideo                      | 7.477          | 15.62 |\n| GPT-3.5 Generated Story+ GPT-3.5 | 34.2           | 27.53 |\n| Vicuna Generated Story+GPT-3.5   | 31.54          | 27.24 |\n| GPT-3.5 Generated Story+Flant5   | 37             | 27.96 |\n| Vicuna Generated Story+Flant5    | 31.13          | 27.32 |\n\n\n\n**Experimental details:**\n- Finetuning details of VideoMAE and InternVideo -\n    - VideoMAE : We finetune VideoMAE with an MLP head of n-way classification task (n depending on the number of labels). We divide each video into 13 non-overlapping parts, and sample 16 frames at 2fps (taken from the VideoMAE paper). Each part of the video is input to VideoMAE to get an embedding of (1,768). Each individual embedding is concatenated to form a (13,768) embedding, which becomes the representation of the video and is input to the MLP head. The MLP has a MaxPoolLayer and 2 fully connected Linear layers. We use ReLU activation to train the model.VideoMAE weights are frozen during training.\n\n    - InternVideo :  We finetune InternVideo with an MLP head of n-way classification task (n depending on the number of labels). The InternVideo returns a (1,768) embedding representation of the video. We use an MLP with 2 fully connected linear layers to finetune for the tasks. \n\n\n\n\n**Shot detection method**\n- We agree that our current method of uniform sampling might discard a few important keyframes. Experimentally, we observe that:\n    - There is a lot of information redundancy in the video: the same/complementary information is repeated through audio, video frames, OCR, etc. We find that removing video frames completely did not alter the performance on the downstream benchmarks significantly. The numbers w/o visual information are given in Table 4 (Ablation)\n    - Uniform sampling captures the most important information. Our sampling rate is 2-3 fps (depending on the native video fps). This might miss a few frames but it captures most important ones.\n    - For shorter videos (with length than <120 seconds), we use GMFlow-based sampling algorithm (Fig.2, Sec.3.1 Line 292).\n    - However, we agree with the reviewer’s new experiment. In the limited rebuttal time, we performed an additional experiment where we compare the verbalisations generated on Video storytelling dataset, we find that using pyscenedetect (https://github.com/Breakthrough/PySceneDetect) the scores were comparable. We will perform more experiments and put them in the camera-ready.\n|Method|Frame Extraction|METEOR|CIDEr|Rouge-l|Bleu-1|Bleu-2|Bleu-3|Bleu-4|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|GPT-3.5|Uniform Sampling|24.8|102.4|24.3|63.8|56.4|47.2|38.6|\n|GPT-3.5|Pyscenedetect|24.17|67.8|21.17|54.59|49.05|41.54|33.88|\n\n\n\n\n**Persuasion strategy dataset zero-shot results, test settings**\n- For zero shot results, we perform evaluation on 1002 videos. For finetuning, we perform evaluation on a test split of 198 videos. The same test-split is used for VideoMAE, InternVideo and Roberta Classifier.\n\n\n**Hallucination and failure examples**\n- For example, where BLIP-2 fails to provide sufficient captions for the keyframes\n    - **Example 1**\n    - Video_url = https://www.youtube.com/watch?v=B63ubKGNqos\n    - Deduplicated BLIP Caption = \n        - a man standing on a red moon many \n        - a desert landscape many \n        - a saturday night live performance worlds \n        - a purple background with the word worlds one by adam sandler one \n        - a character in a black background a boy is playing with a ball in front of a statue a man is standing on a car at night ultraman \n        - ad-screenshot nascar driver nascar nascar nascar n a character's face with a black eye tse seeds\n        - tse seeds devilman cybaby devilman cybaby saint sevyx knights of the zodiac eden vs eden carole and tuesday aggretsu a cartoon character is flying in the sky a woman is holding a satchel se manos - el juego de juegos godzilla \n        - hd wallpaper a man is standing in the snow with a torch Castlevania \n        - a saber-toothed tiger the beginning a girl with big eyes canon busters rilakuma and koru rilakuma and koruru a dragon is standing in front of a blue light ultramarine magnell knights of the zodiac 2 global netflix logo\n\n    - Generated Story = \n        > The advertisement by Netflix, titled \\\"Many Worlds, One Home\\\", is a stunning vision of many different universes. We see a man standing on a red moon, a desert landscape, and a live performance by Adam Sandler. Purple backgrounds with the word \\\"worlds\\\" convey the idea of many possibilities, while a boy playing with a ball in front of a statue shows a more peaceful and homely side to this. As the night falls, a man is standing on a car, and the Ultraman. There are also glimpses of Tse Seeds, Devilman Crybaby, Saint Seiya: Knights of the Zodiac, Eden Vs. Eden, Carole and Tuesday, Aggretsuko, Seis Manos, Godzilla, Castlevania, B: The Beginning, Baki, Cannon Busters, Rilakkuma and Kaoru, Beastars, and Ultramarine Magmell. We see a teddy bear, knights of the zodiac, a character walking in the water, a man in a snowy area, a pair of shoes, a man running away from the fire, the moon, a man wearing a suit with a green light, eyes being the only thing that can be seen, birds flying in the sky, a red and black background with a red light, bats and castles, the eyes, a man holding a gun, a man flying in the air, a face, a girl with long hair, and two hands. All of this culminates to signify the many different worlds that one can find \\\"home\\\", and the advertisement concludes with the Netflix logo. This advertisement for Netflix captures the idea of many possibilities and of being at home in a single place.\n\n    - Remark = Blip2 hallucinates entities as “Adam Sandler” who is not associated with the advertisement. But it causes the generated story to have the same association.\n\n\n    - **Example 2**\n    - Video URL = https://www.youtube.com/watch?v=WCIEmrN8ot4\n    - Deduplicated BLIP Captions = a fox cub is playing in the grass cheetah eats in the wild\n    - Generated Story =  \n        > The sun shone brightly in the savanna as the cheetah cub frolicked in the grass. Suddenly, a cheetah emerged from the shadows. The cheetah and it's cub plan to chase the gazzeles. Despite the presence of death, life in the wild continued on, just as it had for centuries. This story is featured in the advertisement for Walt Disney Studios, which captures the beauty and unpredictable nature of life in the African savanna. This advertisement ultimately serves to promote the film African Cats, which tells the story of a cheetah family as they strive to protect their respective prides against a variety of obstacles in the wild.\n    - Remarks = The captions are insufficient to capture the overall information of the video. However, due to presence of ASR and world-knowledge of the LLM.The LLM is able to generate a coherent story.\n\n- For example, Lack of speech and only music resulting in no transcription\n    - **Example 1** \n    - Video URL = https://www.youtube.com/watch?v=Lh5n4HcLvCU\n    - Generated Story = \n       > The woman in the Ulta Beauty advertisement skillfully applied makeup to her face, creating a beautiful pastel tint. With each brush stroke, her look became brighter and more dreamy. She beamed with confidence and joy, her face glowing with beauty. As the advertisement continued, she highlighted her eyes and cheeks with subtle pastel shades, adding to the stunning effect. With each addition, her face lit up with a vibrant luminosity. When she was finished, her face was a work of art, the perfect combination of bright and dreamy. The Ulta Beauty advertisement showcased the power of a perfect pastel tint, and how it can enhance any look. This advertisement was for Ulta Beauty.\n    - Remark = The video contains only music. However, the generated verbalisation is able to pick the gist of the advertisement.\n\n   - **Example 2**\n   - Video URL = https://www.youtube.com/watch?v=VdhRmVnbqQo\n   - Generated story = \n        > Jackie had been wanting diamond hoop earrings for what seemed like forever. She had been dreaming of the perfect pair that she could wear on special occasions and that would bring out the best in her. One day Jackie happened to come across an advertisement for Costco, which featured a pair of Round Brilliant Diamond Hoop Earrings in 14kt White Gold. She immediately knew this was the perfect pair for her. The advertisement had a voice-over that said, “These earrings sparkle with glamour and sophistication.\" Jackie was mesmerized by the sparkle of the diamonds, and she was sure anyone who saw them would also be taken in by their beauty. \\n\\nJackie quickly decided to purchase the earrings. She was sure they would make the perfect accessory for any special occasion. From date nights to family gatherings, she knew these earrings would make her look simply stunning. The advertisement for the Costco Round Brilliant Diamond Hoop Earrings in 14kt White Gold proved to be the perfect choice for Jackie. They not only added a touch of glamour and sophistication to her wardrobe, but made her feel extra special when she wore them. \\n\\nThe advertisement for the company Costco was for a pair of Round Brilliant Diamond Hoop Earrings in 14kt White Gold.\n    - Remark = The video has no audio transcription, and minimal visual information. However the LLM hallucinates a story, maintaining the essence of the advertisement. It also attributes a false voice-over which is not present in the video.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Multi-stream extraction of textual information through various multimodal models, including scene descriptions, embedded text (OCR), and transcripts (speech).\n* Aggregation of multiple forms of text information through large-language models for generating a coherent description.\n* Usage of coherent descriptions for multiple media-centered video-understanding tasks", "reasons_to_reject": "* Hallucination examples are provided, but failure examples of individual pipeline components and how they were handled are not mentioned/shown\n     * For example, where **BLIP-2** fails to provide sufficient captions for the keyframes\n     * Lack of speech and only music resulting in no transcription\n* Details regarding the persuasion strategy dataset are missing wrt following:\n     * How were the brands selected?\n     * Details regarding the pool of human annotators, including how many unique annotators were used for the entire task?\n     * In the case of the multi-label nature of persuasion strategies, top-5 accuracy, and **mAP** should be reported as metrics for comparing different methods (**Table 2**)\n* Experimental details:\n     * Finetuning details regarding the usage of **VideoMAE** and **InternVideo** are lacking in Section 4.2\n* To test the utility of the proposed method as a purely zero-shot alternative for video description-driven understanding, comparisons should be shown on complex media-based video tasks, including **LVU** and **HVU**.", "questions_to_authors": "* For **BLIP2** caption extraction, keyframe selection was used as a pre-filtering step. \nAs an alternative, have the authors explored shot detection followed by BLIP-2-based caption extraction for center frames in shots?\n\n* For the persuasion strategy dataset zero-shot results, it is not clearly mentioned that the evaluation was performed on the complete set of 1002 videos or a separate test split.", "paper_topic_and_main_contributions": "The paper presents a zero-shot method for video understanding by extracting long coherent descriptions through various large language models.  The major contributions of this paper are:\n* Usage of various multimodal models to extract textual information\n* Aggregation of textual information through large language models followed by text-based reasoning/classification tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some relevant references that are missing:\n\n[1] **LVU dataset**: https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Towards_Long-Form_Video_Understanding_CVPR_2021_paper.pdf\n\n[2] **HVU dataset**: https://pages.iai.uni-bonn.de/gall_juergen/download/HVU_eccv20.pdf\n\n[3] **LLAVa**: https://arxiv.org/pdf/2304.08485.pdf", "Typos_Grammar_Style_and_Presentation_Improvements": "* Overall, the paper is well written with clear motivation.\n* If dataset names are mentioned in the table captions, It will be easier to understand the improvements wrt different tasks.\n     *  In **Table 2**, it should be clearly mentioned that the persuasion strategy section is from the in-house collected dataset, whereas the emotion, topic and action+ tasks are associated with the advertisement dataset from Hussain et.al"}}, {"review_id": "Ti3Wch1LLE", "reviewer": "Reviewer_Rk8R", "summary": "", "questions": "1. \"Uniformly Sampled BLIP-2 Captions\" in Table 1 has no method description.\n2. Why do the three methods (i.e., GPT-3.5, Vicuna, and Flant-t5-XXL) in Table 1 have large performance gaps in multiple metrics?  Is it proportional to the base model capability?\n3. Should the downstream tasks in Table 2 add the fine-tuning setting of (Generated story + normal text classifier)? Moreover, why is vicuna's performance so different from the other methods?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison wrt the Visual Clues, Xie et al paper**\nThank you for pointing us to the visual clues paper. There are certainly similarities between our work and visual clues. We find that both works differ in the following ways:\n- Focus for both papers is quite different. Visual clues focuses on generating good image paragraph captioning. That’s why they compare their performance with other visual captioning models. We focus on solving downstream tasks for which the video domain doesn’t have many annotated samples to learn a task. We show that we get much better generalization across tasks by converting visual information to stories and then finetuning a language model on the generated text as opposed to finetuning a task-specific vision model on raw videos. For us, good story generation is an intermediary step, for visual clues, that’s the end goal. \n\n- The methods are quite different. While both our paper and visual-clues use captioner and object detectors, visual-clues rely on a visual-language model (like BLIP) to detect which paragraph is the best candidate. We don’t do that. Further, we train an LM (Roberta) on the generated stories to perform downstream tasks better, visual-clues don’t do that. We also show that just concatenating “visual clues” doesn’t perform well on the downstream tasks (Table-4).\n\n- The data that both papers deal with is quite different. Visual clues focuses on single-frame images; we focus on short (<120 seconds) and long (12.4 minutes avg) videos. The videos we deal with have audio, music, speech, on-screen text, fast-moving frames, animations, etc, needing world knowledge. The Visual genome data and the Stanford dataset (Krause et al 2017) contain camera-shot images in the natural surroundings. They contain pictures of monuments, outdoors, indoors, etc. Our videos, on the other hand, are both camera-shot videos (Li et al) and computer-generated graphics (Hussain et al and persuasion strategy datasets).\n\n\n\n**Performance on other datasets:**\n- Reviewer 2 pointed us to the HVU and LVU datasets. Thanks to that, in the limited rebuttal time, we were able to test out our best models on the following tasks across the two datasets. Similar to the tasks reported in the paper, we obtained significant improvements in them.\n\nWe observe that on 8/9 tasks in the LVU benchmark and 6/6 tasks in the HVU benchmark, we get state-of-the-art results.\n\n\n1. LVU Benchmark (https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Towards_Long-Form_Video_Understanding_CVPR_2021_paper.pdf):\n\n| Training         | Model                                            | relationship | way_speaking | scene | like_ratio | view_count | director | genre | writer | year  |\n|------------------|--------------------------------------------------|--------------|--------------|-------|------------|------------|----------|-------|--------|-------|\n| Trained          | R101-slowfast+NL [ICCV,2019]                     | 52.4         | 35.8         | 54.7  | 0.386      | 3.77       | 44.9     | 53    | 36.3   | 52.5  |\n| Trained          | VideoBert, Sun et al, 2019, ICCV                 | 52.8         | 37.9         | 54.9  | 0.32       | 4.46       | 47.3     | 51.9  | 38.5   | 36.1  |\n| Trained          | Xiao et al, 2022, CVPR                           | 50.95        | 34.07        | 44.19 | 0.353      | 4.886      | 40.19    | 48.11 | 31.43  | 29.65 |\n| Trained          | CVRL, Qian et al, 2021, CVPR                     | 50.95        | 32.86        | 32.56 | 0.444      | 4.6        | 37.76    | 48.17 | 27.26  | 25.31 |\n| Trained          | Object Transformer, Wu et al, 2019, CVPT         | 53.1         | **39.4**         | 56.9  | 0.23       | **3.55**       | 51.2     | 54.6  | 34.5   | 39.1  |\n| Zero-shot (Ours) | GPT-3.5 generated story + Flant5-xxl [zero shot] | 64.1         | **39.07**        | 60.2  | 0.061      | 12.84      | 69.9     | **58.1**  | **52.4**   | 75.6  |\n| Zero-shot (Ours) | GPT-3.5 Generated story + GPT-3.5 classifier     | **68.42**        | 32.95        | 54.54 | **0.031**      | 12.69      | **75.26**    | 50.84 | 32.16  | **75.96** |\n| Trained (Ours)   | GPT-3.5 generated story + Roberta                | 62.16        | 38.41        | **68.65** | 0.054      | 11.84      | 45.34    | 39.27 | 35.93  | 7.826 |\n\n(like ratio and view count are MSE scores, (less is better), all others are top-1 accuracies, (more is better))\n\n\n2. HVU benchmark (https://pages.iai.uni-bonn.de/gall_juergen/download/HVU_eccv20.pdf)\n| Training  | Model                             | Scene | Object | Action | Event | Attribute | Concept | Overall |\n|-----------|-----------------------------------|-------|--------|--------|-------|-----------|---------|---------|\n| Trained   | 3D-Resnet, Diba et al, 2020, ECCV | 50.6  | 28.6   | 48.2   | 35.9  | 29        | 22.5    | 35.8    |\n| Trained   | 3D-STCNet, Diba et al, 2020, ECCV | 51.9  | 30.1   | 50.3   | 35.8  | 29.9      | 22.7    | 36.7    |\n| Trained   | HATNet, Diba et al, 2020, ECCV    | 55.8  | 34.2   | 51.8   | 38.5  | 33.6      | 26.1    | 40      |\n| Trained | 3D-Resnet, Diba et al, 2020, ECCV [Multitask] | 51.7 | 29.6 | 48.9 | 36.6 | 31.1 | 24.1 | 37   |\n| Trained | HATNet, Diba et al, 2020, ECCV [Multitask]    | 57.2 | 35.1 | 53.5 | **39.8** | 34.9 | 27.3 | 41.3 |\n| Trained   | GPT-3.5+Flant5 | **59.66** | **98.89** | **98.96** | 38.42 | **67.76** | **86.99** | **75.12** |\n| Zero-Shot | GPT-3.5+GPT3.5 | **60.2**  | **99.16** | **98.72** | **40.79** | **67.17** | **88.6**  | **75.77** |\n\n(All the tasks are MAP scores, (more is better))\n\n**Method description of \"Uniformly Sampled BLIP-2 Captions\"**\n- We sample every 10th frame at native fps of the video and generate captions through Blip2. These captions are then deduplicated among themselves, and consecutive captions having 80% word overlap are dropped.\n- This final set of captions is used to prompt the LLM to generate verbalisation for the video. \n\n**The three methods of GPT-3.5, Vicuna, and Flant-t5-XXL having large performance gaps in multiple metrics. More discussion of Vicuna’s results:**\n- GPT-3.5, because of its bigger size (13x bigger), generates better stories than Vicuna. We observe this across all models.\n- We find that Flan-t5 generally performs better than other models for all classification tasks. Though this phenomenon is new and we would like to investigate this phenomenon more, but this is consistent with a few other papers that have observed similar results:\nhttps://arxiv.org/abs/2005.14165 - they observe that in several tasks, other models beat GPT-3 (various sized) models in few-shot settings. - Further, in some tasks, smaller models outperform larger models\n- https://arxiv.org/abs/2204.03954 - they observe that in-context learning based GPT-3 does not achieve higher results than much smaller models like BERT.\n- Vicuna, as a classifier, performs worse than all other methods.\n\n**The fine-tuning setting of (Generated story + normal text classifier) in Table 2**\n- We have shown the performance of Roberta-base trained for the classification tasks in Table-2 ( Generated Story + Roberta Classifier ).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides a feasible framework for long video understanding and generation, and describes the design in the framework in detail. The experiment further proves its effect and feasibility.\n2. This paper releases a dataset for advertising video persuasion strategy classification.", "reasons_to_reject": "1. The framework proposed in this paper bears a relatively large similarity to a previous visual understanding framework (e.g. [1]), as it is a common practice to convert visual and audio information into text and then utilize prompts to construct it with input into LLM. Therefore, this paper lacks a sufficiently novel contribution.\n2. Video understanding is a commonly undertaken task, and numerous works and datasets have been proposed in recent years. Among these, longer videos are typically seen as more intricate tasks for video understanding, given the characteristic multi-camera transitions and the greater length of the video. Whilst this paper highlights its ability to handle long videos, it does not delve into these aspects above.\n\n[1] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning, Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, Michael Zeng", "questions_to_authors": "1. \"Uniformly Sampled BLIP-2 Captions\" in Table 1 has no method description.\n2. Why do the three methods (i.e., GPT-3.5, Vicuna, and Flant-t5-XXL) in Table 1 have large performance gaps in multiple metrics?  Is it proportional to the base model capability?\n3. Should the downstream tasks in Table 2 add the fine-tuning setting of (Generated story + normal text classifier)? Moreover, why is vicuna's performance so different from the other methods?", "paper_topic_and_main_contributions": "This paper proposes a comprehensive framework that aims to extract and convert information from a lengthy video and generates a corresponding story text in a zero-shot setting. Additionally, the effectiveness of the framework is discussed with respect to the quality of the generated story text and its performance in downstream video understanding tasks, providing evidence of its superiority over previous methods. Furthermore, the paper also presents a classification dataset of advertising video persuasion strategies.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The caption of Figure 2 lacks a full stop.\n2. \"Figure\" is mixed with \"Fig.\" in this paper.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper that did extensive experimentation on using LLMs for zero-shot video understanding tasks, showing an interesting result beating supervised baselines. They demonstrated the utility of language generation systems in domains where a strong data signal could be limited such as video-based data that heavily relies on human annotation for quality control.

Pros:
- Clear description of their framework, pipeline and task formulation. Would be essential for reproducing such a setup on other tasks
- Their method of multi source text data extraction and comprehension via LLMs is sound
- Experiments are sufficiently detailed and paper is well written with extensive literature review

Cons: 
- Representing non-text modalities via text has been covered by other papers, reducing novelty of the idea
- Video understanding is an umbrella term comprising several complex understanding tasks, handling missing data, and varying noise levels/durations of data. Tasks covered in this paper do not necessarily reflect takeaways towards complex video understanding problems.
Decision: Accept-Main