Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Authors: Mingfei Han
Linjie Yang
Xiaojun Chang
Lina Yao
Heng Wang
Venue: ICLR 2025 Poster (2025)
Keywords: vision language model
video question answering
video captioning
multi-shot videos
Primary area: datasets and benchmarks
TL;DR: Shot2Story presents a large-scale dataset with 43,000 multi-shot videos and 188,000 manually annotated shots, offering detailed visual/audio captions, summaries, and QA pairs to advance multi-shot video understanding.

Abstract: A short clip of video may contain progression of multiple events and an interesting story line. A human need to capture both the event in every shot and associate them together to understand the story behind it. In this work, we present a new multi-shot video understanding benchmark \dataset with detailed shot-level captions, comprehensive video summaries and question-answering pairs. To facilitate better semantic understanding of videos, we provide captions for both visual signals and human narrations. We design several distinct tasks including single-shot video captioning, multi-shot video summarization, and multi-shot video question answering. Preliminary experiments show some challenges to generate a long and comprehensive video summary for multi-shot videos. Nevertheless, the generated imperfect summaries can already achieve competitive performance on existing video understanding tasks such as video question-answering, promoting an under-explored setting of video understanding with detailed summaries.

Published as a conference paper at ICLR 2025
SHOT2STORY: A NEW BENCHMARK FOR COMPREHEN SIVE UNDERSTANDING OF MULTI-SHOT VIDEOS
Mingfei Han 12,,,5t Linjie Yang" Xiaojun Chang Lina Yao Heng Wang'
Bytedance Inc. Rel Lab, AAII, UTS DDpartment of Computer Vision, MBZUAT
University of Science and Technology of China Data61, CSIRO
ABSTRACT
A short clip of video may contain progression of multiple events and an interesting story line. A human need to capture both the event in every shot and associate them together to understand the story behind it. this work, we present new multi- shot video understanding benchmark Shot2Story with detailed shot-level captions, comprehensive video summaries and question-answeriig semantic understanding of videos, we provide captions for both visual signals and human narrations. We design several distinct tasks including single-shot video captioning, multi-shot video summarization, and multi-shot video question answering. Preliminary experiments show some challenges to generate a long and comprehensive video summary for multi-shot videos. Nevertheless, the generated imperfect summaries can already achieve competitive performance on existing video understanding tasks such as video question-ansee answering, promoting an under- explored setting of video understanding with detailed summaries.
pairs. To facilitate better
INTRODUCTION
Video captioning is long-standing video understanding task facilitate open-world video analy- sis with the help of human-annotated captions. Since a video may contain multiple events, dense captioning benchmarks (Ego4D GGruman et al., 2022). YouCook2 (Zhou et al., 2018). ActivityNet- Caps (Kishhaa et al., 2017)) capture information of multiple events in videos ranging from 3-20 minutes. However, even within seconds. more than one event often occurs in daily videos such as news broadcasts, tutorial videos, and movies. Specifically, shot transition, which is common technique to transfer from one event to another, or to switch the viewpoint of a single event, happens less than every 4s for average English movies after 2010 (Cutting et al./ 2011). Although some existing captioning benchmarks (Xu et al., 2016 Krishna et al. 2017) Zhou et al, 2018) already use multi-shot videos, they often annotate the captions in coarse- grained manner, either providing holistic caption or asking annotators to subjectively choose the boundary of each event. To better accommodate the multi-shot formation of videos, we believe a new video benchmark with rich textual descriptions based on video shots is favored in the research community.
On the other hand, multi-shot videos are often accompanied by rich narrations that relate to the different events happening in the video. A model needs to capture both the visual and audio signals understand the underlying story. Specifically, narrations may contain key information that cannot be inferred from pure visual information only. See Figure [| without the narration, a viewer is unable to capture the relationship between the man's 's action and the avocado product in the first shot.
In this work, we propose new benchmark Shot2Story for audio-visual understanding of multi-shot videos. We collect dataset of 42,958 short videos where the average number of shots in each video is 4.4. For each video shot, we annotate detailed textual description for the video frames and another textual description for the human speech. We also leverage a state-of-the-art large language model (LLM) GPT-4 (OpenAI) to generate long textual video summary from the annotated clip descriptions, which are further verified by human annotators. The summary includes additional details such transitions of different shots, progression of multiple events, and mapping of the subject identities in different scenes. An example of one annotated video is shown in Figure [
Work was done during an internship Bytedance. Equal contribution.

---

Published as a conference paper at ICLR 2025
MAri 54.5
Sior ft
Shat M65-750
Captioen: 11's men katenar diappigg seectting vathy wash earaer The vidro to blace and shite. the vellew verds
Captien: is clese of a po with
Coptisa: EL's wama in pank
waccoell Captine: stoe.
with wos
catis
also wiite soul next to IT.
arit, The wont "OREE" in purple taped
plactic container of puacencle NaLt a purple package rext 2t.
charges to bevi of guacaole with chips so the side.
Narradion Caption the tackground vosce saus prddect refragarates pre-scooped the costoners.
Narratio Captiee: The beckgrtand usice enys don't cisk
getting Injores
Video sumoariztiinn
The video begins with a man in kitchen, wearing 1 T-shitt with the mumber on He is chopping somehing with kmiie, but he scems to her himself and appar in pein
This scene presented in black and white for drumatic effect As cautonary message. the yellow woods "DON'T RISK IT" appea the screen. The video tten transitions to
bonel of grscamani with clips the vile, and the padeage of prolist is ahoo shown The mallo during llks segmeni scooping aveendo with
not rik getting injunai from cuiling
evecados. The video then shiftss close-pp shat of
wooden cutting beard white bowl is placed
the cutting boord. altemative
The andio during this signent informs viewen that product refrigeaned and presocoped for the customers, implynng thet sofer segesting
prepring avncaros mamsllly
pink shint injured amn possilyy indicating that she approves the pee scceped avocado product. word "SWEET" purple is typel on screen. pottuee empilsiiz satisslatiion with the product The video concluilss with
might injural bereelf
propeing fiod
holds thumb
ofa bool quacamnl with chips on pupple paciage also duven, lideyy the product has discussed throughour the video. video scons to be promotional musertal pre-scooped, retiigeratod ovocada product, highlighting risks proparing avecades manuilly and comveniones of their producs
plaatic contiines guacamole
Hideo gwestire onswering- Mul! shot
Facee paettlo--auneniii asoneering Muti shot and tempores roloted:
O What overarching
conveying the proparing avecades?
A: video sccoped product.
preparation risyy
injury chosping, introduces the
scroped product,
Figure I: An annotated example of our Shot2Story with sing-shot visual captions and narration captions. Moreover, we provide coherent and reasonable video summaries, and question-anee -nnweeing pairs to facilitate comprehensive understanding of multi-shot videos.
To benchmark the advances of multi-modal video understanding, we designed several distinctive tasks using our dataset, including single-shot video captioning. multi-shot video summarization, and multi-shot video question answering. We design and implement several baseline models using a frozen vision encoder and an LLM, by prompting the LLM with frame tokens and ASR (Automatic Speech Recognition) text. Through extensive experiments, we show that: (1) the ASR text is critical to joint understanding of visual and audio content, (2) processing the video as whole without the shot structure degenerates the model's s capacity of understanding the multi-shot video, (3) the summarization model trained shot QA benchmark and generalized to other datasets with longer durations (A (ActivityNet(Krisi) al. 2017) and out-of-domain topies (MSRVTT(Xu et al., 2016)), validating the quality of our annotated summaries. Without any bells and whistles, we attain competitive results on zero-shot video question-answering by converting the problem into pure text-based QA with the generated video summaries.
multi-shot summaries can be used on the proposed multi-
2 THE SHOT2STORY BENCHMARK
Our new benchmark Shot2Story contains 42,958 vidcos. The length of cach video is ranging from 10s to 40s. We first use an off-the-shelf shot detection method TransNetV2 (Součck & Lokoë 2020) to split each video into shots. For each video shot, we annotate captions for both visual and audio information. Then we further annotate video summaries based on the annotated shot captions. Figure 2 shows overview of our dataset with some key statistics.
2.1 DATA PREPARATION
We source videos for our dataset from the public video benchmark HD-VID II-VILL--O0MM (uue et al. 2022). It offers a large collection of narrative videos, comprising 3M YouTube videos segmented into 100M clips, each about 13 seconds long. We choose this data source for its concise yet complex multi-shot formats, diverse topies, and abundant ASR content. Since we prefer videos with both rich visual and ASR information, design several filtering techniques to exclude those videos with either low iisull-ASR correlation or static visual content.
We start with keeping video clips with durations between 10 to 40 seconds, since we observe that the majority of the video clips from HD- VILA- fall in this range. Then we remove videos with more than 8 shots due to the heavy annotation cost. We also notice that the video segments with too many shots in ED-VILA- 100M tend to be slideshows or image collages that deviates from our focuses. Further, to harvest videos with rich visual-ASR correlations, set up metric between
2

---

Published as a conference paper at ICLR 2025
2000
20000
Norbae eddess
0000
10000
12000
Minberaf Whate
12000
20000
12000
4000
2000
Number
8000
0000
2000
0000
2000
Number shotss per video
Shot duratton range
Langth caption
Longth shat nernel on cxption
Thnd Dvant Valides
1000
Statistics on video summary phrases
4.1%
I Video Progrosion
Sdivar 6.1%
Scana Tranetione
20000
Camera Shots
Narration Decriiptios
Education, 8.1%
deemub
10000
6.0%
4.5%
vilto laginn with
falloin till
the Binal scome
Serre transtioee to
video shilte to
ond close
the surtiy remnals the rore aftor exphirr the and suggesss
amores
Astilen
horprefits &
New to
Enteraament.
12.5%
13.0%
2.4%
6.75
Carning
Length video summary
Figure 2: Statistics of Shot2Story Our dataset features detailed visual captions and narration captions, and video summaries, highlighting video progressions, transitions, camera cuts and narration descriptions, with statistics of frequent expressions depicted in the figure.
video shots and ASR texts. Specifically, we uniformly sample 4 video frames for each shot and obtain the cosine similarity score between the video shot embedding and the text embedding using CLIP (Radford et al. 202I) encoders. We only keep the videos containing at least one shot that is visually correlated to ASK with a threshold of 0.25. Next, in order to obtain videos with diverse shot contents, we set up an inter-shot metric to filter out the videos with similar adjacent shots. We compute the cosine similarities between embeddings of adjacent shots and keep the videos with all inter-shot similarity scores smaller than 0.9. Finally, to further remove the videos with static contents, we adopt an intensity-based scene change detector in PySceneDetecf with a low threshold of 11 on our segmented shots. If the filter is unable detect scene changes at this low threshold, it is conceivable that the shot contains static contents. We only keep the video clips in which all shots contain static content based our filtering method.
As a result, from a total of MM sampled video clips from HD-VILA-IOS we obtain 42, 958 video clips that meet our quality standard. The number of shots in each video is from 2 to 8. These videos are then shared with our annotators for further annotations.
2.2 ANNOTATION OF SINGLE-SHOT CAPTIONS
After using TransNetV2 to divide the target videos into video shots, we ask annotators to annotate both visual-only captions and audio-related captions for each shot. We split these two caption annotation to facilitate separate modeling of these two types of information source. For visual-only captions, we require annotators to describe the major subjects and events in the video. Since it is an open- world setting, the videos can be quite diverse and hard to describe. In order to reduce the difficulties of annotating caption from scratch, we generate initial video caption using MiniGPT- (Zhu et al. 2023) by sampling 4 image frames from the video clip and prompting the model using below prompt.
Frame2</Img' <Img>Frram Frame3 Tramee<//mg><<< <//m/>//m/>>>>>>>>//> Faamss<mm<>>//<///// Frame4</Im ### Jemmlmmmnnmmm <Ing::mmmmaann Framel </Img <elggmFrame22/iii
Please describe this video. Do not include details that you are not sure of. For example, if there is text in the image, do not include the content of the text if they are not clearly shown. ###Assittant:
Although MiniGPT-4 is originally designed for image understanding, empirically it is able to generate captions for short video clips, both comprehensively and reasonably. It is able describe different subjects including persons, animals, food, tools, and virtual objects like animated characters. Annota- tors are first instructed correct any errors in the original captions. The mistakes include incorrect descriptions of the object categories, attributes, actions, facial expressions, etc. Also, there might be some subjective descriptions generated by MiniGPT-4 such as emotion and atmosphere. We ask annotators to remove all these subjective descriptions. We then ask annotators to supplement the
hhtps//wwww.ccnccdeee..oom
3

---

Published as a conference paper at ICLR 2025
Table High-level comparison of our dataset to previous ones. The summary length of ActivityNet and YouCook2 are combined length of captions in one video. M and G denote manual and generated.
Multi-shot Multi-event Detailed Video
Summary Length
Avg.
Dataset
Annotation
#Videos
Descriptions Summary
Duration
MSRVTT Xu et al.
2016
M
K
x
10K
15s
ActivityNet Caps KKishna et al. 2017
/
52.4
20K
3min
VidcoStorytelling (L. ot | 20019)
M
/
162.6
105
12.5min
Ego4D (Grauman al. 2022
10K
23min
YouCnok2 (Zhou al. 2018)
M
/
67.8
2K
6min
VAST Chen 2024)
X
32.4
27M
5m30ss
Shot2Story
M+G
/
218.3
43K
17.1s
information about the major subjects, actions, and backgrounds present in the video. The goal is for the resulting captions accurately capture the key elements of each video shot. Statistics shows over 80% of single-shot visual captions are manually corrected. For narration captions, annotators should watch the video and interpret the audio content that visually correlates into descriptive narration, including information sources. For example: "According to the woman in white, the room is not very clean." " All narration captions are manually drafted from scratch. An example this annotation process is shown in Appendix A.T where the annotator corrects the caption from "standing in front of the car" to "getting close to the car", , and adding a missing detail of "a close-up shot of the front". In this way, we find the annotation speed significantly faster (~ 3x) compared to writing caption from scratch. On the other hand, we find the captions generated this way has more coherent style and tend to cover more details of the video.
In contrast to the traditional video captioning benchmarks (Xu et al. 2016 Krishna et al. 2017 Zhou et al. 2018), we also annotate narration captions in addition to the visual- captions. Different from existing audio captioning benchmarks (Gemmeke et al., (017), we focus more on human speeches rather than acoustic events. Annotators are required to associate the human speech with the video content and summarize the main idea of the speech. We require annotators to describe the source of the speech using visual information. For example, if someone is talking, the annotators need to describe which person in the video is talking. If the human speech refers to some object in the video, the annotator required to describe which object the video the speaker is referring to. Note that the speaker identity and reference of visual objects are critical information for understanding a video that cannot be trivially obtained using existing algorithms. There are existing research on speaker identification (Kim et al. (02T)) and visual grounding (Anne Hendricks et al. 2017 Zhou et al. 2019), but they only work well on constraint scenarios.
2.3 ANNOTATION OF VIDEO SUMMARY
To create video summaries with annotated video-shot captions, we leverage an L.MM-based approach. Specifically, we form text prompt with incorporating all shot captions and ASR text included, and uses GPT-4 (OpeAA) to generate a cohesive summary. The text prompt we use is shown in Appendix The quality is assured through further review and correction by our annotators.
We prompt GPT-4 to produce coherent, fluent text summaries with transition expressions such as "the video begins", "following this", and "in the final scene" connect video-shot descriptions. The generated annotations also encompass higher-level understanding of shots, using key phrases such as "scene shifts back" and "returns the scene" denote recurring scenes across shots. Notably, GPT-4 often identifies and links the same subjects across scenes without relying on explicit re- re-identification models. draws on descriptive and attributive text from shot captions like "a newsroom" or "a man wearing a black suit" to infer scene or subject identity. To ensure quality, annotators carefully review and correct any inconsistencies in scene or subject references within summaries. Since our shot-level captions for generating video summaries video summaries merely have factual errors, with exceptions for some subject identity and scene mismatches. We require annotators to pay more attention to these errors and ensure holistic summary is accurate and comprehensive. Statistics show that over 40% summaries are manually corrected.
manually checked and annotated, the initial
Despite the rigorous verification process, the reliance on automated generation introduces certain inaccuracies and biases. Common pitfalls in the generated summaries include the omission of minor yet contextually important details and bias towards emphasizing more prominent actions or objects,
4

---

Published as a conference paper at ICLR 2025
potentially overlooking less conspicuous elements. This is partly due our dataset predominantly featuring
activities, as a result of our video filtering process that selects videos rich in visually related audio information and sourcing from HDVILA DOVILAXXue et al. (2022), which primarily curates content related to human activities. Consequently, our annotations tend to highlight salient
events and large-scale objects essential to the video's storyline, mentioning smaller objects only when they directly contribute to the narrative.
2.4 ANNOTATION OF QUESTION- ANSWERING PAIRS
We annotate the question-ansse answering pairs on videos validation and testing splits. To construct this benchmark, we begin with the numan-annotated video summaries from Section 2.3 which is detailed in video content. We then prompt GPT-4 (Achiam al. 2023) to generate candidate question-ance answer pairs in three predefined categories: temporal-related relattdd (e.g., Does the woman appearing at the end of the video wear any accessories?) ,), multi-shot holistic understanding (e.g.. What is the overarching theme of the video?), and audio content related (e.g., Which specific model does the background voice mention, and what visual features confirm its identity?). The text prompt we use is shown in Appendix B.3
The quality is then assessed through further automatic filtering and manual checks. Annotators instructed verify each question-answer pair against the video content and discard any with mistakes. Simultaneously, they are asked to categorize the questions, where a single question might fall into multiple categories, facilitating evaluating different aspects of multishot understanding i.e. understanding sequences of events and actions (temporal related), integrating information across multiple shots (holistic understanding), correlating audio content with visual elements (audio related), and others (directly discarded). After this verification, annotators carefully review the quality of the questions to ensure they solely correlate with the video content and that the answers are not revealed in the question texts. This step includes a thorough manual check to address potential mistakes and biases from the initial verification. Through these sequential annotation stages, we ensure high-quality annotations. The process starts with detailed manual single-shot captioning, followed by careful verification and correction at each stage. Even with LLMs introduced to reduce workload, the human-involved process and thorough procedures keep the resulting annotations well-aligned with human labeling.
Subsequently, optimize clarity and reduce unnecessary information in the QA pairs, we remove QA pairs with questions exceeding 28 words answers exceeding 20
Helri//rmppr DUPS
12.2%
and
words. Then, we employ Vicuna-13B (Chiang et al., (023) to attempt answering the questions without video con- text and discard those answerable without accessing the videos. Finally, the pairs are tested against popular meth- ods such as Video-LLaVA (Lin et al. 2023), LLaMA- VID LLi al. 2023e), Video-ChatGii
12 200.2222200000
Length ofouration
(Maaz et al. (023), and VideoChat2 (Li et al. 2023c). Questions correctly an- swered by more than two models are excluded to guarantee that our dataset poses substantial challenge. Finally, we obtain 4905 QA pairs for validation and 6465 QA pairs for testing set.
Figure 3: Distribution of multi-shot video QA benchmark. Questions from different cat- egories overlap. All-shared means questions fall under all three categories.
2.5 COMPARISON TO EXISTING BENCHMARKS
Compared to existing video description datasets, our dataset is more challenging due to the explicit modeling of the multi-shot nature of web videos. Our textual description includes both shoe- captions and vidco-level summaries, combining visual and audio understanding, which provides unique test bed for multi-modal video understanding. Table [ shows a high-level comparison of our new dataset with existing video captioning benchmarks. Most existing video captioning benchmarks, such as MSRVTT (Xu et al., 2016), YouCook2 Zhou et al. 2018) and ActivityNet Caps (Krishna et al. 2017). also use multi-shot videos as annotation source, but they either annotate a holistic caption for the video (MSRVTT) or ask annotators to decide the boundary of different events. In our study, observe that video shots naturally create sequence of related events, motivating to annotate distinct captions for each shot. Ego4D (Grauman et al. (022) only annotates dense visual captions but not audio captions for relatively long egocentric videos. Video Storytelling (Li et al.
5

---

Published as a conference paper at ICLR 2025
Table Performance of models on video shot captioning using different modalities, following the settings of VASTiChen et al. 2024). The models are fine-tuned on Shot2Story video shot captions. V, and S are abbreviated for vision, audio and subtitle (ASR text) respectively.
Model
FT Modality
B4
VAST
V4S
10.7
16.1
30.3
33.8
VAST
V+A+S
34.0
MiniGPT4-C
9.2
14.7
27.9
25.1
MiniGPT4-C
V4S
11.8
16.7
30.1
35,9
VideoChat2-C
8.8
16.1
27.9
23.7
VideoChat2-C
V4S
13.8
18.7
32.1
43.9
Table 3: Performance of models on video summarization. The models are fine-tuned on Shot2Story video summaries. V and are abbreviated for vision and subtitle (ASR text) respectively.
Model
FT Modality
B4
M
R
Video-ChatGPT w/o ASR Maaz et al. 2023
4.8
17.3
21.3
1.5
Mido-ChattPPT (Maaz et al. 2023
V+S
3.6
17.8
19.7
1.0
MiniGPT4- SURGPTT--UUM-AA SUM-holistis
V+S
7.8
16,9
23.4
2.8
MiniGPT4-SUMM-N w/o ASR
V
10.4
18.5
25.8
4.8
MiniGPT4-SUM-shot VideoChat2-SUMM
V4S
12.4
19.7
27.6
7.6
V+S
12.7
19.8
28.3
9.0
2019) a small-scalc dataset with annotations of multiple events a videos and provides a summary of the video by concatenating all captions.
recent work VAST (Chen et al., 2024) feeds generated video and audio captions into an LLM to generate video summary. However, it processes multi-shot video whole and lacks the granularity of the events in different shots. Moreover, VAST directly uses predicted captions without any human verification, leading to potentially noisy and biased summaries towards the captioning models. Our dataset stands out from VAST with its accurately annotated visual and narration shot captions. Although our video summary also generated using an LLM, is further verified by annotators to make sure there is no hallucinated details from the LLM. Our dataset has an average length of 218.3 words for the video summary, which much longer than existing benchmarks, and longer than the combined length of captions in one video in ActivityNet and YouCook2.
Furthermore, our Shot2Story-Q0 introduces unique and complex challenges through its emphasis on shot transitions and multi-event progression, setting it apart from benchmarks like MSRVTT-QA Xu al. 2017) and ActivityNer-QA Yu et al. (2019). For instance, unlike existing benchmarks Xu et al. (2017); ; Yu et al. (2019) that assess understanding
single time point, e.g.. "Who do three judges talk to?" (MSRVTT-QA, in Figure 19), or general inquiries like "What is the person in the video doing?" (ActivityNet-QA. QA, in Figure 20), Shot2Story-Q2 includes "Temporal-related" and "Multi- shot Holistic Understanding" questions. A temporal-relatee question, such as "What is the man's san's immediate action after handling the skewer?" shown in Figure 21 requires models comprehend the sequence and progression of events, linking consecutive actions meaningfully. Similarly, multi- progression questions like "How does the setting change from the start the end of the video?" necessitate understanding multiple concurrent events within their temporal context. This provides more rigorous and nuanced evaluation framework for temporal understanding.
3 TASKS AND EXPERIMENTS
3.1 BASIC SETTINGS
For all the tasks described in this section, we follow the same training/validationneet split. Specifically, the number of videos for training, validation, and test set are 36951, 1982 and 4025, respectively. We
resize the frames to 224 x 224. We adapt two popular VLMs to accomodate our tasks: MiniGPT-
4 (Zhu et al, 2023) and VideoChat2 (Li al. 2023c For MiniGPT-4, we employ ViT-G/14 (Fang et al. 2022) and Q- Forrerr (Li et al. 2023a) visual encoder, and Vicuna vO-7B (Chiang al. as the language model. We load pretrained Q-Former and MLP from MiniGPT-4 (Zhu et al.
2023
2023
In training, update only Q-Former and MI.P parameters, keeping the VIT and LLM frozen.

---

Published as a conference paper at ICLR 2025
This is video of shots. The first
Q-former
CLIP encoder
N
Multi-shot Video sumamry
Large Language Model
shot 1
i
The second shot is ... The third shot is The fourth shot is
Q-Former
CLIP encoder
VN=
In the audio, hear that:
shot
"Have you suffered an avocado related injury, risk it holy guacamole is refrigerated and pre scooped your protection ..."
Input to LLM
ASR
Porams updated
Visual takens
Sing-shot tokens
Describe this video in detail.
Figure Model structure for multi- video summarization model SUM-shot. We arrange visual tokens sequentially for each single shot and in multi-shot format to encapsulate multi-shot information. Additionally, ASR text is incorporated for audio-visual video summarization.
For VideoChat2, we employ UMT-L et al. 2023d as backbone and load pretrained Q-Former and MLP from VideoChat2 (Li et al. 2023c). During training, we adopt LoRA (uu et al. 202 and Adam (Loshchilov & Hutter 2017) with learning rate 8e-5. We train both models for epochs with batch size of 128 for single leglle-shee shot video captioning. We finetune
video summarization models on the single-shot captioning models with a batch size of 32. For captioning and summarization, we evaluate the models using BLEU @4 (Papineni al. 2002) (B), METEOR (Denkowski & Lavie 2014) (M), ROUGE (in 2004) (R), and CIDEr (Vedantam et al. 2015) (C).
3.2
SINGLE-SHOT VIDEO CAPTIONING
This task involves generating descriptions for individual video shots, where the target description a concatenation of the visual-only and narration caption for vidco shot. This task requires a adopt similar structure as we generate pseudo captions for data annotation in Section 2.2 First, we sample N, frames from a video shot, encode them using a fixed vision encoder, then feed the encoded features a Q-Former to produce visual tokens. Further, we combine the visual tokens and an optional ASR text into a unified LLM prompt facilitate both visual and narration understanding. We adapt the framework of MiniGPT-4 (Zhu et al. 2023) and VideoChat2 (Li et al.] 2023c), with the two models denoted as MiniGPT4-C and VideoChat2-CC for brevity. We compare with baseline model VAST (Chen et al. which is able to processes audio, vision, and subtitle inputs simultancously.
joint understanding of visual and speech information. Specifically,
2024).
The results are shown in Table Benefiting from extensive pretraining, VAST achieves 34 C and 30 on R, comparable to MiniPP44-C. Incorporating an additional audio modality results in negligible performance difference, indicating that audio content only contributes marginally given the ASR text as input on our benchmark. We then assess variants of our models, MiniGPT4-C and VideoChat2-C, with and without the additional ASR text. shows that including ASR texts significantly enhances performance across all metrics, with notable boost in R and C, highlighting the relevance of audio content to our video captions. Furthermore, VidcoChat2-C. featuring a superior visual backbone and extensive video pretraining, consistently outperforms MiniGPT4- C. This superiority highlights the importance of advanced visual backbone and video pretraining, confirming our benchmark's robustness. However, despite these advances, the results also indicate room for improvement. Figure (a) showcases output examples of our model's single-shot video captioning, detailing visual elements and audio content effectively, capturing actions like "gesturing to explain her fear" and secondary elements such as "a red stuffed doll next to her".
3.3 MULTI-SHOT VIDEO SUMMARIZATION
Multi-shot video summarization is new task that is distinct from existing video description tasks. It requires the model to understand the shot structure of the given video and to provide a coherently paragraph deseribe the progression of events in the different shots. Due to the complexity of this task, we adopt GPT-V (Achiam et al. 2023) to generate a supplementary training set, with video summaries for another 90K videos, sampled in the same approach as descrihed in Section 2.1 Please

---

Published as a conference paper at ICLR 2025
Inimiii Int di mimiii inmmmmmmn mnrimim antium
Human speech: explains to the fear she has.
Human speech: But when you look at something like this, how many man hours have gone into peeling back the motrik?
Single-shor video captioning:
Single-skoo vidco captioning:
the video, a pregnant woman is sitting on black couth. she is wearing black halter there red stuffed dall next her. she gesturing to explain her fear.
the video, man a blue shirt and man in hat are looking shirt querying how many
large room. the man man hours have spent.
Fxample predictions single-shot captioning, 2 $78 JMMGjONNOut:5R 28_
InseiIIIIIA DE IIT BEITER intera TREEEEI IIEE Irin inti mnstrii wwr.enttiiaaaaae IREEIE FEEIEREI EiADeEen FIGAEREEI EITRERI TEREEEEEEEARREI
Human speeck: Putter pocket holds two putters for quick and easy access the front. We have flap that you can close to protect your discs, but theres also wecro there, that when youre reay you have access to your dives.
MideoChat-SSUM---ee video summarization:
the video begins with close-up features a putter pocket, as mentioned by male voice in the audio. the scene then transitions to a man sitting at the same wooden table. he dressed
black backpack resting wooden
backpack adorned with logo
black shorts, revealing its contents, as he does this, the male voice in the audio explains that the backpack has flap that can be closed to protect the dises final part the video, the focus shifts close-up
is surrounded by various backpacks. man proceeds to open one
backpacks,
man's hand opens the backpack. backpack black, with a logo on the front, the male voice in the audio explains that there velcro in the backpack, which makes it easy for people to access their
they are ready to play.
(9) bample predictions mult-shoz shottshtottee video summarization. Video: L1yf00299MM222
Temporat-related Question. What initially shown Ground-truk anwwsr Blue, graw and white.
visible the backpack that
Multi-shot Holistic Understanding Question. What the theme of video?
nisbes?
Ground trath answer: Showcasing disc storage backoocks.
Answer based on FideoChar2-Si S0arr-SUM-shhan summnary: X
Answer based on VideoCCat2-StAN SUM-sho summary: ovararching theme of the video the demonstration backpack with features that make suitable for playing disc golf.
The backpack in the video has pattern of blue, black, and white.
[c) Example questions and predicted answers of multi shot vidao. Video: ETу/Q0ISMM..
Figure 5: Example predictions of our models. (a) shows single-shot video captioning results of VideoChat2-C, C, which incorporates audio and visual content correctly (b) shows multi-shot video summarization of VideoChat2SSUM-shhn with accurate descriptions in green and errors in red, illustrating the model's 's ability narrate event sequences (c) shows two sample questions of the video in (b). The answers are based on the generated summary of VideoChat--SUM--Shi
check the annotation prompt and data samples in Appendix B.5 First, we finetune an existing video caption model Cideo-ChatGPT (Maaz al., 2023) by instruction-tuning it on our video summary data, with and without the additional ASR text input. Then, we experiment with three different architecture designs based on MiniGPT4. The first model MiniGPT4- S0M-14--SM-holistti uses a similar pipeline MiniGPT4-C. We uniformly sample 16 frames from the full video clip and prompt the LLM with frame tokens and ASR text. The second model MiniGPT4-SUN neglecting ASR input, uses more refined framework by sampling 4 frames in each video shot and prompting the LLM with frame tokens from different shots, as is shown in Figure 4 The third model, MiniGPT4-SU SUM-sso further incorporates ASR text input as an additional input. Further, we replace the backbone of MiniGPT4-SUM-- model variant. Compared SUM-shot, SUM-holistic does not have explicit shot information and relies on the LLM to parse the video shots using the provided frame features.
w/o ASR,
with the more advanced VideoChat2 model, resulting in the
VideoChat2-SUM-
Table 3 shows the results of the models. It is shown that MiniGPT4-SUM-hoolise is worse than MiniGPT4-UUM-sho showing the importance of the shot structure in predicting a video summary matching the transition of shots. MiniGPT4-SU SUMTT-- w/o ASR underperforms compared to MiniGPT--SUM-holisse
highlighting the significance of both audio information and shot structure in multi-shot video understanding. Compared VideoChat2- the former model achieves the best performance, indicating
MiniGPT4-SUM-sin and outperforms
SUM-shot and MiniGPT4-SUMM- the benefit of advanced vision backbone and video pretraining. Video-ChatGPT obtains much worse performance comparing to our models, potentially due to their weakness in processing multiple scenes and building the correlation between visual frames and ASR texts. It directly encodes the whole video into sequence of tokens, potentially losing significant frame details and essential correlation between ASR and visual frames, while ours directly feed frames tokens into the LLM without compressing them.
8

---

Published as a conference paper at ICLR 2025
Char2-Ca Char2-5//MMMhhot vidao summariration
the opening scene of the vidio, man stands on ping pong tuble indear sports faciliyy dressed in black palo shirt with logo on the left chest and holds nacket his right hand, the barkground a vibrant blue, and thene's reads welcome world ping he speaks, introduons the topic of the
video, which the
transttoning
funda acdres audience. the
plng pong hand.
emphisstes
undorstanding
suggecting
sing pong tubli,
preparing
he demonsretas
appears ping pong
embleroond plays game finelly, fitth tret ping pong ping pong. highiighting
pingnonginto
from scone roappears, impontence priper
the ping song toble, poses qucetion the viewer, asking
quection about sing pong this serves
melittion the aucience to empage with contant and potendally learn more about sport. throughout the video, the speateer, including man the ping porg and the man the white t-shirt, provide infor motrrmathv commentary on basiox ping pong. emphosirig the Importance undentanding the fundsmenbals of the game.
Questions:
Answers:
Q1: what person the video doing?
play
02: the person in the video wearing a witch?
G72:yts
Q3: is the person the video woorinn a hat?
GT
video?
GT4: table tennis table
CS what hasponed before playing zable tonnis?
GTS: introduciion ball
Vicuna andwors using summartes
evaluation:
Al: The per sen sideo demonstrating the badis of ping pong, including proper grip, stanie, serve, footwork.
Al: No, the person in the video not wairing watch. X
Al: Nin
in the video weering t
person Aã: Bload on video, it not clear what happwned before started plaping table tannis
ping pony tablee indlar sports
ActivityNettCUA case:v_ www.fmmsSUPAA.mp duration: 104 seconds
Figure 6: Example predictions of our model on zero-shot question answering an ActivityNet-QA video. More questions and explanations can be found in Appendix C.2
Figure (b) showcases predictive capabilities of our VideoChat2- SUS-ssott model. The model adeptly narrates event sequences with appropriate emphasis. For instance, it details the backpack's 's colour and location, and rationalizes the item in the beginning shot, ie.. "putter pocket", aligning with the ASR with "by a male voice in the audio". However, some predictions that marked in red are erroneous, such the incorrect "black" colour and the non-existent "with a logo on the front" in the ending shot. These inaccuracies likely stem from the LLM's tendency to "hallucinate" plausible yet non-factual details. Despite these errors, the model demonstrates proficiency generating consistent and nuanced summaries, highlighting our model's 's potential and the challenges our dataset presents.
3.4 VIDEO QUESTION-ANSWERING WITH VIDEO SUMMARY
Generated video summaries are supposed to be grounded and detailed, covering rich elements like event progression, holistic topics and audio elements, making them suitable for other vision tasks such video question-ansswrring Existing work (Guo et al. 2023 Zhang et al. 2023) uses image or video frame captions as input to an LLM to generate question responses. However, little work has been done for the capacity of video summaries. We directly apply video summarization model on video QA benchmarks, Le. MSRVTT-QA (Xu et al. 2017), ActivityNet-QA (Yu et al., 2019) and our Shot2Story-QA.
Specifically, we first split the testing videos into video shots, and then feed the videos into our SUM-shot models. The generated summaries and the associated questions are then fed into a Vieuna model to derive the answers with the prompt shown in Appendix [.4.I] Note there is no adaptation or finetuning conducted for the Vicuna model. Since the original answers in the QA benchmarks are very short and the responses generated by LLM tend to be long sentences, we leverage the 5-turb model generate a binary decision on whether the answer correct, following Video-ChatGPT (Maaz et al. 2023
Zero-shot video question-answering. As shown in Table our results with VideoChat2-SUU shot surpass 5 out of 6 existing video-VLMs on MSRVTT-QA and 4 out of 6 existing models on ActivityNeI-QAA Furthermore, our results are comparable to the SOTA performance on MSRVTT-QA with Video-LLaVA (Lin et al. 2023). Note that these models require extensive instruction-tuning data to learn to directly generate answers from visual features and the text prompt, whereas our model bypasses instruction tuning by distilling the video information into summary. Our model also follows the zero-shot QA setting since the model only uses Shot2Story as training data. Note that MSRVTT contains large portion of videos with out-of-domain topics such as TV shows

---

Published as a conference paper at ICLR 2025
Table 4: Performance on video question answering on MSRVTT-QA and ActivityYeet-AA IT means
whether the model uses video-text instruction tuning data. All methods follow the zero-shot manner.
QA
MSRVTT QA
ActivityNet
Model
Training Data
IT
Input
QA
VideoChat Li el
2023b
Cap.+QA
/
V+I
45.0
26.5
Video-ChatGPT Maaz
2023
Cap.+QA
/
V+T
49.3
35.2
MovieChat Song
2023
Cap.+QA
V+T
52.7
45.7
LLaMA- (L] ct al. 2023c
Cap++QA
V+T
57.7
47.4
VideoChat2 (Li ct al. 2025c
Cap.+QA
V+T
54.1
49.1
Video-LLaVA Lin
2023
Cap.+QA
V+I
59.2
45.3
MiniGPT4-SUM-shhe
Summary
X
T
57.7
45.6
VideoChat2 SonChhat--SUMshoo
Summary
X
T
58.5
47.1
Table 5: Benchmark on Shot2Story-QA. IT means usage of video-text instruction tuning data. Summary, Cap. and QA denote video summary, captions and question pairs.
QA Input
Temporal related
Holistic understanding
Audio related
Model
Training data
IT
Overall
LLaMA-VID Li al. 2023cb
CA
V+T
7.9
9.7
11.4
9.7
Video-ChatGPT (Maaz at.
2023
Cap.+QA
V
V+T
13.1
15.5
14.3
14.2
VideoChat2 [Li al. 2023c)
Cap.+QA
V+T
15.1
15.4
13
14.5
Video-LLaVA (Lnn et al.
2023
Cap.+QA
V+T
16.4
14.8
11.7
14.3
MiniGPT4-SUM
Summary
X
T
28.9
31.9
36.7
32.5
VidenChar2-SUMM-sho
Summary
x
T
36.1
41.5
43.8
40.5
(e.g., Figure 19 and food, while ActivityNet has much longer videos than our training videos (c.g.. Figure This validates the robustness and transferability of our model across different topics and longer videos. This surprisingly good result indicates that a comprehensive and detailed video summary is a high-quality abstraction of the video, facilitating wide range of tasks including video QA and video-based conversation. Moreover, our model achieves strong results ActivityNet- QA, which predominantly consists of single-shot long videos, validating that models trained with multi-shot videos can effectively generalize single-shot videos.
Multi-shot video aecettin-answeringg As shown in Table 5 we benchmark existing and our proposed video summary models on Shot2Story- QAA. Specifically, four popular video- VLMs
video-VLMs
compared, ie., Video-ChatGPT (Maaz et al. 2023). LLaMA- (Li et al. 2023e). VideoChat2 (Li al. 2023c) and Video-LLaVA (Lin al. shot and VideoChaI2-SUM--hh zero-shot VQA. Accuracies on temporal-related, multi-shot holistic understanding and audio-related are reported, with the "overall" metric showing the average score from these three sub-tasks. The current video-Vo VLMs present unsatisfying results, potentially due to factors: (1) The current models does not have audio or ASR as input, lacking capacity with audio-related understanding. (2) Current models not have training data with detailed descriptions based on multi-shot videos, weakening their performance on holistic understanding and temporal modeling. For our models, SiteoChatt-SUMM--- achieves an overall score of 40.5, surpassing the compared models and BHnGPT4-SUU--hhe on all three subtasks. This performance underscores the benefits of video pretraining and the advanced visual backbone of VideoChat2. These baseline results highlight the complexities and demanding nature of our Shot2Story-O0 -AA task. We show some example predictions in Figure clc.. Please refer to Appendix C.2 for more qualitative results.
2023 The predicted summaries of MiniGPP44SSUM-
are used to tackle the QA task with the same configuration as
4 CONCLUSION
In this work, we present Shot2Story, large-scale benchmark for comprehensive multi-shot video understanding. provide detailed shot-level captions for both visual signals and human narrations. Furthermore, provide comprehensive video summaries based shot-level captions and design a challenging video guestion-answering benchmark for multi-shot video understanding. With the rich and diverse descriptions, our benchmark serves playground for future multi-modal video understanding models, ready to be extended for range of other video understanding tasks, such as visual grounding and video-based conversation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (tt::23333008774,, 2023.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pp. 5803-5812, 2017.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu, Vast: A vision-audio--ubiitleeeex omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https
www.s.a.............. og/ 2023-03- 30 -vicuna
James E Cutting, Kaitlin L Brunick, Jordan E DeLong, Catalina Iricinschi, and Ayse Candan. Quicker, faster, darker: Changes in hollywood film over 75 years. i-Perception, 266))569 2011.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pp- 376-380, 2014.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10867-10877, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09685, 2021.
You Jin Kim, Hee-Soo Heo, Soyeon Choe, Soo-Whan Chung, Yoohwan Kwon, Bong- Lee, Youngki Kwon, and Joon Son Chung. Look who's talking: Active speaker detection in the wild. Interspeech, 2021.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. Proceedings of the IEEE international conference on computer vision, pp. 706-715, 2017.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Video storytelling: Textual summaries for events. IEEE Transactions on Multimedia, 22(2):544-555, 2019.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language aangagge-mmag pre- training with frozen image encoders and large language models. arXiv preprint 201.57230.. 2023a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. 2023b.

---

Published as a conference paper at ICLR 2025
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi- modal video understanding benchmark. arXiv 2023c.
preprint ar arXiv:2311.17000
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint arXiv:2303. 16058, 2023d.
Yanwei Li, Chengyao Wang, and Jiaya Jia Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.1704. 2023e.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.00122, 2023.
Chin-Yew, Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01101, 2017.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05524 2023.
OpenAl. Gpt-4, URL https:// www.aii......n...........c.. www.ppawaa..a.aaaaaeooamm.oeeeee ?model -gtt
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp- 311-318, 2002.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307
2023.
Tomáš Souček and Jakub Lokoč. Transnet v2: An effective deep network architecture for fast shot transition detection. arXiv preprini 2000v:2008..44: 04838, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 10XX::2302.1397,,, 2023.
Ramakrishna Vedantam, C Lawrence Zimick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. CVPR, June 2016.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, pp. 5036-5045, 2022.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference Artificial Intelligence, 2019.
12

---

Published as a conference paper at ICLR 2025
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range vidco question-answering: arXiv preprint arXiv:2312.17255, 2023.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, volume 32, 2018.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded video description. In CVPR, pp. 6578 6587, 2019.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13


Reviews:
[{"review_id": "1uktail1L6", "reviewer": "Reviewer_5bm5", "summary": "The paper introduces Shot2Story, a benchmark aimed at improving multi-shot video understanding through a rich set of annotations, including shot-level captions, detailed video summaries, and question-answer pairs. The benchmark focuses on tasks such as single-shot captioning, multi-shot summarization, and multi-shot question-answering, each designed to leverage both visual and auditory cues in video data. The authors provide a comprehensive dataset of nearly 43,000 videos, annotated with visual and audio information at each shot level, as well as summaries and QA pairs to facilitate high-level semantic understanding of video content.", "questions": "The paper mentions that models sometimes produce hallucinated details in the video summaries. Did you explore any methods to mitigate these issues during model training or fine-tuning? Additionally, were there specific patterns observed in these hallucinations (e.g., adding non-existent objects, exaggerating actions)?\n\nThe distinction between single-shot captioning and multi-shot summarization is well-explained. However, were there cases where summarizing at the shot level led to information redundancy or fragmentation when compiling multi-shot summaries? If so, what strategies did you use to address this issue?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Shot2Story addresses the limitations of existing benchmarks by providing fine-grained shot-level annotations, narration captions, and comprehensive video summaries. This detailed annotation setup allows for a more nuanced understanding of multi-shot videos.\n\nThe dataset's scale and quality are commendable, with 42,958 videos annotated both visually and auditorily. The authors employ advanced techniques, including GPT-4 for generating initial summaries, followed by human verification, ensuring data reliability.\n\nThe authors present solid baseline models across tasks, demonstrating the benchmark's challenges and providing insights into areas where current models struggle, particularly in complex video summarization.", "weaknesses": "The reliance on automated generation may introduce inaccuracies or biases, despite human verification. Clarifying any common pitfalls in the generated summaries would strengthen the paper.\n\nThe baseline models appear to struggle with integrating visual and auditory information effectively, as noted in the single-shot captioning task. It may be beneficial to explore more advanced audio-visual fusion models as baselines.\n\nA deeper analysis comparing Shot2Story with related benchmarks would contextualize its contributions, especially in terms of unique challenges posed by shot transitions and multi-event progression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UjIVKX8oGb", "reviewer": "Reviewer_G1Gp", "summary": "- This paper presents Shot2Story, a large-scale benchmark for comprehensive multi-shot video understanding. \n\n- The dataset contains detailed shot-level captions for both visual signals and human narrations, and comprehensive video summaries based on shot-level captions.\n\n- The authors further design a challenging video question-answering benchmark for multi-shot video understanding.", "questions": "I think the main concern is on its contribution of methodology, as it doesn't involve any architecture/method design for tackling the multi-shot video understanding problem.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a new multi-shot video understanding benchmark (Shot2Story), with detailed shot-level captions, comprehensive video summaries and question-answering pairs. This will benefit the community, as a benchmark for multi-modal, multi-shot video understanding benchmark.\n\n- The paper has made significant efforts for curating the datasets, to maintain high quality standard, getting the data from 2.1M video clips to 42,958 video clips. The descriptions of the filtering stages are clear and reasonable.\n\n- For baseline establishment, the authors have extensively evaluated a series of open-source models.", "weaknesses": "- My main concern is on its scientific novelty, as the paper does not include new designs on tackling the multi-shot video evaluation problem, only providing a benchmark, it seems to be more suitable for a benchmark track.\n\n- For baseline evaluation, the focus has been mainly on open-source models, what about models, for example, gpt4v, claude3.5 ?\n\n- In Table 1, \\hline is incorrect ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYkNOy3Eux", "reviewer": "Reviewer_n4p7", "summary": "For multi-shot videos, current video benchmarks tend to annotate captions in a coarse-grained manner, either by providing a holistic caption for the entire video or by allowing annotators to subjectively define the boundaries of each event. But multi-shot videos often include rich narrations that correspond to distinct events occurring within the video. Given this context, this work introduces a new benchmark, *Shot2Story*, aimed at enhancing audio-visual understanding of multi-shot videos through the detailed textual annotation of each individual shot. The authors conduct extensive experiments from three perspectives: single-shot video captioning, multi-shot video summarization, and video question-answering with video summaries. These experiments demonstrate significant limitations in current models’ abilities to understand multi-shot content. Additionally, the authors design two baseline models based on MiniGPT-4 and Video-Chat2. Experimental results reveal that the multi-shot instruction data constructed by the authors substantially improves the performance of multi-shot video understanding.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addressed here underscores a critical issue in current Video LLM work, where shot-level temporal understanding in video instruction data has largely been neglected. Most existing video instruction datasets lack the fine-grained, shot-level annotations necessary to capture nuanced transitions between events within videos.\n\nTo address and evaluate this issue, this work introduces a challenging multi-shot video understanding instruction dataset and benchmark. The dataset construction process involves substantial human oversight, which significantly reduces the likelihood of inaccuracies and biases often introduced by external models (e.g., GPT).\n\nIn the experiments, the authors analyze and compare several existing Video LLM approaches, highlighting the benchmark’s applicability and challenges. Additionally, they propose two suitalbe baseline models ,which achieve considerable improvements in the understanding of multi-shot video content.", "weaknesses": "The methods compared across the three tasks—video shot captioning, video summarization, and multi-shot video question answering—lack consistency, making it difficult to observe a clear, uniform improvement in performance across these tasks with the proposed dataset. To improve consistency, I suggest that the authors consider using the same set of baseline models across all tasks. \n\nThe constructed multi-shot dataset appears to resemble a series of single-shot descriptions loosely connected by simple transition words, such as \"begin with,\" \"then,\" and \"next.\" The annotation process seems more akin to first splitting multi-shot videos into individual shots, generating isolated descriptions for each, and then using an external tool (e.g., GPT-4) to concatenate these descriptions. This method does not accurately represent the interconnections between shots, which may pose challenges in teaching large language models (LLMs) multi-shot temporal reasoning skills using the constructed instructional data. Although the actual content contained in different shots is inconsistent, there is an obvious time sequence and event relationship between them. I suggest that the authors improve the annotation process by having annotators explicitly describe relationships or continuity between shots, rather than relying on simple transition words. This approach would better capture the temporal flow and coherence between shots, which is essential for developing multi-shot temporal reasoning capability in LLMs.\n\n\nMoreover, it raises the question of whether multi-shot understanding is at odds with simple video understanding and long-video comprehension. I noticed that the average length of videos in the dataset is only 17.1 seconds, and shots with minimal variation between them were deliberately excluded during construction. Although the videos are relatively short, they contain numerous individual shots with significant differences between them. This could lead models to develop a bias toward segmenting any input video into multiple shots for interpretation, which may pose challenges for understanding both simpler and longer videos. I recommend that the authors conduct additional experiments to evaluate whether models trained on this dataset struggle with simple single-shot videos or longer videos, as this could help clarify any potential biases introduced by the dataset construction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zsd3ed3Ij", "reviewer": "Reviewer_YdUF", "summary": "The paper presents a new benchmark for understanding multi-shot videos, named Shot2Story. The benchmark provides a dataset of 42,958 short videos, each consisting of an average of 4.4 shots, with detailed textual descriptions, video summaries, and question-answering pairs for multi-modal understanding. It aims to enhance the understanding of videos by providing shot-level captions, human narrations, and generated summaries. Several distinct tasks are defined using this dataset: single-shot video captioning, multi-shot video summarization, and multi-shot video question answering. The paper also presents baseline models and demonstrates the challenges of long, comprehensive video summaries.", "questions": "1. Is it an optimal approach to use large language models (LLMs) to annotate datasets and then use those annotations for reasoning tasks by the LLMs themselves? This self-referential process might introduce biases or limitations in understanding. Is there a better approach to improving the depth of reasoning and multi-modal alignment, perhaps involving a more collaborative method of training distinct specialized models (e.g., one focused on annotation and another on reasoning)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, providing clear explanations of the benchmark, tasks, and dataset construction, making complex ideas accessible.\n2. Shot2Story sets a new standard for multi-shot video understanding, with detailed shot-level captions, audio narrations, and summaries that advance research in video analysis and multi-modal understanding.\n3. The benchmark's unique focus on multi-shot videos with rich shot-level annotations differentiates it from previous benchmarks, contributing to a better understanding of event transitions in multi-shot videos.\n4. The authors conduct thorough experiments using baseline models like MiniGPT-4 and VideoChat2, providing valuable insights into current models' strengths and limitations, and establishing a solid foundation for benchmarking future models.", "weaknesses": "1. There is a format issue in Table 1 that affects its readability. Consistent formatting is essential for clarity and for the accurate presentation of comparative data.\n2. The motivation for annotating audio text descriptions is not clearly articulated. Understanding audio alongside video is crucial for comprehensive multimedia understanding, especially in tasks involving multi-modal large language models (MLLMs). Merely using textual descriptions without integrating auditory context reduces the potential for deep reasoning capabilities across modalities, which limits the advancement of MLLM applications.\n3. The experimental section lacks baselines from well-regarded models such as the Qwen-VL series. Including such comparisons would provide a stronger benchmark and enable a more effective evaluation of the proposed model’s performance.\n4. The paper does not propose any self-developed model tailored specifically for the Shot2Story benchmark, which would demonstrate the specific advantages and limitations of the benchmark through a purpose-built approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)