SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
Authors: Zhenyu Yang
Yuhang Hu
Zemin Du
Dizhan Xue
Shengsheng Qian
Jiahong Wu
Fan Yang
Weiming Dong
Changsheng Xu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Multimodal large language model
Streaming video analysis
Video understanding
Primary area: datasets and benchmarks
TL;DR: A benchmark with temporal multi-turn dialogues specifically designed to thoroughly assess the capabilities of long-context streaming video understanding of current LVLMs.

Abstract: Despite the significant advancements of Large Vision-Language Models (LVLMs) on established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To address these limitations, we introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent a series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. We also construct a StreamingChat model, which significantly outperforms open-source LVLMs on our SVBench and achieves comparable performance on diverse vision-language benchmarks. We expect SVBench to advance the research of streaming video understanding by providing a comprehensive and in-depth analysis of current LVLMs. Our benchmark and model can be accessed at https://yzy-bupt.github.io/SVBench.

Published as a conference paper at ICLR 2025
SVBENCH: A BENCHMARK WITH TEMPORAL MULTI- TURN DIALOGUES FOR STREAMING VIDEO UNDER- STANDING
Zhenyu Bongl2.;yuhann Hut, Zemin Du², Dizhan Xuel2, Shengsheng Qian Jiahong Wu?, Fan Yang', Weiming Dong Changsheng X IIstitute of Automation, Chinese Academy of Sciences, UUiversity of Chinese Academy of Sciences, Kuaishou Technology, "Zhengzhou University, "ShanghaiTech University, 'Peng Cheng Laboratory yangzhenyu20222@1 ac. en shengsheng
ia an.
ABSTRACT
Despite the significant advancements Large Vision-Language Models (LVLMs) established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically em- phasize isolated single-instanes text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To ad- dress these limitations, we introduce SVBench, a pioneering benchmark with tem- poral multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design semi-automated annotation pipeline obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our ex- perimental results, obtained from 14 models in dialogue and streaming evalua- tions, reveal that while the closed-source GPT-4o outperforms others, most open- source LVLMs struggle with long-context streaming video understanding. We also construct StreamingChat model, which significantly outperforms open- source LVLMs on our SVBench and achieves comparable performance on di- vision-language benchmarks. We expect SVBench to advance the re- search of streaming video understanding by providing a comprehensive and in- depth analysis of current LVLMs. Our benchmark and model can be accessed htttps://yzy-bppe bupt. btt.t//yyybbbutt...tttt.bbbb github. 20/SBBench
INTRODUCTION
In recent years, the rapid advancements in Large Language Models (LLMs) Touvron et al. (2023 Achiam al. (2023) and visual processors Radford al. (202I): Dosovitskiy al. (2020) have sig- nificantly enhanced the performance of Large Vision-Language Models (LVLMs) Zhu et al. (2023); Ataallah et al. (2024): Maaz et al. (2023). These powerful models have been instrumental in pushing the boundaries of artificial intelligence, showcasing exceptional progress in domains such visual reasoning and dialogue, particularly in the field of video understanding.
Furthermore, there is a growing interest in applying these advancements to the emerging field of streaming video understanding Qian et al. (2024); [Chen et al. (2024a). In conventional video under- standing tasks, models are given the entire video and can leverage past, current, and future content to comprehend specific video segment. In real-world scenarios, such as live streaming and secu- rity monitoring, video content is continually streamed, necessitating that dialogues should update
Work done during an internship at Kuaishou Technology. "orresponning author.

---

Published as a conference paper at ICLR 2025
SVBench Skills
Intesdien Inforence
Paturrialyy Assazament
Counterfecte Reasoning
Spatio Tesporaa Specaledon
Aotetanshipp
Chorashe State
Comparionn and Trand Analyal
Conmoo deece Ieforence
Evene Cecole Arslyab
Istorence
Tramation
Inter Chain: Temporal Reasoning Capability (Challenge2)
Mull-Tam Dialogue Chaie
Multi-Ture Dialogue Chaln
Diatogue Chain
Intra Clai-ihiin Mulii Buti--Bundd Dialogus
Capability (Challenge 1)
TRoi they a help?
Tanaad Fligeing SOSTarn darting Tma.
artaepted to Modd The TIME parsoe
Mi Iacar black clemter lping
rrectiin the other neerst!
other taller resert
art the trecie))
[ ottempte woald
Temporal Linkage;
<Q1
A00,
OZA2>
OZA
Poasible Temporal Dislogue Pathe: Intra- Intra-Chein Chein a Inter-Ghain
Clebo
<22
Chub
400
Illustration of temporal multi-turn dialogues. A temporal dialogue path represents a conversation within video progressing over time. Our SVBench evaluates the capabilities of LVLMs in long-context streaming video understanding by constructing temporal dialogue paths to assess 9 critical skills.
Figure 1:
concurrently with the temporal flow of the video without knowing the future information. Re- cently, streaming videos have become increasingly popular with diverse sources, including online video platforms Zellers et al. (2022), live streaming services Gao et al. (2023). ). and wearable camera footage Grauman et al. (2022). Therefore, researchers have begun to study LVLMs with the ability to interpret and interact with streaming content Qian et al. (2024); Chen et al. (2024a). Given these research advancements, it becomes imperative to establish a comprehensive evaluation benchmark specifically tailored to assess the progress in streaming video understanding achieved by LVL.Ms.
Existing video benchmarks primarily focus single-turn text input and cannot measure the ability to conduct temporally sequential reasoning, which falls short of capturing the complexity of stream- ing video understanding. For instance, several question-answering benchmarks Yu et al. (2019); Zhang et al. (2023); Xiao et al. (202T) typically comprise disjointed QA pairs tied to individual video clips, ignoring the continuous and dynamic nature of video streams. However, in real-world scenarios, users usually ask multiple questions during the entire duration of a video stream, with potential relevance among questions and video clips. Such complex human-computer interaction necessitates mastering fundamental skills, including engaging in multi-turn dialogues and compre- hending extensive contextual histories to foster coherent and contextually relevant conversations. Although some current benchmarks partially assess essential interaction abilities, they are limited by relying on either multiple static images Feng et al. (2022) or short clips Zhao et al. (2018) to simulate multi-turn dialogues, rather than using long streaming videos. Therefore, appropriate benchmarks are required to evaluate the ability of streaming video understanding. To tackle these problems, propose the first benchmark for streaming video understanding, named Streaming Video under- standing Benchmark (SV Bench), which aims at comprchensively evaluating the temporal multi-turn dialogue capabilities of LVLMs for streaming videos.
First, we introduce a novel and challenging task named temporal multi-turn dialogue for streaming videos. We define a QA chain to represent series of consecutive multi-turn dialogues over a video segment. Subsequently, we define Temporal Linkages between successive QA chains for video segments, which can be established based on the common people, events, objects, etc. LVLMs should understand the current video segment, dialogue, as well as historical video segments and dialogues to answer the current question. For example, as shown in Figure to answer the question "Did any other racers fall?", an LVLM should recall the previously called racer in historical video segments and dialogues. Our proposed task aims to comprehensively evaluate the capability to leverage historical content and conduct multi-turn dialogue throughout a real-time streaming video.
Second, we construct a large-scale dataset with temporal multi-turn ceesstoo-annweriig chains for the proposed streaming video understanding task. We compare our dataset with existing video datasets in Table [] Our dataset comprises 1,353 diverse videos from 6 streaming platforms, each undergoing thorough filtering and meticulous selection. Coupled with streaming videos are anno-
2

---

Published as a conference paper at ICLR 2025
Table 1: The comparison of different datasets. Avg. Q/V: the average number of QA pairs per video. Open-Domain whether the video sources are diverse. Long: whether the average video length greater than minutes. Dialogue: whether there are contextual connections between QA pairs. Streaming: whether the QA pairs can be tested in syne with the video over time.
Dataset
#QAs
Avg. Q/V
Open- Dommann
Dialogue
Streaming
Amnolation
EgoSchema Mangalam et 2024
5,063
1.00
K
Auto&Marual Manual
ActivityNee-QA et et 2019
800
1,00
X
MVBench Li 2024
4,000
1.10
Auto
How2QA Li et 2020
44,007
2,00
V
Manual
Perception Text Patrauceun et 2024
44,000
3.79
X
Auto&&Manual
Social 10
Zudec et al 2019
7,500
6.00
X
Auto&Maaual
MSVD QA Xu al 2017
13,157
6.68
Auto
TVQA Lei al. 2018)
152,545
7.00
X
Manual
NEXT-QA Nino et al
202
52,044
9.57
X
Manual
MovieChat Song
2024
13,000
13.00
XX
X
Manual
LVBench Wang et
20244
1,549
15.04
X
Manual
TGIF-QA Jang al
2017]
165,165
17.25
X
Mutu&Manual
MSRVTI-QA Xu
2017
72,821
24.35
X
Auto
SVBench (Ours)
49,979
36.94
Mamual
tations comprising 49,979 QA pairs, where, on average, each video contains 36.94 pairs, which is the highest number among known video datasets. Moreover, we build temporal dialogue paths that occur in syne with a video progressing over time, as shown in Figure T designed specifically to assess the capability to reason effectively through time.
Third, we conduct extensive evaluations, in terms of dialogue and streaming performance, of various prevalent LVLMs our SVBench. Our results provide the first overview insight into the streaming video understanding capability of existing LVLMs. Surprisingly, these state-of-the-art LVLMs are far from satisfactory, in terms of streaming video understanding. These results motivate us to de- velop a stronger LVLM, namely StreamingChat, which significantly improves the overall dialogue evaluation score by 9.41% and the streaming evaluation score by 3.30% our SVBench, compared to the po-pefforming open-source LVLMs, while achieving comparable performance on conven- tional image and video benchmarks. Our benchmark and model will be publicly available, in order to catalyze the progress in streaming video understanding.
RELATED WORK
2.1 LARGE VISION-LANGUAAE MODELS FOR VIDEO
Recent advancements Large Language Models (LLMs) have paved the way for a significant re- search focus on Large Vision-Language Models (LVLMs) aimed at improving multimodal under- standing, such as multimedia retrieval Yang et al. (202b a) particulariy video content. Currently, in addition to the popular closed- source LVLMs such as GPT-4V Yang et al. (022). 4PT-4o/Aciaa al.) (2023). and Gemini 1.5 Pro Reid al. (2024), an increasing number of open-source LVLMs, including Video-ChatGPT Maaz et al. (2023), VideoLLaMA2 Cheng et al. (204)), and VILA Lin jet al. (2024), also have demonstrated the impressive capability video understanding tasks. Ad- vanced human-compuln interaction in everyday life requires the ability to engage in multi-turn dialogues and understand extensive contextual histories to maintain coherent and contextually ap- propriate conversations Qian al. 2024); Chen al. (2024a). However, these LVLMs are still not fully adept at handling the intricacies of streaming videos and do not completely grasp the com- plexities of real-world contexts. To rigorously evaluate the capabilities of these models, propose SV SVBench to measure the performance of LVLMs in video-related tasks that imitate the complexity of real-world interactions.
2.2 VIDEO UNDERSTANDING BENCHMARKS
In recent years, the exponential growth video data has elevated video understanding to a crucial area within computer vision. To rigorously assess the performance of LVLMs, researchers have
3

---

Published as a conference paper at ICLR 2025
huppeead?
Constructing QA Chains
2. Scone Splitting
1. Filtering Y
Video Source
racer
ACTIVITYNET
New end reser
YouCook2
MaveChel
racer felled
HQ
EGG
Panda 70M
G2: What was the
racare
Adlentilyingg S Temporal Linkages
Linking QA Chains
P
7. Temporal Dialogue Paths
Manual Annotation
DA
Modify OAs
just huppoeed!
Modil the GA the later duin socording to r
X Delea Linaages
at: fir the rRcet....
ELL Now dLd the recer...
ald
Quality Evalvatioon
/1:0bjact
no-Event
Sa-SSaddc-- Temporal
Speculation
Fally
Figure 2: Overview of the proposed SVBench framework: (1) Filtering raw videos from di-
verse streaming sources; (2) Detecting scenes and splitting videos accordingly; (3) Constructing QA chains for dialogues within videos; (4) Performing manual annotation and quality assessment; (5) Identifying temporal linkages between QA chains; (6) Connecting QA chains to facilitate temporal reasoning; (7) Building temporal dialogue paths for evaluating LVLMs.
introduced a range of standardized benchmarks. These benchmarks provide not only comparative evaluation criteria for models but also catalyze advancements in video understanding models. The recent benchmarks, such as TGIF-QA et al. (2017), MSVD-QA[Xu et al. (2017), and MVBench Li al. (2024), primarily comprise relatively brief videos capturing single events, thereby overlook- ing the temporal dependencies inherent in longer videos. To address the intricacies of long video comprchension, benchmarks using longer videos like movies have been developed. For instance, LLaMA-Vid Li et al. (2023), based on MovieNet Huang et al. (2020), has developed movie QA dataset to identify character relationships. Similarly, MovieChat Su et al. (2020) employs a diverse set of videos and avoids specific character names or plot details within its questions. However, these benchmarks often fall short in addressing the intricate challenges posed by streaming videos, which encompass extended temporal contexts and dynamic scene variations. Therefore, we estab- lish SVBench, a novel and comprehensive benchmark that aims to bridge this gap by offering an elaborate evaluation framework for long-context streaming video understanding.
DATASET
We develop a comprehensive data collection pipeline to construct a high-quality streaming video dataset, tailored for annotating temporal multi-turn dialogues. We split the SVBench dataset into training set and an evaluation set, ensuring that videos and their corresponding QA pairs appear in only one split. This results in 42,605 QA pairs for training and 7.374 QA pairs for evaluation, with 1,153 videos and 200 videos in each set, respectively.
3.1 DATA FILTERING AND SCENE SPLITTING
We source 12,989 raw video data from a variety of publicly available datasets, including YT- Temporal-1B Zellers al. (0022), YouCook2 Zhou al. (2018), ActivityNet Caba Heilbron et al. 2015). MovicChat Song et al. (2024). Panda-70M Chen et al. (024c)) and Ego4D Grauman et al.
2022 These datasets offer wide spectrum of video content, ensuring a rich diversity that essen- tial for streaming video understanding. We then filter high-quality videos of adequate lengths, high aesthetic scores, and appropriate optical flow scores. Subsequently, we employ DeSceneDetec to identify and enumerate scenes within filtered videos. These results are crucial in determining whether the video content exhibits adequate variation and complexity. Further filtering is conducted to retain only those videos containing 5 to 15 scenes, thereby excluding content that is either ex- cessively monotonous or overly intricate. Moreover, only videos with an appropriate average scene duration are chosen, ensuring fluidity and rhythm. Finally, 1,353 videos are selected. Ultimately, we split each video into clips = (8; i [ based on timestamps, merging clips shorter
4

---

Published as a conference paper at ICLR 2025
than 2 seconds with their adjacent ones. Notably, to prevent disjointedness and ensure continuity between scenes, we add an extra 0.5 seconds to both the beginning and end of each clip, resulting in one-second overlap between consecutive clips. More details are included in Appendix A
We propose semi-automated annotation pipeline for streaming videos, as shown in Figure in- cluding multi-stage LLM-assisted generation process with several rounds of manual annotation. See Appendix H for detailed prompts. The annotation takes about 3 months and involves over 30 professional annotators.
3.1.1 CONSTRUCTING QA CHAINS FOR VIDEO DIALOGUES
We propose creating multi-turn question-answeriig dataset videos to evaluate the ability of LVLMs conduct multi-turn dialogues and comprehend video-related contextual information. This ability is essential for the coherence and relevance of conversations various contexts. Initially, define a series of multi-turn question-ant aan--and-aae and-answer interactions on video clips as QA chains. For every video clip 8; V, we build a QA chain C; {(2). A3) 00 S < [Cl), where Qj and A represent the j-th question and answer generated on the i-th video clip. To achieve this, we harness the video understanding capabilities of existing LVLMs (e.g. GPT-4o):
C [C, (PL s;) 0 0 M).
where P. represents the prompt that generates 5 to 6 consecutive rounds questions and answers, and M |V is the number of clips segmented from the video. However, due the limitations of the video understanding and text generation capabilities of current LVLMs, we have to employ human annotators manually augment, delete, and modify the QA pairs so that the QA in the chain is connected and aligned with the video. For instance, specific persons or objects mentioned in questions should be modified to utilize third-person pronouns (e.g. he/she/it/they).
3.1.2 IMPLEMENTING QA QUALITY EVALUATION
Due to the inconsistent quality of manually annotated QA chains, devise a comprehensive evalu- ation mechanism to guarantee their high quality. We utilize GPT-4 to assess QA chain quality across 7 dimensions accuracy. completeness, relevance, fluency, contextual comprehension, logical con- sistency, and temporal understanding, scoring each from 0 to 100. In addition to the 7 dimensions, the QA chain quality also has overall score from 0 to 100, with 90-point minimum for high standards. Any QA chain failing to meet this threshold must undergo manual revision again. This iterative process ensures the production of highly reliable and insightful QA chains, contributing to the utility of our SVBench.
3.1.3 IDENTIFYING TEMPORAL LINKAGES
Given that adjacent QA chains are derived from sequential video clips, they inherently contain over- lapping entities such as objects, scenes, and events, well inherent temporal linkages. To effec- tively establish these temporal linkages, we initially employ LLMs (e.g. GPT-4) to search for and identify potential linkages between adjacent QA chains. For the junction of the i-th and the (i+ 1)-th QA chains, we construct set of relations R, < i < M 1) as follows:
R; = {ryl 0 j < R = LLM Ci, Ci+1),
where Pt is the prompt that stimulates the generation of candidate relations and contextual cate- gories between two given QA chains. Here, an individual relation ry is represented as a quintuple, structured as:
(3) which suggests that relation exists between the z-th QA pair within the QA chain of the i-th clip and the y-th QA pair within the subsequent QA chain, characterized by the relationship category Rc. Accordingly, we have delineated a range of relationship types to classify these relations, the outer ring as shown in Figure ((b)
ry A2 1 AAt RR),
3.1.4 LINKING QA CHAINS FOR TEMPORAL REASONING
To evaluate the ability to reason through time, we need to establish temporal linkages between successive QA chains, which facilitates multi-tur QA interactions between the chains throughout
5

---

Published as a conference paper at ICLR 2025
Disar
Cal Eackawe
Elffer wece
Canaua
lat Let
Freed
farh
Bualysis
Pytertlality
Exeat
lcseueent
Stanu Trannttene
disrecter
RAAA
yeforginc
Enoopria
Gertect
Video Distribution
(b) QA Distribution
Figure 3: Distributions of videos and QA categories.
the duration of the video content. Identifying potential connections between two consecutive QA chains allows us to create coherent narrative thread by adjusting the QA pairs the following chain. This process helps to maintain smooth multi-turn QA interaction across the entire video. The modified relation ry is as follows:
7 A2.
A 1++1 Rcc,
(4)
where Re, A and Q++1 respectively represent the modified relationship category, the question and answer in the subsequent chain after modification. During the modification phase, the fol- lowing criteria are met: (1) Contextual coherence within each QA pair of single chain (e.g., (Q:-::4-1) + (0t+1 .A**1) + (Q A+t1)) has to be preserved. (2) Logical links should be forged both within individual chains and between different chains, facilitate cross-clip reason-
ing, as illustrated by (Qi, A Ris (Q=+1, A (3) Through analytical reasoning and harnessing inter-clip information, repetitive, similar, and simple QAs should be changed into more in-depth and complex QAs. Given the difficulty of simultaneously meeting these stringent criteria, utilizing LLMs establish these linkages proves be impractical. Therefore, manual annotation is required, with the specific modification guideline outlined in Appendix
4 STATISTICAL ANALYSIS
We present a comprehensive statistical analysis of our dataset, which is the first dataset annotated with temporal multi-turn dialogues for streaming videos, to the best of our knowledge. Our dataset comprises 1,353 videos from distinct sources and stands out with an average of 36.94 semi- automatically annotated QA pairs per video, which is the highest among existing datasets. No- tably, it also consists of long videos with an average length exceeding 2 minutes. Moreover, each multi-turn dialogue in our dataset contains an average of 4.29 QA pairs and every video contains an average of 8.61 multi-turn dialogues. The details and comparisons are illustrated in Table [
Video Categories. Our dataset contains videos organized into 12 primary categories and 36 sub- categories, depicted in Figure ((a) which illustrates the the diversity and inclusiveness of video types within our dataset.
Question Categories. To facilitate a more comprehensive evaluation of the capabilities of LVLMs, we classify the questions into 9 distinct categories as shown Figure ((b Each category corre- sponds to the assessment for one specific skill of LVLMs. The criteria for these categories are as fol-
lows: (1) Intention Inference (II): Discerning the underlying intention behind actions characters.
(2) Potentiality Assessment (PA): Evaluating the feasibility of an action under certain conditions.
(3) Counterfactual Reasoning (CR): Analyzing outcomes by hypothesizing alternative scenarios.
(4) Temporal Speculation (STS): Understanding the spatial and temporal relationships within the video. (5) Relationship Inference (RI): Identifying and interpreting the relationships between
entities. (6) Character State and Transition (CST): Tracking the emotional states and transitions

---

Published as a conference paper at ICLR 2025
StreamingChat
Large Language Model
x
LoRA
TaDAa Video Frame Tokens
Tokeniaoe
MLP Projector
Vision Encoder
Queston3
Quesions
Quedbonn
Quatimn
Franies
Video
Figure 4: Architecture of the proposed Streaming model.
of characters under analysis within specific contexts. (7) Comparison and Trend Analysis (CTA);
Comparing different entities and analyzing emerging trends. (8) Common Sense Inference (CSI); Applying general world knowledge to provide logical framework. (9) Event-Centric Analysis
(ECA): Focusing on in-depth examination of significant events. Details of the above categories are provided in Appendix
STREAMINGCHAT
Model Architecture. Built upon InternVL2 Chen et al. (0024d). we develop a streaming LVLM baseline named StreamingChat. It comprises vision encoder (intern Chen all (2023 ) an MLP projector, and an LLM (InternL.M2 Cai et al. 20244), ), as illustrated in Figure 4 For the vision encoder, employ the Intern VIT model (pre-trained OCR-specific datasets) to extract video frame embeddings at 1 FPS. To enhance the efficiency of learning streaming video understanding capabilities, we fine-tune the model using static resolution strategy, which allows the model to handle several minutes of video and context within a 32k context window. The extracted frame embeddings are then fed into an MLP projector generate frame tokens, following the approach used in LLaVA- 1.5 Liu et al. (204aa). These frame tokens are interleaved with language tokens and input into the LLM, InternL Additionally, we incorporate LoRA Hu et al. (202T) in every linear layer of the LLM to facilitate efficient tuning.
combination of image captioning and
Supervised Fine-Tuning Data. We utilize the training data from our dataset for supervised fine- tuning to enhance the performance of StreamingChat Each data entry represents temporal dia- logue path, which converted into conversation data. The data format follows a multi-turn, multi- image structure. Specifically, the video multi-turn dialogue format involves sequentially inputting segments of the video and engaging in several rounds of dialogue for each segment until the en- tire video has been processed: <video>>Segmen 1 </video> <questtinnuuusoonnnn. I>-niweerr I>... <ques- <>>evide>>>>>>>>> Segment 2 </video>... Through this approach, we sequentially input video segments and engage in multiple rounds of dialogue for each segment until the entire video has been processed. Due to the context window limitation, we split temporal dialogue paths exceeding 100 frames into multiple segments for training.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
To effectively evaluate the performance of current LVLMs in streaming video understanding, we meticulously select diverse range of state-of-the the-ahh-rr art models, encompassing both open-source and closed-souree LVLMs. We design two distinct experimental setups within the SVBench evaluation set to rigorously assess the capabilities of these L.L.Ms.

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results of various models on SVBench in dialogue and streaming evaluation.
Dialogue Evaluation
Streaming Evaluation
Model
SA
CC
LC
TU
IC
OS
SA
CC
LC
TU
IC
OS
Open-source LVLMs
MovieChat
20.46
20.05
27.76
21.81
22.21
21.89
17.99
16.42
20.37
15.77
19.08
17.43
31.86
32.58
40.28
35.32
36.26
33.80
27.98
29.54
33.81
27.95
31.00
28.88
LLae
35.62
36.52
42.93
38.63
38.84
37.34
32.22
32.83
36.35
32.46
34.54
32.79
ShareGPT4Videe
39.01
40.42
47.89
41.42
43.18
40.70
34.65
36.70
41.07
35.76
37.22
35.79
Vidcol LaMA2
39.13
40.33
47.60
42.36
41.80
40.60
35.68
36.40
42.23
34.65
36.70
35.84
TimeChat
36.19
37.06
44.72
40.42
37.12
37.22
35.72
37.88
42.65
36.23
36.34
36.32
InternVL2
45.91
46.30
$2.67
49.81
46.25
46.13
43.55
44.10
48.91
40.95
44.17
42.71
VILA
46.83
48.41
54.92
48.30
50.12
48.51
46.19
47.95
51.60
44.84
48.56
46.26
InternI. Miternl.-MCoomeee 2.-poonp0s
51.57
53.93
59.69
51.57
56.28
52.31
52.22
53.39
58.14
48.05
54.79
51.46
MiniCPM-V2..
53.50
55.42
60.88
55.03
55.78
54.30
53.33
54.30
58.97
49.64
54.71
52.19
StreamingChal
39.48
61.31
66.05
58.61
61.09
59.41
55.10
56.66
60.72
51.78
55.87
53,90
Closed-source LVLMs
Gemini 1.5 Pro GPT-4V GPT-40
54.89
56.05
61.45
53.08
56.06
54.29
49.06
50.05
54.62
45.73
49.84
48.02
65.56
68.02
71.78
63.80
68.01
65.19
58.82
59.55
64.29
54.08
60.61
57.35
65.73
68.10
71.95
66.54
68.40
66.29
59.52
60.42
65.45
55.10
61.36
58.17
Dialogue Evaluation. In this setup, we evaluate the capabilities of LVLMs in understanding long- context scenarios within temporal multi-turn dialogues. Each LVLM is provided with contextual history that includes all preceding QA pairs up to the current timestamp in the QA chain. Once dialogue sequence concludes chronologically, the model transitions to the next video clip and addresses its associated QA chain. This process continues until the entire video is played and every question has been answered. This evaluation allows to assess the ability of LVL.Ss to maintain continuity over multiple turns and to respond while considering the accumulated context. The aim to simulate real-world scenarios where users pose a series of related questions while watching a video, requiring the model to track and integrate information over time.
Streaming Evaluation. Building upon the dialogue evaluation, this setup focuses on assessing the ability of LVLMs to perform temporal reasoning by introducing probabilistic transitions between related QA chains. Similar the above setup, models are presented with the context up to the current timestamp. However, when encountering questions that have temporal linkages to subsequent QA chains, there an 80% probability that the model will jump to the corresponding related question in the following chain. This evaluation aims to challenge the understanding of temporal dependencies and its capability to reason about the sequence of events across different but related video segments.
6.2 EVALUATION METRICS
Basic Metrics. Evaluating performance in streaming video understanding requires sophisticated metrics to capture various dimensions. Below, we outline the commonly-used evaluation metrics.
[BaneeBa Lavie (2005) evaluates the precision, recall, and alignment of words and phrases between the references and the ground truth by considering synonymy and stemming.
GPT4-Score assessed by GPT-4, evaluates the accuracy of generated answers solely based on the semantic similarity between single answer and the ground truth.
Dialogue Evaluation Framework. To evaluate the capabilities in both dialogue and streaming evaluations, it crucial to adopt a multidimensional framework that assesses the quality of multi- tum dialogues. We propose an LLM-based evaluation framework encompassing several key aspects that contribute to the holistic assessment of LVLMs. See Appendix H for detailed prompts.
Semantic Accuracy (SA) evaluates the accuracy of the generated answers based on a holistic understanding. It considers not only the direct overlap with ground-truth answers but also the context, coherence, and overall relevance of the response to the question posed.
Contextual Coherence (CC) examines the ability to maintain relevance and context across se- quential questions and answers, ensuring continuity and alignment with the evolving discourse.
Logical Consistency (LC) evaluates the logical progression and consistency of answers, ensur- ing that answers do not contradict each other or previous information.
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results on SVBench across 9 long-context streaming video understanding skills.
Model
PA
RI
CR
CST
CSI
CTA
ECA
STS
Open-source LVLMs
MovieChat Song ct al. (024))
21.91
21.32
18.82
17.78
24.58
19.23
16.77 7.
16.59
15.41
Mateo-ChatGPPMMaazz et 12023
35.85
40.12
31.18
29.37
39.38
29.95
25.77
26.97
22.79
TimeChat Reen al. a (024
33.26
43.62
29.86
30.58
39.97
28.49
27.08
28.10
24.22
VideoLLaMA2 Chene el al 2024
36.76
47.38
30.56
33.08
39.85
37.09
29.67
31.75
26.03
ShareGPT4Video Chen
39,92 47.54
33.61
33.56
43.69
35.97
28.98
31.25
26.43
Video-LLLaVA Lin at 22.2
38.99
46.42
35,86
34.27
46.09
35.65
31.21
31.20
26.67
37.10
51.19
38.25
36.79
36.02
41.66
32.22
33.24
29.07
VILA Lin ald 2024)
42.13
55.48
40.60
39.48
45.93
43.29
35.87
34.22
31.21
IntemLMMXCooppoeee Ziane et (024b)
47.64
57.71
40.49
41.72
50,00
47.20
36.99
39.53
33.90
MiniCPM-V V 2.6 Yao al 2024
43.25
50.45
39.93
39.96
51.61
47.40
37.31
38.67
33.43
Streaming (Ours)
53.94
71.96
50.22
50.49
59.26
53.46
44.60
47.68
37.99
Closed-source LVLMs
Gemini 1.5 Pro Reid et al. (2024)
41.29
43.32
42.24
39.10
47.20
50.80
39.19
34.78
35.98
GPT-4V Yang et al (2025)
56.84
61.57
49 10
51.31
58.57
55.76
47.47
47. .4
42.97
GPT-4o Achiam et al. (2026
57.95
59.47
52.29
49.97
56.63
58.65
49.02
47.47
44.58
Temporal Understanding (TU) assesses the model's proficiency in comprehending and rea- soning about temporal events and sequences depicted in the video content.
Informational Completeness (IC) measures the comprehensiveness to gauge whether the model captures and conveys all relevant elements from the video to provide thorough answer.
Overall Score (OS) is derived by aggregating the scores from each aforementioned criterion.
6.3 OVERALL PERFORMANCE
The dialogue and streaming evaluation results on SVBench, outlined in Table provide a com- prehensive comparison among various LVLMs. Notably, closed-source models such as GPT-4o and GPT-4V attain significantly higher scores across all metrics, with GPT-4o achieving an Overall Score (OS) of 66.29 in dialogue evaluation and 58.17 in streaming evaluation. Among the open-source models, StreamingChat and MiniCPM-V 2.6 stand out as top performers, achieving OS scores of 59.41 and 54.30 respectively in dialogue evaluation, and 53.90 and 52.19 in streaming evaluation. Notably, StreamingChat demonstrates a significant improvement over the original Intern VL2, with a 28. 39% increase in dialogue evaluation and a 26.20% increase in streaming evaluation. This un- derscores the effectiveness of SVBench training data for streaming video understanding tasks.
Additionally, we conduct performance comparisons before and after fine-tuning on 6 image and video understanding benchmarks, as illustrated in Figure The results indicate that the fine-tuned StreamingChat shows substantial improvements over the original VL2 on SVBench. While there are slight decreases in performance on the image benchmark MMBench Liu et al. (2023) and the video benchmark MMBench- Video Fang et al. (2024). there modest gains the video benchmarks VideoMME Fu et al. (2024) and MV NTVBencc]p Li et al. (2024). These findings suggest that StreamingChat enhances streaming video understanding capabilities without compromising funda- mental image and video comprehension skills.
It is evident that scores in streaming evaluation are consistently lower compared those in dialogue evaluation. This discrepancy can be attributed to the inherent complexity of streaming evaluation, which demands seamless comprehension and processing of dynamically evolving video content. Unlike dialogue evaluation, which deals with relatively static and contextually stable inputs, stream- ing evaluation necessitates understanding and integration of extended temporal contexts and dy- namic scenes, posing significant challenges for current models.
6.4 PERFORMANCE ANALYSIS OF VIDEO UNDERSTANDING SKILLS
The analysis of model performance across long-context streaming video understanding skills (see Section 4 in SVBench reveals significant disparities, underscoring the inherent challenges posed by different types of skills. According to Table 3 skills such as Intention Inference (II) and Po- tentiality Assessment (PA) generally exhibit higher performance across most models, particularly in closed-soures LVLMs such as GPT-4V and GPT-4o. This suggests that both open-source and closed- source models relatively adept at deciphering character intentions and predicting potential future

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study single-instance (Sin.)
SVBasch-dreaning
and multi-turn (Mul.) QA evaluation.
53,00
METEOR GPT4-Score
Model
62.73
Sin.
Mul.
Sin.
Mul.
Open-source LVLMs
MovieChat
19.20
23.77
17.18
21.81
Video-NNE
Video-ChatGPT
25.93
30.44
26.83
34.62
Video-LLLaVe VA
26.29
31.51 30.65
40.03
TharGPPT4Vider
26.31
31.38 30.41
39.68
VideoLLaMA2
24.52
30.87 29.67
38,88
TimeChat
25.50
27.47 26.64
34,42
Intern VL.2
26.85
32.74
32.26
42.25
VILA
28.44
33.57
34.63
45.41
23.32
31.02
37.11
49.69
Detore fine-uringg
HHMBanch-iidee
WiDBanch
MiniCPM-VV 2.6
27.36
33.90
36.73
48.57
StreamingChat
32.94
35.58 43.85
57.94
Figure 5: Performance comparisons on 6 image and video understanding benchmarks: Before fine-tuning (InternVI.)) and after fine-tuning (StreamingChat).
Closed-source LVLMs
Gemini 1.5 Pro
26.63 32,91
36.85
48.83
GPT-4V
28.66
36.47
44.94
60.11
GPT-44o
28.81
36,84 45.37
60.70
actions. Conversely, more complex skills like Counterfactual Reasoning (CR) and Spatio-Tempora Speculation (STS) manifest comparatively lower accuracy. This trend is echoed in closed-source models, where even the high-performing GPT-4V and GPT-4o models show relative declines in per- formance, though still maintaining superior scores compared to other models. This drop can be attributed to the cognitive demands of these tasks, which involve abstract reasoning, intricate sce- nario construction, and temporal-spatial awareness-areas that current models struggle to emulate effectively. RI and ECA also pose significant challenges (see Appendix H).. Both involve multi-entity and context-sensitry interactions. leading to performance variability. Notably, Streaming out- performs other open-soure LVLMs across skills, and even surpasses glosed-source models like GPT in PA, CST, and ECA. Nearly all models scored below 60 on 9 long-contex. streaming video understanding skills, suggesting that SVBench poses a challenging task.
6.5 ABLATION STUDY
Ablation studies are conducted to assess the effectiveness of dialogue evaluation (multi-turn QA de- noted as "Mul.") versus traditional evaluation (single-instance QA denoted as "Sin.") in our dataset. As shown in Table models generally exhibit improved metrics on both METEOR and GPT4-Scom when additional contextual information from previous QAs is incorporated. This trend is consistent across both open-source and closed-source LVLMs. These results indicate that dialogue evaluation significantly enhance the performance of models when applied to streaming video understand- ing. Unlike the traditional single-instance QA evaluation, the dialogue evaluation leverages the rich contextual information accumulated from previous interactions, thereby providing more compre- hensive evaluation framework. Despite the overall observed improvements in performance enabled by dialogue evaluation, there are notable instances where models exhibit relatively low scores even in this enhanced setting. These instances provide valuable insights into the limitations and areas for improvement existing frameworks for streaming video understanding.
CONCLUSION
This paper presents SVBench, a novel benchmark for the assessment of long-context streaming video understanding. SVBench comprises a diverse collection of 1,353 streaming videos from streaming platforms and 49,979 meticulously annotated question-answer pairs for temporal multi- turn dialogues. Our experiments reveal that while state-of-the-art LVLMs have made strides in single-instance video QA, their performance on streaming videos falls short of human-level accu- racy. Motivated by this, we develop a StreamingChat model, which significantly outperforms open- source LVLMs our SVBench and achieves comparable performance diverse vision-land -laguuee benchmarks. By providing a challenging benchmark, we hope to stimulate the development of ad- vanced models capable of tackling the complexities of streaming video understanding.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENT
We would like to present our appreciation to the anonymous reviewers and ACs for their constructive suggestions. This work is supported by the National Key Research and Development Program of China D1.20233FF33100200
the Beijing Natural Science Foundation (JQ23018), and the National Natural Science Foundation of China (No.62036012, 62276257).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal 1lms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.00 00440133,,, 2024.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv. Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang. Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang. Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang. Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Joya Chen, Zhaoyang Lv, Shiwei WU, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418, 2024a.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint arXiv:2406.04325, 2024b.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Fun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13320-13331, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint Xiv:23112.14258 2023.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 1682/, 2024d.
11

---

Published as a conference paper at ICLR 2025
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-tempore modeling and audio understanding in video-llms. arXiv preprint arXhv:2406.07476, 2024.
Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12934 12943, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406. 14515, 2024.
Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang. Chongyang Tao, Dongyan Zhao, and Qingwei Lin. Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.0577 2022.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal llms in video analysis. arXiv preprini arXiv:2405.21075. 2024.
Jingsheng Gao, Yixin Lian, Ziyi Zhou, Yuzhuo Fu, and Baoyuan Wang. Livechat: A large-scale personalized dialogue dataset automatically constructed from live streaming. arXiv preprint arXiv:2306.08400 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0966 096885 2021.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 709-727. Springer, 2020.
Yunscok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7758-7766, 2017.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video 2018.
question answering. arXiv preprint
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li. Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et Mybench: comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22200 2024.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language ommi-representation pre-training. arXiv preprint arXiv:2005.0022 2005.00000, 2020.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint 1rXiv:2311.17044 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
12

---

Published as a conference paper at ICLR 2025
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296- 26336, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXivv2307.06281/ 2023.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding arbitrary resolution. arXiv preprint arXiv:2409.122 12961, 2024b.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint wrXiv:2306.05524. 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik, et al. Egoschema: A diagnostic benchmark for very long-fon form-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405. 16009, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763. PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.0555 05530, 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313 -4323, 2024.
Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18233 2024.
Hui Su, Xiaoyu Shen, Zhou Xiao, Zheng Zhang, Ernie Chang, Cheng Zhang, Cheng Niu, and Jie Zhou. Moviechats: Chat like humans in closed domain. In Proceedings the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 6605-6619, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing dision-lanuuee model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
13

---

Published as a conference paper at ICLR 2025
Weihan Wang. Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406. 08035, 2024b.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal ilms to 1000 images efficiently via hybrid architecture, 2024c. URL https://
arxiv. htw/ abs 2409. 02889
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Pro- ceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos. 2024.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint WrXiv:2309.17421, 9(1):1, 2023.
Zhenyu Yang, Shengsheng Qian, Dizhan Xue, Jiahong Wu, Fan Yang, Weiming Dong, and Chang- sheng Xu. Semantic editing increment benefits zero-shot composed image retrieval. In Proceed- ings of the 32nd ACM International Conference on Multimedia, pp. 1245-1254, 2024a.
Zhenyu Yang, Dizhan Xue, Shengsheng Qian, Weiming Dong, and Changsheng Xu. Ldre: Llm- based divergent reasoning and ensemble for zero-shot composed image retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 80 90, 2024b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:2408.08 800, 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8807-8817, 2019.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16375-16387, 2022.
Haoji Zhang, Yiqin Wang. Yansong Tang, Yong Liu, Jiashi Feng. Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406. 20l.:2006.08855, 2024a.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hur Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understand- ing. arXiv preprint arXiv:2312.0488/7 2023.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm- wwwpll--xoomoser-2225 A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03322 2024b.
14

---

Published as a conference paper at ICLR 2025
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Leava-next: A strong zero-shot video understanding model, April 2024c. URL io -4
next video/
Zhou Zhao, Xinghua Jiang, Deng Cai, Jun Xiao, Xiaofei He, and Shiliang Pu. Multi-turn video question answering via multi-stream hierarchical attention context network. In IJCA,, volume 2018, pp. 27th, 2018.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Appendix
Details of Data Collection
16
A.I
Data Filtering
16
A.2
Aesthetic Assessment
17
A.3
Optical Performance Assessment
17
A.4
Scene Detection and Video Splitting
17
A.5
Distribution of Video Lengths
17
Question Categories
17
B.
Intention Inference
17
B.2
Potentiality Assessment
17
B.3
Counterfactual Reasoning
18
B.4
Temporal Speculation
18
B.5
Relationship Inference
18
B.6
Character State and Transition
18
B.7
Comparison and Trend Analysis
B.8
Common Sense Inference
18
B.9
Event-Centre Analysis
18
Modification Guideline
18
C.1
Action
18
C.2
Quantity
19
C,3
Person
20
C.4
Object
20
C,5
Event
21
6
Environment
21
D
Primary Evaluation Metrics
21
Case Study
22
E.1
Benchmark Case
22
Model Case
22
Additional Experiments
22
F.1
Comparison of LLM-based Evaluations with Human Evaluation
22
F2
Correlation Between Inference Speed and Consumption with Video Length
23
3
Additional Evaluation Results
23
F.4
Impact of Model Size on Performance
23
Real- Time Leaderboard
24
H
Prompts
24
A DETAILS OF DATA COLLECTION
A.1 DATA FILTERING
Initially, we source raw video data from a variety of publicly available datasets, including YT- Temporal- YouCook2, ActivityNet, MovieChat, Panda-70M, and Ego4D. These datasets offer a wide spectrum of video content, ensuring a rich diversity that is essential for streaming video understanding. In our initial filtering stage, enforce minimum duration threshold, excluding videos shorter than minute to ensure sufficient temporal information depth. Subsequently, we employ aesthetic assessments to retain videos with aesthetic score of 4 or above, thereby ensuring video clarity and visual quality. Furthermore, we filter videos with optical flow scores within the range of 0.5 to 100, capturing both visual complexity and motion coherence.
16

---

Published as a conference paper at ICLR 2025
A.2 AESTHETIC ASSESSMENT
To further ensure video clarity and visual quality, we apply Open-Sora to assign score the aesthetic appeal of videos. Videos with an aesthetic score of 4 or above are retained. This criterion ensures that videos within our dataset are not only content-rich videos but also with high aesthetic appeal.
A.3 OPTICAL PERFORMANCE ASSESSMENT
Furthermore, we apply Open- -een-Sora for assigning optical flow scores to each video within our dataset as well, which reflects both the visual complexity and the motion coherence of video. We filter videos with optical flow scores within the range of 0.5 100 ensure the quality of videos within dataset.
A.4 SCENE DETECTION AND VIDEO SPLITTING
After the filtering and the assessment above, we employ advanced scene detection algorithms to identify and enumerate scenes within each video. These results are crucial in determining whether the video content exhibits adequate variation and complexity. Further filtering is conducted to retain only those videos containing 5 15 scenes, thereby excluding content that is either excessively monotonous or overly intricate. Moreover, we calculate the average duration scenes for each video. Only videos with average scene duration between and 30 seconds are chosen, ensuring fluidity and rhythm that are essential for effective streaming data analysis. Ultimately, we split each video into clips based timestamps, merging any clips shorter than 2 seconds with their adjacent ones. Notably, to prevent disjointedness and ensure continuity between scenes, we add an extra 0.5 seconds both the beginning and end of each clip, resulting in a one-second overlap between consecutive clips.
A.5 DISTRIBUTION OF VIDEO LENGTHS
Figure 6 shows the distribution of video lengths. The results indicate that 95.05% of the videos in the dataset are longer than minute, primarily ranging from 60 to 240 seconds.
B QUESTION CATEGORIES
All the questions within our dataset fall into nine categories follows, with each category corresponding to the assessment for one spe- cific skill of LVLMs.
Video Length Dievibuttion
B. INTENTION INFERENCE
category aimed at delving into the underly- ing intent behind action. It assesses whether LVLMss truly comprehend the latent motivation behind action, rather than simply observing the action itself.
Leegt
Figure 6: Distribution of video lengths.
B.2 POTENTIALITY ASSESSMENT
A category that involves inferring the feasibility of action under certain conditions its potential outcomes the context changed. It evaluates whether LVLMs understand the nature an action and the conditions required for realizing action.
17

---

Published as a conference paper at ICLR 2025
B.3 COUNTERFACTUAL REASONING
A category that entails hypothetical reasoning within a scenario opposite to the existing facts. assesses the capacity of LVLMs for causal understanding and recognizing the significance of vari- ables.
B.4 SPATIO-TEMPORT TEMPORAL SPECULATION
category concerned with placing events or actions in particular temporal spatial contexts to infer their implications, differences, or changes. It assesses the capacity of LVLMs to comprehend the influence of temporal and spatial factors on entities.
B.5 RELATIONSHIP INFERENCE
category focusing on exploring the links associations between entities (such as persons or objects). It evaluates the capacity of LVLMs to understand the nature of relationships on the basis of observed interactions.
B.6 CHARACTER STATE AND TRANSITION
A category that involves dissecting emotional states and transitions of characters under analysis within specific contexts. gauges the capacity LVLMs to accurately perceive emotional states and the potential impact of contextual changes.
B.7 COMPARISON AND TREND ANALYSIS
A category dedicated to the comparison and trends across different entities (objects, actions, or events). It assesses the capacity of LVLMs accurately discern similarities and differences of different entities and analyse the changes in trends.
B.8 COMMON SENSE INFERENCE
category that employs common sense or established facts to provide logical framework for contextual inference. It evaluates the capacity of LVLMs to understand the essences of entities and utilize common sense for accurate inference.
B.9 EVENT-CENTRIC ANALYSIS
category devoted in-depth examination of significant events, including investigation of trigger- ing factors and their effects on the overall narrative trajectory. It evaluates the capacity of LVLMs comprehend the nature and future trajectories of the events.
C MODIFICATION GUIDELINE
In the modification phase, adherence to specific criteria is essential. To this end, propose comprehensive modification guideline.
Due our categorization of QA pairs into six types of relationships, the modification guideline we propose also divided into six major modules, including Action, Person, Object, Event, En- vironment, and Quantity. Each module contains more detailed methods of modification to assist annotators in revising questions to better evaluate diverse skills of LVLMs.
C. ACTION
adjacent chains contain QA pairs related to action, QA pairs within the subsequent chain can be modified according the methods mentioned in the Action module. The specific method of modification is determined by whether the actions involved in the QA pairs within the adjacent chains the same.
18

---

Published as a conference paper at ICLR 2025
Table 5: Results of Different Primary Evaluation Metrics Various LVLMS
Model
BLEU-4
METEOR ROUGE-L
CIDEr GPT4-Score
Open-source LVLMs
MovieChat 1.87
23.77
17.74
14.68
21.81
4.69
30.44
26.17
51.60
34.62
Video-LLaVA
5.38
31.51
26.61
62.84
40.03
ShareGPT4Video
4.51
31.38
26.64
52.60
39.68
VideoLLaMM2
4.43
30.87
24.40
49.07
38.88
TimeChat
4.56
27.47
25.63
64.52
34.42
Intern VL2
5.96
32.74
27.32
67.57
42.25
VILA
6.37
33.57
29.30
75.31
45.41
4.36
31.02
22.90
39.18
49.69
MiniCPM-V 2.6
5.94
33,90
27.95
73.97
48.57
StreamingChat
8.30
37.58
34.92
91.11
57.88
Closed-source LVLMS
Gemini 1.5 Pro
5.78
32.91
27.03
78.51
48.83
GPT-4V
6.67
36.47
29.58
82.45
60.11
GPT-4o
6.93
36.84
29.77
83.69
60.70
the actions are the same (Q1 is similar to Q2 and A1 similar to A2), the following modification methods can be adopted:
Modify the question based on the purpose of the action. Change Q2 to "What is the purpose or intent of this action?"
Modify the question based on the likelihood of the action occurring. Change Q2 "Is this action possible under the given conditions? If so, what conditions need to be met?"
Modify the question based on the purpose of the action and spatio-temporal inference. Change Q2 "What is the purpose or intent of this action at specific time and place?"
Modify the question based on counterfactuals. Change Q2 to "What different outcomes or impacts might result from taking different action?"
Modify the question based on counterfactuals and spatio-temporal inference. Change Q2 to "What different outcomes or impacts might result if this action were performed at another time and place?"
Modify the question based on the sequence of actions. Change Q2 to "What is the action before/after this one?" or further modify Q2 "How does the action before this one affect the execution of the current action?" to deepen the inquiry.
Modify the question based on the comparison of multiple actions. Change Q2 to "Which action specific time and place more complex/simple compared a specific action at a previous time and place?" or further modify Q2 to "How long is the interval between these two actions in time?" deepen the inquiry.
If the actions different (QI is not similar to Q2 and A1 is not similar to A2), the following modification methods can be adopted:
Modify the question based on sequence. Change Q2 to "What action is taken next?"
C.2 QUANTITY
adjacent chains contain QA pairs related quantity, QA pairs within the subsequent chain can be modified according to the methods mentioned in the quantity module, as follows:
Modify the question based on the comparison of two quantities. Change Q2 to "Com- paring the quantities of two categories/types of things, which is more/less?" For example,
19

---

Published as a conference paper at ICLR 2025
if QI is "How many people are involved the field?" and Q2 is "How many red balls are on the field?" change Q2 to "Which is greater in number on the field, the balls or the people?"
Modify the question based on the comparison of two quantities and the trend of change. Change Q2 "What the trend of change in the quantity of these things un- der certain context/time?" or further modify Q2 to "Does this change have periodicity or regularity, and if so. what the nature of this regularity?" deepen the inquiry.
C.3 PERSON
If adjacent chains contain QA pairs related to people, QA pairs within the subsequent chain can be modified according the methods mentioned in the person module, and the specific method of modification is determined by whether the people involved in the QA pairs within the adjacent chains are the same.
If the people involved are different (P2 is different from PI), the following modification methods can be adopted:
Modity the question based on the social relationship between two people. Change Q2 "From the existing video, what is the relationship between P2 and P1?" ?" or further modify Q2 to "Does their relationship change in different plots of the video?" to deepen the inquiry.
Modify the question based on the relative positional relationship between two people. Change Q2 to "In the most recent scene where PI and P2 appear together, what is the po- sitional relationship between P2 and P1?" or further modify Q2 to "How does the distance and direction between them change in certain context/time?" deepen the inquiry.
Modify the question based on the interaction between two people. Change Q2 to "What is the interaction between characters in certain context/time?"
the people involved are the same (P2 is the same as PI), the following modification methods can be adopted:
Modify the question based on the emotional state of the person. Change Q2 to "Based specific context/time in the video, how does this person's 's emotional state change?" or further modify Q2 "How does the change in emotional state affect behavior?" to deepen the inquiry.
Modify the question based on the person's identity background. Change Q2 to "What is the background of this person?"
C.4 OBJECT
adjacent chains contain QA pairs related to the same object, QA pairs within the subsequent chain can be modified according the methods mentioned in the object module, follows:
Modify the question based on inference. Change Q2 to "How does the existence or
characteristics of this object affect other objects or events?" or further modify Q2 to "If the object no longer exists or characteristies change, how would the event differ?" to deepen the inquiry.
Modify the question based on counterfactuals. Change Q2 to "If certain attributes or characteristics of this object different, what different outcomes or impacts might there be?"
Modify the question based on outcomes. Change Q2 to "What is the contribution of this object to a certain event or phenomenon?"
Modify the question based on counterfactuals and spatio-temporal inference. Change Q2 to "If this object were not present at certain time and place for a certain event or phenomenon, what different impact or outcome might there be?"
Modify the question based on impact and spatio-temporal inference. Change Q2 to "What is the contribution of this object to a certain event or phenomenon at specific time and place?"
20

---

Published as a conference paper at ICLR 2025
Modify the question based counterfactuals and general knowledge. Change Q2 to "If certain attributes or characteristics of this object were different, how might affect our common known facts?"
Modify the question based on impact and general knowledge. Change Q2 "How does the existence or characteristics of this object affect our common sense or known facts?"
Modity the question based on state evolution. Change Q2 to "How does the state of this object change in a certain context/time in the video?"
Modity the question based on the trajectory of action. Change Q2 to "What is the trajectory of this object's action in certain context/time in the video?"
C.5 EVENT
If adjacent chains contain QA pairs related the same event, QA pairs within the subsequent chain can be modified according to the methods mentioned in the event module, as follows:
Modify the question based sequence. Change Q2 to "What happens next?"
Modify the question based on the relationship of events. Change Q2 "What event caused this event?" or further modify Q2 to "What kind of chain reaction did the occurrence this event cause?" to deepen the inquiry.
Modify the question based on event classification. Change Q2 to "To which category of events does this event belong (e.g., natural disasters, social events, etc.)?"
Modify the question based on the impact of events. Change Q2 to "What impact will this event have?"
Modify the question based event prediction. Change Q2 to "Based the current situation, what events might occur in the future?" or further modify Q2 to "Based on the current situation, how likely is it that certain events will occur in the future, and what the basis for this?" to deepen the inquiry.
C.6 ENVIRONMENT
adjacent chains contain QA pairs related to the same environment, QA pairs within the subsequent chain be modified according the methods mentioned in the environment module, follows:
Modify the question based on environmental changes and trends. Change Q2 to "Com- pared to before, what changes there in the current environment?" further modify Q2 to "How will this environment change afterward?" to deepen the inquiry.
Modify the question based on counterfactuals. Change Q2 to "Without this environment, what impact would there be?"
D PRIMARY EVALUATION METRICS
As shown in the Table 5] we employ four primary evaluation metrics to assess the answers generated by various LVLMMs mentioned and compare the evaluation results with those evaluated by GPT- 4. It can be observed that the results from these four primary evaluation metrics are broadly in agreement with the evaluation made by GPT-4, which suggests that our SVBench capable of accurately and effectively distinguishing distinct levels of capabilities possessed by various LVLMs. Just like the use of sentence similarity Di Xie (2024) QAEgo4D as an important evaluation metric for computing distances in semantic space, will subsequently use semantic similarity as an evaluation criterion to assess the consistency between the model's output answers and the ground truth.
21

---

Published as a conference paper at ICLR 2025
123
Figure 8: Correlation between inference speed Figure 9: Correlation between resource con- with video length.
sumption with video length.
E
CASE STUDY
E. BENCHMARK CASE
We employ qualitative comparisons in the necessity of contextual content and the responses within QA chain between our SVBench and answers generated by both GPT-4V and GPT-4o, with details in both the Figure and the Figure [2] Furthermore, we conduct additional comparisons in streaming and nine skills assessment between our SVBench and answers generated by GPT-4o, shown in Figure 13 and Figure [4 These comparisons above present that our SVBench a challenge to the existing LVLMs.
question-answer
E.2 MODEL CASE
As demonstrated by Figure 15 the current best-performing open-source model, MiniCPM-V 2.6, is unable to analyze the questioner's intent based contextual information and provide the most ap- propriate answer. MiniCPM-V 2.6 relies on the most accurate information from the current question and video segments insufficient), but it overlooks the coherence and contextual information between questions.
accurately as possible (and fails respond when information is
F ADDITIONAL EXPERIMENTS
F.1 COMPARISON OF LLM-BASED EVALUATIONS WITH HUMAN EVALUATION
To ensure consistency with LLM assessments, we incorporate human involvement the gen- eration process and human evaluation. We add human evaluations of various LVLMs results (see Figure Z To validate the consistency between human scores and those from open- source and closed-source models, employ human evaluation (10 people annotating 200 videos over a week), open-source model evalu- ation (Intern VL2-Llama33-76B), closed-source evaluation model (GPT-4), and traditional eval- uation metrics (METEOR). We then plot score comparison on Multi-turn QA Evalua- tion. The results indicate: (1) The score variations between Human and GPT-4 are generally consis- tent across different models, though human scores are more discriminative, suggesting that GPT-4 scores are reasonably reliable; (2) Intern LL2--LLma-33333 shows excessive leniency, with more than half of the models scoring above 60 using the same prompt as GPT-4, while METEOR scores are too low, lacking discrimination.
Scam Compurioa 17amm Feslactio))
Figure Score comparison on multi-turn QA evaluation (Mul.).
Regarding the circular dependency issue, we mitigate the reliance on LLMs by incorporating two stages of manual annotations during the construction of annotations. Additionally, the inclusion of open-source and human evaluations enriches the evaluation results, further reducing the dependency on GPT-4 for evaluation.
22

---

Published as a conference paper at ICLR 2025
Table 6: Evaluation results of various models SVBench in dialogue and streaming evaluation.
Dialogue Evaluation
Streaming Evaluation
Model
SA
CC
LC
TU
IC
OS
SA
CC
LC
TU
IC
OS
Human
88.43
87.47
89,58
88.29
76.15
83.93
83.57
80.71
87.86
86.43
75.71
80.24
Open-source LVL.s
MovieChat
20.46
20.05
27.76
21.81
22.21
21.89
17.99
16.42
20.37
15.77
19.08
17.43
Video-ChatGPP
31.86
32.58
40.28
35.32
36.26
33.80
27.98
29.54 54
33.81
27.9
31.00
28.88
LLae
35.62
36.52
42.93
38.63
38.84
37.34
32.22
32.83
36.35
32.46
34.54
32.79
PTareeo
39.01
40.42
47.89
41.42
43.18
40.70
34.65
36.70
41.07
35.76
37.22
35.79
VideoLLaMaa LLMM2
39.13
40.33
47.60
42.36
41.80
40.60
35.68
42.23
36.70
35.84
Flash-VShe VStream
38.23
41.56
51.51
45.99
43.32
42.69
36.78
37.50
40.48
32.46
35.23
35.92
TimeChat
36.19
37.06
44.72
40.42
37.12
37.22
35.72
37.88
42.65
36.23
36.34
36.32
Intern InternVL2
45.91
46.30
52.67
49.81
46.25
46.13
43.55
44. 10
48.91
40.95
44.17
42.71
LLa
VA-NeAT--TTTT Video
43.56
48.33
51.79
48.54
47.40
45.48
40.79
48.84
54.61
43.03
48. 38.32
45.36
VILA
46.83
48.41
54.92
48.30
50.12
48.51
46.19
47.95
51.60
44.84
48.56
46.26
Qwen2-Vi LL
48.52
50.65
54.75
43.04
48.59
48.65
48.94
52.46
53.24
48.32
50.70
49.48
IntemLLM-XCopossee CCoppose2s
51.57
53.93
51.57
56.28
52.31
$2.22
53.39
58.14
$1.46
MiniCPM- V 2.6
53.50
55.42
60.88
55.03
55.78
$4.30
53.33
54.30
58.97
49.64
54.71
$2.19
StreamingChat
59.48
61.31
66.05
58.61
61.09
59.41
55.10
56.66
60.72
51.78
55.87
53.90
Closed-source LVLMs
Gemini 1,5 Pro GPT-4V GPT-40
54.89
56.05
61.45
53.08
56.06
54.29
49.06
50.05
54.62
45.73
49.84
48.02
65.56
68.02
71.78
63 80
68.01
65.19
58.82
59.55
64.29
54.08
60.61
57.35
65.73
68.10
71.95
66.54
68.40
66.29
59.52
60.42
65.45
55.10
61.36
58.17
F.2 CORRELATION BETWEEN INFERENCE SPEED AND CONSUMPTION WITH VIDEO LENGTH
Figure and Figure show the inference speed and resource consumption of multiple LVLMs in relation video length. The inference time generally positively correlated with the video length. For resource consumption, some models show plateau as the input length increases, due to the saturation of input frames and the memory reaching preset limit.
F.3 ADDITIONAL EVALUATION RESULTS
We include comparison with human performance, shown in which can provide valuable in- sights into the gap between current models and human capabilities long-context streaming video understanding. The results show that human performance significantly outperforms various open- source and closed-source models across all metrics in both evaluation settings (Dialogue Evaluation and Streaming Evaluation). Additionally, humans excel in Temporal Understanding (TU) but per- form relatively weaker in Informational Completeness (IC). We also included LVLMs such as Flash- VStream-7BB Zhang et al. (2024a). Qwen2-VL-7B Wang al. (2024)), and LLa LLaVA-NeXT-Videe 7B-DPO Zhang et al. (2024c). We will also add the evaluation results for the models (Oryx Liu al. (2024b), Long- LLa Wang al. (2024c), Long Xue et al. (2024)) to Table 6 as soon as possible.
F.4 IMPACT OF MODEL SIZE ON PERFORMANCE
To ensure a fair comparison, we select base models with 7B 8B parameters. Our bench- mark is an ongoing project, and we plan to update it with different sizes of mainstream Video-LLMs, maintaining real-time updated leaderboard. Additionally, we conduct fur- ther experiments comparing the performance of models with different L.M sizes, specifically Intern
Susees Uifiiennt Indertit Sizes (Siewaming Exaluctio DSI
40.2
G3T-4 Ooers Store
InternVL.2228 VI.2-BB, 4B, 8B, 26B, 40B, and 76B, on SVBench, as shown in Figure [0] The results indicate that model size significantly impacts performance, with larger models demonstrating better accuracy. Interestingly, the performance of Intern VL2 with 76B parameters is actually lower compared the 40B parameter version.
Model's
Figure 10: Impact of model sizes performance of LLte
23

---

Published as a conference paper at ICLR 2025
Both models use the same vision part, InternViT-6B. but the LLM has been changed from Yi-based to Llama3-bart based. This suggests that Llama3 may have weaker visual language processing capabili- ties, we also observed similar decrease in performance MVBench.
G REAL-TIME LEADERBOARD
Our benchmark is ongoing project, and plan to continuously update it with various mainstream Video-LLMs maintaining a real-time updated leaderboard. Given the resource-intensivv nature of evaluating large number of newly emerging models using GPT-4o, have switched using the latest and best open-source model, Swen2.5572B--nstmman for our assessments and scoring. The real-time updated leaderboard can be found at https://gtthu.....tb/t/ www.cc///y/y.ceeee wutyyyyytt///eenenee
H PROMPTS
Below the prompts used for generation of QA chains within our dataset and evaluation of answers generated by all the LVLMs mentioned.
24

---

Published as a conference paper at ICLR 2025
Prompt for generating QA chains
Task:
Please construct chain of 5-6 consecutive open-ended QA pairs based on a series of video frames arranged in temporal order. 'The chain should follow the rule that, except for the first question, the content of each subsequent question must continue from the answer to the previous question. Vague questions like "What is he doing?" are permissible. If the correct answer to the previous question is not provided, the current question cannot be answered. The chain must not leapfrog to content beyond the answer to the previous question. Generated example:
{{ ""questions"::
"Did any other competitors fall?",
"How did he fall?",
"What about the other fallen competitor?",
"What does he look like?",
"answers":
"Another rider dressed in red, white, and blue also fell.",
"He tried to avoid the first person who fell but failed.",
"The other fallen competitor off the track and sat down by the side of the track.", "He
looks seriously injured.".
}}}
Required output:
Generate the QA chain following the format of the provided example, and output the chain in JSON format. Do not produce any additional content. The QA chain should contain two key-value pairs: "questions" and "answers". "Questions" is list of all the questions generated in the QA chain; "answers" is list of all the answers generated in the QA chain. Ensure that the questions in the QA chain are open-ended and can explore multiple different aspects of the video content without focusing too much on the details. Reasonable inquiries should be made about the following content: complex plot understanding, such "What the little boy's motive throughout the video?"; inference of implied information, such as "How did the little boy and little girl in the video meet?"; analysis of emotional coherence, such as "At which moments in the video does the boy feel the happiest?"; understanding the magic trick accomplished?"; association and
of complex actions, such "How memory of details, such as "What is the connection between the first and last scenes in the video?"; multi-level plot analysis, such as "What is the turning point of the plot?".
Ensure that the questions and answers in the QA chain are strictly based on the video content itself, constructed only from the direct information in the video, avoiding any un- necessary speculation over-association.
Ensure that the questions in the QA chain are clear and precise, directly corresponding to specific information events in the video, and can be answered by watching the video content without the need for a video description or inference, avoiding questions that require assumptions reasoning.
Ensure that the text of the questions and answers in the QA chain does not include specific time descriptions such "which second".
Ensure there are no references to the source of information in the QA, avoiding expressions like "from the image", "sequence of pictures", "which frame". or "which photo"; you should understand the input as a video and describe it using video footage.
Ensure that all questions and answers are output in Chinese. Video frame input: (video frames)
25

---

Published as a conference paper at ICLR 2025
Prompt for generating relationships hetween consecutive QA chains
Task:
You will be given two QA chains, each consisting of 5 consecttive open ended QA pains, generated for two different but consec. utive segments the video. Your task identify the related QA pains between the QA chains.
Generated example:
[f "question-Refore [
"What kind of gear are the racers wearing?" "What kind gear are the racers wearing?"". "What they prepuring do!".
"answersBcfore",
racers are wearing motocross
helmers",
"The racers are wearing motocross attire and helmets.",
"They are gotting ready to start race.".
"qusttionsAterr"
"What kiod of outfits are the racers wearing?".
"What kind outfits racers wearing?", "What were the racers doing the start of the race?".
"answersAffer" After":
"The wearing racing suits various colors, most equipped safety helmets.",
"The racers are wearing racing suits of various colors, most equipped with safety helmets.". "The rocess were rushing down hillside, ready to jump.".
"relationship11
"Person":
"Object".
"Action".
11} Required output:
Note that when you looking related pairs, you should search from the following aspects: relationship between actions, that is, when two QA pairs involve related actions: the relationship between quantitics, that is, when two QA pairs involve number related people objects; relationship between persons, that
persons; the relationship between objects, that is, when two QA pairs invalve related objects; the relationship between events, that is. when the
QA pairs
events activities; the relationchip hetween enviromments, is, when two QA pairs changes the Generate results following the format of the provided example and output all results in JSON format without producing any additional content. The outpur should include five key-value pairs, where "questionsBofore" is list of questions from the first chain within related QA pairs; "answersBefore" is list of answers from the first QA chain within related QA pairs; "questions After" the second QA
two QA pairs invalve the
QA pairs:"anseer "amnwersAfler" chain within related QA pains; "relationship" is the list of relationships corresponingg to the related QA within the two QA chains. should be consistent, and number chains.
questions
list
QA the
Note the number related QA pains
"questionsBefor"" and "questionsAfier"
relationships in the "relationship" list should match the number related QA pairs the
Note that for the two QA chains, you should output least six related QA pairs, and each related QA pair should conform to one types relationships mentioned above.
The "relation" list should only include the following relationships:
relationship between actions related QA puirs. 2. "Quantity", when there is relationship hetween quanitties in related QA pairs.
"Action".
"Person". when there selationship berween people in related QA pairs.
4. "Ohject", when there relationship between objects related QA pairs.
5. "Event", when there is relationship between occurrences in related QA pairs. "Environment". when
relationship between environments related pairs.
The linst QA chain: chhiial
QA chain: (chain2)
26

---

Published as a conference paper at ICLR 2025
Prompt for generating answers of questions within our dataset by both GPT-4V and GPT-4o
Task:
You are video comprehension expert, and you need to answer the questions posed in se- quence based on the provided video image sequence. The generated answers should be concise, clear, with emphasis on the key points, and summarized in one sentence.
Generated example: {{ "'They are smiling and looking the camera." }}
Required output:
Ensure that the content the answer closely related to the topic, avoiding unnecessary expansion and redundancy to provide concise, direct, and relevant information. Summarize the answer clearly one sentence, ensuring conciseness and emphasis on the key points.
Ensure that the answer precisely targets the posed question, providing comprehensive and direct information. When answering, clearly articulate your viewpoint and ensure all content is closely related to meet the requirements of the posed question. Answers should be given following the provided examples, only output the answer, and do not output any text irrelevant to the answer.
Do not provide information sources the answer, avoid expressions like "from the image," "picture sequence," "frame number," or "picture number." You should understand the input as video and describe it using video footage.
Video frame input: {video frames] Posed questions: {question}
27

---

Published as a conference paper at ICLR 2025
Prompt for Gcor-Score
Task Description:
You are expert judge evaluating the accuracy of answers question about scenes in streaming video. For each scene, there is a specific question and its ground truth answer. Several models have provided responses
these questions. Your task evaluate the scale from () to 10, where:
accuracy of each response
-10: The response is completely accurate and matches the ground truth all relevant details,
providing any necessary context.
8-9: The response mostly accurate but may miss minor details or context.
6-7: The response is somewhat accurate but lacks significant details context.
4-5: The response provides some relevant information but misses key aspects of the ground truth.
2-3: The response has little relevance or severely misconstrues the ground truth.
011: The response completely inaccurate off-topic.
Additional Requirements and Considerations for the Evaluator:
Thoroughly Understand the Question: Ensure that you fully grasp the context and nuances of the question before evaluating the response.
Accurate Comparison: Compare the model's response against the ground truth answer with high degree of precision. Pay attention to the correctness, completeness, and relevance of the information provided.
3. Objective Scoring: Assign score scale from 1 to 10, focusing solely on the accuracy of the response. Do not consider style, grammar, or additional information that unrelated to accuracy.
4. Detailed Explanation: Provide clear and concise explanation for the score you assign.
This explanation should justify your scoring by pointing specific accurate or inaccurate details in the model's response.
Consistency: Apply the same criteria uniformly across all evaluations ensure fairness and consistency in scoring.
Be Neutral and Unbiased: Do let any prior knowledge, assumptions, or personal opinions affect your judgment. Only use the provided ground truth and the response when making your decision.
For the following QA, please evaluate the model's performance according to the criteria mentioned above and provide a detailed justification for each score.
Questions:
{question}
Ground Truth:
(ground truth)
Model Responses:
smodel_responne
Please provide your evaluation score and detailed comment below:
Accuracy:
Score:
Comments:
28

---

Published as a conference paper at ICLR 2025
Prompt for dialogue evaluation
Task Description:
You are an evaluation expert for video multi turn dialogue evaluation. Each video contains different timestamps that are followed by series of QAs. Your task is to evaluate the quality of model responses series of open-ended questions the same timestamp within streaming video. You will assess the responses based on several specified dimensions:
1. Semantic Accuracy: evaluates the accuracy of the generated answers based on a holistic understand- ing. It considers not only the direct overlap with ground-truth answers but also the context, coherence, and overall relevance of the response the question posed.
points: Completely accurate, directly reflecting the video with no apparent errors. 7-9 points: Mostly accurate, with few minor detail errors. 4-6 points: Several error, but generally conveys most of the content. 1-3 points: Only a small part of the content is accurate or mostly incorrect. 0 points: Completely inaccurate, unrelated to the vidco.
Scoring Guidelines:
Contextual Coherence: examines the ability of LVLMs maintain relevance and context across sequential questions and answers, ensuring continuity and alignment with the evolving discourse. Scoring Guidelines: 10 points: Highly coherent, natural transition hetween scenes. - 77-9 points: Mostly coherent, with minor issues in transition points. 4-6 points: Some coherence, but loose or partially disjointed transitions. 1-3 points: Poor coherence, most transition points unnatural. 0 points: Completely incoherent, response appears to be unrelated or independent content.
Logical Consistency: evaluates the logical progression and consistency of answers, ensuring that answers do not contradict each other or previous information.
Scoring Guidelines: 10 points: Logically consistent, sense of incongruity. 7-9 points: Mostly consistent, with few minor inconsistencies. 4-6 points: Several logical issues, but the response is somewhat understandable. 1-3 points: Logically chaotic, difficult to understand or largely unreason- Completely illogical, contradicts video content.
able. 0 points:
4. Temporal Understanding: assesses the model's proficiency in comprehending and reasoning about temporal events and sequences depicted in the video content.
points: The response accurately reflects the timeline and causal relationships events. - 7-9 points: The response largely reflects the correct timeline with minor errors or omis- sions. - 4-6 points:
Scoring Guidelines:
The response reflects the event sequence partially but has significant time-related errors key omissions. 1-3 points: The response has little correct temporal understanding, with many time errors. points: The response entirely fails reflect the correct time sequence events process. points: The question not involve temporal understanding
5. Informational Completeness: measures the comprehensiveness to gauge whether the model captures and conveys all relevant elements from the video to provide a thorough answer.
Scoring Guidelincs: 10 points: Fully comprehensive, covering all necessary details. 7-9 points: Mostly comprehensive, with some missing details. 4-6 points: Partially informative, but incomplete. 1-3 points: Largely incomplete, containing only a few details. points: Contains no useful infor- mation.
Overall Score: is derived by aggregating the scores from each aforementioned criterion, ranked as follows:
Low quality, with no major errors but not meeting requirements. Scores 5-6: Moderate quality, meets basic requirements hut performs poorly in some aspects. Scores 7-8: High quality, performs well in most dimensions. Scores 9-10: Excellent performance, fully addressing the questions and all criteria, significantly exceeding the reference answers.
Scores 1-2: Irrelevant, factually incorrect, or harmful content. Scores 3-4:
Additional Requirements and Considerations for Evaluator: Unbiased Evaluation: Ensure an unbiased assessment by focusing purely on the content and quality of the responses compared to the ground truth. 2. Consistency: Maintain consistency in scoring across different responses by adhering strictly to the detailed scoring breakdown provided. 3. Detail and Justification: Provide detailed foodback for each criterion, explaining why a particular score was assigned help identify strengths and weaknesses in the responses. 4. Thoroughness; Avoid rushing through the evaluation. Ensure each response is carefully reviewed and scored based on all aspects of the criteria.
For the following QA chain, please evaluate the model's performance according to the criteria men- tioned above and provide detailed justification for each score. Questions: {question}
Ground Truth: {ground truth)
Model Responses: (model ]
Please provide your evaluation scores and detailed comments for each criterion below:
Semantic Accuracy:
Score:
Comments:
29

---

Published as a conference paper at ICLR 2025
Prompt for streaming evaluation
Task Description:
You are evaluation expert for streaming video evaluation. Each video contains different timestamps that are followed by series of QAs. Your task is to evaluate the quality of model responses to series of open-ended questions different timestamps within streaming video. You will assess the responses based on several specified dimensions:
1. Semantic Accuracy: evaluates the accuracy of the generated answers based on a holistic understand- ing. It considers not only the direct overlap with ground-truth answers but also the context, coherence, and overall relevance of the response the question posed.
points: Completely accurate, directly reflecting the video with no apparent errors. 7-9 points: Mostly accurate, with few minor detail errors. 4-6 points: Several erros, but generally conveys most of the content. 1-3 points: Only a small part of the content is accurate or mostly incorrect. points: Completely inaccurate, unrelated to the video.
Scoring Guidelines:
Contextual Coherence: examines the ability of LVLMs maintain relevance and context across sequential questions and answers, ensuring continuity and alignment with the evolving discourse. Scoring Guidelines: 10 points: Highly coherent, natural transition hetween scenes. - 77-9 points: Mostly coherent, with minor issues in transition points. 4-6 points: Some coherence, but loose or partially disjointed transitions. 1-3 points: Poor coherence, most transition points unnatural. 0 points: Completely incoherent, response appears to be unrelated or independent content.
evaluates the logical progression and consistency of answers, ensuring that answers do not contradict each other or previous information.
Logical Consistency:
Scoring Guidelines: 10 points: Logically consistent, sense of incongruity. 7-9 points: Mostly consistent, with few minor inconsistencies. 4-6 points: Several logical issues, but the response is somewhat understandable. 1-3 points: Logically chaotic, difficult to understand or largely unreason- able. points: Completely illogical, contradicts video content.
4. Temporal Understanding: assesses the model's proficiency in comprehending and reasoning about temporal events and sequences depicted in the video content.
points: The response accurately reflects the timeline and causal relationships events. 7-9 points: The response largely reflects the correct timeline with minor errors or omis- sions. - 4-6 points:
Scoring Guidelines:
The response reflects the event sequence partially but has significant time-related errors key omissions. 1-3 points: The response has little correct temporal understanding, with many time errors. points: The response entirely fails reflect the correct time sequence events process. points: The question not involve temporal understanding
5. Informational Completeness: measures the comprehensiveness to gauge whether the model captures and conveys all relevant elements from the video to provide a thorough answer.
Scoring Guidelincs: 10 points: Fully comprehensive, covering all necessary details. 7-9 points: Mostly comprehensive, with some missing details. 4-6 points: Partially informative, but incomplete. 1-3 points: Largely incomplete, containing only a details. points: Contains no useful infor- mation.
Overall Score: is derived by aggregating the scores from each aforementioned criterion, ranked as
tollows:
Low quality, with no major errors but not meeting requirements. Scores 5-6: Moderate quality, meets basic requirements hut performs poorly in some aspects. Scores 7-8: High quality, performs well in most dimensions. Scores 9-10: Excellent performance, fully addressing the questions and all criteria, significantly exceeding the reference answers.
Scores Irrelevant, factually incorrect, or harmful content. Scores 3-4:
Additional Requirements and Considerations for Evaluator: Unbiased Evaluation: Ensure an unbiased assessment by focusing purely on the content and quality of the responses compared to the ground truth. 2. Consistency: Maintain consistency in scoring across different responses by adhering strictly to the detailed scoring breakdown provided. 3. Detail and Justification: Provide detailed foedback for each criterion, explaining why a particular score was assigned help identify strengths and weaknesses in the responses. Thoroughness; Avoid rushing through the evaluation. Ensure each response is carefully reviewed and scored based on all aspects of the criteria.
For the following streaming QAs, please evaluate the model's performance according to the criteria mentioned above and provide detailed justification for each score. Questions: {question]
Ground Truth: {ground truth)
Model Responses: (model response)))
Please provide your evaluation scores and detailed comments for each criterion below:
Semantic Accuracy:
Score:
Comments:
30

---

Published as a conference paper at ICLR 2025
Without Contextual Content
With Contextual Content
Go--Westtodiondees poseeooooooeeee-eooooeoeeeeooooo por-oooooo-ooooooiii-i--e-e effer-he the bite the p-22e
perform
after he bite the pizza?
SyBench:-hhe hoot hare
SVBench: host pushes up glasses.
What action does he perform than? SVBench: The host first finger of the left hand, then licks the thumb of the
Q1:
Q1: What action does he perform then? SVBench: The host finger of the left hand, then licks the hand.
middle
middle
hand.
GPT-4v Ans: He moves his fingers to demonstrate a point.
GPT-4v Ans: The host performs hand gesture.
GPT-4o Ans: He picks slice of pizza and prepares to take bite.
GPT-4o Ans: The host speaks while
gesturing with his hands.
Comment As evident from the case, without contextual content, both GPT-4v and GPT-4o fail to comprehend the sequential relationship of actions within the question. However, when contextual content is provided, both of them demonstrate improved capacity grasping the temporal order actions, although they still exhibit the capacity to accurately identify actions themselves.
Case study between single QA pair with contextual content and the same QA pair without contextual content: The red text highlights inaccuracies in the answer generated by both GPT-4v and GPT-4o.
Figure 11:
31

---

Published as a conference paper at ICLR 2025
Q0: How does the attacking team advance the ball?
SVBench: attacking team advancce GPT-4v Ans: The attacking team advances the ball with series of short passes outmaneuver the defenders beforesending sending a cross into the penalty area. The attacking team advances the by passing through
through series passes movementss
before
Q1: goalkeeper reection?
SVBench: The goalkeeper tries block but unsuccessful GPT -4v Ans: The goa keeper dives to the left an attempt to block the incoming shot but fails to stop the ball from entering net.
GPT-4o Ans: The goalkeeper's reaction is to dive toward the ball in attempt save it.
Q2: reaction
goal?
SVBench: He still on the ground.
GPT-4v Ans: He displays joy and excitement by raising his arms and running towards his teammates. GPT-4o celebrates joyfully by raising
running.
Q3:
competitor?
SVBench: The competitor very excited after scoring, running to the sidelines to celebrate. GPT Ans: His competitor, the goalkeeper and several defenders dark jerseys, are actively trying to block shot
goal ultimately GPT-4o Ans: His competitor appears to be engaged the soccer training session, actively participating and practicing along with others on the field.
Comment: As depicted in the case, both GPT-4v and GPT-4o exhibit shortcomings in comprehending the contextual content, exemplified by their inability to discern that 'him' in Q2 refers to the goalkeeper mentioned in Q1. Consequently, both of them fail to provide accurate responses to Q2 and subsequent questions within the QA chain.
Figure 12: Case study of a QA chain: The red text highlights inaccuracies in answers generated by both GPT-4v and GPT-4o.
32

---

Published as a conference paper at ICLR 2025
Streaming Questlon-Answeriin
Whtwh the boy looking at tire?
Q2: is lodknng the shonogmphhe vomiing
Q1: low risecho he use the showe the cheerin?
phottggeete sogresheee
00:00
CPT-4o Ave: The boy at the photographer to saek instructions.
looking
the
GaT-to Ars: At fimst the boy looking
GPT-4a Are. The boy use the shovel by
camara the parson taking
nepord baing expres
scraping back
Qc: Doea the ahowelin sersewe rion un bullor??
5i dhzelin the way firs sod 1?? SyBeech Vou. comt wus use
O2: Wher ouour the final result? SuBeecht Finalyy clars the snow from
CP Arm the
(DUST)
the GPT-4o Are: No. the boy now
GPT Ane The final dignffiaat portion the snow iie has dleared from windsfiees
showe
the shovel with more force und in locused, seraping motion chear
applying the shovel with grip
glass
windehiedd thonughyy.
Comment evident from that GPT-4o exhibits performance streaming Questton-Asswering.
Figure 13: Case study of Streaming Question-Answering: The red text highlights inaccuracies in answers generated by GPT-4o.
33

---

Published as a conference paper at ICLR 2025
toots
SyBench Secouse and first wiped the floor with stff brush, he now neaded map dean the din that had rubbedifft
Intention inference
GPT 4o Ans Secause te thoughttt cenerr.
+ What the possbelty handiing g ten poopert in thes way under normal dhoumstancee?
unilety
GFf 4o Ans decends form dish.
Q: there other boy. can wesh the dishes aore? SyBerch Hi cant, because has hands and can hold the brudlt
Counterfactual Reasoning
SVBenchu
Spadio Temponsa Specolltiin
with hards
GFT-4o Ans He chopped the tot with intte and then mbxdd the toll with the vogetablss
Q: What her herrocatiorste with the man begingg in the vicee? man Eagininn
SVBerehs
Rolationship
SvBench Sectuse he rod first wiped the floor with riff bruch he now necded mop dean the dir that had rubbedoftt
Why dhangehee tnck?
Chanecter State and Transition
Swasool thought cesner.
Q What's offere to between those two ticttmens of butter and daa'?
pan half cookee
Compartson
frssh favor both which make the dish
cous.
GFT-4o Ane Theas vun di Herrer tnnemmentt speati to the cifierent steps proparing the dish. which can make the disht more delidious.
Q: Wi she duing with th toof? SViterett the unes thes toat purme garte and adds using
Common Serse Inference
SUBarchh Actiing
lquad stidky the onion
Event- Centric Analysta
rigs scoted with the lcuid on eggy smell
GFT-4o Aes Acding at 020 thicsen Bouid
Comment:Ar can discemed from these cases, GPT-4o exhibits suboptimal performance across the nine skills assessed SVBench.
Case study of skills assessment: The red text highlights inaccuracies answers generated by GPT-4o.
Figure 14:
34

---

Published as a conference paper at ICLR 2025
Q0: What does video mainly show?
SVBench: mainly shows road conditions in Thara-Nite
MiniCPM-V 2.5 Ans: The video mainly shows a news broadcast segment.
StreamingChat Ans: This video mainly shows the road conditions in Thara-Nithi.
What the current condition of this road?
SVBench: it under maintenance, with pile of soil in the middle of the road.
MiniCPM-V Ans: video shows that the is unpaved and dusty, visible potholes uneven surfaces. StreamingChat Ans: The road is under construction, with some areas having already been paved.
Q2: What effect do the mounds of dirt the read have on the vehides passing by?
SVBench: The mounds
obstruct the normal passage vehicles.
and dusty MiniCPM-V 2.6 Ans: The mounds of the road causing environment for vehicles passing by.
StreamingChat Ans: The mounds of dirt obstruct the normal passage of vehicles.
Why this construction necessary?
SVBench: it likely to improve traffic conditions.
MiniCPM-VV Ans: caption does provide information about necessity of construction.
StreamingChat Ans: it improve traffic conditions.
Comment: As demonstrated by this case, current best-performing open-source model, MiniCPM-V 2.6, unable to analyze the questioner's intent based contextual information and provide the appropriate answer. MinICPM-V 2.6 relies on the most accurate information from the current question and video segments to answer as acourately as possible (and fails to respond when information insufficient), but it overlooks the coherence contextual information between questions.
Case study: Comparison of StreamingChat and the best-performing open-source LVLM on SVBench. The red text highlights inaccuracies in answers generated by MiniCPM-V 2.6.
Figure 15:
35


Reviews:
[{"review_id": "gbiI2cZYBU", "reviewer": "Reviewer_YPvF", "summary": "This paper introduces SVBench, a benchmark designed to evaluate Large Vision-Language Models (LVLMs) on streaming video understanding. A single data point in SVBench consists of a video segment with its corresponding temporal dialogue chain. Each chain contains 4-5 question-answer pairs that are contextually connected, meaning each subsequent question builds upon previous answers. Additionally, each chain has \"temporal linkages\" to chains from adjacent video segments based on common elements (people, events, objects). \n\nThe authors employed a semi-automated annotation pipeline where GPT-4 generates QA pairs for each video segment, identifies related QA pairs based on six relationship types, followed by human annotators who modify the result. The evaluation framework uses two modes - dialogue evaluation (testing multi-turn QA within the same video segment) and streaming evaluation (testing understanding across temporally linked segments with an 80% chance of jumping to questions linked with temporal linkages) - while scoring responses across five dimensions using LLMs. \n\nThe authors also introduced StreamingChat, a LVLM built on InternVL2 that uses an InternViT vision encoder, MLP projector, and InternLM2 language model. Regarding the results, GPT-4o leads with the highest scores (66.29% in dialogue and 58.17% in streaming evaluation) while among open-source models, StreamingChat performs best (59.41% in dialogue and 53.90% in streaming), with all models scoring below 60%, indicating significant room for improvement in streaming video comprehension.", "questions": "- How reliable are LLMs evaluations? \n- Did you try different LLMs to evaluate the models?\n- How did evaluation depend on LLMs hyperparameters like temperature?\n- Does it change if run several times?\n- How does it change over different prompts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel Technical Contribution: The paper introduces a semi-automated pipeline that combines LLM-assisted generation with human verification to create temporal multi-turn QA chains, representing a methodologically sound approach to dataset creation.\n\n- Comprehensive Empirical Validation: The evaluation spans 14 different models (both open and closed-source), uses multiple metrics (METEOR, GPT4-Score, etc.), and includes detailed ablation studies comparing single-instance vs. multi-turn QA performance.\n\n- The temporal linkage: The temporal linkage concept between QA chains is interesting and addresses a real gap in video understanding benchmark by forcing models to maintain context across time segments, mimicking real-world streaming scenarios.", "weaknesses": "The paper's heavy reliance on LLMs for evaluation is a significant methodological concern. While authors use GPT-4 to assess models’ performance across multiple dimensions (semantic accuracy, contextual coherence, etc.), there's no validation of whether these automated scores align with human judgments. The fact that LLMs are being used to both generate the annotations and evaluate the results creates a circular dependency that could mask real limitations or biases in the evaluation process. Without human validation studies, it's difficult to trust the reported performance gains or confidently compare different models using this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oECM4PRYKA", "reviewer": "Reviewer_k44W", "summary": "In this paper, the authors introduce SVBench, a novel benchmark designed to evaluate the capabilities of LVLMs in long-context streaming video understanding. SVBench features temporal multi-turn question-answering chains comprising 49,979 QA pairs across 1,353 streaming videos. The study reveals that while the closed-source GPT-4o model outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. \n\nThe authors also develop StreamingChat, a model that outperforms existing open-source LVLMs on SVBench and achieves comparable performance on diverse vision-language benchmarks.", "questions": "Q1: The videos of the benchmark come from some publicly available datasets; how do you ensure that the models used for evaluation have not encountered this data during their training?\n\nQ2: How do QA chains address the issue of information sparsity in videos? For lengthy videos with few key events, how does SVBench construct meaningful QA chains? Is there a mechanism to handle or generate question chains that effectively deal with scenarios where information is sparse or low-density?\n\nQ3: In your benchmark, do the videos retain their audio tracks? Multimodal information significantly aids in understanding videos. If audio is included, does it genuinely assist in answering the questions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: The paper introduces SVBench, a benchmark explicitly designed for evaluating LVLMs in long-context streaming video understanding. I believe this fills a notable gap in existing benchmarks, which typically focus on isolated text inputs rather than sustained temporal reasoning across video streams. The comparison between the benchmarks also shows the advantages of SVBench.\n\nS2: The QA pairs in this work are annotated semi-automatically, making it a large-scale dataset with high-quality annotations.\n\nS3: The experiments and evaluation provide insights into open-source models' struggles with long-context video understanding.\n\nS4: The paper is well-organized, with clear explanations of SVBench, the QA chain structure, and the temporal linkages created for sustained reasoning tasks.", "weaknesses": "W1: The paper does not include a comparison with human performance. Incorporating such a comparison would provide valuable insights into the gap between current models and human capabilities in long-context streaming video understanding.\n\nW2: The paper does not analyze the impact of language model size on performance. Considering that models like InternVL2 have versions with 1B, 2B, 4B, 8B, 26B, 40B, and 72B parameters, and Video-LLaMA2 also have 72B versions, expanding experiments to include these variations and providing more detailed analysis would enhance the work. Additionally, exploring the number of frames the model can process would offer valuable insights. In addition to model size and the input length, analyzing the amount and diversity of training data used for each model would provide a more comprehensive understanding of their performance in long-context streaming video understanding. Training data quality and quantity are crucial factors influencing model capabilities.\n\nW3: The evaluation overlooks several important models capable of long-video understanding, such as LLaVA-OneVision [1], Qwen2-VL [2], LongVILA [3], Long-LLaVA [4], and Oryx [5]. Including these models would enhance the comparison, providing a more comprehensive view of current capabilities in this area.\n\n[1] https://github.com/LLaVA-VL/LLaVA-NeXT\n\n[2] https://github.com/QwenLM/Qwen2-VL\n\n[3] https://github.com/NVlabs/VILA/blob/main/LongVILA.md\n\n[4] https://github.com/FreedomIntelligence/LongLLaVA\n\n[5] https://github.com/Oryx-mllm/Oryx", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "penOkpLdc0", "reviewer": "Reviewer_6qmJ", "summary": "To make up for the gap of lacking attention in streaming video understanding, this paper introduces an novel designed SVBench which try to assess how well large multi-modal language models handle temporal multi-turn question-answering dialogues over streaming videos. SVBench includes 49,979 QA pairs derived from 1,353 videos and all\nannotations are interconnected through temporal linkages, ensuring that the model needs to consider previous and current video segments to answer questions correctly.\nThe authors developed StreamingChat, which significantly improved performance by incorporating long-context reasoning abilities specific to streaming videos. They also leveraged advanced training techniques like fine-tuning with LoRA (Low-Rank Adaptation) to handle long video contexts efficiently", "questions": "1. According to the paper, the length of video data used by SVBench is between 5 and 30 seconds. Has the author considered the maximum video processing time of StreamingChat?\n2. Can the author provide the test results of the latest streaming video understanding model Flash-VStream and Video-online on SVBench? This will further highlight the advantages of the article.\n3. Can the author provide some data on the model's inference speed and resource consumption? For example, the change curve of inference speed and current consumption under different video lengths, which may be what I am more interested in.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The SVBench proposed fill the gap between video benchmarks and streaming video understanding. In the real world, streaming video is a more challenging data form, so this benchmark has very important practical significance.\n2. The authors proposed a semi-automatic annotation process and integrated multiple types of video data, which not only maintained the diversity of the benchmark, but also the accuracy of the annotation information and prevented hallucinations from affecting the benchmark results.\n3. In response to the observed limitations of current large multi-modal langeuage models, the authors introduce StreamingChat, which significantly improves performance on SVBench.\n4. The paper conducts an extensive evaluation of 14 models, comparing both open-source and closed-source models,  provides valuable insights into the current state-of-the-art models in handling streaming video understanding.", "weaknesses": "1. Although the authors provide an intuitive expression of the proposed SVBench in terms of video types and the diversity of annotation information through visualization results, the authors lack a description of the distribution of video lengths, which may be important for a benchmark.\n2. The training method of the StreamingChat architecture proposed by the author is similar to the recently proposed streaming video understanding model Video-online. I hope the author can explain the difference between them in more detail.\n3. The author's experimens are rich, but lacks measurements of the overall system's latency and inference speed like FPS, which are also very important matrics for streaming video understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9R8rU6yZs1", "reviewer": "Reviewer_xWou", "summary": "This paper focus on evaluating LVLMs on streaming video understanding tasks. The authors claimed that existing benchmarks for video understanding merely emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. Therefore, they proposed SVBench, a comprehensive benchmark created from a semi-automated annotation pipeline to obtain QA chains that represent consecutive multi-turn dialogues. The new benchmark is closer to the real-world scenarios and the evaluation results reveal that most open-source LVLMs struggle with long-context streaming video understanding. The authors also developed StreamingChat, a novel model that could significantly outperform open-source LVLMs on SVBench and could be a good starting point to inspire future research.", "questions": "1. Can the authors provide some data samples to demonstrate the significance of the proposed benchmark?\n2. Is it possible to utilize open-source models (suggested in the weakness part) to perform evaluation in SVBench? If yes, how do the results align with the existing metrics (METEOR and GPT-4 Score) and human evaluations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the motivation is clear and reasonable, i.e., to develop a complete solution (benchmark + model) for streaming video understanding.\n2. The paper is well-written and easy-to-follow.\n3. The proposed benchmark seems to have good quality, and it's closer to real-world scenarios.", "weaknesses": "I only have minor concerns regarding the quality of the collected annotations and the evaluation metrics.\n\n1. The proposed semi-automated (LLM + human) annotation pipeline is novel and reasonable. However, the authors did not provide sample data to demonstrate the high quality of the benchmark. Although they mentioned in the paper that GPT-4 was utilized to score and rank the QA chain annotations, it still cannot fully convince me that such an LLM-based check can ensure the quality of the benchmark.\n2. SVBench leverages METEOR and GPT-4 Score to evaluate model outputs, which either cannot fully reveal the semantic consistency between prediction and ground truths (METEOR) or is costly to call APIs (GPT-4 Score). It would be better to explore the possibility of leveraging open-source language models (e.g., computing distances in semantic space like [1] or prompting open-source LLMs) to perform evaluation.\n\n[1] Di et al. Grounded Question-Answering in Long Egocentric Videos. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)