CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
Authors: Guo Chen
Yicheng Liu
Yifei Huang
Baoqi Pei
Jilan Xu
Yuping He
Tong Lu
Yali Wang
Limin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Long Video Understanding
Primary area: datasets and benchmarks

Abstract: The existing video understanding benchmarks for multimodal large language models (MLLMs) mainly focus on short videos. The few benchmarks for long video understanding often rely on multiple-choice questions (MCQs). Due to the limitations of MCQ evaluations and the advanced reasoning abilities of MLLMs, models can often answer correctly by combining short video insights with elimination, without truly understanding the content. To bridge this gap, we introduce CG-Bench, a benchmark for clue-grounded question answering in long videos. CG-Bench emphasizes the model's ability to retrieve relevant clues, enhancing evaluation credibility. It includes 1,219 manually curated videos organized into 14 primary, 171 secondary, and 638 tertiary categories, making it the largest benchmark for long video analysis. The dataset features 12,129 QA pairs in three question types: perception, reasoning, and hallucination. To address the limitations of MCQ-based evaluation, we develop two novel clue-based methods: clue-grounded white box and black box evaluations, assessing whether models generate answers based on accurate video understanding. We evaluated multiple closed-source and open-source MLLMs on CG-Bench. The results show that current models struggle significantly with long videos compared to short ones, and there is a notable gap between open-source and commercial models. We hope CG-Bench will drive the development of more reliable and capable MLLMs for long video comprehension.

Published as a conference paper at ICLR 2025
CG-BENCH: CLUE-GROUNDED QUESTION ANSWER- ING BENCHMARK FOR LONG VIDEO UNDERSTANDING
Guo Chen Yicheng Liu Yifei Huang? Yuping He', Baoqi Pei Jilan Xu2.5
Yali Wang", Tong Lu I Limin Wang
State Key Laboratory for Novel Software Technology. Nanjing University
Shanghai Artificial Intelligence Laboratory "The University of Tokyo
*Zhejiang University, Fudan University
thenguoo1178gmmaii coo
ABSTRACT
The existing video understanding benchmarks for multimodal large language models (MLLMs) mainly focus on short videos. The few benchmarks for long video understanding often rely on multiple-choice questions (MCQs). Due to the limitations of MCQ ovaluations and the advanced reasoning abilities of MLLMs, models often answer correctly by combining short video insights with elimi- nation, without truly understanding the content. To bridge this gap, we introduce CG-Bench, benchmark for clue-grounded question answering long videos. CG-Bench emphasizes the model's ability to retrieve relevant clues, enhancing evaluation credibility, It includes 1,219 manually curated videos organized into 14 primary. 171 secondary, and 638 tertiary categories, making it the largest bench- mark for long video analysis. The dataset features 12,129 QA pairs three ques- tion types: perception, reasoning, and hallucination. To address the limitations of MCQ-based evaluation, we develop two novel clue-based evaluation methods: clue- gre-ggrouded white box and black box evaluations, assessing whether models generate answers based on accurate video understanding. We evaluate multiple closed-source and open-source MLLMs CG-Bench. The results show that current models struggle significantly with long videos compared to short ones, and there is a notable gap between open-source and commercial models. We hope CG-Bench will drive the development of more reliable and capable MILMs for long video comprehension. All annotations and video data available wtw.bu...iooeeee otl.ub..o//edderaarrd
INTRODUCTION
Recently, video understanding has made significant progress with the advent of multimodal large language models (MLLMss) To evaluate these models, many recent efforts have been made to create video understanding benchmarks (Li et al. 20235] Mangalam et al. 2024 Liu et al.) (2024)), providing assessments of model comprehension capabilities and clues for future improvement.
Since early benchmarks only focus on short video clips, recent works have started to create bench- marks (Fu et al.) 2024a Wu et al. 2024b Zhou al. 2024 Huang al., 2024) for longer videos (2 10 minutes). However, these works employ multiple-choiee questions (MCQ), where the difficulty level is heavily influenced by the configuration of negative options. In such scenarios. models (Chen)
et al. 2023d [i et al. (2244a] ZZhang et al. 2024b] Lin et al.) (024) tend to focus only general video knowledge and use elimination to avoid selecting the negative options. As result, the models can achieve correct answers without genuinely engaging with the relevant video content, leading to lack of trustworthiness. One illustration can be found question 2 of Figure [ the option 'A' can be easily eliminated based purely textual information. Recently, the NExT-GQA (Xiao et al.) 2024) benchmark tries to address the problem of credible models by incorporating temporal grounding into MCQ. However, NEXT-GQA is limited to the NextQA (Xiao et al. 202T) dataset, which lacks di-
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Clue- grouuddd Annotation for Long Videos
CG-Bench Evaluation Suite
Queaioon vides, many photo does the peotagomist touch viile sirdling lrnggh morming marker Anstonland
vidar.
Quastion Ia the videa,
Muttipled hake Qurelion Evaltation
busband offer cass desateal aingord
proegonis and her
capture samosedlal puopli??
fomous panninng the protigomnt
Long AC22
A. Drive to Lancra Paris
"Laborty leating the Beople"
Clicked glases drnkk legether
"The Coeeration Naguion""
Cie lains HCQ
Clacssc.
C. Embraced each other
"The Wedding at Cana"
A.1 B.2
C.3
D.4 E
D. Tock ploto with the singer
Made tost logelier
E. "The Gleaness
"Mean Liu""
Crodibilit Evalustion
wix
OF-Clee
Viic Bis
Slvck
Opra-ended I Esolitation
Gjs Radit g+
Figure 1: Left: examples of CG-Bench's clue-grounded annotation. To correctly answer the questions, models need to ground their reasoning into the correct clue. Right: CG-Bench provides an evaluation suite with two novel credibility evaluation criteria while supporting both MCQ and open-ended evaluations.
versity and primarily consists of short videos. A comprehensive benchmark for credibly evaluating generalist MLLMs for long video understanding, is still missing in the research community.
To make up this gap, we introduce CG-Bench, illustrated in Figure [ novel benchmark designed to evaluate gree-grounded question answering in long videos. In contrast to traditional benchmarks that focus primarily on the accuracy of question answering. CG-Bench goes a step further by eval- uating whether the model bases its answers on relevant clues within the video. CG-Bench designs two novel clue-based evaluation methods to provide more reliable model performance assessments. 1) litt-grouded white box evaluation requires the model to directly provide the clue interval corre- sponding to the question while selecting the correct answer. 2) clue-grouded black box evaluation requires the model to align the accuracy of video-level and clue-level MCQ. Furthermore, we pro- pose a novel heuristic method, aided by munantandotated clues, for open-ended QA evaluation, to effectively balance the cost and performance.
CG-Bench features 1,219 meticulously curated vidcos and 12,129 human-annotated question- answer-clue (QAC) triplets, establishing it as the largest and held-out VideoQA and question ground- ing benchmark for long videos. It employs a highly detailed manual classification system, organizing each video into 14 primary categories, 171 secondary categories, and 638 tertiary categories. The benchmark includes three main question types: perception, reasoning, and hallucination. Perception questions further divided into 10 subcategories, such object and attribute recognition, while reasoning questions are categorized into 12 subcategories, including relation reasoning, etc.
The com- mercial models, GPT-4o [OpenAl 2024) and Gemini-1.5 Pro (Anil et al. (202)) achieve scores of 53.9 and 43.4, respectively, with 128 frames for long-video multiple-choice questions. The leading open-source MLLM,, Qwen2-VL VL-7BB (Wang et al. 2024b), scores 51.4 .44 under the same conditions, indicating its initial benchmarking against GPT-4o. However, our credibility assessments and open- ended evaluations reveal significant drop in accuracy for existing MLLMs, with scores decreasing from 53.9 to 21.7. This underscores the considerable room for improvement in current MILMs for long video understanding. We hope this benchmark can become a vital tool for advancing research and development of more reliable and capable MLLMs.
We evaluate a range of closed-source and open-source MILMS using this benchmark.
2 RELATED WORK
Multimodal Large Language Models (MLLMs) have rapidly gained popularity due to their pro-
ficiency in integrating visual and textual information (Liu et al. 2024a 2023, Chen et al. 2023d
Wang et al. 2022 2024d). Recent advancements, such as LI VA-Next- Video (Zhang et al. 2024b)
LLaVA- Vision (Li et al.) 2024a), InternVL2 (Chen et al., (224c) and Eagle-2 (Li ct al. 20255,
focus on enhancing MLLMS by integrating LLM backbones with visual encoders and specialized adapters, creating higher-quality multimodal instruction data. This results in improved perfor- mance across tasks that involve both text and images.
Another area of focus is multimodal video understanding. Most models CCenn et al. 2024e| Li et al. 2023a Maaz et al.) (2023,1e Pei et al. 2024 Huang et al. [2018) (0206) are optimized for short videos, typically few seconds or at most few minutes, without exploring their visual understanding with
2

---

Published as a conference paper at ICLR 2025
Art cuture
Corteng (115)
Event Purcaption (2382)
(221))
Eairyy
Life
Perouption
cepplio on
Music & TV
(216)
Perrestion
(9273)
Show
(110)
Video (1219)
Quastion
Insturction Knowledge (977
Timre
(12129)
Anonee &
Spari
Reasoning
Specea Scoves
(9412)
GUI (81)
Erecpee proa PRu
(81)
(ht)
Driving
Entrrv SOSEIIO
Event Roasoning (1480)
Figure 2:
Distribution of video root categories, dis- playing the number of videos within each category.
Figure 3:
Distribution of question root types, illus- trating the frequency of different question types.
longer context. response, researchers have explored methods such as compressing video frames into fewer visual tokens to allow for the handling of longer videos, as seen in models like LLaMA-
Vid (Li et al.; 2023c), MovieChat (Song et al. (002). MA-LMM (He et al. 2024), VideoChat- Flash (Li et al. 2024b) and Oryx (Liu al. 2024T). In addition, Long VA (hang et al., (024a) and
Long ViL.A (Xue et al. 2024) explore the system-level optimization for long-context MLLMs which
natively support long video understanding. Despite the continuous proposal of various MILMs, their real-world performance in long video understanding is still under explored.
MLLM Benchmarks. The development of benchmarks is becoming increasingly essential, espe- cially for evaluating the MLLM performance in video understanding tasks, As the field develops, various benchmarks have been established assess MLLMs across different modalities and video lengths. Previous efforts primarily focused on short videos, with traditional specialized VideoQA datasets like TVQA (Lei al. 2018), NexiQA (Xiao et al. (021), and benchmarks for MILMM like VideoBench (Ning et al. 2023), MVBench (Li et al. 2023b) and EgoSchema (Mangalam al., 2024). MVBench provides a comprehensive framework for evaluating general temporal understand- ing capabilities through question aawoonnnnnnwwenngg short clips, while EgoSchema focuses egocentric video understanding with multi-choice questions. The videos in these benchmarks typically range from few seconds to several tens of seconds, making them similar to image benchmarks and thus hindering the development of general video LLMs.
Recently, several works such as VideoMME (Fu et al. 2024a), CinePile (Rawal et al. 2024
MIVU Zhouu et al. 2024), Long VideoBench (Wu et al. HourVideo (Chandrasegaran al. 2024), and LV (Wang et al. 2024c). have introduced long video benchmarks to evaluate MLLMs. VideoMME constructs a diverse video MCQ dataset, in- corporating multimodal evaluations with visuals, subtitles, and audio. MLVU designs range of tasks that focus on granular detail understanding to assess long video comprehension capabilities. However, a common limitation of these benchmarks their reliance on MCQs, where the difficulty is heavily influenced by the construction of negative options. This allows MLLMs to often elimi- nate incorrect answers using sparse frames and common sense reasoning, which can inflate perfor- mances. With our clue interval annotation, CG-Bench enhances the evaluation quality of MILMs in long video understanding by introducing new evaluation mechanisms on credibility.
2024b), MoVQA (Zhang et al. 2023h
CG-BENCH
3.1
DATASET CONSTRUCTION
The dataset construction process of CG-Bench consists of three steps: video collection, question- answering-clue annotation, and quality review iteration. We provide details as follows.
Video Collection. To avoid using videos that have been used for pre-training by existing MLLMs, we manually collect videos from the internet and provide new annotations on them. To facilitate the collection of raw videos from the Internet, During the collection process, we manually assign a brief tag (4-8 words) to categorize the content of each video. This supplementary tagging helps to ensure the diversity of the videos. We define
define 14 root domains as listed in Figure 2
3

---

Published as a conference paper at ICLR 2025
video to be long if it exceeds 10 minutes in duration. Accordingly, we collected videos longer than 10 minutes while considering the distribution of video duration. Furthermore, we retain the accompanying subtitles and audio to provide multimodal information. We carefully review and filter the videos manually for 7 rounds. More details about the video collection can be found in the supplementary material.
Question-Ansser--Cuee Annotation. After collecting the raw video data, we annotate it with high- quality question-answer--uue (QAC) triplets. To ensure question diversity, we establish a taxonomy with three types: Perception, Reasoning, and Hallucination. As shown in Figure 3 Perception and Reasoning questions are further divided into 10 and 14 subcategories, respectively, while Halluci- nation questions combine elements of both. Annotators are instructed to include negative options to create a multiple-choice QA format, facilitating straightforward and cost-effective assessments. To minimize expression loss, annotators use their native language during the annotation process. Each video annotated with 6 to 15 QAC triplets, depending on its duration. To ensure consistency in QAC triplets, we standardized the annotation process by first annotating the QA pairs and then iden- tifying the clues. Annotators must watch the entire video, select a question type from the predefined categories, and then annotate more intervals from the video to form a QAC triplet. Since the actual clue intervals often consist of multiple short moments, annotating each fragment is costly. Therefore, annotators required to mark intervals that cover these short moments while ensuring the completeness of each event.
question and its corresponding answer. Next, they select or
To ensure the difficulty and quality of the dataset, conduct repetitive review and iteration process to enhance annotation quality. We reject annotations that do not meet our quality standards and request annotators revise them. Our quality requirements for annotations and the measures taken to ensure them are as follows: 1) The rationality of the question, options, and answer: we conduct manual reviews; 2) The video dependency of the question, options, and answer: we input questions and options into GPT-4 and filter out QA pairs that can be answered solely based on pure text; 3) The difficulty of negative options in multiple-choice questions: we input the video, questions and options into MLLMs and filter out QA pairs that can be answered using only sparse frames and small models; 4) The positional diversity of clue intervals: We monitor the distribution of clue duration and position and provide timely guidance to annotators.
Review Iteration.
3.2 DATASET STATISTICS & COMPARISONS
We present the detailed statistics of our dataset to provide a more comprehensive understanding, including meta-information, QAC triplets, qualitative analysis, and comparison previous works.
3.2.1 DATASET STATISTICS
Video Meta. Our dataset comprises total of 1219 videos with multimodal information, including vision, audio, and subtitles. The duration of the videos varies between 10 and 80 minutes, with distribution illustrated in Figure 4 Notably, videos that last between 20 and 30 minutes are the most prevalent. This selection process is manual, based on content relevance, which mirrors real- world duration distributions and highlights a long-tail effect for longer videos. As illustrated in Figure [ each video is classified using a three-tiered tagging system that succinetly encapsulates its content and assigns it to fundamental categories. The primary classification augmented by secondary layer of 171 tags and a tertiary layer consisting of 638 tags. This multi-level tagging mechanism guarantees broad diversity of data content. For a more detailed classification of tags, please consult the supplementary materials.
QAC Annotation. CG-Bench includes 12,129 annotations consisting of questions, answers, and clues. Table [ presents the sentence lengths and totals for the annotated questions and answers, highlighting the linguistic diversity within our dataset. Each QAC triplet is annotated with negative samples, resulting in an approximately uniform distribution with ratios of options A to H of 12.4%, 14.7%, 12. 1%, 14.8% 15.1%, 1., 16.1%, %, 11.6%, and 3.1%. There are a total of 14,362 clue intervals across all QAC triplets, with average duration of 19.24 seconds each. Additionally, we conduct further analysis of the positions of clue intervals within the video. Figure 5 illustrates the frequency with which each normalized timestamp is represented by intervals. This demonstrates the unbiased nature of our interval annotations and highlights the diversity of our QA content in temporal position.
4

---

Published as a conference paper at ICLR 2025
Video Duration Distribution
Clue Time Coverage
Annotation Statistics
Timos Coverage
#QAC Triplets
12129
#Avg/QAC per video
9.95
#Avg/Option per QAC
6.96
#Avg/Clue per QAC
1.18
#Avg/Words Questions
20.07
#Avg Words Options
22.88
Druration (minutes)
Time Bins (1:88)
#Avg/Duration Clues
Video duration distribu- tion, showing number of vidcos for different duration intervals.
Figure 4:
Figure 5: Cluc time coverage, il- lustrating the frequency of clues across diffcrent time bins.
Table 1:
Annotation statistics, de- tailing the number of QAC triplets, questions, options, and clues.
Comparison of benchmarks across key aspects: number of videos (#Video). average duration (#Du- ration), number of QA pairs (#QA Pairs), number of clues (#Clue)) annotation method (M/A for ual/automatic), Open- Domain (OD), Open-Ended (OB), Multi-modal (MME), and Credibility (CE) Evaluation.
Table 2:
Benchmark
#Video
#Dur.(s)
#QA Pairs
#Clue
Anno.
OD
OE
MME
CF
Question Clac Grouding
NexGQA Xiao
2024
Ego4D-NL
Grumal ot
2022
415
499.7
4,554
M
Bgo4D-NI.Qeod
Limuman el
ATZ2
333
493.7
4,005
M
MultiHop-EgoQAA
1,080
A&M
E.T. Benchees
Liu ctal. 2044
129.3
2,011
M
RexTimee-cr eat
A&M
CG-Bench-QG
1,219
1624.4
14,362
M
Short Video OA
TVQA Tei et
2018
2,179
11.2
15,253
15,253
M
STAR
914
7,098
7,098
NexiQA Xigo
[1
1,000
44.0
8,564
EgoSchema Mangalam
5,063
TempCompass Liu cl 20249
410
11.4
7,540
A&M
Rex RexTime-,rr Clen
20206
141.1
2.143
A&M
MVBench
L 20233
3,641
16.0
4,000
A&M
MMBench- Viden (Pang er 2024
600
165.4
1,998
M
CG-Bench-Clue
12,129
22.8
12.129
Long- Vong-Video QA
EguTineQAcees
Di Xie
2024
492
500
MovieChat- IK Stng at
130
500.0
1,950
M
Video-MME [Fi
20744
1017.9
2,700
LongVideof VideoBencca WB et
2024b
966
1408.0
6,678
M.
CG-Bench
1,219
1624.4
12,129
14,362
M
3.2.2 COMPARISON WITH PREVIOUS BENCHMARKS
CG-Bench is characterized by its diverse features, allowing it to be compared with three distinct types of benchmarks, as depicted in the three sections of Table 2 Question Clue Grounding, Short-Video QA, and Long-Vin Video QA benchmarks. For the question clue grounding benchmarks, NextGQA Xiao et al. 2024). Ego4D-NLQ (Grauman et al. (022). MultHHop-EgoQAA (Chen et al., 2024ch. E.T. Bench (Liu al. 2024d). and RexTime (Chen et 2024a) are primarily centered around action and egocentric domains. Their videos are sampled from academic datasets. In com- parison, the question clue grounding part of CG-Bench, CG-Bench-CC QG, stands out with the highest number of videos and the longest average length, the diversity of which fosters broad spectrum of drestton-grounding queries.
Furthermore, we transform QAC triplets to our novel Short-Video QA benchmark, termed CG- Bench-Clue. When contrasted with prior short video benchmarks such as TempCompass (Liu al.) 2024c), MVBench (Li et al. 2023b) and MMBench- Viieeoc--VVeoo (Fang al.) (224)). CG-Bench-Clus emerges as the largest, held-out, open-domain and multimodal Short- Video QA benchmark.
As for the Long-Vide Video QA benchmark, CG-Bench excels in the number of videos. length, quantity of questions, and annotation quality. Owing to our clue interval annotations, CG-Bench further facilitates reliable evaluations for long videos and open-en open-ended evaluations with clue assistance, feature that sets it apart from existing long video benchmarks like Video-MME (Fu et al. 2024a) and MLVU (houu et al. 2024).
5

---

Published as a conference paper at ICLR 2025
3.3 EVALUATION
In this section, we describe the evaluation tasks of our CG-Bench which include traditional MCQ. the unique credibility evaluation, and clue-aided open-ended QA evaluation.
3.3.1 MULTIPLE--HOICE QUESTION EVALUATION
We assess the accuracy of MCQ in two settings: Long-Vide Video MCQ and Clue-based MCQ. In the Long-Video MCQ setting, the model receives the entire video as input and required to select the correct answer based on the video, the question, and the candidate options. For the Clue-based MCQ setting, the model is given only the video within the annotated clue interval as input. The model has access only to the clue clip, the question, and the candidate options. It does not have access to the original long video. Since single QA may correspond to multiple clues, we merge these clues and treat the combined clue single, cohesive clue segment.
3.3.2 CREDIBILITY EVALUATION
The ability of a model to identify relevant clues related to questions is crucial factor in determining its reliability. Therefore, we define model's reliability based on its proficiency locating accurate clues when addressing problems. To achieve this, we introduce two clue-grounder mechanisms for credibility assessment: white-box evaluation and black-box evaluation.
White-Box Evaluation requires the model to directly output the intervals of clues that can accu- rately answer the question. This task is similar to video temporal grounding (Lei et al. |021]Huang et al. 2023). Therefore, we use tloU (Temporal Intersection over Union) as the evaluation metric. Since each question may correspond to multiple intervals of clues, we allow the model to predict multiple possible intervals. Given set of prediction P and ground truths G, the tloU is defined as:
DiegJeP max(0, min(b, dz) max(as,
tloU=
Eieg a:) (de EiEg.JE max(0, min(bi, d,) max(ei, X 100%, (1)
where Ggs b; the start and end timestamps of the i-th ground truth interval of g. Cj. dj the start and end timestamps of the j-th predicted interval of P. We calculate the mean loU (mIoU) by averaging the tloU scores obtained by the model across all question queries. To further improve the robustness of question grounding evaluation, we introduce the rec. @ IoU metric. This metric measures the probability of successfully recalling clue intervals at various IoU thresholds.
Additionally, we propose acc. @oU that evaluates both MCQ accuracy and clue stue-grounding ability. A response is considered correct only if the selected answer is accurate and the tloU exceeds (>) a predefined threshold т. Since locating short-duration clues in the long videos in CG-Bench is inherently challenging, we set the default 7 to be 0 for the more obvious comparison on ablation 0 ensures that acc. @ loU requires the model to select the correct option and
studies. Setting produce a time interval that overlaps at least slightly (tloU > 0) with the annotated clue interval, rather than reducing to naive MCQ accuracy. We calculate the rec. @ IoU and acc. IoU at IoU thresholds of 0.1, 0.2, 0.3, 0.4, and 0.5 to determine the final result.
Black-Bi Box Evaluation aims to evaluate the model's ability to seek out clues implicitly. Understand- ing long videos involves the retrieval of clues distributed across various spatiotemporal locations within the entire video. Therefore, an effective model for long videos should naturally focus cap- turing human-annotated clue intervals in its hidden states. However, beyond the explicitly annotated clue intervals, there are likely hidden clues scattered throughout the video that can also help to de- termine the correct answer. Thus, a model with access to the full video should yield higher accuracy compared to solely relying on the clue interval. In other words, the accuracy of Long- Video MCQ ( long-acc.) should be greater than or equal to the accuracy of Cluc-based MCQ (clue-acc.).
With this insight, for the black box evaluation, we define new metric called Clue Recovery Rate (CRR). This metric evaluates the model's robustness to context dilution, i.e., how stable model can find related clues from long but diluted video context. CRR is calculated by:
CRR = min(long-acc., cluc-acc.)
x 100%,
(2)
clue-ace. A CRR of less than 100% suggests that the MLLM's ability to retrieve short clues from long video representations is not optimal.

---

Published as a conference paper at ICLR 2025
Question: la the video, who alc the oraage
Question: In the video, how does protagorist show curying today?
jolly?
GT: girl in black clothes
GT: Show it while standing sideways.
Prediction: with blonde hair
Prediction: Show by taking selfie front of the miner.
Figure 6: Two examples illustrating the ambiguity challenge of using LLMs for open-ended evaluation. While in different expressions, GT and prediction should both be treated as correct answers.
3.3.3 CLUE-AIDED OPEN-ENDED QA EVALUATION
CG-Bench supports open-ended QA evaluation for more comprehensive assessment. Previous works like MM-Vet Yuu et al. 2023) and MMBench- Video (Fang et al. (024) have used LLMs to evaluate open-ended QA for images and short videos. However, long videos contain more com- plex information, leading to ambiguous user-generated questions. This can result in discrepancies between LL.M-evaluate scores and the actual QA ability of models, as shown in Figure 6
To address this, we use a low-hallucination MLLM to evaluate the similarity between text output and visual information. We selected GPT-4o (OpenAl 2024 as the multimodal evaluator due to its high ranking in benchmarks like OpenCompass (Contributors 2023) and the Lmsys leaderboard (Chiang) al. 2024). and its relatively low hallucination rate compared to other MLLMs. Since using GPT-4o directly for multimodal judging can still introduce hallucination errors and incur high API costs, propose a heuristic evaluation method to mitigate biases and reduce costs. First, GPT-4o determines if the output requests visual cues by stating, "I need visual clues." This prompts the inclusion of supplementary visual data to aid GPT-4o in its evaluation. By using pre-annotated time intervals with question clues, we sample frames as visual aids, further reducing hallucination errors and costs. We analyze this evaluation method Sec 4.3 with more details available the supplementary materials.
be evaluated based solely the text answer. It outputs yes no; if not, it
EXPERIMENTS
In this section, we evaluate wide range of MLLMs using CG-Bench. We first introduce the evalua- tion setup, followed by quantitative results for both closed-source and open-source models. Finally, we analyze some key factors in the evaluation.
4.1 SETTINGS
We first briefly describe the settings used our experiments. The supplementary material provides more detailed settings.
Models. We evaluate the performance of three mainstream commercial models on our CG-Bench: GPT4o (OpenAI 2024), Gemini-1.5 (Anil al. 2023), and Claude-3.5, including their different versions. Also, we assess the representative open- image-MLLMs, such as LLaVA-OV (Li al. 2024a), QW (Wang et al. 2024b) and Intern VL2 (Chen et al. 2024e). video-MLLMS.
such as VideoChat2 (Li et al. 2023b).
Frame Sampling, For long video understanding, the frame sampling strategy significantly impacts evaluation results. For open-source MLLMs, we make the best use of our computational resources to use as many frames as possible. For closed-source MLLMs, since the local computational resource is no longer bottleneck, we can use even more frames. We uniformly sample (Wang et al. 2019) 128 frames for Long-video MCQ, and use 32 frames as the for Clue-based MCQ.
Modality. We also explore other modalities: subtitles and audio. For subtitles, we employ uniform sampling method. the timestamp of a sampled frame falls within the time interval of subtitle, that subtitle will be included in the analysis. Each subtitle is considered only once avoid redundancy.
Prompt. For MCQ tasks, the model prompted to provide the uppercase letter corresponding to the correct option. In Open-Ended QA tasks, the model responds freely based the questions. For the Clue Grounding task, we append the timestamps of each frame and subtitle to enhance the model's lime-awareness requiring it to return nested lists in the format [ss, el], [s2, e2], ..... J. For open-ended evaluation, we require the model to assess the correctness between the predictions and the ground truth and respond with yes no.

---

Published as conference paper at ICLR 2025
Table 3: Performance of various open-source and closed-source MLLMs CG-Bench. We provide human
evaluation for showing annotation agreements the difficulty of our benehmark.
Models
L.M
#F
MCQ
Cred. Eval.
OF
#param
clue
long
clue-acc.
long-acc.
mioU
rec. IoU
acc. nec.61ol MoU
CRR
acc.
Random
14.2
14.2
0.13
0.16
0.09
100
Human (full-video)
92.2
90.3
35.5
51.2
29.8
97.9
83.7
Human (sparse frurnes)
128
59.9
GPT40 (text)
16.8
16.8
0.14
0.2
0.15
100
2.1
Open-source MILLMs
Video-LLAVA Lin
2023
715
8
34.2
16.2
1.13
1,96
0.59
47.4
12.3
VidcoLLAMA Zhang stal
2023g
7B
32
32
3688
18.4
1,21
1.87
0.84
50.0
15.8
Videochar2 a
16
Qwen VL Chal Maa et 2023
7B
4
38.3
21.6
0,89
1.19
0.42
56.4
19.4
21744
2,86
20.7
ShareGPT4Vicss (C hon
2024b
16B
16
16
41A
26.7
1.85
2.65
1.01
64,5
22.0
Chat-UniVi-vl.5
2024
13B
64
41.5
25.9
2.07
2.53
1.21
62.4
21.4
ViLA (Lnn
2024
8B
14
14
41.8
28.7
1.56
2,89
1,35
68.7
24.0
GrundVQq (Tali ci
clal..
20244
0.25B
1200
27.3
1.33
1.37
Chen
2.81
ET Chal Lib al
41
1fps
17.6
1.38
1.43
InterVVI-Chaavvi
Chen
2023d
2.38
68.0
23.1
MiniCPM-22.6 Yao 0f
8B
32
32
44.6
30.1
2.35
2.61
1.04
67,5
26,6
Long Zhang 21
7B
32
128
42.8
28.7
2.94
3.86
1.78
67.1
25.1
LLaVA One
[L]
2024
71
16
43.2
31.1
1.63
1.78
1,08
72.0
25.4
Videa-CCAM Fei 2.
20744
14R
32
96
43.6
29.7
2.63
3.48
1.83
68.1
25.3
Kanguroo al
VITA Fu ct M724n
8x7B
32
32
47.8
33.3
3,06
3,53
2,06
69.7
27.5
QwenZ VL
Wang 2024b
72B
56,2
5,32
73.5
33.6
LoterVL2 Teea
78B
32
32
38.5
42.2
3.91
5,05
2.64
72.1
32.5
Closed-source MLLMs
(OpenAl 2024
77.5
GPT-4mini-08- (OpenAl
2024
32
128
48.3
33.4
3.75
5.18
2.21
69.2
25.4
Gemini-1.5-Pro (Ahil 20 2125
50.1
37.2
5,81
74.3
29.3
Gemini- Comini-1155FFi Fla.Flash
128
47.0
32.3
3.67
5.44
2.45
68.7
26.3
So-onct5-Sonnet
32
50
56.2
40.5
3.99
5.67
2.79
72.1
35.2
4.2
MAIN RESULTS
As shown in Table the closed-source MLLM GPT4o (OpenAl 2024) leads significantly, sur- passing other models across all metrics. Notably, GPT4o's long-acc. reaches 45.2%, much higher than Gemini-1.5-Pro (Anil al. (023). demonstrating its strong capabilities in long video under- standing. Among open-source MLLMs, Qwen2-VL (Wang et al.] 2024b) performs impressively, achieving results comparable to GPT4o long-acc. and clue-ace. Other models underperform due to insufficient context support or inadequate video training. While these MLLMs perform well on MCQ tasks, they experience significant performance drops credibility and open-ended evalua- tions on CG-Bench. For instance, GPT-4o's long-acc. falls from 45.2 to 4.38 in Acc@ loU and 39.5 in OE-acc. Additionally, with the same number of sampling frames, GPT-4o achieves CRR of 77.5, whereas Gemini-1.5-Pra only reaches 74.3, indicating its weaker ability to retrieve short-term clues from long videos. Overall, current MLLMs do not perform well on CG-Bench, suggesting considerable room for improvement in their capability and credibility.
Since is difficult to input more than 128 frames due to the hardware limitations, we alternatively conducted human evaluation experiment under constrained visual conditions, to how severe the "undersampling" issue is for longer video. We uniformly sampled 30 videos from CG-Bench, resulting in 296 questions. For each video, uniformly sampled 128 frames and asked volunteers to perform an MCQ testing. The resulting accuracy was 59.85% (row in Table 3 This result indicates that our dataset is indeed challenging and that it is difficult to derive solutions from limited number of frames. It also highlights that even the most advanced models, such as GPT-4o, have ample room for improvement in long video comprehension.
4.3 ANALYSIS
Furthermore, perform comprehensive analysis of the two leading closed-source MLLMs, GPT4o (OpenAl 2024) and Gemini-133 Pro (Anil et al. 20233, as well as the best performing open- source MIL.M, Qwen2-VI. (Wang al. 2024b), CG-Bench. In this analysis, use 1000
8

---

Published as a conference paper at ICLR 2025
Table 4: Impact of different prompts and modalities. Each prompt can be composed of frames (F), frame timestamps (FT), subtitles (S), subtitle timestamps (ST), and audio (A). We conduct main experiments with GPT40-0806 (OpeaAl 2024) while studying the audio modality with Gemini-1.5 Pro (Anil et al. 2023).
model
prompt & modality
clue-acc.
long-acc.
mlot
Aot@Iot
CRR
OF-acc.
GPT4e
S (128 frames)
31.5
GPT4e
S (full video)
34.3
GPT40
F
65.8
5188
3.39
10.7
78.7
35.4
GPT4e
F+FT
(-0.5)
(1.6(-0.2)
(.7.44))3))
(0.4(+9.7)
(9.0(-
(6.8(+1144)
GPT4o
F48
66.7(10.9)
53.4 1.0)
3,96; (0.07)
11.2 10.5)
80.1 1.44
38.2 (2.8)
GPT4o
F+S+ST
67.1
54. 54.1(+2.3)
5.19(+1.80)
13.2(42.55
80.6(-1.9)
38, 38.4(+3.0)
GPT4o
F4S+FT
67.4)116))
53.2(1 1::4)
7.80; (4.41)
22.3( (11..))
78.9 0.23
37.9: (2.0)
GPT4o
F+S+ST+FT
67.5 (4-177)
54.9 (43.0)
9.68 (40.2))
26.7 (41660)
81.3 (-230-
39,5
Gemini
F+S-ST+FT
45.1
9.16
20.7
Gemini
F48-ST+ET+A
62.3 2)
45.0(-0.13
9.10
19.8(
(2.2(-0.4)
23.5
53-
Ovenbi..
Gersasd.Shrs Oven2ess
32
35-
Germia
44-
23
Nunba hrttcs
Muite of humio
Nurber Of framcss
Rumberr of Namrcc
Numbe of Humms
(a) Long Acc
(b) mloU
(c) Acc@IoU
(d) CRR
(c) Open ended
Figure 7: Impact of sampling frame numbers on different metrics for GPT-4o-0806 OpenAl 2024). Gemini-
1.5 Pro Anil et al. 2023) and Qwen2VL-72B (Wang el al. 2024b).
QAC triplets sampled uniformly from all annotations for fast experiments. We report acc. Ioo with r = 0 for a more obvious comparison.
Impact of Prompt & Modality. As shown in Table conduct the ablation studies the subset that contains subtitles and explore the impact of different prompts on GPT4o and the effect of the audio modality on Gemini-1.5 Pro. Our findings indicate that all prompt types (FT/S/ST), except video frames (F), provide performance benefits across most metrics. Subtitles contribute more to long-acc. than they do to clue-acc.. Additionally, the inclusion of timestamp information (FT/ST) is critical for interval prediction. Timestamps from both frames and subtitles enhance loU-related metries, revealing a complementary effect. When both FT and ST are added simultaneously, mloU increases from 3.39 to 9.68, and Ace@Iol rises from 10.7 to 26.7. When S, FT, and ST are all used in the prompt, the model achieves the best performance across all metrics. contrast, our exploration of the audio modality (A) revealed that audio does not yield significant performance gain and, in some cases, even slightly degrades the results, as shown in Table 4 Finally, we con- duct experiments using only subtitles from 128 frames versus the full video. The results show that while subtitles offer useful semantic cues, their impact is significantly reduced when visual input is included. This suggests that our benchmark favors visual signals.
Impact of Frame Number. As illustrated in Figure we conducted experiments to analyze the performance across various metrics as the number of frames increases. Overall, the performance of all three MLLMs gradually improves with the addition of more frames, with GPT-4o consistently outperforming the others across all metrics. For long-acc. and OE acc., Qwen2VL achieves per- formance comparable to GPT-4o. However, compared with Qwen2VL, Gemini excels in terms of mloU and Acc@ IoU, Regarding CRR, GPT-4o demonstrates greater consistency between clue-ace. and long-acc. across more frames, indicating its superior reliability in long video understanding. For open-ended QA, Gemini's higher refusal rate results in a noticeable decline in performance.
Open-ended Evaluation Quality. To assess the stability and accuracy of various MLLMs as evalu- ators, we utilized four Mondels-Geming Qwen2VL, Claude, and GPT-4o-each which evaluated GPT-4o's predictions five times. Human evaluations of GPT-4o's predictions are also conducted for reference. The results, shown in Figure 8, indicate that GPT-4o has the highest stability and the smallest deviation from numan-assigner different evaluation methods. When evaluators were provided only with ground truth (col. "GT"") or visual information (col. "Vis"), the scoring bias (absolute difference) between human and model-
scores. Furthermore, Table 5 explores the impact of

---

Published as a conference paper at ICLR 2025
Gemini
GT
GT+Vis
Vis
Ours
Qwen2VVL-
Biast ( (4)
12.4
6.4
17.0
1.0
19,640
Price (S)
0,05
6.1
6
CPT4o
Human Score
Trigger Trigger Recall Rate (1517
100
100
88
Figure 8: Comparison of using different LLMs as open-ended evaluators for GPT-4o's outputs.
Table 5: Comparison of different modes: GT--only, visual-only. GT+vision and heuristic (Ours).
Video Duration vs Accuracy
54
55.6
Accuracy (%)
#Frames
Resolution Sampling Strategy long-ace.
53.6
128
Low
Uniform
53,9
50.7
53
49.4
50
Low
Uniform
46.7
50
Low
Keyframe
45.7
50
High
Uniform
51.0
10.23mm
20-00mm
40.55m
20.53mm
30.6mm
60-mi
Figure 9:
Long-Video -ige Accuracy grouped by video duration for GPT4o 0806 with 128 frames.
Table 6: Impact of different frame sampling strategies long-acc. for GPT40-0800
based evaluation increased. While fully leveraging visual information (col. "GT+Vis") improved evaluation accuracy, it also significantly increased the time and cost required. Our proposed heuris- tic evaluation method achieves the lowest evaluation bias. Additionally, we manually annotated 200 evaluation samples to determine the necessity of visual request triggers. From the bottom block in Table 5 the statistics show that our method achieved a visual request trigger rate (the probability that the model triggers "visual clues required") of 14%. The recall rate of this triggering achieves 88%. This proves that our approach effectively balances cost and performance.
Performance grouped by Video Duration. We grouped videos by duration and evaluated the long- acc. performance GPT--4-08806 using 128 frames. Figure 9 shows that the model struggles with undersampling, especially for longer videos.
Impact of Frame Sampling Strategy. We investigate how different frame sampling strategies affect performance. To expedite testing, primarily evaluated GPT40-0806 using 50 uniformly sampled frames, focusing on the long-acc metric. The experiment consists of three parts: 1) low resolution, 2) high resolution, and 3) keyframe extraction (via FFmpeg) combined with low resolution. As shown in Table 6, higher resolution offers some improvement, while keyframe extraction has no significant impact.
5 CONCLUSION AND FUTURE WORK
In this paper, we introduce CG-Bench, a novel benchmark designed to evaluate clue-grounded ques- tion answering capabilities in long video understanding. Unlike existing benchmarks that focus on short videos or rely solely on multiple-choice questions, CG-Bench emphasizes the importance of models retrieving and grounding their answers in specific video segments, enhancing evaluation credibility. CG-Bench includes 1,219 manually curated videos organized into detailed three-tier system, with 12.129 QA pairs covering perception, reasoning. and hallucination question types. This provides a comprehensive and diverse dataset for assessing MLLMs. We propose two clue-based evaluation
white-box and black-box evaluations determine whether models genuinely comprehend video content or merely rely on superficial cues. Extensive experiments with various closed-soume -ource and open-source MLLMs reveal that current mod- els significantly underperform in long video understanding compared to short videos. We hope that CG-Bench will serve as a valuable resource for the research community, driving the development of more trustworthy and capable MILLMs for long video understanding.
offer novel ways to
Acknowledgement. This work supported by the National Key R&D Program of China (No. 2022ZD 0160900), the National Natural Science Foundation China (No. 62372223 and U24A20330), Jiangsu Fron- tier Technology Research and Development Program (No. BF2024076), Postgraduate Research & Practice Innovation Program of Jiangsu Province (No. K KYCX24_0260), and Nanjing University-Chin Mobile Com- munications Group Co., Ltd. Joint Institute.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- Cean-Baptiste Alayrac, Jiahui Yu, Radu Sorieut, Johan Schalk- wyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Mclvin Johnson, loannis Antonogiou, Julian Schrittwieser, Amelia Glaesc, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Ruther- ford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/23 12. 1805, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin. Chang Zhou, and Jingren Zhou, Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Fyzaguirre, 7ane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video-language understanding. In Advances in Neural Information Processing Systems, volume 37, 2024.
Aozhu Chen, Ziyuan Wang, Chengbo Dong, Kaibin Tian, Ruixiang Zhao, Xun Liang, Zhanhui Kang, and Xirong Li. Chinaopen: A dataset for open-world multimodal learning. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 6432 6440, 2023a.
Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin. Yu-Chu Yo,, Yea-Chun Chen, and Yu-Chiang Frank Wang. Rextime: A benchmark suite for ceasoning-across-iinn in videos. arXiv preprint arXiv:2406.1925 2024a.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegnt4video: Improving video understanding and generation with hetter captions. arXiv preprint arXiv:2406..e4 04325, 2024b.
Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. arXiv preprint arXiv:2408. 14469, 2024c.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: laino-aadio-language omni- neni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 2023b.
Sihan Chen, Handong Li, Qunbo Wang. Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vibion-audio-subiilleeexxx omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842-72866, 2023c.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, tesi Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming- Hsuan Yang, al. Panda-70m: Captioning 70m videos with multiple cross-mode mosdattylltty teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13320-13331 2024d.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Intervll: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprint 23X1: 14238, 2023d.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are to gpt-4y? closing the gap commercial multimodal models with opeo-poeeneeeooo source suites, arXiv preprint arXiv:2404. 16821, 2024e.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprini arXiv:2403.04132 2024.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https:
//github.com ccn http:///ww.ccpppoooooooo////eeee http://ppwpss/ppp.ooopppps..sssp 2023.
Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12934-1299 12943, 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li. Dahua Lin, and Kai Chen. Mmhench-video: A long-form multi-shot benehmark for holistic video understanding. arXiv preprint arXiv:2406.1450 2024.
11

---

Published as a conference paper at ICLR 2025
Miquel Farré, Andi Marafioti, Lewis Tunstall. Leandro Von Werra, and Thomas Wolf. Finevideo. https: / wwwgggiggaaccco co/ htt.aaaatttetgggaaaaaaesssssseee wwt.a/gggoo...aa///aaaaaaeaaaaaa (te.eeiieeeeeeoeoooooo 2024.
Jiajun Fei, Dian Li, Zhidong Deng. 7ekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint vrXiv:2408.14033 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms video analysis. CoRR, abs/2405.21075, 2024a.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng. Ran He, Rongrong Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint 2xxiv:2488.0522 2024b.
Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. pp. 18995-19012, 2022.
Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for compre- hensive understanding of multi-shot videos. arXiv preprint arXiv:2312.10300 2023.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefci Cao, A Ashish Shah, Abhinav Shrivastava, and Ser- Nam Lim. Ma-Imm: Memory- sugmere- large multimodal model for long-term video understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514, 2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. pp. 961-970, 2015.
Gabriel Huang, Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video captioning. CoRR, abs/201 1.11760, 2020a.
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task- dependent attention transition. In Proceedings of the European conference on computer vision (ECCV), pp- 754-769, 2018.
Yifci Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal rea- soning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14024- 14034, 2020b.
Yifei Huang, Lijin Yang, and Yoichi Sato. Weakly supervised temporal sentence grounding with uncertainty- guided self-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 18908-18918, 2023.
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang. Lijin Yang. Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang. Limin Wang. et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22072 2207722222622222 2024.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual rep- resentation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700-13710, 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video datasct with long durations and structured captions. arxiv preprint arXiv:2407.00 06358, 2024.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaanarasimman, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint 1809000090019 2018
Jie Lei, Tamara I.. Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. pp. 11846-11856 2021.
12

---

Published as a conference paper at ICLR 2025
Bo Li. Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.055 03326, 2024a.
KunChang Li, Yinan He, Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305. 06355, 2023a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprchensive multi-modal video understanding benchmark. CoRR, abs/231 1.005 2023b.
Xinhao Li, Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li. Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint axXiv:2511.00574 2024b.
Yanwei Li. Chengyao Wang. and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. CoRR, abs/2311. 1. 10143 2023c.
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang. Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et Eagle 2: Building post-trat training data strategies from scratch for frontier vision-language models. arXiv preprint arXiv:2501.1455
2025.
Bin Lin, Bin Zhu, Yang Yc, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shocybi, and Song Han. Vila: On pre-trat pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689-26699 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, 0ass/2310.037444 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 36, 2024a.
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408. 15542, 2024b.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. ST-LLM: large language models are effective temporal learners. CoRR, abs/2404,00328 003000 2024c.
Ye Liu, Zongyang Ma. Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open- ended event-level video Languaan understanding. arXiv preprint wxXh:2449.118111 2024d.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint urXiv:2403.000 2024e.
Zuyan Liu, Yuhao Dong, Ziwci Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409. 2024f.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards de- tailed video understanding via large vision and language models. arXiv preprint 2rXiv:2306.05424. 2023.
Karltikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac Makarand Tapaswi, Tvan Laptev, and Josef Sivic. Howto100m: Learning a text-viden embedding by watching hundred million narrated video clips. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 8630-26400 2019.
Kepan Nan, Rui Xic, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijic Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- Im: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02377 2024.
Munan Ning Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, arxiv preprint arXiv:2311.16108, 2023.
13

---

Published as a conference paper at ICLR 2025
Andreea-Maria Oncescu, Joao F Henriques. Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: video dataset with high-quality text and audio narrations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2265-226622222 2269. IEEE, 2021.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774 2023.
OpenAL Hello gpt-4o. 2024. URRptttss////peeeee
Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406. 18070, 2024.
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405. 08813, 2024,
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hicrarchical vidco dataset for fine- aiee-graind action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 2616-2625, 2020a.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Intra-and inter-action understanding via temporal action parsing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 730-739,
2020b.
Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang. Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video under- standing. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 18221-18233 2024.
Min Sun, Ali Farhadi, and Steve Seitz. Ranking domain-specific highlights by analyzing edited videos. In Computer Esion-ECCYy 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 787-802. Springer, 2014.
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. Advances neural information processing systems, 35:38032-38045, 2022.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional vidco analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1207-1216, 2019.
Du Tran, Heng Wang, Matt Feiszli, and Lorenzo Torresani. Video classification with channel-separated convo- lutional networks. pp. 5551-5560, 2019.
Han Wang, Tan Rui Yang, Usman Naseem, and Roy Ke-Wei Lee. Multihateclip: multilingual benchmark datasct for hateful video detection on youtube and bilibili. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7493-7502, 2024a.
Limin Wang. Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. 41(11):2740 2755, 2019.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen. Xuejing Liu. Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's percoption of the world any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Qi, Shiyu Huang, Bin Xu, Yuxiao Dong. Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08033 2024c.
Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10785, 2021.
Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang. Limin Wang. and Yu Qiao. Intern- video: General video foundation models via generative and discriminative learning. CoRR, abs/2212.03191, 2022.
14

---

Published as a conference paper at ICLR 2025
Yi Wang, Yinan He. Yizhuo Li, Kunchang Li. Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Intervvid: A large-scale vidco-text dataset for multimodal understanding and generation. arXiv preprint ar Xixx:2307.06944 2023.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan Hc, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, 7un Wang, et al. Intervvide22: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 15377, 2024d.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real- world videos. CoRR, abs/2405.097 2024a.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: benchmark for long-context inter- leaved langu-la understanding. arXiv preprint arXiv:2407.15754 2024b.
Junbin Xiao, Xindi Shang, Angela Yao. and Tat-Seng Chua. Next-qa: Next phase question-answerigg to explaining temporal actions. pp. 9777-9786, June 2021.
Junbin Xiao, Angela Yao. Yicong Li, and Tat- Tat-Seng Chua. Can trust your answer? visually grounded video question answering. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 13204-13214, 2024.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang. Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint vrXiv:240.110671 2024a.
Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivie, and Cordelia Schmid. Vidchapters 7m: Video chapters at scale. Advances in Neural Information Processing Systems, 36, 2024b.
Dongjic Yang, Suyuan Huang, Chenggiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. arXiv preprint arXiv:2406.060400 2024c.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408 01800, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02499 2023.
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hi- crarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23056-2000 200666226652222 2023.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and lan- guage and sound. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16375-16387, 2022.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pp. 543-553. Asso- ciation for Computational Linguistics, 2023a.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zuen-Hua Ling. Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile canesiin-ansseriin for long-form movie understanding. arXiv preprint 2311v22312.048// 2023b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Hao- ran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024a. URL https: : arxiv..iivvvvi.iii/eaaaaaaiiaiva org/ abs 2406 1020066668... 16852]
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liange Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Lexva-next: A strong zero-shot vidco understanding model, April 2024b. hollhhttss///lavaa
gi git www..b..cccllgg/. 20000////22204400-00000000000000 1-
Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. HACS: human action clips and segments dataset for recognition and temporal localization. pp. 8667-8677. IEFE, 2019.
15

---

Published as a conference paper at ICLR 2025
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong. Bo Zhang, Tiejun Huang. and Zheng Liu. Mivu: A comprchensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 04264, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instruc- tional vidcos. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Dimitri Zhukov, Jean Jean-Baptinn Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2019.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
Annotation
A.I: Quality Control
A.2: Statistics
A.3: QA Examples
B: Model Inference and Evaluation
B.1: Common Prompts
B.2: Inference Prompts
B.3: Evaluation Prompts
Video
C.I: Video Collection
C.2: Video Tags
Statisties
D: Additional Experiments
A ANNOTATION
A.1 QUALITY CONTROL
During the annotation process. implement a quality control system illustrated in Figure 10 batch increment method for data iteration, reviewing each batch of about 1,000 items.
We
QAC Botch
Fust Monaal
Check
yes
Pure Text Cleck
yes
Sparse Frame Check
Small Model
yes
Second Mannal
Check
yes
Over
no
annotare
Ro-Annotated
Figure 10: Annotation Quality Control Flowchart.
First, manual review checks for typos and ensures question quality. We focus two main aspects: clarity and granularity. Questions must have clear anchor point, such an event scene, avoid confusion. The granularity should be appropriate; overly broad questions provide too many easy clues, which undermines our goal of testing the model's ability to pinpoint clues.
Next, to ensure question difficulty, we conduct tests using LLM, such as GPT4 (OpenAl 2023 and Qwen2.5 (Yang et al. 2024a), with pure text and small MLLM, like Intern Intern VL2-2B (Chen et al. 2024c) and LLtemmVL2-4B, with sparse frames. The pure text test ensures that questions and options don't reveal too much information, allowing models answer without visual data.
Finally, the second manual review catches other remaining issues, resulting in the final test set.
We provide two examples of filtered samples for "Small Model & Sparse Frame Check" in Figure |1 and Figure 12 For Figure [I] the protagonist cycling in first-person view, and the outfit appears throughout the video. For Figure 12] the climbing wall is a prominent target, and the distinctions between the options very clear, requiring minimal comprehension.
And here is another example of filtered sample for "Pure Text Check":
In the video, according to the content shown in the PPT, the teacher talked about ethylene. So what will be produced after ethylene is oxidized by potassium permanganate? acetone
acetic acid
C. acetaldehyde
D. carbon dioxide
E. carbon monoxide
This QA is essentially simple chemistry question and therefore did pass the check.
17

---

Published as a conference paper at ICLR 2025
User: bat color is the protagonists eycling outfit the video?
Red
B. Blue
Green
D. Black
White
Figure 11: Example of filtered QA "Small Model & Sparse Frame Check".
Use: What does the climbing wall in the video look like? Indoor artificial climbing wall
B. Climbing wall with various color markings
Outdoor lot-tlti climbing wall
D. Training wall with multiple climbing routes
Cliff
Figure 12: Example 2 of filtered QA by "Small Model & Sparse Frame Check""
A.2 STATISTICS
Question Categories and Definition. We list all question categories in Table We also provide a rough definition of each question type:
Entity Recognition: Identifying entities within context, focusing on recognizing specific objects or entities present in the scene.
Entity Counting: Addressing the quantity of entities, focusing counting the number of specific objects.
Entity Attribute:
Exploring the attributes of entities, such as shape, color, material, etc.
Entity State: Investigating the of object, including its current condition and any changes time, focusing on the status of the entity and evolution.
Event Recognition: Identifying events within the context, focusing on recognizing specific occurrences or actions taking place in the scene.
Event Counting: Counting the occurrences events, focusing on many times a particular event takes place within the context.
Scene Recognition: Identifying and understanding the seene where takes place. Questions may explore details about the setting, such its characteristies, background elements, or overall atmosphere.
Text Recognition:
Identifying the content of text, focusing recognizing specific text elements within
the context.
Text Counting:
Addressing the number of specific aspects of the text, focusing on the number of certain elements or directions within the text.
Time Localization: Identifying the temporal range specific time points of
Time-grounded Question: Exploring questions based on the time points or intervals of specific entities, such as when certain entities appear events occur.
Spatiotemppoal-groundde Question: Exploring questions based on both the spatial and temporal peets of specific entities.
Entity 2D Spatial Perception: Inquiring about the 2D spatial position of entities within the video frame, referencing specific such top, bottom, left, right, center, lower-right, lower-left, etc.
18

---

Published as a conference paper at ICLR 2025
Table 7: 3-level Question Categories.
Level-1
Level-2
Level-3
Example
Entity Recognition
Figure
Entity Counting
Figure
Entty Perception
Entity Attribute
Higure
15
Emity State
Figure
Event Recognition
Higure
Event Perception
Perception
Eveni Counting
Figure
Scene Perception
Scene Recognition
Figure
Text Perception
Recognition
Figure
Text Counting
Figure 21
Time Localization
Figure 22
Time Perception
Time-grounded Question
Higure 25
Spatialtemporal-gron Question
Figure
Spatial Perception
Entity Spatial Perception
Higure
Character Identity Reasoning
Figure
Character Emotion Reasoning
Figure
Emity Reasoning
Character Intention Reasooing
Figure
Character Relationship Reasoning
Figure
Eniity General Reasoning
Figure 50
Entity Spatial Relationship
Figure
Reasoning
General Reasoning
Figure
Event Reasoning
Event Time Relationship
Figure
Event Causal Reasoning
Figure
Scene Reasoning
Scone Time Relationship
Figure 35
Text Reasoning
Text General Reasoning
Figure 36
Text Spatial Relationship
Figure
Time Interval Reasoning Duration Time Reasoning
Figure
Time Reasoning
Figure
Hallucination
Hallucination
Hallucination
Higure
Character Identity Reasoning:
Inquiring about the identity of a character, focusing on deducing or identifying who a character is within the context.
Character Emotion Reasoning: Understanding a character's emotions, focusing on analyzing or terpreting the feelings or emotional state of character.
Character Intention Reasoning: Reasoning about character's motivations intentions within the video context, exploring the underlying purpose actions by analyzing situational details and moti- vations.
Character Relationship Reasoning: Delving into questions regarding social or interpersonal re- lationships between characters, focusing on the type of relationship or connection shared based on observed interactions and context.
Entity General Reasoning:
Examining general relationships between entities, including person-object and object-object interactions, clarifying connections or associations beyond spatial or social rela- tionships.
Entity Spatial Relationship: Understanding spatial relationships between objects or entities, focusing on relative positioning to form a mental map of the scene's layout.
Event General Reasoning: Answering questions requiring deeper reasoning cognitive understand-
ing events, encouraging a comprehensive interpretation actions, motivations, and consequences.
Event Time Relationship: Understanding the temporal sequence of events, focusing ordering events correctly identifying particular event's position within sequence grasp the flow of actions the viden.
Event Causal Reasoning:
Exploring cause-and-effect relationships within events, facilitating derstanding why event occurred by linking
underlying causes.
Scene Time Relationship: Exploring the sequence in which different scenes occur, focusing chrono- logical order progression between different backgrounds.
Text General Reasoning: Answering questions involving inferential content within text in the video, deducing implied meanings, identifying underlying messages, or drawing conclusions from textual information.
19

---

Published as a conference paper at ICLR 2025
Text Spatial Relationship:
Addressing the positioning of text within the video frame, such identify- ing specific locations (top, bottom, left, right, center) clarify the visual layout.
Time Interval Reasoning: Addressing the time interval between two events, focusing on the time gap separation between occurrences.
Duration Time Reasoning: Exploring the duration specific event, inquiring about how long an event lasts or the time span of an action within the context.
Hallucination: Evaluating multiple statements related to video content. Unlike single-statee statement hal- lucination questions, the multiple hallucination class involves listing statements and judging which ones are correct. emphasizes attention to detail and careful assessment of options, distinguishing between accurate and subtly altered statements.
A.3 QA EXAMPLES
We provide an example for each problem category from Figures 1o
User what did the black throw the person the yellow platform?
Bag
R. Microphone
D. Camera
E. Keys
Figure 13: An example of QA in CG-Bench for Entity Recognition.
B MODEL INFERENCE AND EVALUATION
In this section, we list the prompt use in inference and evaluating existing models.
B. COMMON PROMPTS
Subtitle Prompt (Denoted <Sub>):
The subtitles of the video are as follows: <Subtitles>
Subtitle Time Prompt
Uñem: When the video author to sheep on the mad, how many sheep crossed road?
0
B.2
C.
E
Figure example of QA in CG-Bench Entity Counting.
20

---

Published as a conference paper at ICLR 2025
video, when introdacing kitten's toys, rolling the kitten playing with?
B. Orange
Green
D. Y Yellow
E. Puple
Figure 15: An example of QA in CG-Bench for Entity Attribute.
the protagonist cutlery while eating pasta?
hand, spoon right B. Hold spoon left hand, and fork in right hand
Only use tock in
D. Use chopsticks
E. Only use fork right hand
Figure 16: An example of QA CG-Bench for Entity State.
Whet author making the cream?
Prepared cheese platter
B. Made caraeel coolue crumbs
Cooked vegetable stew
D. Stured fruit smoothie
Baked a ple
Figure 17: An example of QA in CG-Bench for Event Recognition.
User: How many people fell in video?
B.3
D.5
E.2
Figure 18: An example of QA in CG-Bench for Event Counting.
21

---

Published as a conference paper at ICLR 2025
User: bat the weather the ead of the author's last Tripel?
Cloudy
B. Sunny
Reiny
D. Fogay
Snowy
Figure 19: An example of QA CG-Bench for Scene Recognition.
User: when scrubbingg the induction cooker, what brand induction
cooker being scribbed??
Cuisinart
R. Tefal
Kenwood
D. Zojizushi
Taagroo
Figure 20: An example of QA CG-Bench for Text Recognition.
CTT
many gmom the bride?
B.
15
D. 10
Figure 21: An example of QA in CG-Bench for Text Counting.
User the video, what did first goal of the match occur?
22nd
half
8. 60th minute of the second half
D. 34th minute of the first half
E. 12th micute the first
Figure 22: An example of QA in CG-Bench for Time Localization.
22

---

Published as a conference paper at ICLR 2025
User bat the video author doing the interval [00:14:59, 00:15:17]P
Watering plant
B. Reading book
Cooking a meal
D. Playing the guitar
Playing with
Figure 23: An example of QA in CG-Bench for Time-grounded Question.
User: the video, from 3:18, what object center of video?
Fruit
R. Meet
Plastic
D. Dining Table
Cats
Figure 24: An example of QA in CG-Bench for Spattootempora-ggronnddd Question.
What pattem appeared
the
rectangle
B. green triangle
C.A red circle
D. blue circle
red rectangle
Figure 25: An example of QA in CG-Bench for Entity 2D Spatial Perception.
Use:: beginning the video, front of the clock tower buiding. a person
dressed in all black squats down to take a photo of someane?
wearing rabbit headdress
florl shirt
B. Women with long red hair and blue top
Women long hair weering lolita
D. Woman with brown hair and black pants
Woman white dress
Figure 26: An example of QA CG-Bench for Character Identity Reasoning.
23

---

Published as a conference paper at ICLR 2025
User: Way did the protagenist keep laughing aftersinging "Red Peach Sister Rocks Tagether?
A. Because the lipetick stuck to his teeth
Because forgot the lyries halfway through the song.
Because her microphone stopped working
D. Because voice cracked during high note
noticed her reflection and found amosing
Figure 27: An example of QA in CG-Bench for Character Emotion Reasoning.
STEAM
User: After the protagonist video republited the game, clicked into the STEAMWORKS platform egain. W What was he doing?
update the game's description
B. In order to adjust the game's release date
to price game
D. In order to configure the gume's graphics settings
E. order to manage game's download size
Figure 28: An example of QA in CG-Bench for Character Intention Reasoning.
Fe
Usar: la the video, when the first-person perspective protagonist pulls small pile of what is relationskup between the person who appears
boxes the store and the protagonist?
They employes employee relationship
B. They are old friends
They are neighbors
D. They are colleagues
They are strangers
Figure 29: An example of QA CG- Bench for Character Relationship Reasoning.
Use:: the video, the protagonist intriducing what can controlled remotely controlled in the left anneest of the seat?
A. Seats, Curtains, TV, In-fiight engettteertimmeent system
B. Table, Doors, Charning port
C. Reading light, Safety Instructions, A Amres
Temperature, Lights
E. Reclnning feature, Storage bin, Table
Figure 30: An example of QA in CG-Bench for Entity General Reasoning.
24

---

Published as a conference paper at ICLR 2025
video placed use?
table
B. In the sink
On shelf
D. Nexct the microwave
On counter
Figure 31: An example of QA in CG-Bench for Entity Spatial Relationship.
pama
User:
Which ties emerged first game?
4:4
B. 6:6, 9:9
4:4, 6:6
D. 5:5, 8:8
5:5, 9:9
Figure 32: An example of QA in CG-Bench for Event General Reasoning.
what third- last protagonist perfors before cleaning the first squid?
Removing
Squid
B. Palling Out the Intemall Organs
Detaching Squid's Tentacles
D. Cutting Open the Squid's Head
E. Removing the Squid Eyes
Figure 33: An example of QA in CG-Bench for Event Time Relationship.
Usar: Why did step the protagonist ugust decrsese video?
Because the protagonist injured and needed
home
B. Because the protagonist joined a new gym and focused on indoor workouts
protagonist decided dedicate
reding books
D. Because the protagunist quit his job and made Y YouTube videos home
Because protagonist took vacation traveled
Figure 34: An example of QA in CG-Bench for Event Causal Reasoning.
25

---

Published as a conference paper at ICLR 2025
Use:: During the ride, what terrain does protagonist the video experience and in
what order?
Snow Mountain- Forest- Grassland
B. Snow Mountain Lake Crassladd
Snow Mountain Grassland Forest
D. Snow Mountain- Crussland- Lake
Snow Mountain Forest Cliff
Figure 35: example of QA in CG-Bench for Scene Time Relationship.
User: In the video, if you buy 100 wood-grain tiles, how much will you have pay price from last, year?
A. 150 yuan
yuan
C. 480
yuan
E. 840
Figure 36: An example of QA in CG-Bench for Text General Reasoning.
Usem: In the video, when the protagoniist was getting massaige, what was writhon in red the bottom row on the blue sign next to
A. Le Minerale
B. Le Mineral
Le Mimable
Le Minale
F. la Minerale
Figure 37: An example QA CG-Bench for Text Spatial Relationship.
Use:: In the video. protagemist was holding mushroom in band. How long did It take for him to pick the fourth mushroom?
About
B. A bout three minute
D. bout four minute
bout one
Figure 38: An example of QA in CG-Bench for Time Interval Reasoning.
26

---

Published as a conference paper at ICLR 2025
Use:: In the video, the prolgeenist captured the some fireworks. How did
scene firewors last in total?
hout 52 seconds
B. A bout 65 seconds
C.uu seconds
D. D.A bout 30 seconds
E. bout seconds
Figure 39: An example of QA in CG-Bench for Duration Time Reasoning.
User: video, After humans fed parrots potato chips, the following statements are incorrect 1. There are parrots the ground T There are 2 wild sunflower partots the ground There are crimson rosella on the ground There are 2 wild sunflower parrots fighting 4.A crmsso mestll flew away. S.A wild sunflower parrot flew away.
A. and
B. and
1 and
2 and
E. and
Figure 40: Example 1 CG-Bench for Hallucination.
language the game, which following options are correct? 1. Thai une of the languages selected by the protagonist 2. Bulganan is one languages selected Romaniaa is one of the languages selected by the protagonist Pinnish was selected by the protagonist first then cancelled 5. Danish was selected by the protagonist first and then cancelled. A. and B. and
protagonist
prolagonist 3.
C. and
1, and
E. 2 and
2,
Figure 41: Example 2 in CG-Bench for Hallucination.
27

---

Published as a conference paper at ICLR 2025
<subtitle>
[start, end]
itlb (Optional)
Frame Time Prompt (Denoted as <FT>)
total of <n> frames uniformly sampled from the video, and their corresponding timestamps are <frame_timel>, <frame_time2>,
<frame_timer>
Choices Prompt (Denoted as <Choices>>
A. ChoiceA
B. ChoiceB
E/H. ChoiceE/Choicet (5"8 choices)
B.2 INFERENCE РROMPTS
Long- V & Clue-based-MCQ
Task description:
You will watch video and read miltiple-choice question based the video content. You need to choose an answer that best matches the video content from five to eight options.
<Framel>, <Frame2>, ... <Framen> <Sub> (Optional) <PT> (Optional)
Multiple-choice question:
<Question>
<Choices>
Important:
You only output the uppercase letter corresponding the correct answer.
Do not include any additional text, punctuation, or explanations in your response.
Your output is:
Blind-MCQ
Task description:
You will be read a mittiple-hhoiee question related to visual task. However, no visual context or information will be given. Please do your best to answer the question based solely on the textual information. Choose the most likely answer from the given options, even if the question appears to require visual input.
Multiple-choice question:
<Question> <Choices>
Important:
You only output the uppercase letter corresponding the correct answer.
Do not include any additional text, punctuation, or explanations in your response.
28

---

Published as a conference paper at ICLR 2025
Your output is:
Question-Clue Grounding
Task descripe inscription:
You will watch a video and read a miltiple-choic question based on the video contert. You need to output each clue interval that can answer this question in nested list format.
<Framel>, <Framc2>, <Sub> (Optional) <FT> (Optional)
<Framon>
Multiple-choice question:
<Question>
<Choices>
Important:
The output must strictly follow the format: [[startl, endl], [start2, end2], ...]
where start and end are the timestamps in seconds. Any output that does not conform to this nested array format will be considered incorrect.
Your output is:
Open-Ended QA
Task description:
You will watch a video and read a question based on the video content. Please answer this question directly based on the frames sampled from the video.
<Franel>, <Frame2>,
<Framen>
<Sub> (Option)
<PT> (Option)
Question:
<Question>
Important:
You must provide answer, If explicit clues are lacking, make inference. your best based on the given frames.
Failure to provide an inferred answer will be considered incorrect.
Your output is:
B.3 EVALUATION PROMPTS
Heuristic Evaluation Method for Open-ended QA: Step
Task Description:
You are judge. You will read question, a model's prediction. and the ground truth answer to this question. You need to judge whether the model's prediction is correct. In most cases, this juagment can made by determining whether the meaning of the two texts is consistent. That is, it the meaning of the model's prediction consistent with the meaning of the ground truth answer, the prediction
29

---

Published as a conference paper at ICLR 2025
is considered correct; otherwise, it is considered incorrect. However, there are sone special cases among the incorrect ones, where inconsistencies may juat focus on different details of the same visual scene and don't t have fundamental differences, In this case, the problem cannot be judged only by text, and additional visual information needs be introduced.
Therefore, I hope you:
Output "yes" it the meaning of the two texts of the model's prediction and the ground truth answer is consistent.
Output "no" the model's prediction and the ground truth answer are not consistent, and their meanings are fundamentally different. Output "need visual clue" if the model's prediction and the ground truth answer are not consistent but the model's prediction does not appear to be fundamentally different from the ground truth answer. is possible that the two focus on different details of the same visual scene. Visual information is needed for further judgment. You are required to give an explanation as to why they might focus different detalls.
Question: <Question>
The ground truth is: "<Answer>" The model's s prediction is: "Preediction""
Important:
The "model's prediction" has already been made based visual information. So "need visual clue" means that you need visual information to make the next judgment, not that the model needs it. The "ground truth answer" is annotated by a human, so it is ABSOLUTELY RIGHT.
Therefore, for relatively simple problems such as counting, if the model's predi ction is different from the ground truth, just output "no" directly and don't need additional visual information. The only fffrrnncce between the "ground truth anawer" and the
"model's s predi cti that requires further judgment based visual information maybe the different details of the same visual scene they focus on.
Your output is:
Heuristic Evaluation Method for Open-ended QA: Step
Task description:
You judge. You will the sampling frames of the clue intervals this question. You need
question, model's preaiction, and
to determine whether the model answered the question correctly based on the visual information. I hope you:
Output "yes", if the model's prediction correctly.
this question
Output "no", if the model's prediction doesn't t answor this question correctly.
Question:
<Question>
The model's prediction is: "<Frediction""
<Franel>, <Frame2>, <Framen> <Sub> (Option)
30

---

Published as a conference paper at ICLR 2025
<ET> (Option)
Your output is:
Pure Text Evaluation Method for Open-ended QA
Task description:
You are a judge. You will read question, a model's s prediction and the ground truth answez to this question. You need to determine whether the model answered the question correctly.
hope you:
Output "yes", if the model's prediction answers this question correctly.
Output "no", if the model's prediction doesn't answer this question correctly.
Question:
<Question>
The ground truth answer is: "<Answer>"
The model's s prediction is: "sPrediction>" Your output is:
Full Vision-aided Evaluation Method for Open-ended QA: With Ground Truth Answer
Task descript interpption: on:
You judge. You will read question, model's preciction, the ground truth answer to this question, and the sampling frames of the clue intervals of this question. You need to judge whether the model has answered the question correctly based on the sampling frames of the clue intervals.
<Framel>, <Frame2>, <Framen>
<Sub> (Option) <FT> (Option)
Question:
<Question>
The ground truth answer is: "<Answer>"
The model's prediction is: "sPrediction>" Your output is:
Full Vision-aided Evaluation Method for Open-ended QA: Without Ground Truth Answer
Task description:
You are judge. You will read question, model's prediction, and the sampling frames of the clue intervals of this question. You need judge whother the model has answered the question correctly based on the sampling frames of the clue intervals.
<Framel>, <Frame2>,
<Framen>
<Sub> (Option) <FT> (Option)
Question: <Question>
The model's prediction is: "<Predict i Your output is:
31

---

Published as a conference paper at ICLR 2025
C VIDEO
C.I
VIDEO COLLECTION
Our vidcos primarily come from Bilibiii and YouTube. We initially constructed broad Level-1 and Level-2 video tags and used these tags for manual searches. During this process, we expanded the Level-2 tags and annotated Level-3 tags. the manual filtering process, applied the following criteria:
Videos must exhibit sufficient dynamism.
2. For nowweddge-rlltted videos, retained those with some visual dynamism and excluded those that were purely sreech-based,
3. We prioritized selecting the most recently uploaded videos to ensure they are as held-out as possible.
4. For each Level-3 tag, we retained only 1-2 vidcos.
5. We developed checking program to ensure that the selected video IDs do not overlap with those in major existing video datasets, including COIN (Tang et al. 2019). YouCook2 (Zhou al.
2018), ActivityNet (Heilbron al. 2015). HACS (Zhao al. 2019), CinePile (Rawal al. (2024)
CrossTask (Zhukov al. 2019), FineCiym (Shao al. 2020aa Fine (Farré et [2024), , HD-
VII.A-100M (Sun ct al. 2022). HiREST (7ala ct al. ). NowTo100M (Miech ct al. (219)) Intern-
Vid [Wang et 2023), Kinetics (Kay 2017). Mira Data (Ju et al.
2024).
V VidIM (Nan
et
2024).
Panda70M (Chen al.
2024d). QueryD (Oncescu al.
2021). QVHlighlght (Lei
et
2021 Shot2Story (aan et al.
2023). Sports (Tran et al.] 2019). TAPOS (Shao al.
2020b),
UVO (Wang ct. (021), VALOR (ALOR al. 2023b), VAST (Chen at 2023e). VidChap-
ters
Yang ct al. [2024b), VITT (Huang ct al. 20200a). Vript (Yang ct al. al.| 2022cc). YubTubeHL (Sun)
ct al. 2014), YT-Temporal-1B IB (Zellers al. 2022). MultiHateClipp (Wang et al. 2024a), and Chi-
naOpen (Chen
2023a).
By this means, approximately 20M video IDs were excluded to ensure that our video data are held out to the largest extent.
C.2 VIDEO TAGS
We collected 1219 videos on the two platforms, of which 570 videos were collected on YouTube, accounting for 46.8%; and 649 videos collected Bilibli, accounting for 53.2%. 50.12% of the videos have subtitles. In addition, we assigned level-2 or level-3 tag to video, of which there are level-2 tags and 638 level-3 tags. The specific categories and quantities of tag 2 and tag-3 are shown in Tables and 9
Vilra
Lif Racknt
197
Figure 44: Distribution of video langauge.
Figure 42: Distribution of video root categories, displaying the number of videos within each category.
Figure 43: resolusion.
Distribution of video
C.2.1 TAG-1
The categorics and quantitics of Tag-1 (root categories) are shown in Figure
C.2.2
TAG-2
The specific categories and quantities of Tag-2 2 are shown in Tables
32

---

Published as a conference paper at ICLR 2025
Table 8: Categories and counts of the level-2 video tags.
Category
Category
Category
Category
Category
Knowlolge shariing
Variety shows
Fins: person work
40
Forest
3
Traditional sports
41
Boand gumes
Travel
34
Extreme
Simulation games
Russian
Movies/TV domas
Software
28
Racing games
Wildlide
24
MOBA games
Social games
demonstraion
Festivlss
Driver's license
Documentay
Waterside
Coding
22
InDesign
2
Hunor/oomedy
20
Designhmilder
Learning
21
Working
18
Illustnir
Makeup
ZBrah
Eating
games
Diginal peoduct reviews
Shopping sharing
Pers
14
Reality challlage
Publicc safety
13
Kariing
Finess
Cooking
12
Excavator
Housekesping services
12
Social news
Animation
Stratogy games
Helicepter
Handioralt
Panay videos
10
leficiency tool software
Shopping
10
Ruins
Underwater
Music
Aschitecturre
Political
Humonities
Fashion
Insects
Dunce
First person aegunented reality experience
Technology
Business news
School
Chemistry
Open world games
NuntefieldCouuneer
lemorrim
Shooting games
competition
Medical came
Human unimal relationship
Art
7
Auction
Stage perfommance
strategy gomes
Firsi-person live-action CS
Rosed games
Note-taking software
Desen
Raccoon
Clothing
Primates
Test drive
Firss-pesson sports
Barrefield
Packing
sum
Fiss--pesoon cooking
Aquatic animals
Installation
Storage
Robuis
Cave
Trucks
Texas Hold'en
Gmmplic design
Laboratory
Trin
softeare
Cars
Game: Cities Skylines
Fisst drivang
Driver's license
Space
Knowlodge management software
Photogmaphy
culitare
Compeehhnssive
convention
Suiling Airplane
Religion
Health and wellnoss
First-penon adventure
Motoneycle mainteninnee
Repair
photography
Selection
Canyening
Video editing
Cycling
Asimation and image generation softesee
Fist-person homewoek
Street interviews
wollware
Rconemic news
Finst pensen wook: Coffee
Ensertaimment news
Driving
Diet and wellness
Reseue
First pessi-peenn reality experience
Envirommental
Sports gumes
production sofware
relief
Jade carving
Hiss gensen work: Burger shop
Detective
Military news
Drawing techniques
International news
Polar
First-peroon games
C.2.3
TAG-3
The specific categories and quantitics of Tag-3 are shown Tables g
Table 9: Categories and counts of the level-3 video tags.
Category
Category
Category
Light Cuisines
16
Photogmpby Tips
2
Cat
Python
TV Series
5
Chinese Postries
Rail Survival
2
Short Film
Psychology
Merchandise
5
Tea
Opera
Drama
Flectrc Vehiele
MasteeChef
Pottery
Rood Explorarion
Raskerball
33

---

Published as a conference paper at ICLR 2025
Category
Categury
Categury
Category
Cutegury
Cleaning
Action Film
Football
MaiLah
Ballet Journal
Skelch
The Amazing Race
Motongycle
Distory and Calture: Museum
Parenting
3
Shoppung
Desective Chinatowa
Public Service Short
Launch
Keep Running
Food Delivery
Unity
Triwan Trivel
Prison Deeamentury
Dog
Rescue and Disaster
Kung
Monopoly
Golf
Tennis
Organication Tips
Pandemic Response
Groding Hlomwwokk
Human Animal
Lide and Seek
3
Symbiosis
Challenge
Long
Bake Oif
Comedy
Shark
SkeschU
Stable Diffusion
3
Prep Tips
Dumplings
Winenskinn
Driving
Phosoehop
3
Gua Sha
Ecunomy
Candboard
Japan
Korca Shopping
Japan
Divas Hit
Face Painting
Gourmet Food
Special Ellects Makeup
Cream Cake
Everyday Makeup
Campus Life
Giddutionn
Hlology/Chemissty
'Tap
Experiments
Nunsing Procedures
2
Biology Expetiment
Escape Room
Special Forces
Underwater Esploration
2
Training
Kacing
Surting
Climbing
Wingsuit Hlynn
Paragliding
2
Foundation Makeup
Gymnustics
Cake
DOTA2
2
Civlization
Subway
Plants Zombies
Pop-up
Energy Vehicle
Novice Highway
Handmade Soup
CSGO
2
Milk Tea Shop
GIAS
2
Driving
Solo Dising
Cheesocuke
Night
Esperience
Housework
Puff Pistry
Wodk Life
Annial Competttonn
Craft Making
Music MV
Belly Danne
Symphony Orchestra
Thmma Cere
Castle
Underwater Salvage
Pyramid
Skiing
Eyebroo Drawing
Baseball
Skating
Coumia-Teroriem
Sulway Operations
Action
Man's Sky
Sushi
Stadde Valley
Nail
Supermarket
Restocking
Amusement Park
2
Meal Prop
Family Feast
Underwater Fishing
Procuneentt
2
Magic
Underwater Welding
Music Festival
Street Dance
Going,
Cave
2
Rabbit
Freediving
2
Biology
Cooplay Makeup
2
Valvel Flowers
Festval
Madicine
Car
Cultural District
Truck Driver's Duily Life
Healthy Living Habits
Restaumant Waiter
Mountaia
Badumajaa
Trash
Elephant
Behind the Scenes
Latin Dance
Lion
Medical Equipment
Meerial
Callege Entrance
Use
Exam
Solvice
Rochninton
Mallnerrmeen
Long-Distance Runsing
Fitness Plan
2
Makeup Removal
Truth or Dare
Koreun Makeup
Leather Craft
2
Hanfu
Shoe Making
Red Alait
Freetancer
Cooking
Shopping
Mountain Biking
Theme Park
Red Pands
Librarian
Concert
Brows Beir
Earthquake Dvill
2
Walf
Snowmohile
2
Oolong
Embroidery
Paper Cutting
Cuisine
Archaeology
Luxury Car Text
Collage
Hearthstone
Vinity
Negetirianism
2
Drive
Microdilm
Mushroom Picking
Streei Dunce
Arab Robe
Emergency Evacuation
2
Beoding
Space
Bescheumbing
Skatebounding
Diving
Fishing
Truck
Duck House
Skyline
2
Ocean Park
Violin
Rebahiltutiun Training
Dungeun
Real Battkfield
2
Water Splasting
Polur Animals
Minseraft
Traditional Chinese
Cloud Notes
Festival
Medicine
GoodNotes
Dlorioon
Shopping
Delivery
Antique
Shoyping
34

---

Published as a conference paper at ICLR 2025
Category
Categury
Categury
Category
Cutegury
Volleyhall
Canvasience
Boad
Board Game: Who
Sculyture
Bix
2
Bourd Game: Storgteling
Valorant
Making Small Books
Notion
2
City
Eyebeow Shaping
Supethero Movies
Repair
Fried Chickon
Caneraler
7asero
Eaprop
Duty-Free Tuiy-FieeSShhpping
Waterside
Tskoyaki
CPR
Creative Market
Fighting
Beschcombing
Temple of lleaven
2
Variety Show
National Day
Beund Game:
Halloneen
2
Redemption Journyy
Dragon Festival
Tacit Challenge
Acupuncture
Supemarket Challeage
Temples
Go Karting
Elephants Wild
Yacht
Airplane
World of Warcraft
Digital Product Review
Obsidian
Composer
Furnitur Assembly
Digicl Product
Digitl Painting
Shopping in Europe
Digial Product Review:
Home
Abandoned
Digital Product
Fir Loss Training
2
Chocolate Making
Ab Wockout
Baildings
Review: Ergonomic
Chuir
Hockey
2
DIY Mini House
Spring Festival
2
Wasenside Life: Fishing
Easter
2
Digital Product
Delivery
Duwing Techoiques
Review: Smartphone
High-Altien Nigh-Alttadde Woek
Braised Pock
Farming
Pish Pond
Shopping
Museum
2
Italy
Fiex Marker
Happy Old Friends
Art Gallery
Ace
Wilderness Survival
Am Singer
Medieval Dynasty
Firefighting
Milltary Exercise
Camping
Dumbbell Training
2
Aircrah Loading
Bowling
Reali life CS
Finness Ball Training
2
Ialian Chisine
Repair
Jepanesa Chissin
Job
Flden
Water Obstacle
Ergonomic Chair
Markdown
Basemen:
Word
Course
CapCul
Glacier Climhing
Ruby
Pallerfish
VSCade
Blender
Jade Carving
Australian Travel
Ancient Greek Philosophy
Baking Techniques
Wollligg
Train
Drowning
Theory Relativity
Rains Exploration
Simulatoe
Archery
Used Cars
Colosseum
2
Thiwan Shopping
Thanksgiving
2
Autonomous Driving Experience
Painting
Fishing
Duty
Adobe Acrobat Pro
Farm
Summer Outfits
2
Duily Life After
Southeast Asia Travel
2
Retumiing Home
Camping
Home Tour
Disney
Village School
Massage Thempy
2
The Tonight Show Jumay
Desert
Fire Dvill
2
Parkour
Fire Eescustion
Fallon
Oipso
Buddhissa
French Cuisine
Great Wall
Helicopter
Mance Lond
Subway
Fallont Shelter
Mixol Noodles
Mower
PPT
2
Eрony Resin
SQL
Kniting
Spring Outfis
2
Scafoed Ruffer
Paris
Studio
Yoga
North American
Helicopter Skiing
Calligraph
Quii Festival
2
Thriler
Spanish Cuisine
2
Cuisine
Real
in7Ot
Chineso Paiming
Vivion
Terrorism
Mailing Packaggng
Making Hot
Lugsage
LaTeX
Steam
2
Digital Product Review: Flectric
Family Feud
3
Mythical Funtasy
Thai Cuisine
2
Film
Tocchbeush
Christinity
Strange House
Kingdom of Cirder
Mabjoug
Plaats vs. Zombies Hybrid
2
Sunny
Café
Grounded
Kimano (Japan)
Coffee
Cleaning
Quicker
Editing Tipe: Movie Commenary Editin
Hunting
Depurtment Shopping
Chicago
[ome Gardening
Simulator
35

---

Published as a conference paper at ICLR 2025
Table 10: Impact of different prompts and modalities the full test set. Each prompt be composed of
frames (F), frame timestamps (FT), subtitles (S), subtitle timestamps (ST), and audio (A), We conduct the main
experiments with GPT40-0806 (OpenAl 2024) while studying the audio modality with Gemini- 1.5 Pro Anii
ct al. 2023
model
prompt & modality
clue-acc.
long-acc.
mloU
Acc@Iot Toll
CRR
OF-ace.
GPT4o
S (128 frames)
28.9
GPT4n
S (full-video)
31.2
GPT4o
F
66.0
52.4
3.41
10.2
79.4
35.8
GPT4e
F+IT
65. (-.99)
(2.2(40.2)
6.10(+2.09)
20.6;-10.4)
80.2
(6.5
GPT4o
66.1(+0.11
53.4(+1.2)
3.54
11.0(-0.8
80.8(+1.4)
(7.2(+1.4)
GPT4o
F+S+ST
66.3
52.4(40.0)
4.63(411.22)
16.3 (-.3
78.8; (8.8
36.8
GPT4o
F4S+FT
(
52.2
6.45;
21.3 11.33
78.5; 0.93
36.9 (1.1)
GPT4o
F+S+ST-FT
(6.5))
53. (39.4).))
8.33 (-3.9)
21.7 (-1.8-)
81.(++1.9)
(7.2
FTT
Gemini
F
61.2(+002)
(401)
7,56
18.6
(0.5
18.9 (40.9)
Category
Category
Category
Category Covering
Category
Trailers
Snow Monstain Adventure
Equestrian
Organic Chemistry
Dexert Off Off-Roading
Street Food
Ponoelain
2
Driving
Drawing Tips:
Dhawing
Cloching
Ieeland
Dishwasshngg
Americal's Gox Talens
Olympics
Rughy
Jonmey the
Sculpture
Days Die
West
Barieder
Rafling
Radiomics
Battlefield
European Travel
2
Delivery
Hikking
Ping Pong
Christmas
2
Teu Set
Col and Mouse Game
Thailand
Frosupunk
2
Myth:
Design
Fiest-Person Cooking
Hengdian
PC Ruilding
Wilknng
Rainforest Survival
Who's the Undercover
[lgh Insensity Training
Real Life lide and
The Sinking Land
C.3
VIDEO STATISTICS
We provide overview of the dataset's characteristics through two statistical visualizations Figure which demonstrate the distribution of video resolutions, and languages.
+ and
Figure 44
Figure illustrates the distribution video resolutions. The majority of videos (1,065) have resolution between 720p and 1080p, while 120 vidcos are exactly 720p. Only 34 vidcos have resolution below 720p.
Figure 44 shows the distribution video languages using a logarithmic scale. The most frequent languages are Chinese (730 videos) and English (432 videos). Additionally, 38 videos have no speech. Other languages such German, Korean, and Japanese are also represented but in smaller quantitics.
D ADDITIONAL EXPERIMENTS
We further report the ablation studies of different prompts and modalities the full subset in Figure I0
36


Reviews:
[{"review_id": "ZRL7nE8SA7", "reviewer": "Reviewer_3P2q", "summary": "This paper presents a newly collected dataset that is intended for evaluating long-video (10min+) understanding capability of current models. The key is to introduce clue-grounded questions as question-answer-clue triplet, to minimize the possibility of models \"cheating\" by combining short-video understanding and elimination to \"answer correctly\" without genuinely understand long videos. The dataset contains 1219 manually selected videos and 12K QAC triplets with fine-grained category annotations. The authors also designed two clue-based metrics as complementatry to the multiple-choice accuracy metric, to better assess whether model understands the video before they can answer correctly. The authors evaluated the proposed dataset with multiple open-sourced and commercial LLMs and showed that existing LLMs still have much room for improvement on this novel dataset.", "questions": "* See my comments about weakness for questions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Originality \n\nFor long video understanding, the lack of appropriate benchmark dataset has been a main problem for properly evaluating modeling progress, and actually many concurrent work is aiming for filling this gap. This paper presents a novel dataset with high quality (human annotation) and novel setup (clue-guided), essentially a finer-grained segment-level annotation, which enabled more possibility such as the finer-grained evaluation (the two clue-based evaluation metrics). The idea is not entirely new but the dataset can be highly impactful for the community.\n\n* Quality\n\nThe dataset is curated in mutiple rounds with a lot of human efforts, and the authors have done extensive experiments with modern MLLM models as well as human evaluations to prove its usefulness. \n\n* Clarity\n\nThe paper is well-written and is rather easy to follow.\n\n* Significance\n\nThe proposed benchmark dataset could be a useful addition to the community.", "weaknesses": "* In L151 the authors mention that videos are manually collected from Internet to \"avoid using videos that have been used for pre-training\", but no details is provided. Actually I'm curious how this can be guaranteed. Is it done by license? The human-annotated QAC might be novel but I feel it's hard to ensure that Internet videos are excluded from LLM pre-training.\n\n* I fully understand the difficulty (e.g. high requirement for hardware) to evaluate long videos especially some long-tailed videos (longer than 60min), but using 16/32 or even 128 frames probably is far from enough. One possible mitigation is to perform human eval under the same constraints, e.g. only feeding human with 16/32 frames and ask them to answer the question. This probably can give a more informative sense about how challenging the dataset could be. Also it would be great to have finer-grained analysis, e.g. grouping video/questions based on the duration (10min, 10-20min, etc.) and show how severe the \"undersampling\" issue is for longer video. Actually this could be a quite important missing ablation.\n\n* The two proposed metrics (white-box and black-box evaluations) are intuitive but really simple; it feels more like some analysis to provide insights but not as novel, e.g. the white-box evaluation is clearly an extension from video temporal grounding. Also even though the authors claim the acc@IoU to be a combined metric, but since $\\tau=0$ it's effectively degraded to MCQ accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mx8E0Z2BD3", "reviewer": "Reviewer_6Ngt", "summary": "The paper introduces CG-Bench, a benchmark for assessing multimodal large language models on long video understanding, moving beyond traditional multiple-choice questions (MCQs). CG-Bench emphasizes “clue-grounded” question answering by focusing on models' ability to retrieve relevant video segments that support accurate question answering. The dataset includes 1219 long videos annotated with over 12K Q&A pairs spanning categories like perception, reasoning, and hallucination. CG-Bench not only support standard evaluation metrics such as accuracy, but also incorporates two novel metrics for credibility to measure the reliability of model predictions: (1) white-box evaluation, where models need to identify specific intervals related to questions, and (2) black-box evaluation, which measures the discrepancy in model performance when with access to the full video compared to solely relying on the clue interval.", "questions": "- What is the distribution of the resolution of the raw videos used in this dataset?\n- Can the authors provide more details about the video collection process? How to prevent the videos from overlapping with the pre-training data of existing MLLMs, especially if they are collected from the internet? The reviewer would imagine this as a difficult process especially with closed source models.\n- As subtitles are provided with the videos, are all subtitles human uploaded or machine transcribed? Are all subtitles in English? \n- The reviewer is also curious about how the performance of both closed-source and open-source MLLMs can be affected by (1) the frame sampling strategy (e.g., uniform vs. key frames) and (2) the frame resolution (as it also affects how many frames can be fit in memory for open-source MLLMs given the computational limit). \n- For Black-Box Evaluation, the authors claim that \"a model with access to the full video should yield higher accuracy compared to solely relying on the clue interval.\". The reviewer find it a little counter-intuitive, though the full video may contain more information, however, it is a harder task (even for human) to search for the clue first then answer the question, than answer the question with clue provided. \n- How to think of the white-box credibility measures especially when the meta information such as timestamps is not available to the model?  For example, evaluation setting with only frames as inputs (F in Table 4), the accuracy is on par or slightly higher than the setting with both frames and frame time prompt (F+FT in Table 4), while the scores for white-box evaluation are significantly lower. \n- Regarding the quality control process, especially for the \"small model & sparse frame check\" step, can the authors provide more insights in terms of what kind of questions are being filtered?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Originality:  The paper presents CG-Bench, a new benchmark specifically designed for clue-grounded question answering in long video understanding, which fills a notable gap in multimodal large language model (MLLM) evaluation on video understanding. In addition, the paper also proposed two novel metrics for measure reliability of model predictions specifically to the clue-grounded QA scenario in CG-Bench. \n- Quality: The quality of this work is reflected in the construction of the dataset, with 1,219 long videos, over 12,000 Q&A pairs, and a detailed three-tier tagging system, ensuring comprehensive coverage across categories like perception, reasoning, and hallucination. \n- Clarity: The paper is easy to follow.\n- Significance: The proposed benchmark has great potential for MLLM research, as it exposes the limitations of current models in long video comprehension and sets a new standard for assessing MLLMs in real-world, temporally extended contexts.", "weaknesses": "The paper can be improved by incorporating more details regarding the construction of the benchmark, and more thorough evaluation of existing models to provide more insights on how and why existing models fail to solve CG-Bench, to inspire future works. Please refer to the Questions section for detailed concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yd6ys4mzHC", "reviewer": "Reviewer_eKZm", "summary": "This paper propose a new video language question-answering evaluation benchmark, featuring that each QA pair is grounded into a time interval. The QA and the time interval (as clues for the question) are manually annotated by human raters which are useful to answer the question. Having the correct time interval prediction ensures the MLLM finds the correct evidence to answer the question. The authors propose a various set of evaluation metrics, including overall QA accuracy, QA accuracy given ground truth time intervals, time intervals prediction IoUs, and GPT-guided open-ended evaluation. The authors evaluated a wide range of existing MLLMs and observed performance gaps between the models.", "questions": "Overall this paper puts good effort in an important direction, with solid dataset designs and experiments. I have some concerns on the mIoU evaluation as discussed in paper weakness, and expect the authors to reply them. My current rating is a week accept, and I am happy to further raise the ratting in my concerns are addressed in the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of evaluating multi-modal large language model is important, and I believe this paper provided a unique perspective of evaluating the temporal grounding for video question answering.\n\n- The authors provided a comprehensive set of evaluation metrics which all make sense to me. The clue-acc, long-acc, and CRR metrics smartly shows how the model leverage temporal information without explicitly look at temporal grounding prediction. I also like how the authors use GPT-4o as a guide with optional visual inputs to save evaluation budgets.\n\n- The authors evaluated a wide range of existing MLLMs (Table 3) and ablated several interesting settings (Table 4 and Figure 7), with interesting insights (e.g., how the models take use of more frames, how additional modality helps).\n\n- The fact that Audio does not help much of the dataset is a good surprise to me, where I was under the impression that many existing long video datasets are heavily ASR-biased.\n\n- I like the fact that the authors collect videos and questions all from scratch rather than reusing existing datasets or create by using LLM. This is important to reduce biases in existing datasets.", "weaknesses": "- The mIoU scores in Table 3 for most models are low (<9%), and even the human performance is only 35.5%. This raises a question that how reliable are these temporal interval annotations are. At a high level, a model does not need to look at all frames within the interval to gather the necessary information, but only a few frame or a much smaller interval is sufficient. Therefore metrics like recall may help here (this is optional for the rebuttal).\n\n- Following the above comment, it is unclear to me how good the current MLLMs are at following the specific output format instruction in supplementary Line 552 \"Question-Clue Grounding\". It will be helpful if the authors report how often the case \"Any output that does not conform to this nested array format will be considered incorrect\" happens, to get a better understanding of why mIoUs are low.\n\n- Since the authors collect the videos manually, please make sure there are no license issues of releasing the videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JBAINbo7pN", "reviewer": "Reviewer_a6oo", "summary": "The paper introduces a new benchmark dataset for long video understanding called CG-Bench. In contrast to the few existing long video benchmarks, CG-Bench includes new metrics (white box and black box) that not only measure its ability to answer multiple choice questions, but also the models’ ability to ground its answer in the video by providing time intervals containing clues to the answer. With >12k multiple choice questions, the dataset is the largest of its kind and is fully manually annotated. It includes videos from a wide variety of domains, making it suitable for predicting model performance in many real-world applications. The paper provides an evaluation of >20 models, both open and closed source, across several metrics, providing interesting insights into the state of the art in long video understanding.", "questions": "Overall I have no major concerns about this paper. I appreciate the effort the authors put into creating such a large and diverse benchmark and the contributions they are making to video understanding metrics. So I would recommend the paper for publication, but hope the authors could help answer my questions and fix the issues I found in the paper. Also a public release would be much appreciated by the community and increase the paper’s impact.\n\n* Will CG-Bench be publicly released? This would be a major contribution and should be mentioned in the text.  (Though I understand links might need to be omitted due to anonymity reasons.)\n* l. 161: Could authors describe what the “Hallucination” question type is and how its questions were designed? Some examples would be helpful too.  \n* Fig. 5: Why is the clue coverage increasing almost linearly with the time bin index? Is there a reason that clues are more likely to be located towards the end of the video than the beginning?  \n* l. 176 “To minimize expression loss, annotators use their native language during the annotation process.“ Are questions translated into English? If yes, how? If not, how many languages does the benchmark contain and what is their distribution?  \n* Do the annotators have access to audio and/or subtitles of the video? If yes, were any measures taken to make sure the answers are grounded in visual content and not only in spoken / audio content?  \n* l. 298: If \\\\tau is 0, doesn’t acc.@IoU degenerate to MCQ-acc? Is this a typo?  \n* l. 305: “In other words, the accuracy of Long-Video MCQ (long-acc.) should be greater than or equal to the accuracy of Clue-based MCQ (clue-acc.).” This contradicts Tab.3, where clue-based accuracy is always greater than long video accuracy. Also the formula for CCC assumes that long video accuracy is smaller than clue based accuracy. My guess is that this is due to context dilution, i.e. the model not being able to extract relevant information when more irrelevant information is added to the context. So, CRR is actually a measure of robustness to context dilution and not a measure of the model’s ability to seek out clues outside the provided clue windows, as the “Black box evaluation” paragraph says. I would suggest rephrasing this paragraph accordingly.  \n* l. 323: “To address this, we leverage a low-hallucination MLLM, such as GPT-4o” I’m curious why GPT-4o is called a low-hallucination MLLM. Is there any literature to back this up?  \n* Tab. 3: Why are mIoU scores \\> 1? According to Eq. 1, they should be \\<=1.  \n* Tab. 4:  \n  * I was surprised that accuracy does not vary much as modalities are added. clue-acc changes from 66.0 to 66.5 and long-acc from 52.4 to 53.9 as subtitles and timestamps are added. I would assume that subtitles would give models a clear advantage since they are a strong semantic signal. Why do they have so little effect?  \n  * It would be interesting to have an ablation that uses only subtitles and no frames.  \n* Tab. 5:  \n  * What is the unit for bias? What does a score of 1.0 mean?  \n  * How are trigger rate and recall rate defined? This explanation could be expanded in the text.  \n* Fig. 7d: GPT-4o’s CRR decreases at first and stays below other models until it catches up at 128 frames. I’m curious of authors have an explanation for this.  \n* l. 458: What is the meaning of “refusal rate” here?  \n* l. 464: “When evaluators were provided only with ground truth (GT) and visual information, the bias between human scores and model-based evaluations increased.“ Does this mean when only one of the modalities is provided?  \n* l. 465: “While fully leveraging visual information improved evaluation accuracy \\[...\\]” What does “fully leveraging visual information” refer to here? Does this refer to the “GT+Vis” columns?\n\n**Minor points:**\n\n* l. 107: “Despite the continuous proposal of various MLLMs, their real-world performance in long video understanding is still unknown.” This statement is not entirely fair since other long video benchmarks such as Video-MME have provided evaluations of these models. I would suggest toning down the statement a little.  \n* l. 192: “we align with annotators in real-time“ I am not sure I understand what this means.  \n* l. 214: What are “option probabilities”?  \n* In related work, it would help if all method/dataset names were mentioned alongside citations, e.g. Llava-Next (Zhang et al., 2024b).  \n* In tables, please right-align numerical columns for easier comparison.  \n* l. 232, use \\\\citep  \n* l. 372: Tab. 3 has many more open source models. Maybe add “among others”.  \n* Tab. 4: In row 2, long-acc column, the sign of the diff should be minus. In the following row, the diff is 1.0.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* There is a large gap between benchmark scores and human performance, showing that there is plenty of room for innovation.  \n* CG-Bench is the largest long video understanding benchmark to date.  \n* The ground truth is fully manually annotated.  \n* Good video lengths: 10-60 mins with mode around 20 mins.  \n* Good variety of video types and question types.  \n* The Evaluation of \\>20 current methods is a good reflection of the SotA in long video understanding.  \n* Clue based credibility evaluation is a great contribution to video understanding metrics.  \n* The paper is clearly structured and well written.", "weaknesses": "* The source(s) of the videos are not disclosed and the collection process is not described in enough detail. It would be interesting to know what website(s) the videos come from, what search queries were used. The paper says that videos were manually filtered, but filtering criteria were not mentioned. This information would be critical to understanding the composition and applicability of the benchmark.  \n* Evaluations on closed-source MLLMs are only performed with up to 32 / 128 frames even though ablations show that performance keeps improving as more frames are used (Fig. 7). Note: Because cost might be a prohibiting factor here, and 128 frames are enough for the provided SotA analysis, I am not considering this shortcoming in my rating. If the benchmark gets publicly released, this could easily be done by groups with sufficient resources.\n* Only three examples are provided in the text. It would be good to see a few more example questions from the benchmark to get a feeling for its quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9UzaXcQmx8", "reviewer": "Reviewer_mbFL", "summary": "This paper focuses on the evaluation of MLLMs. The authors claimed that existing VideoLLM benchmarks merely focus on either short videos or black evaluation, i.e., simply leveraging MCQ accuracies to measure the model performance without further detailed analysis. Therefore, they proposed CGBench, a novel MLLM benchmark that not only requires the models to predict correct answers for MCQs, but also ground clues through predicting temporal boundaries. The CGBench was created by manually collecting videos and annotated by human annotators, ensuring high quality. Comprehensive evaluations on such a benchmark reveals the limitations of existing MLLMs.", "questions": "Please refer to the weakness part for my concerns and questions. In general, some further discussions shall be conducted.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Overall, the motivation is clear and reasonable: providing a more detailed evaluation for MLLMs rather than simply measuring MCQ accuracies.\n2. The created benchmark seems to have good quality, as this was manually annotated by human annotators.\n3. The evaluation settings (white box + black box) are reasonable and extensive, jointly provides a good starting point to encourage future research in this direction.", "weaknesses": "I have some minor concerns regarding the writing, comparison with existing work, and experiments.\n\n1. The presentation about 'clue' is not very clear. To my understanding, this should be the temporal boundaries (represented by start and end timestamps) of the relative video segments that provide hints to correctly answer the question. If yes, it would be better to state such definition clearly in the main paper.\n2. In table 2, a number of existing benchmarks/datasets that also contain the setting of QA + grounding are missing. Further discussion on the differences between CGBench and existing QA + grounding datasets (EgoTimeQA [1], MultiHop-EgoQA [2], ReXTime [3]) and a benchmark (E.T. Bench [4]) is required. And the methods proposed by these papers should also included in table 3, as they are specially designed for such settings.\n\n[1] Grounded Question-Answering in Long Egocentric Videos. CVPR 2024\n[2] Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos. arXiv 2024\n[3] ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos. NeurIPS 2024 D&B Track\n[4] E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding. NeurIPS 2024 D&B Track", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)