LVBench: An Extreme Long Video Understanding Benchmark

## Abstract

Abstract Recent progress in multimodal large language models has markedly enhanced the understanding of short videos (typically under one minute), and several evaluation datasets have emerged accordingly. However, these advancements fall short of meeting the demands of real-world applications such as embodied intelligence for long-term decision-making, in-depth movie reviews and discussions, and live sports commentary, all of which require comprehension of long videos spanning several hours. To address this gap, we introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction.
LVBench is designed to challenge multimodal models to demonstrate long-term memory and extended comprehension capabilities. Our extensive evaluations reveal that current multimodal models still underperform on these demanding long video understanding tasks. Through LVBench, we aim to spur the development of more advanced models capable of tackling the complexities of long video comprehension.

## 1 Introduction

Recently, the rapid advancements in large language models (OpenAI, ; Anthropic, ; Du et al., ) and visual feature extraction models (Radford et al., ; Sun et al., ; Zhai et al., ) have led to significant improvements in the performance of multimodal large models on open-domain video question-answering tasks. These multimodal understanding models have also empowered various downstream tasks, such as embodied intelligence, video generation, and subtitles for the visually impaired. However, most current end-to-end video understanding models are limited to processing videos of only a few minutes in length. More complex tasks require the capability to understand much longer videos, which presents a significant challenge to existing multimodal models.

Despite numerous video understanding benchmarks being proposed in the past, the field of long video understanding remains underdeveloped due to the difficulties in data acquisition and annotation. To address this gap, we introduce LVBench, a benchmark designed to evaluate the capabilities of models in understanding long videos. We collected a substantial amount of long video data from public sources, with annotations provided through a combination of manual effort and model assistance. Additionally, we carefully designed a series of evaluation tasks. Compared to previous video understanding benchmarks (Li et al., ), LVBench offers the following unique features:

- •
We define six core capabilities for long video understanding, which can be flexibly combined to create complex and challenging questions. This multifaceted approach enables a comprehensive evaluation of a model’s ability to process and comprehend lengthy video content.
- •
We have collected a diverse range of long video data from various sources, with an average duration approximately four times longer than the longest existing dataset. The categories of videos in LVBench are illustrated in Figure 1. This extensive collection of long-form video content provides a robust foundation for testing models on extended temporal contexts.
- •
Through meticulous human annotation and multi-stage quality control processes, we ensure the high quality of our dataset, providing a reliable benchmark for assessing long video understanding capabilities.

Figure: Figure 2: Examples from LVBench. LVBench covers problems involving various temporal and spatial dimensions.
Refer to caption: extracted/5947880/figures/DatasetSample.jpg

## 2 Related Works

Multi-modal Large Language Models. Building upon the achievements in Large Language Models (LLMs), the field has shifted towards Multi-modal Large Language Models (MLLMs) to enhance multi-modal understanding and generation capabilities (Wang et al., ; Hong et al., ; Alayrac et al., ; Li et al., ; ; Liu et al., ; Xu et al., ). Early advancements in this area include models like Flamingo (Alayrac et al., ), which fused text and vision to perform exceptionally well in multimodal tasks. Subsequent models such as VideoChat (Li et al., ) and VideoChatGPT (Maaz et al., ) began exploring the video modality, using ChatGPT (Achiam et al., ) to generate video instruction-tuning data for improved instruction-following capabilities. VideoChat2 (Li et al., ) advanced the field by introducing a dedicated video encoder, requiring extensive training on large-scale datasets to optimize performance. The ST-LLM (Liu et al., ) model streamlined this process by leveraging LLMs for visual sequence modeling, thereby reducing training complexity and enhancing performance. PLLaVA (Xu et al., ) introduced a resource-efficient method for adapting image-language pre-trained models to dense video understanding through a novel feature pooling strategy, achieving state-of-the-art results. Gemini 1.5 Pro (Reid et al., ) further pushed the boundaries with a mixture-of-experts architecture, excelling in long-context reasoning and multi-modality across extensive multimodal benchmarks. These advancements underscore the significant progress and potential of MLLMs in advancing multimodal comprehension and generation. Despite the progress made, our experiments indicate that current video understanding models still fall short on tasks requiring long-range comprehension, highlighting an urgent need for the development of models tailored for long video understanding.

Benchmarks for MLLM. Recent advancements in vision-language (VL) benchmarks have largely focused on images and short videos, as seen in datasets like MMBench (Liu et al., ), SEED-Bench-2 (Li et al., ), TGIF-QA (Jang et al., ) and MVBench (Li et al., ). For long video understanding, previous benchmarks such as Perception Test (Pătrăucean et al., ) have explored multi-modal video perception and reasoning but often with shorter video clips and limited temporal context. Datasets like How2QA (Li et al., ) and ActivityNet-QA (Yu et al., ) are domain-specific and do not adequately capture the complexity of long-term video understanding. EgoSchema (Mangalam et al., ) and MovieQA (Tapaswi et al., ) provide insights into narrative and thematic understanding but are constrained by shorter video durations and limited granularity.
While LongVideoBench (Wu et al., ), MovieChat (Song et al., ), MoVQA (Zhang et al., ), and Video-MME (Fu et al., ) utilize longer videos to test models, their average duration is still limited to around 10 minutes. In contrast, LVBench features significantly longer video segments averaging 4101 seconds, pushing the boundaries of long-term video understanding with comprehensive tasks and detailed annotations.

**Table 1: Comparison of different datasets. Open-domain represents whether the video source is diversified. Multi-type represents whether the types of questions are greater than 2 categories.**
| Dataset | Num QA | Avg sec | Open-domain | Multi-type | Annotation |
| --- | --- | --- | --- | --- | --- |
| TGIF-QA (Jang et al., ) | 165,165 | 3 | $\usym{2713}$ | $\usym{2717}$ | Auto |
| MSVD-QA (Xu et al., ) | 13,157 | 10 | $\usym{2713}$ | $\usym{2717}$ | Auto |
| MSRVTT-QA (Xu et al., ) | 72,821 | 15 | $\usym{2713}$ | $\usym{2717}$ | Auto |
| MVBench (Li et al., ) | 4,000 | 16 | $\usym{2713}$ | $\usym{2713}$ | Auto |
| Perception Test (Pătrăucean et al., ) | 44,000 | 23 | \usym2717 | \usym2713 | Auto&Manual |
| NExT-QA (Xiao et al., ) | 52,044 | 44 | $\usym{2713}$ | $\usym{2717}$ | Manual |
| How2QA (Li et al., ) | 44,007 | 60 | $\usym{2713}$ | $\usym{2713}$ | Manual |
| ActivityNet-QA (Yu et al., ) | 800 | 111 | \usym2717 | \usym2717 | Manual |
| CinePile (Rawal et al., ) | 303,828 | 160 | $\usym{2717}$ | $\usym{2713}$ | Auto&Manual |
| EgoSchema (Mangalam et al., ) | 5,000 | 180 | $\usym{2717}$ | $\usym{2713}$ | Auto&Manual |
| MovieQA (Tapaswi et al., ) | 6,462 | 203 | $\usym{2717}$ | $\usym{2713}$ | Manual |
| LongVideoBench (Wu et al., ) | 6,678 | 473 | $\usym{2713}$ | $\usym{2713}$ | Manual |
| MovieChat-1K (Song et al., ) | 13,000 | 564 | $\usym{2717}$ | $\usym{2713}$ | Manual |
| MoVQA (Zhang et al., ) | 21,953 | 992 | $\usym{2717}$ | $\usym{2713}$ | Manual |
| Video-MME (Fu et al., ) | 2,700 | 1018 | $\usym{2713}$ | $\usym{2713}$ | Manual |
| LVBench(Ours) | 1,549 | 4,101 | $\usym{2713}$ | $\usym{2713}$ | Manual |

## 3 LVBench

In this chapter, we primarily discuss the construction of the original dataset for LVBench and the generation and optimization of the question-answer pairs.

### 3.1 Dataset Collection

We define a long video as having the following properties:

- •
A duration exceeding 30 minutes.
- •
Highly dynamic content with rich visual information.

To curate our dataset, we sourced publicly available videos from YouTube, covering a diverse range of topics such as sports, live streams, TV shows, documentaries, animations, and more. By using various search terms and YouTube’s auto-suggestion feature, we gathered an initial collection of 500 videos, each with a minimum duration of 30 minutes.
Subsequently, our annotators carefully screened these videos based on the following criteria to select a subset of 103 high-quality, diverse videos:

- •
The presence of one or more protagonists (possibly in a first-person perspective) who serve as narrators, appearing on-screen for a significant portion of the video and interacting with the environment.
- •
A complete video structure with a coherent logical flow.
- •
The occurrence of multiple minor events throughout the video, following a chronological order and exhibiting completeness.
- •
Visuals that are relatively easy to comprehend without overly fragmented information.
- •
Video content that can be understood independently of audio cues.

This multi-stage filtering process ensures that our dataset comprises diverse, high-quality long videos suitable for evaluating complex video understanding tasks.

### 3.2 Task Definition

To comprehensively evaluate long video understanding, we propose a benchmark testing six core capabilities. Example questions for each capability are presented in Figure 2. The proportion of each capability is shown in Figure 3. Questions are designed to flexibly combine multiple skills to construct complex queries that probe a model’s capacity to:

- 1.
Temporal Grounding(TG): Questions focus on understanding sequences and dynamics within the video, such as identifying specific events at designated times (e.g., "What happened at 29:30?").
- 2.
Summarization(Sum): Annotators are required to produce an abstractive summary that encapsulates the entire video content, demonstrating a cohesive understanding of the sequence from start to finish.
- 3.
Reasoning(Rea): This involves the application of advanced reasoning skills to interpret the video content:
•
Causal: Determining causal relationships between events.
•
Emotional: Understanding the emotional developments of characters.
•
Intentional: Interpreting the underlying intentions of characters.
•
Prospective: Making predictions about future events based on current evidence.
- 4.
Entity Recognition(ER): This capability requires the identification and continuous tracking of key entities (such as people, places, and objects) throughout the video:
•
Entity Detection: Identifying mentions of entities and resolving their identities across different instances.
•
Relation Extraction: Extracting the relationships among identified entities.
•
Action Recognition: Observing and understanding the progression of an entity’s actions over time.
•
Entity Association: Linking entities to relevant events.
- 5.
Event Understanding(EU): Comprehending overarching semantic concepts in the video:
•
Video Classification: Classifying the genre of the video (e.g., news, film).
•
Event Detection: Identifying significant occurrences within the video (such as goals scored or conflicts).
•
Scene Perception: Recognizing changes between different scenes or settings.
- 6.
Key Information Retrieval(KIR): Extracting specific, detailed information, such as text displayed in the video (e.g., "What revenue growth did the firm report at the conference?").

By composing questions that test combinations of these temporal, abstractive, reasoning, entity-centric, event-based, and detail-oriented skills, our benchmark enables robust evaluation of a model’s ability to understand long videos across multiple modalities. This multifaceted typology covers the key cognitive capabilities required to comprehend complex, open-ended video.

Figure: Figure 3: The proportion of different core capabilities.
Refer to caption: extracted/5947880/figures/categories.png

### 3.3 QA Generation

The total number of questions for each video is positively correlated with the video duration, averaging 24 questions per hour. After constructing a question, annotators are asked to provide four options, including one correct answer and three distractors.
Annotating long videos is significantly more challenging than annotating short videos or image data, making quality control a substantial challenge. To ensure the quality of the evaluation set, we encourage annotators to follow these principles:

- •
Question Diversity: Construct at least one question for each question type in a video.
- •
Specificity: Questions should refer to unique scenes, events, or characters, avoiding vague descriptions. For example, if a video contains two arguments, a well-constructed question might be: "When did A and B start arguing?" or "How did the person in red’s expression change during the hallway argument?". Less specific questions would be: "Why did they start arguing?" or "Who are the people arguing?"
- •
Temporal Coverage: Questions should cover multiple events throughout the video, avoiding repetition of a single event.
- •
Consistency: Constructed correct answers should precisely address the questions. Answers should match the content of the questions, avoiding irrelevant information. Correct and incorrect answers should be constructed consistently, avoiding obvious differences in length, form, or format.

By following these principles, we aim to create a high-quality, diverse, and challenging evaluation set for long video understanding.

### 3.4 Data Quality Control

During the annotation process, we observed that annotators had a tendency to label most questions as temporal grounding, i.e., specifying a time range to limit the referent of the question. This practice may inadvertently reduce the difficulty of the questions and unfairly disadvantage video-understanding models that lack the ability to perceive the temporal dimension. To address this issue, we instructed annotators to minimize the number of temporal questions while still ensuring the uniqueness of the referents, effectively converting temporal grounding questions into other question types.

Upon constructing all the questions, we discovered that for certain questions, a language model could generate answers without any visual input. As highlighted in MMstar (Chen et al., ), many multimodal benchmarks can be effectively solved using pure text input alone. To mitigate this issue, we employed a straightforward yet effective approach. We utilized two powerful large language models, GLM-4 (Du et al., ) and GPT-4 (Achiam et al., ), to independently generate answers for all the questions. In cases where the outputs from both models were identical and matched the ground truth answer, we removed that particular data sample from the dataset. This filtering process successfully eliminated the majority of questions that did not rely on video content for answering. Following this filtering step, we obtained a refined set of 1,549 question-answer pairs.

## 4 Experiments

In this chapter, we report the experimental results of different video understanding models on LVBench and also compare them with human performance.

### 4.1 Settings

We evaluated the performance of nine models that support multi images or short video input: TimeChat (Ren et al., ), Video-ChatGPT (Maaz et al., ), PLLaVA (Xu et al., ), LLaVA-Onevision (Li et al., ), CogVLM2-Video (Hong et al., ), LLaVA-NeXT (Zhang et al., ), GPT-4o (OpenAI, ), GLM4V-Plus (Hong et al., ) and InternVL2-40B (Chen et al., ). To adapt these models for long video inputs, we sample a fixed number of frames from the original video, such as 32 or 96 frames, to maintain consistency with the model’s training sequence length. Additionally, we assessed six models that natively support long videos: LLaMA-VID (Li et al., ), MovieChat (Song et al., ), LWM (Liu et al., ), Gemini 1.5 Pro (Reid et al., ), Kangaroo (Liu et al., ) and Qwen2-VL-72B (Wang et al., ). We processed the videos at a rate of 1 frame per second and fed them into the models, only performing downsampling when the video’s length exceeded the model’s maximum processing capability. It is worth noting that although Gemini 1.5 Pro can handle videos up to 10 hours long, its publicly available interface is limited to processing videos of up to 1 hour in length.
For each question, we provided the following prompt as input to the models:

> Question (A) Option1 (B) Option2 (C) Option3 (D) Option4. Please select the best answer from the options above and directly provide the letter representing your choice without giving any explanation.

After obtaining the model responses, we first attempted to extract the answers using regular expression matching. For questions where the matching process was unsuccessful, we employed a GLM-4 model to extract the answers from the responses.

### 4.2 Evaluation Results

#### 4.2.1 Performance across Core Capabilities

To comprehensively evaluate the performance of various long video understanding models across core capabilities, we conducted extensive experiments on the LVBench dataset, testing multiple representative models, including both non-native and native long video support models. The experimental results are presented in Table 2.

Overall, Qwen2-VL-72B achieved the best performance, outperforming other models in multiple tasks such as entity understanding (EU), temporal grounding (TG), reasoning (Rea) and summarization (Sum). Interestingly, some models that do not natively support long videos still managed to achieve competitive results compared to native long video support models. In terms of the overall score, InternVL2-40B ranked second only to Qwen2-VL-72B, and GLM4V-Plus ranked third, tying with the native long video support model Kangaroo.

However, the results of three widely used long video models, LLaMA-VID, MovieChat, and LWM, were nearly equivalent to random selection, highlighting the significant challenges that current models face when processing extremely long videos. This suggests that despite the ability to input long videos through model structure optimization, the performance and effectiveness of these models have not substantially improved.

**Table 2: LVBench evaluation results regarding each core long video understanding capability. The highest score are highlighted with green, and the second highest are highlighted with purple. All the numbers are presented in % and the full score is 100%.**
| Model | LLM | ER | EU | KIR | TG | Rea | Sum | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Non-Native Long Video Support Models |  |  |  |  |  |  |  |  |
| TimeChat (Ren et al., ) | LLaMA2-7B | 21.9 | 21.7 | 25.9 | 22.7 | 25.0 | 24.1 | 22.3 |
| Video-ChatGPT (Maaz et al., ) | Vicuna-1.5-13B | 22.9 | 22.6 | 22.7 | 25.5 | 23.4 | 24.1 | 23.1 |
| PLLaVA (Xu et al., ) | Yi-34B | 25.0 | 24.9 | 26.2 | 21.4 | 30.0 | 25.9 | 26.1 |
| LLaVA-OneVision (Li et al., ) | LLaMA3-70B | 25.0 | 26.9 | 29.2 | 30.9 | 25.4 | 31.0 | 26.9 |
| CogVLM2-Video (Hong et al., ) | LLaMA3-8B | 28.3 | 26.9 | 31.0 | 25.1 | 25.5 | 38.9 | 28.1 |
| LLaVA-NeXT (Zhang et al., ) | Yi-34B | 30.1 | 31.2 | 34.1 | 31.4 | 35.0 | 27.6 | 32.2 |
| GPT-4o (OpenAI, ) | GPT-4 | 35.9 | 30.8 | 35.5 | 28.3 | 33.5 | 34.5 | 34.7 |
| GLM4V-Plus (Hong et al., ) | GLM-4 | 39.9 | 35.8 | 34.8 | 37.7 | 40.0 | 32.8 | 38.3 |
| InternVL2-40B (Chen et al., ) | Nous-Hermes-2-Yi-34B | 37.4 | 39.7 | 43.4 | 31.4 | 42.5 | 41.4 | 39.8 |
| Native Long Video Support Models |  |  |  |  |  |  |  |  |
| MovieChat (Song et al., ) | Vicuna-7B | 21.3 | 23.1 | 25.9 | 22.3 | 24.0 | 17.2 | 22.5 |
| LLaMA-VID (Li et al., ) | Vicuna-13B | 25.4 | 21.7 | 23.4 | 26.4 | 26.5 | 17.2 | 23.9 |
| LWM (Liu et al., ) | LLaMA2-7B | 24.7 | 24.8 | 26.5 | 28.6 | 30.5 | 22.4 | 25.5 |
| Gemini 1.5 Pro (Reid et al., ) | Gemini 1.5 Pro | 32.1 | 30.9 | 39.3 | 31.8 | 27.0 | 32.8 | 33.1 |
| Kangaroo (Liu et al., ) | LLaMA3-8B | 38.6 | 37.9 | 29.6 | 35.0 | 41.3 | 36.2 | 38.3 |
| Qwen2-VL-72B (Wang et al., ) | Qwen2-72B | 38.0 | 41.1 | 38.3 | 41.4 | 46.5 | 46.6 | 41.3 |

#### 4.2.2 Answer Distribution

Figure: Figure 4: Distribution of answers generated by different models.
Refer to caption: x1.png

To understand why some native long video support models perform poorly on LVBench, we evaluated the distribution of answers generated by different models on LVBench and observed that existing long video understanding models struggle with precisely following instructions. For example, despite explicitly constraining the output in the prompt to be one of four provided answer choices, Gemini 1.5 Pro generated responses outside of the specified options 20.9% of the time, such as "None of the above options are correct" or "I cannot answer this question". This occurred even though manual validation confirmed that the questions were indeed answerable from the given choices. MovieChat and LWM exhibited a strong bias towards selecting option A, regardless of the question. In contrast, InternVL2-40B demonstrated the strongest instruction-following capability, never generating responses outside the constrained options and producing a nearly uniform distribution over the answer choices.

We hypothesize that this discrepancy arises from the relatively higher quality and diversity of image-based instruction fine-tuning data compared to video instruction data. As InternVL2-40B ingests fewer image inputs, it can more readily generalize the learned capability of precisely following instructions from the image modality to video.

### 4.3 Performance across Video Categories

We conducted a comprehensive evaluation across various video categories. We selected two state-of-the-art models, InternVL2-40B and Qwen2-VL-72B, for testing and compared their results with human performance.

As shown in Table 3, humans achieved very high accuracy across all video categories, with an average of 94.4%. In contrast, the overall performance of InternVL2-40B and Qwen2-VL-72B was relatively lower, at only 39.5% and 41.3%, respectively. This indicates that there is still a significant performance gap between current multimodal models and humans in video understanding tasks, suggesting considerable room for improvement in understanding and analyzing long video content.

**Table 3: LVBench evaluation results across different video categories.**
| Model | Sports | Documentary | Event Record | Lifestyle | TV Show | Cartoon | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Human | 96.3 | 89.8 | 87.4 | 98.4 | 97.2 | 95.8 | 94.4 |
| InternVL2-40B | 43.5 | 45.2 | 38.9 | 41.6 | 32.8 | 36.4 | 39.5 |
| Qwen2-VL-72B | 43.0 | 42.6 | 40.8 | 41.0 | 42.0 | 38.9 | 41.3 |

Further analysis of the results for each category revealed that InternVL2-40B performed best on documentary videos, reaching an accuracy of 45.2%, while performing worst on TV shows, with only 32.8%. Qwen2-VL-72B, on the other hand, achieved the highest accuracy of 43.0% on sports videos and the lowest performance of 38.9% on cartoon.

### 4.4 Impact of LLM Filtering Method

**Table 4: Ablation study on LLM filtering method.**
| Model | w/ LLM | w/o LLM |
| --- | --- | --- |
| LWM | 25.5 | 32.7 |
| LLaVA-NeXT | 32.2 | 48.9 |

Table 4 demonstrates the effectiveness of using large language models to filter question-answer pairs. Despite instructing the annotators to watch the entire video before labeling, a significant proportion of questions could still be answered correctly by inferring from the matching degree between the question and options, as well as the differences among the options. The experimental results show that after applying the LLM filter, the score of LWM decreased from 32.7% to 25.5%, while the score of LLaVA-NeXT, which employs a more powerful language model, dropped from 48.9% to 32.2%, with a decline of 16.7 percentage points. This indicates that stronger language models have a higher probability of inferring the correct answer solely from the natural language context, highlighting the importance of the data filtering step in the process.

### 4.5 Impact of Video and Clue Length

We investigated the impact of different video durations and clue durations on the experimental results. As shown in Figure 5, the performance of GLM4V-Plus, InternVL2-40B, and Qwen2-VL-72B remains relatively stable across various video length intervals, demonstrating overall strong performance.

Clue durations refers to the time span of video content needed to answer a specific question. Figure 5 illustrates that most models perform well when the clue duration is between 0-10 seconds or greater than 60 seconds. This may be attributed to the fact that questions with clues longer than 60 seconds tend to focus more on analyzing and summarizing the relationships between multiple events. These models are equipped with stronger language modeling capabilities, giving them an advantage in tackling such problems.

Figure: Figure 5: The impact of different video and clue durations.
Refer to caption: extracted/5947880/figures/video.png

## 5 Discussion

##### Conclusion.

In this paper, we introduced LVBench, a benchmark designed to advance long video understanding. LVBench comprises a diverse collection of lengthy videos and a meticulously annotated question-answer dataset, presenting a robust evaluation framework for assessing multimodal models on complex video understanding tasks. Our experiments revealed that while state-of-the-art models have made strides in short video understanding, their performance on long videos falls short of human-level accuracy.
By providing a challenging benchmark, we hope to stimulate the development of advanced models capable of tackling the complexities of extended video comprehension.

##### Limitations.

A limitation of our benchmark is the exclusion of audio data. While audio can provide valuable context, we did not include it because most current models lack effective audio processing capabilities. Future work will aim to incorporate audio information to enhance the evaluation framework for multimodal understanding.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * In Figure 5, MovieChat, which is designed to tackle long videos, performs better on short videos than long ones. The results appears counterintuitive. 

* In L184, terms such as “complete video structure” and “coherent logical flow” are not clearly defined
soundness: 2
strengths: * Serveral baseline methods are adopted and tested on the new benchmark.
* The quality control process is well designed to avoid the imbalance of question types and language bias.
summary: In this paper, the authors introduce LVBench, a new MLLM benchmark designed to evaluate performance on long-term videos. LVBench comprises 103 high-quality videos sourced from YouTube, each with an average duration exceeding one hour. These videos are manually annotated using a multiple-choice question-answer format. The question types are diverse, including grounding, key information retrieval, and complex reasoning across events. In the experimental section, several representative MLLMs that support both short and long videos are evaluated and compared using the LVBench benchmark.
weaknesses: I have serveral concerns regarding the quality of the benchmark and insufficent exploration of the bottleneck of current video MLLMs.

* Diversity of Videos: Although the benchmark includes 103 extensively long videos from YouTube, concerns remain regarding the diversity of scene categories, filming techniques, event relationships, and the variety of objects and motions depicted. adding a statistical chart to illustrate the diversity of the videos is necessary.

* Double checking. The dataset would benefit from manual cross-checking to identify any annotation errors raised by a single annotator.

* Few-Frame Bias: A key feature of this benchmark is the extended duration of videos, which leads to complex events and dense question-and-answer pairs. Unlike short-term video benchmarks that can often be addressed with a few frames (resulting in a single-frame bias), long-term benchmarks contain more extensive information. An analysis of the number of input frames and the impact of "few-frame bias" would provide insights.

* Ambiguity in Questions: I think the utimate goal of such benchmarks is to assess real-world interactions between humans and AI when engaging with long videos. However, human questions are usually ambiguous, such as “Why did they start arguing?” rather than more detailed questions like “How did the person in red’s expression change during the hallway argument?” Incorporating the ability to handle such references could be more practical.
 
* Missing important details. The details of the calculation of human scores in Table 3 is not given. How to determine the clue length in the video given a free-form question? Details are not given.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: Please see the weaknesses listed.

Minor questions:
- Clarify how the "Clue duration" ground-truth was collected.
- InternVL2-40B results: Why's there a discrepancy between Table 2 (39.8) vs. Table 3 (39.5).
- Table 3 -- why can't these numbers be provided for all method studied as opposed to just the two best methods?
soundness: 3
strengths: + The paper introduces a video understanding benchmark that emphasizes long videos, longer than prior works. 
+ The paper does a commendable job of enumerating different capabilities required for long video understanding
+ The paper evaluates a total of 15 different MLLMs, which is a reasonably comprehensive list.
summary: The paper proposes a long video understanding benchmark for multimodal large language models (MLLMs), focussing on videos over 30 minutes. The benchmark includes ~100 diverse videos with ~1550 QA paiers that test six core capabilities: temporal grounding, summarization, reasoning, entity recognition, event understanding, and key information retrieval. The paper studies several state-of-the-art MLLMs, both that focus on long videos natively and ones that don't. They key focus is on understanding the capabilities (or lack thereof) of current models in understanding lengthy video content.
weaknesses: Benchmark:
- While I appreciate the effort to create a benchmark with longer videos, as emphasized in the paper and Table 1, without enough scale, the benchmark can prove to be of limited utility. As noted in Table 1, the benchmark as 1549 QA pairs, making it the second smallest benchmark in terms of number of QA, with only the older ActivityNet-QA (Yu et al., 2019) having fewer QA pairs. It only has 103 videos, which is considered quite small by current standards. In fact, number of videos should have been a column in Table 1 to compare with existing  benchmarks, which would reveal how small the dataset is. While the videos in the proposed benchmark are longer, a large number of videos is still necessary for a credible video understanding benchmark, especially since the QA pairs in this paper are video-level rather than fixed-time-clip-level.
- Although the proposed benchmark includes 21 video subcategories, it still lacks many common and significant types of videos, such as video games, non-cartoon educational videos, travel videos/vlogs, health and fitness videos, and technology videos, etc. Not necessarily a negative, but the paper should discuss other potential categories it does not cover so folks who use the results on benchmark should understand what it works or doesn't work on.
- On a similar note, the benchmark is heavily skewed towards object and action recognition -- arguably two of the most studied tasks in recognition (Figure 3, >80% of questions come from Entity and Event recognition categories). 
- Another baseline missing is the following. Using video (or clip) caption generation methods to describe the video and then use LLMs to answer the questions using those descriptions/captions as input. 

Exposition
- The paper emphasizes several times that the questions are designed to be compositional, which can flexibly combine multiple skills to construct complex queries. However, from what I gather, all the questions belong to one of the categories listed in Section 3.2, and there's no composition of questions. It might be a language issue but compositionality of questions has a specific meaning and I'd recommend paraphrasing to not imply compositionality.
- I like the "Clue duration" analysis, however, I don't understand how they ground-truth clue duration was collected? Were annotators asked to label this or was this estimated in an automated way?

Experimentation
- The paper discusses some details in L302-4 regarding how they adapt models not designed long videos. Example, by sampling 32 or 96 frames to provide as input. Now, I understand that if the models are not designed for long videos and/or have limited context length, then it's not easy to adapt MLLMs for long videos. However, providing 32 or 96 frames for avg. ~4000 seconds, without any sampling of frames, is a futile exercise when it comes to understanding long videos. The paper should either propose ways in which to adapt models that natively do not support long videos that is more meaningful or not have so many of them and focus on models that do support longer videos.
- I like the analysis in Section 4.2.2, but I it is still unclear how much of the answer/distribution comes from actually understanding videos vs. LLMs inherent biases. It would help to use the LLMs for each model to answer the questions and provide that as lower bound for the results. I understand the authors did try filtering out questions using two LLM models, but for this to make sense, it has to be done using the LLM underlying the MLLM.
- InternVL2-40B results: Why's there a discrepancy between Table 2 (39.8) vs. Table 3 (39.5).
- Table 3 -- why can't these numbers be provided for all method studied as opposed to just the two best methods? It would help understand the video category axes of performance for all methods, and doesn't require any more experimentation.

Findings:
- While a pure benchmark and result paper is suffice, it would be good if the paper provided some discussion on potential directions to improve current MLLMs using the proposed benchmark. Another shortcoming is straight foward application of MLLMs not designed for long video understanding. I'd have preferred to see some effort to make these MLLMs work better on long video understanding.

Minor typos:
Section 3.2, all enumerated items are missing a space before the parenthesis (Grounding(TG) --> Grounding (TG)).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please discuss the video license issues.
soundness: 3
strengths: - It is a new benchmark with longer videos than other existing datasets.
summary: This is a benchmark dataset paper. The paper provides multiple choice question annotations of 30+ minute videos, designing them to be only used for the evaluation. The videos in the dataset are certainly longer than previous datasets.
weaknesses: - The annotation process is not 100% clear. The paper does not reveal the detailed process used to collect the annotations. For instance, what instructions were given to the annotators to prepare the question and the four options in multiple choice questions? Example annotations provided in the paper are also extremely limited (just one example in Figure 2 per type), making it very difficult to judge the quality of the dataset. We also do not see any supplementary material or appendix with such examples.

- It seems the dataset only has multiple choice questions.

- There is no video license consideration with the videos in the dataset. In fact, the paper does not mention licensing at all. How do the authors plan to distribute the videos in this dataset? The YouTube videos get deleted occasionally, and the benchmark accuracy will be influenced by what videos are still remaining on YouTube. This happened in previous datasets like Kinectic400 before, and this is concerning. In addition, this will restrict people from using this dataset for any model training.

- Since the paper mentions movies as some of its videos, please include other relevant datasets like Short Film Dataset (SFD).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The additional explanations regarding various sections under weaknesses
2. How would alternate evaluation schemes like likelihood based selection affect performance? This ties into Q5 above: is performance on the dataset actually correlated to long video understanding ability or instruction following ability of evaluated models?
3. Possibly related to LLM filtering idea; how would the naive baselines (suggested under weaknesses) perform on the benchmark? 

This proposed dataset / benchmark would definitely be valuable for the community to better evaluate long video understanding approaches. It would be really great if the authors could clarify / address the above points.
soundness: 3
strengths: 1. Timely and useful benchmark for long-video QnA containing diverse and long videos 
2. Paper is quite well written clearly outlining the reasoning for building this benchmark and how it differentiates from existing works. Table 1 in particular is quite useful for the latter. 
3. Several interesting analysis on dataset statistics.
summary: Authors propose a long-video QnA evaluation benchmark consisting of human annotated video question answer pairs. They evaluate multiple state-of-the-art approaches on this dataset, highlighting the difficulty of the benchmark for existing approaches. The authors clearly highlight the distinctions of this benchmark compared to prior work, focussing on video length, diversity, and question complexity. Several interesting analysis is provided on the dataset.
weaknesses: 1. L143: Consider adding hours, i.e.  … 4101 seconds, … -> … 4101 seconds ( > one hour), … 

2. **On L316 (answer matching to choices):** The authors use the following setup for selecting the correct choice with a given VLM: “After obtaining the model responses, we first attempted to extract the answers using regular expression matching. For questions where the matching process was unsuccessful, we employed a GLM-4 model to extract the answers from the responses.” 
    1. Could you try likelihood selection / cloze-prompting similar to most NLP benchmarks (see likelihood selection in [1] or cloze-prompting in [2])? This could be more efficient to evaluate (as opposed to generating tokens iteratively) and also avoid any negatives due to difficulty in answer matching. 
    2. Could you explain how the GLM-4 model is used exactly to extract answers? Are you directly prompting GLM-4 to select one answer given the outputs of the model? 
    3. If yes (to 2 above), could you also try using some text-encoder feature-space similarity between generated answer and choices (i.e. standard zero-shot setup in CLIP)? 

3. **Naive Baselines:** Can we include two naive baselines with only text and a single center frame? These may give some interesting insights about the benchmark.
    1. For only text, evaluate with a SOTA LLM (feeding no images, using only the question and choices) similar to what is done in [1, 3]. See Just LLM in [1] and blind variant in Table 4 of [3]. 
    2. For single center frame, using some SOTA VLM and feed only the single center frame (this is motivated by Single Frame VLM in [1]). 

4. Table 2: Could you repeat the abbreviation names (ER EU KIR TG Rea Sum) in the caption? This would make it easier for a reader. 

5. Is performance on the dataset actually correlated to long video understanding ability or instruction following ability of evaluated models? 
    1. This is motivated by L379-382 mentioning “MovieChat and LWM exhibited a strong bias towards selecting option A, regardless of the question. In contrast, InternVL240B demonstrated the strongest instruction-following capability, never generating responses outside the constrained options and producing a nearly uniform distribution over the answer choices.”
    2. Maybe a secondary set of results using likelihood selection (see point 2.1 above) may resolve this problem. Integrating such an evaluation scheme into the benchmark could help it better stand out from existing long-video benchmarks making evaluation both faster and more robust.  

6. Can you explain LLM filtering (Section 4.4) in more detail? If this is related to point 3, adding a full row of that to Table 2 could provide a lot more insights, strengthening the benchmark. 

7. “Clue Duration” : can you explain how this is calculated? Is this part of the dataset ground-truth provided by humans? 

8. Similar to Figure 2, maybe consider adding visual examples for each question type in the appendix. 
 
9. Also consider adding visual examples for each of the 6 major categories in the appendix. And possibly link to examples of all 21 sub-categories also in some external website. 

10. L072 mentions “Through meticulous human annotation and multi-stage quality control processes …” for dataset curation. Please explain this process in more detail. 

11. Compute, Inference Time, and other details
    1. For a reader, it will be useful to know the compute required to evaluate a model on this benchmark. Consider elaborating on compute used for evaluation and time taken. 
    2. Details on dataset size (i.e. storage required) and also any copyrights details for any videos used will be useful for those using benchmark in future.   

&nbsp;

[1] “Understanding Long Videos in One Multimodal Language Model Pass.” ArXiv abs/2403.16998

[2] “Leveraging Large Language Models for Multiple Choice Question Answering.” ICLR 2023

[3] “Memory Consolidation Enables Long-Context Video Understanding.” ArXiv abs/2402.05861