InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding

Authors: Kirolos Ataallah, Chenhui Gou, Eslam Mohamed BAKR, Khushbu Pahwa, Jian Ding, Mohamed Elhoseiny
Primary area: datasets and benchmarks
Keywords: video understanding, benchmark, long video benchmark, long video understanding

Abstract:
Understanding long videos, ranging from tens of minutes to several hours, presents unique challenges in video comprehension. Despite the increasing importance of long-form video content, existing benchmarks primarily focus on shorter clips. To address this gap, we introduce InfiniBench a comprehensive benchmark for very long video understanding which presents 1)very long video duration, averaging 52.59 minutes per video 2)The largest number of question-answer pairs, 108.2K 3) Diversity in questions that examine nine different skills and include both multiple-choice questions and open-ended questions 4) Memory questions, such as Global Appearance that require remembering and tracking the visual aspects through the video. Using InfiniBench, we comprehensively evaluate existing Large Multi-Modality Models (LMMs) on each skill, including the commercial models such as GPT-4o and Gemini 1.5 Flash and the recent open-source models. 
The evaluation shows significant challenges in our benchmark.
Our findings reveal that even leading AI models like GPT-4o and Gemini 1.5 Flash face challenges in achieving high performance in long video understanding, with average accuracies of just 56.01 % and 43.32 %, and average scores of 3.25 and 2.79 out of 5, respectively.
Qwen2-VL matches Gemini's performance in the MCQ skills but lags significantly in open-ended question tasks.
We hope this benchmark will stimulate the LMMs community towards long video and human-level understanding.

Decision: reject
Reviewer scores: 5, 3, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730010230918

thread_type:
official_review

<thread_object>
creator: reviewer_dcjq

<content>
summary:
The paper presents a benchmark, called InfiniBench, for the evaluation of long video understanding. The dataset consists of 1219 videos. The average length of the videos is 52.59 minutes. There are 108.2K (video, question) pairs. The questions are divided into 9 categories. Some categories require the ability to make associations across a longtime span. Some categories require in-depth understanding and reasoning capabilities. It is a very interesting new benchmark for long video understanding.

strengths:
The videos are very long with an average length 52 minutes.

The number of (question, answer) pairs is large (108k)

Some of the questions are unique such as spoiler questions, global appearance, and scene transitions. 

Compared to the existing benchmarks, this benchmark contains much longer videos and contains some new interesting types of questions. It'll be very useful to the researchers who work on long video understanding.

weaknesses:
The variety of TV show sources is limited since there are only 6 different TV shows.

questions:
Can the authors comment on the limited variety of TV shows? What about sports events like NBA, NFL, Tennis, etc. 

On GPT-4o evaluation, only 250 frames are selected. Are the 250 frames selected uniformly? Have you tried to reduce the frame size and squeeze more frames into GPT-4o?

Will all the videos be released to public? Are there any legal issues?

Are there text scripts (screenplay) associated with all the videos (movies and TV shows)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454807959

thread_type:
official_review

<thread_object>
creator: reviewer_ijs3

<content>
summary:
In this work, the authors propose an InfiniBench for very long video understanding. To contain local/global events and understand visual/contextual content, they define a long video understanding covering nine skills through four critical aspects in movies and tv shows.

strengths:
1 Important Topic. Long-form video understanding is a challenging but important problem. Hence, how to develop benchmark to evaluate this problem is critical.

2 Experiments. The experimental results are sufficient to support the claim of benchmark.

weaknesses:
1 Similar work has been proposed in the literature. For example,  [MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding, arXiv:2312.04817].  Please clarify the difference. 
 
2 The writing and paper organization is not good. Please refine it for easy reading.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567275033

thread_type:
official_review

<thread_object>
creator: reviewer_ozkf

<content>
summary:
The paper proposes InfiniBench, a novel benchmark for long video understanding based on movies and TV shows. The benchmark has 108.2k question-answer pairs on 1,219 videos that average 52.59 minutes in length. The benchmark tests 9 different reasoning abilities including visual, long-context and local reasoning. This makes InfiniBench the largest-scale long video understanding benchmark to date. InfiniBench was constructed by combining and augmenting from two existing video benchmarks, TVQA and MovieNet. Most question types were generated by prompting GPT-4 with the transcript of the video while a custom pipeline was used to generate questions on changes in character appearance. The paper presents benchmark results of 8 long video understanding models, including 6 open source ones and 2 commercial ones, and discusses insights into their performance across various tasks.

strengths:
* The presented benchmark has an impressive scale with 108.2k questions on 1,219 videos that average 52.59 minutes in length.  
* There are 9 different question types that test long video understanding models across a variety of skills.  
* The paper presents results of 8 long video models and draws interesting conclusions on their performance.  
* There is a large gap between human performance and model performance, suggesting the benchmark has ample room for improvement.  
* The paper has a good in-depth discussion of related work.

weaknesses:
* The question-answer pairs in the benchmark were generated fully automatically without any human intervention. This raises questions about soundness and of the questions and potential bias. A human evaluation is performed on a subset of the data, but good human performance is no proof that questions are well-formed and free of hallucinations.  
* Most questions are generated from transcripts that authors obtained online, but it is unclear what information these transcripts contain, whether they are complete and error-free. It is also unclear how much visual information the transcripts contain and therefore it is unclear to what degree this is a multimodal benchmark.  
* The use of movies and TV shows raises questions about generalizability. Most MLLMs likely know the plots of popular movies and shows because their summaries or transcripts were part of their training data. So, they may be able to answer the questions in the dataset without any context, which is not the case for most videos from the web. The effect of this is not examined.  
* It is unclear how much the benchmark relies on multimodal reasoning. Questions about movies and TV shows could often be answerable from subtitles alone, which are provided as context in the evaluation. It would be interesting to see an ablation that uses (1) No context, only the question itself (2) Only the question and subtitles (3) the question, subtitles and video frames.  
* The copyright implications of using movies and TV shows and possibly releasing the dataset are not discussed and raise ethical concerns.  
* Since the dataset has \~100 questions per video, it is likely that there are (near) duplicate questions. However there is no analysis of this and no mention of a filtering stage to remove duplicates.  
* There are several issues with the presentation such as redundant figures, tables that are not referenced, and wrong references. The limitations section also exceeds the 10-page limit.

questions:
Given the concerns listed above, I have doubts that this paper is suitable for publication at ICLR. I hope that authors can provide evidence to address my concerns as well as answers to the following questions.

* Could authors provide evidence of transcript quality? How accurate and complete are they? How much focus do they have on vision? Could authors provide examples?  
* Why are multiple-choice questions evaluated by asking the model to generate an answer and then using GPT to match this answer to the options? Authors state in the appendix that the reason is that models often do not follow the prescribed answer format, but from my experience at least the larger VLMs are good at following instructions about the answer format.  
* I am worried that using GPT for option matching introduces additional bias. I believe this could be measured by evaluating GPT or Gemini again by giving it the answer options in the prompt and asking it to respond with only the answer letter. Results could then be compared against the GPT-matched results.  
  * Also to the above point, did authors verify that event ordering type questions get matched correctly with GPT? These answers only differ in their ordering of options, so I am wondering whether GPT matches them correctly.  
* The benchmark was constructed using GPT, and GPT is the best performing model across all tasks. It would be interesting to quantify if there is bias towards GPT, e.g. by generating part of the data with Gemini and checking if relative model performance is consistent with the original benchmark.  
* How are copyright concerns handled? Did authors obtain permission from the copyright owners to use the video material for this purpose and to reproduce this content in a publication? If the dataset will be publicly released, how are copyright concerns handled?  
  * l. 198: “To address this limitation, we transformed the TVQA dataset from a collection of short clips into a long video dataset by gathering and sequencing the clips corresponding to each episode thereby reconstructing the full episode frames.“ How was this done and what data source was used?  
* Appendix l. 12: “The remaining two skills, i.e., local visual questions and summarizing, do not need human verification, as the first one is adopted from the TVQA dataset, and the latter is scrapped from human responses on the web.” I do not fully agree with this statement since existing benchmarks and the humans writing the summaries that were pulled from the web could still contain errors. Do authors have evidence to the quality of TVQA annotations and summaries obtained from the web?  
* How does the number of video frames provided affect the model accuracy?  
* Appendix B is quite important to understand the evaluation results presented, so I think it would be better suited to be in the main text.  
* Appendix B mentions that the benchmark videos have no audio, so video and subtitles are provided to the model separately. Does this mean that alignment between frames and subtitles is missing? Did authors measure the effect of this?  
* Could authors explain how spoiler questions are generated and provide the prompt used?  
* How does the “I don’t know” option affect results? How accurately does GPT match model answers to this option?  
* Fig. 5 (left) is redundant with Tab. 3, so one of them should be removed.  
* l. 363: The explanation of local vision and text questions is not clear. It is not explained what these questions are nor how they were generated.  
* It would be good to have random accuracy in Tab. 5 for direct comparability. Then, Tab. 4 could be omitted.  
* l. 482: “As shown in the table 5, MiniGPT4-video and LLaVA-NeXT-Interleave match lower than the random performance” What random performance is being compared to here? It would help to add this to the table as suggested above.  
* l. 482, l. 505: How can a model’s performance be lower than random?  
* l. 488: “One reason may be that eliminating the noisy information and focus on only the related information helps more in answering the questions“ How does the Goldfish model eliminate noisy information?  
* For the human verification, how were human responses on open-ended questions evaluated?

Minor points

* Tab. 1: I would not agree with the “human” checkmark for InfiniBench since questions were generated fully automatically.  
* Tab. 2 is never referenced.  
* Appendix B: It would be helpful to express this in tabular form so readers can see at a glance how many frames and what modalities were used in each model.  
* Tab. 5.: I would suggest to organize this into one big table with one column per task type. Also would be nice to visualize as a radar chart.  
* It would be helpful to annotate question types in Sec 3.2.2 and Fig. 1 with whether they are MCQ or OE.  
* It would be helpful to see a listing of modalities (vision, summary, transcript) used to generate each question.  
* Please use \\citep for citations to place citations in parentheses.  
* In tables, please right-justify numerical columns and use a consistent number of digits after the decimal point.  
* Fig. 4: The font size in these charts is very small in print. I suggest increasing it. Also I would suggest to change the pie chart into a bar chart for easier readability.  
* Fig. 5: Same concern as above about the font size.  
* l. 373: Here, the reference to Fig. 4 is repeated, but Fig. 5 is wrongly referenced. Suggest correcting this sentence to refer to Fig. 3\.  
* l. 406: Broken reference.  
* l. 413: The reference should point to Sec. B in the supplementary material.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613870276

thread_type:
official_review

<thread_object>
creator: reviewer_147d

<content>
summary:
This paper introduces InfiniBench, an innovative and comprehensive benchmark focused on evaluating large multimodal models' performance in understanding very long videos. InfiniBench is notable for its ultra-long video duration (averaging 52.59 minutes per video) and massive question-answer pairs (108.2K), covering nine different skills including multiple-choice and open-ended questions. These questions are designed to be both diverse and human-centric, with videos primarily sourced from movies and TV shows. Experimental results show that even leading AI models like GPT-4V and Gemini 1.5 Flash face significant challenges in long video understanding, achieving average accuracies of only 49.16% and 42.72%, with mean scores of 3.22 and 2.71 (out of 5) respectively. This indicates that while these models perform relatively well on local skills, they still have limitations in skills requiring global reasoning and deep contextual understanding, such as scene transitions and movie spoiler questions. Open-source models generally perform below random chance on multiple-choice questions, highlighting long-sequence global reasoning as a major challenge for existing models. Additionally, models relying on both video and text information perform poorly without caption input, emphasizing the importance of processing both visual and textual information for long video understanding. The introduction of InfiniBench aims to fill the gap in long video understanding benchmarks, drive the development of open-source large language models, and motivate multimodal large models toward more human-like long video understanding and reasoning capabilities, despite current limitations such as video source restrictions and script dependency.

strengths:
1. InfiniBench provides a comprehensive evaluation of large multimodal models' capabilities in long video understanding through including the longest video duration and a large number of question-answer pairs, as well as designing diverse question types (multiple-choice and open-ended questions) covering nine different skills, thus thoroughly examining models' performance across multiple dimensions of long video understanding.

2. By evaluating various models including both commercial and open-source models, InfiniBench reveals the challenges and limitations of existing models in long video understanding, especially in tasks requiring deep contextual understanding and critical thinking. This in-depth assessment helps identify model deficiencies and provides clear directions for future research and model improvements.

3. InfiniBench's design not only tests models' technical capabilities but also drives models toward more human-like understanding and reasoning abilities. Through proposing human-centric questions, such as movie spoiler questions, it promotes model performance improvement in long video understanding tasks, which is significant for achieving more advanced AI applications and advancing the field of artificial intelligence.

weaknesses:
1. The benchmark only uses movies and TV shows for testing, which is too limited. It should include more types of videos that show different parts of real life, like nature documentaries or home videos. The problem is that movies and TV shows follow certain storytelling patterns, so AI models might just learn these patterns instead of truly understanding the videos. They should add more casual videos like vlogs and livestreams to make the testing more realistic.

2. The benchmark needs written scripts to create its questions and answers. This is a big problem because most real-world videos don't come with scripts. Without scripts or captions, the benchmark can't test how well AI models understand regular videos that people actually watch and share online.

3. InfiniBench's testing does not cover current mainstream open-source models such as Qwen2VL, LLaVA-Onevision, and InternVL2. This makes it difficult to obtain a more comprehensive and in-depth comparison between open-source and closed-source models.

4. In Table 1, the benchmark comparison is insufficient, especially regarding some recent video benchmarks such as Video-MME and LongVideoBench. Additionally, the authors' definition of "very long" is problematic - MLVU and MovieChat have only a 3-minute gap, yet MLVU is defined as very long. This is not reasonable.

questions:
1. How is GPT-4V's scoring aligned with human evaluation?
2. Why weren't the latest models tested, and why wasn't there comparison and discussion of the latest benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628877621

thread_type:
official_review

<thread_object>
creator: reviewer_6qyh

<content>
summary:
This paper introduces InfiniBench, a video understanding benchmark dataset featuring the longest video duration (average 52.59 minutes per video) and the largest number of question-answer pairs (108.2K) to evaluate 9 different video understanding tasks.

The authors conducted comprehensive evaluations of existing large multimodal models (including commercial models like GPT-4V, Gemini 1.5 Flash, and open-source models). Experiments show that even leading AI models still face challenges in long video understanding, with the best models GPT-4V and Gemini 1.5 Flash achieving average accuracy rates of only 49.16% and 42.72% respectively.

strengths:
1. The questions are comprehensive and well-structured, covering multiple dimensions and employing diverse construction strategies for different types of questions.
2. The evaluation methods are reasonable, adopting different assessment metrics for multiple-choice and open-ended questions.

weaknesses:
1. The paper lacks discussion of related work. For example, benchmarks proposed in Video-MME, LVBench, and Long VideoBench published in June 2024 are very similar to InfiniBench.

2. Most of the question-answer pairs are generated by GPT-4o. Although multiple information sources were used as input, it's difficult to guarantee the quality of the dataset.

3. Part of the data comes from IMDB content, which likely appeared multiple times in the training corpus of LLMs used by video models, potentially leading to dataset leakage issues.

questions:
1. Add references and discussions of related work.
2. It would be better to evaluate more long-video models (e.g., Qwen2VL) and different input frame rates (1, 8, 32, 128, and more).
3. Since most question-answer pairs are generated by GPT-4o, could this lead to inflated evaluation results for GPT-4o? Analysis is needed regarding dataset quality, hallucination rates, and potential information leakage issues.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733045951158

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our Rebuttal

comment:
We sincerely thank the reviewers and area chairs for their thoughtful feedback and constructive comments. Your insights have been instrumental in strengthening our work, improving its clarity, and providing additional evidence to support our claims.

In the revised manuscript, we have incorporated all feedback, with changes clearly highlighted in blue for your convenience. 
Below, we summarize the key updates and experiments added to address your concerns:

1. Inclusion of More Models:

    * We have incorporated additional recent long-video understanding models into our benchmark to ensure they are comprehensive and holistic.
    * As a result, we have updated the findings in the benchmark, providing the community with more robust insights to guide future research directions.

2. Human Verification of the Benchmark:

    * We conducted a human verification of 10% of the benchmark and evaluated all models on this subset.
    * The strong correlation between model scores on the verified subset and the full benchmark demonstrates the reliability of our dataset.

3. Human Evaluation of Models:

    * We evaluated the correlation between our benchmark scores and human preferences, observing a high agreement of 95%.
    * This further validates the benchmark’s effectiveness in assessing model performance.

4. Dataset Leakage Analysis:

    * We performed additional experiments to assess dataset leakage and found it to be minimal due to the careful design of the benchmark.
    * This ensures the integrity of our evaluation and strengthens confidence in our results.

5. Influence of Subtitles:

    * We added ablation studies to analyze the role of subtitles, showing that their impact is limited.
    * This reinforces that our benchmark primarily evaluates visual understanding, focusing on vision-based reasoning rather than textual cues.

6. Enhanced Writing and Visuals:

    * We replaced nearly all figures with enhanced versions for better presentation and clarity.
    * Additionally, we revised several sections to improve readability and ensure a smoother flow of ideas.

We hope these updates comprehensively address your concerns and further demonstrate the robustness and value of our contributions. Thank you again for your invaluable feedback, which has greatly improved the quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presented a benchmark for long video understanding which is an important and challenging problem in the field. The paper received mixed ratings from the reviewers. There are some critical concerns raised by the reviewers. First, one reviewer is concerned about the diversity of the testing long videos. The videos are mainly from TV shows or movies which largely restricts the applicability of the benchmark to be used to model realistic video understanding problems. Another reviewer also mentioned that the testing videos are from only a limited number of channels which may bring bias in testing the performance of different video understanding models. Second, the reviewers are also criticizing the benchmark baseline models being used. There are several important video understanding models not tested in the benchmark. It brings limitations to understanding how challenging or useful the proposed benchmark is. Finally, there are also comments regarding the presentation quality of the submission. Based on these key points, AC decided to recommend a rejection for this time. The authors are encouraged to further polish the paper to submit it another time.