StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

Authors: Junming Lin, Zheng Fang, Zihao Wan, Fuwen Luo, Chi Chen, Peng Li, Yang Liu, Maosong Sun
Primary area: datasets and benchmarks
Keywords: Benchmark, Streaming Video Understanding, Multimodal Large Language Models, Video Benchmark, Evaluation

Abstract:
The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on ofﬂine video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a signiﬁcant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the ﬁrst comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features ﬁve questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 15 open-source and proprietary MLLMs and ﬁnd that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform signiﬁcantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

Decision: reject
Reviewer scores: 3, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730604701573

thread_type:
official_review

<thread_object>
creator: reviewer_rih2

<content>
summary:
The authors propose a new benchmark, StreamingBench, for evaluating MLLMs in streaming video understanding. It assesses three aspects of streaming video understanding: real-time visual understanding, omni-source understanding, and contextual understanding. There are 18 tasks in total. They evaluate 13 open-source Video MLLMs and 3 proprietary MLLMs on this benchmark and analyze the results.

strengths:
S1: In this paper, the authors propose a benchmark to evaluate the MLLMs' capabilities of streaming video understanding, which is novel and unexplored previously. I believe this will facilitate the advancement of Video MLLMs.

S2: The benchmark considers both video and audio modalities, which have been absent in most previous benchmarks.

S3: The experiments and analysis are comprehensive and detailed, effectively highlighting the limitations of current Video MLLMs in understanding streaming video.

S4: The writing is clear and well-structured.

weaknesses:
W1: The impact of language model size on performance has not been analyzed. For instance, models like InternVL-V2 come in 1B, 2B, 4B, 8B, 26B, 40B, and 72B parameter versions, while Video-LLaMA2, LLaVA-OneVision, and Qwen2-VL also have 72B versions. Expanding your experiments to include these variations and providing a more detailed analysis would enhance your work. Additionally, exploring the number of frames the model can process would offer valuable insights.

W2: Several significant models are missing from the evaluation, such as LongVILA [1], Long-LLaVA [2], and Oryx [3]. Including these would provide a more comprehensive comparison.

[1] https://github.com/NVlabs/VILA/blob/main/LongVILA.md

[2] https://github.com/FreedomIntelligence/LongLLaVA

[3] https://github.com/Oryx-mllm/Oryx

questions:
Q1: Do you plan to extend your benchmark to support additional streaming video understanding tasks, such as dense streaming video captioning or grounding? Currently, it only supports QA.

Q2: Could you provide more details on how audio is utilized in the MLLMs? I am aware that Video-LLaMA2 supports audio input, but what about the other models?

Q3: Have you considered memory constraints?  I believe this is a crucial factor in streaming video understanding. It would be more consistent and fair to have a fixed memory limit applied across all models, as retaining all input frames throughout a task—particularly in real-world applications—may not be feasible. If some models retain all frames, it could lead to an unfair advantage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648327315

thread_type:
official_review

<thread_object>
creator: reviewer_winw

<content>
summary:
The paper introduces StreamingBench, a benchmark designed to evaluate the streaming video understanding abilities of Multimodal Large Language Models (MLLMs). Traditional MLLMs are effective in offline video comprehension but struggle with real-time, streaming scenarios that require instant processing, synchronizing visual and audio inputs, and understanding context over time. StreamingBench addresses this by presenting 900 videos across diverse real-world scenarios, structured into 18 tasks and 4,300 human-curated question-answer pairs. These tasks test MLLMs on real-time visual, omni-source, and contextual understanding, aiming to bridge the gap between MLLMs and human-level comprehension in streaming contexts. Testing 13 MLLMs, including state-of-the-art proprietary models, revealed significant limitations in current models, especially in omni-source and contextual tasks, suggesting that MLLMs need further development to match human performance in real-time understanding. In general, it is a solid paper and I would recommend acceptance to it.

strengths:
1. It is the first valid benchmark on streaming long videos. The questions are designed properly to reflect the information gained in a streaming long video, and highly resembles what human will ask when continuously watching a video.
2. The evaluation and discussion are both very solid.
3. Human performance is another plus.

weaknesses:
1. The real-time understanding part is nice, but seems a little bit trivial. From all kinds of NIAH evaluations, all models can best answer questions near the ending part of the input, and questions related to "current moments" (which is actually the ending part of input as implemented) might not be so important. Would love to see the understanding on "remembering earlier moments" and the discrepancy from "current moments" for LMMs.

2. The omni-source (visual+audio) part is good. However, how are LMMs without audio abilities evaluated? As these `audio'-related questions seem to be mostly about speeches, do authors plan to interleave text ASR into the model for evaluation? At present, sadly we only see a black-box Gemini-1.5-Pro (for which we do not know how they integrate audio and video) being evaluated with audio.

3. A minor suggestion: the omnisource part of the benchmark is related to "referring reasoning" part of LongVideoBench, an interleaved benchmark for frames and ASR texts, which also needs to judge between concurrent video and audio information in a video. As some other long video benchmarks discussed in Tab 1, please also try to discuss it in the revised paper.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694012098

thread_type:
official_review

<thread_object>
creator: reviewer_szqs

<content>
summary:
This paper introduces StreamingBench, a benchmark designed to evaluate the capabilities of MLLMs in understanding online streaming videos. Key features of StreamingBench include the ability to pose questions at any point during the video, rather than requiring the full video to be viewed first. The benchmark also considers both visual and audio inputs, and it takes into account the influence of historical interactions in multi-turn dialogues.

strengths:
- StreamingBench addresses a relatively unexplored area in MLLM research—real-time video understanding. By allowing questions to be asked at any moment and incorporating both audio and visual data, it expands the scope of existing benchmarks.

weaknesses:
- The methodology for collecting 900 videos from YouTube lacks sufficient detail.
- Given that the study focuses on a model's capability that is seldom addressed—real-time video understanding—it would be beneficial to create or curate a specific supervised fine-tuning (SFT) dataset. This would allow for an evaluation of model performance post-SFT.
- There is a lack of exploration into the model’s ability to generate proactive outputs. Designing a corresponding SFT dataset to assess whether the model performs better with prior exposure to similar outputs would provide valuable insights.
- Clarification is needed on how open-source models like Qwen2-VL tackle omni-source understanding problems, particularly in the absence of audio inputs. This comparison could shed light on the robustness of the proposed benchmark.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798581333

thread_type:
official_review

<thread_object>
creator: reviewer_7trd

<content>
summary:
This work proposes a benchmark called StreamingBench to evaluate video LLM capabilities in streaming settings. StreamingBench introduces several tasks tailored to streaming scenarios, including real-time visual understanding, omni-source understanding, and contextual understanding.

strengths:
- This work introduces a new benchmark designed to evaluate video models in streaming scenarios.
- It conducts insightful experiments, such as "Does Redundant Information Affect Contextual Understanding?", which provide valuable perspectives in this area.

weaknesses:
- Although this benchmark focuses on streaming scenarios, a standard video LLM can handle it effectively with simple preprocessing. For instance, whenever a question arises, the model can process all frames up to that timestamp. With this approach, the benchmark may not differ significantly from traditional video benchmarks. Therefore, it is essential for this benchmark to identify scenarios that cannot be simplified to an offline setting.

- While handling redundant information is indeed critical for video LLMs, this challenge is not exclusive to streaming scenarios; it is a general issue for any long-video task. As a result, the insights from this paper may be overshadowed by findings from benchmarks specifically focused on long-video understanding.

- The annotation process lacks clarity. Specifically, how do human annotators manually label QA pairs for omni-source understanding and other contextual understanding tasks? What measures are in place to ensure the quality of each question, and what specific strategies were employed?

questions:
As mentioned in the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732771199831

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We appreciate that most reviewers acknowledge the value of StreamingBench as the first benchmark to evaluate streaming video understanding capabilities, which are underexplored by previous video benchmarks. In this response, we address the common questions raised by the reviewers. Detailed replies to individual comments from each reviewer are provided within their respective reviews.

**General response 1: About audio input for Omni-Source Understanding tasks**

In the current Omni-Source Understanding experiments in the paper, we did not perform additional processing for models that do not support audio input; these models could only rely on video information to answer Omni-Source Understanding tasks. 
We also evaluate all open-source MLLMs across all tasks using a combination of video and ASR information, with detailed results presented in the table below. The results show that while the performance of these models on Omni-Source Understanding tasks has improved, it remains significantly lower than that of Gemini 1.5 Pro. This highlights the critical importance of advancing current MLLMs to natively support audio input in order to meet the demands of omni-source understanding capabilities for understanding streaming videos. 

**Table: Model performance with and without ASR input information from videos.**

| Model                        | Params | Frames      | All (RTVU) | All (OSU) | All (CU) | Overall |
|------------------------------|--------|-------------|------------|-----------|----------|---------|
| Gemini 1.5 Pro | -   | Video | 75.69      | 60.22     | 47.79   | 66.90   |
| LLaVA-OneVision (w/ ASR)     | 7B     | 32          | 73.47      | 41.50     | 34.58    | 58.79   |
| LLaVA-OneVision (w/o ASR)    | 7B     | 32          | 71.12      | 38.40     | 32.74    | 56.36   |
| LLaVA-NeXT-Video (w/ ASR)    | 32B    | 64          | 70.27      | 43.20     | 35.46    | 57.49   |
| LLaVA-NeXT-Video (w/o ASR)   | 32B    | 64          | 66.96      | 34.90     | 30.79    | 52.77   |
| Qwen2-VL (w/ ASR)            | 7B     | 0.2-1 fps†  | 70.68      | 41.20     | 36.33    | 57.43   |
| Qwen2-VL (w/o ASR)           | 7B     | 0.2-1 fps†  | 69.04      | 34.90     | 31.66    | 54.14   |
| MiniCPM-V 2.6 (w/ ASR)       | 8B     | 32          | 67.94      | 39.60     | 38.08    | 55.79   |
| MiniCPM-V 2.6 (w/o ASR)      | 8B     | 32          | 67.44      | 35.00     | 34.97    | 53.85   |
| InternVL-V2 (w/ ASR)         | 8B     | 16          | 68.30      | 37.60     | 33.58    | 54.70   |
| InternVL-V2 (w/o ASR)        | 8B     | 16          | 63.72      | 35.80     | 32.42    | 51.40   |
| LongVA (w/ ASR)              | 7B     | 128         | 63.77      | 37.60     | 34.46    | 52.23   |
| LongVA (w/o ASR)             | 7B     | 128         | 59.96      | 35.40     | 29.95    | 48.66   |
| Kangaroo (w/ ASR)            | 7B     | 64          | 65.85      | 31.90     | 28.84    | 51.06   |
| Kangaroo (w/o ASR)           | 7B     | 64          | 64.60      | 34.20     | 30.06    | 51.10   |
| Video-CCAM (w/ ASR)          | 14B    | 96          | 57.84      | 35.90     | 27.72    | 47.13   |
| Video-CCAM (w/o ASR)         | 14B    | 96          | 53.96      | 29.70     | 22.88    | 42.53   |
| Video-LLaMA2 (w/ ASR)        | 7B     | 32          | 53.19      | 30.40     | 26.72    | 42.96   |
| Video-LLaMA2 (w/o ASR)       | 7B     | 32          | 49.52      | 32.40     | 21.93    | 40.40   |
| VILA-1.5 (w/ ASR)            | 8B     | 14          | 50.61      | 24.00     | 24.34    | 39.53   |
| VILA-1.5 (w/o ASR)           | 8B     | 14          | 52.32      | 33.10     | 27.35    | 43.20   |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presented a benchmark for stream video understanding. The paper received mixed ratings from four reviewers. Although some of the reviewers appreciated the importance of stream video understanding and the creation of a corresponding benchmark to advance this research area, there are some concerns still remaining in the current paper. First, as pointed out by reviewer 7TRD, the paper needs more discussion on particular insights of real-time stream video understanding, which is considered a different direction from the existing long-video understanding in an offline setting, including from both the benchmark, the method, and the evaluation levels. Both reviewers szQS and wiNw also mentioned in the weakness part that the unique capability of the model for handling online real-time stream video understanding is seldom addressed. Second, the data processing details are not clearly presented in the paper, and some additional analysis experiments should be added to better show the effectiveness of the model. Based on these significant comments, AC finally decided to reject the submission for this time.