Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

Authors: Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, Huchuan Lu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Streaming Video Understanding; Video MLLM; Hierarchical Memory System

Abstract:
Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. StreamChat leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks.  Extensive evaluations on StreamBench and other public benchmarks demonstrate that  StreamChat significantly outperforms existing
state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730475431848

thread_type:
official_review

<thread_object>
creator: reviewer_yvyo

<content>
summary:
This paper studies streaming video understanding. The authors propose a new benchmark named StreamBench to evaluate streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. They also propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction, with a complex memory mechanism. Their method enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Extensive evaluations on StreamBench and some public benchmarks demonstrate that StreamChat outperforms the selected
baselines.

strengths:
1. The proposed StreamBench may be the first benchmark for streaming video understanding.
2. The proposed StreamChat outperforms the selected baselines on StreamBench.
3. The processing speed of StreamChat significantly outperforms those of baselines.

weaknesses:
1. The proposed dataset is too small, only 306 videos and 1.8K  question-answer pairs are collected. The current video benchmark typically has at least thousands of videos and tens of thousands of QA pairs. This version of the benchmark is not ready for release.
2. State-of-the-art video LLMs are not included in the benchmark, such as MiniCPM-V 2.6 [1], InternLM-XComposer2.5 [2], VILA [3], and InternVL2 [4]. The effectiveness of the proposed method is unclear.
3. The Selective Frame Stacking seems may ignore small objects in the video and only focus on the global frame feature.
4. The proposed memory mechanism is very complicated and discards a large amount of information in the video.
5. The authors test the processing speed of models on two NVIDIA Tesla A800 80GB GPUs, which is not a typical scenario of model development in real-world applications.


[1] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., ... & Sun, M. (2024). Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
[2] Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., ... & Wang, J. (2024). Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320.
[3] Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024). Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 26689-26699).
[4] Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., ... & Qiao, Y. (2024). How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

questions:
Please respond to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666438586

thread_type:
official_review

<thread_object>
creator: reviewer_bg29

<content>
summary:
In this work, the authors propose a training-free framework, StreamChat, for streaming video understanding. StreamChat uses a hierarchical memory system to efficiently process long video sequences for real-time, multi-turn dialogue. With parallel scheduling, it improves speed and reduces latency. 

The authors also introduce a new benchmark, StreamBench, which tests video understanding across diverse scenarios, and results show StreamChat outperforms current models in accuracy and response time.

strengths:
S1: The paper introduces a novel hierarchical memory mechanism that compresses video representations over long sequences, enabling efficient video feature retrieval in real-time multi-turn dialogue contexts.

S2: I believe the introduction of StreamBench as a benchmark for streaming video understanding is a significant contribution that can fill a critical gap in the field. By offering a standardized evaluation framework, StreamBench enables more rigorous comparisons across models, driving advancements in streaming video understanding.

S3: The proposed StreamChat achieves the best performance on StreamBench, surpassing other state-of-the-art methods, which is impressive.

weaknesses:
W1: StreamChat was compared only with several open-source models. Including proprietary models like GPT-4o and Gemini-1.5 in the StreamBench evaluation would provide a more comprehensive comparison. 

W2: Human performance is absent in the comparison, which would help illustrate the gap between current models and human capabilities if included.

W3: The authors compare their new benchmark only with older datasets or benchmarks, such as MSVD, MSRVTT, and ActivityNet. It would strengthen the evaluation to include comparisons with more recent benchmarks for video understanding, such as Seed-Bench [1], Video-Bench [2], MVBench [3], and LVBench [4].

W4: The paper lacks an analysis of factors that impact performance on StreamBench, such as the input frame sequence length or the language model size used by the models.

[1] Li, Bohao, et al. "Seed-bench: Benchmarking multimodal llms with generative comprehension." arXiv preprint arXiv:2307.16125 (2023).

[2] Ning, Munan, et al. "Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models." arXiv preprint arXiv:2311.16103 (2023).

[3] Li, Kunchang, et al. "Mvbench: A comprehensive multi-modal video understanding benchmark." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Wang, Weihan, et al. "LVBench: An Extreme Long Video Understanding Benchmark." arXiv preprint arXiv:2406.08035 (2024).

questions:
Q1: Will the example in the prompt impact the models' prediction? For example, your example in the prompt is "{'pred': 'A'}". Does it increase the probability of predicting "A"?

Q2: The StreamBench benchmark supports only QA tasks now. Are there any plans to extend it to include additional streaming video understanding tasks, such as dense streaming video captioning or temporal/spatiotemporal grounding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718342594

thread_type:
official_review

<thread_object>
creator: reviewer_qlqa

<content>
summary:
This paper presents a benchmark, named StreamBench, and as well a model, named StreamChat, for real-time streaming video LLM answering. The StreamBench comprehensively evaluates assorted capabilities of video LLMs, including spatial search, long & short memory recall, common knowledge reasoning, etc. The size of StreamBench is also larger than previously-proposed benchmarks. The authors also paid a lot of efforts in building the training-free StreamChat model, which bases on current video LLM LongVA and consists of novel techniques including hierarchical memory storage, optimized system scheduling. Evaluations on various benchmarks show that StreamChat enjoys better performance with lower latency compared to previous streaming models.

strengths:
1. The proposed benchmark, StreamBench, is a comprehensive benchmark testing various capabilities of real-time video LLMs. As far as I know, this is the first benchmark designed for online streaming video task evaluation.
2. The proposed method, StreamChat, integrates frame selection, memory building and retrieval, and several optimizations into a video LLM in a training-free manner. Though designed complicatedly, the latency of StreamChat is still under control (under one second). Several benchmarks, including online StreamBench and offline video understanding tasks, show the effectiveness of the proposed StreamChat. The authors also put a detailed inference example, every components illustrated, in Fig 6. Also, the detailed ablations on threshold and memory parameters should be appreciated.

weaknesses:
My major concerns comes from two sides:
1. **The presentation**. The presentation of StreamBench is good but the part of StreamChat seems to need to be further improved. It is hard to tell what StreamBench is designed to do when understanding Fig 4 and Fig 5. Though there are both captions and texts written in the method section, it is still hard to relate text descriptions to components in the figures (especially for ones less familar with memory mechanism and RAG).
2. **The complexity**. StreamChat seems too complicated: there are selective frame stacking based on motion vectors, vision-based short-term memory, caption-based long-term memory (also a clustering mechanism in it), dialogue memory (whose language encoder is different from video LLM), and FAISS indexing library. Although being tuning-free, all these components make this 'system' super heavy. The authors should justify every components used in building the memory system in StreamChat. Moreover, this looks more like a RAG system specifically designed for online videos. The authors should compare with other basic RAG methods.

questions:
See weaknesses. I am just at the borderline due to 1) the contribution of StreamBench and StreamChat, 2) the over-complicated design of StreamChat (being too engineering and RAG-like).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730958309459

thread_type:
official_review

<thread_object>
creator: reviewer_9hdr

<content>
summary:
This paper introduces a method for online interactions with videos through an LLM, and a benchmark suitable for evaluating streaming-style scenarios where the dialogue happens in real time with the video.

Contributions:
- Video-based LLM which processes high framerate video (typically LLMs need to subsample severely the video due to computational constraints). This method uses optical flow magnitude to assess whether to process any particular frame (w.r.t. previous), therefore saving considerable amount of computation needed. The LLM also has a "short/long" term memory system.
- New benchmark that should be useful since it's one of a kind, simulating multi-round interactions centered on the video content. 25h of content was used in the creation of this benchmarks, making it one of the larger ones available.

strengths:
Strong empirical results. This is a systems pape which is tied to a particular benchmark which was designed for it, so it's unsurprising that the proposed method works best. With that said, given the engineering effort put into it, it still is impressive, particularly because of the fact that this system could be used in a realtime scenario. 

I am personally excited by the new benchmark that is proposed in this paper perhaps even more so than the method, because it captures a whole class of interactions which were under-represented before.

weaknesses:
I am kind of baffled by the language of "our method can further improve our foundation model LongVA [20] by 5.1% in accuracy". Does this imply that you're the authors of this method? (Please don't answer this question, but please be mindful of the language which might accidentally de-anonimize you).

The hierarchical design seems a bit ad-hoc. It would have been nice to spend a bit more time to motivate the various design choices. Currently, it sounds like the paper is in the "this is what we did" style, without spending too much into the motivation which led to the various choices made. I could easily imagine many other ways in which the hierarchical memory could have been implemented.

The abstract claims "compression" of video features, but reading the paper reveals that this is actually not at all the case. I find the abstract to be misleading and should be updated to reflect the reality of the methods involved.

Minor comments:
- I am having a hard time reading figures. For example, in Figure 1 I had to use a color picker to figure out which method was which line. In the interest of color blind people, (or partially color blind people like me), please use symbols (*, triangle, diamond, etc) or line styles in addition to colors to make the figures more accessible (either to color-challenged people like me, or when printed out in grayscale).
- There's a typo "Gine-grained" in the appendix.

questions:
- Do you plan to make this benchmark publicly available?
- Do you plan to open source your method so that others may benefit from the engineering effort you've put into your method? (I am of course talking about academic researchers)
- What the process of identifying the various categories STREAMBENCH encompasses? I am of course talking about the domains and classes of videos. 
- In the appendix, you very briefly mention that you're using a LLM to determine accuracy. The formula you present is unclear. (What's N, T?). What are you getting out of the LLaMA model? Also, do you use any particular prompt for the LLaMA model?
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1731272902896

thread_type:
official_review

<thread_object>
creator: reviewer_lmv2

<content>
summary:
This paper introduces StreamChat, a training-free framework for streaming video reasoning and conversational interaction, and StreamBench, a comprehensive benchmark for evaluating streaming video understanding across various media and interactive scenarios. Extensive evaluations on StreamBench and other public benchmarks show that StreamChat significantly outperforms existing state-of-the-art models in accuracy and response times, highlighting its effectiveness for streaming video understanding.

strengths:
Nice paper. This work is comprehensive, provides significant contributions, and is well-written.

The biggest contribution is StreamBench. Previously, there was no suitable benchmark for streaming video understanding. StreamBench is a high-quality addition.

Furthermore, StreamChat is also very interesting, demonstrating that an agent-like system can also achieve state-of-the-art results in streaming video understanding.

weaknesses:
Overall, the paper is satisfactory, but I have some concerns regarding StreamChat:

- What motivated the selection of LongVA as a foundation? What are the essential capabilities required for other Video-LLMs to serve as the base model?

- Would fine-tuning LongVA with some streaming vision-language data be better than the current system?

- Is it feasible to integrate the proposed hierarchical memory storage into a trainable framework?

questions:
Please refer to the part of weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732369285862

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We sincerely thank all reviewers for their thoughtful feedback. **We appreciate the positive comments on the significance of our benchmark and the compelling results of our streaming video understanding system.**  We have carefully addressed all comments and revised the manuscript accordingly. Changes are highlighted in blue.  Key updates include:

**1. Clarity and Presentation:**  Based on the feedback from reviewers **Lmv2**,**9hDR** and **qLQa**, we clarified the selection and utilization of foundation models (main text and appendix).  Section 3 has been reorganized to better explain the motivation behind our memory component design.  Figures 1, 4, and 5 have been revised with distinguishing icons and improved structure for enhanced clarity.

**2. Additional Results:** As suggested by reviewers **Bg29** and **yVyo**, we expanded the evaluation in Tables 4 and 5 to include results for both closed-source models (GPT-4o and GPT-4o-mini) and open-source models (MiniCPM-V 2.6, InternLM-XComposer 2.5, VILA, and InternVL2), as well as human performance, providing a more comprehensive comparison.

**3. Expanded Model Evaluation:**  In response to reviewer **yVyo's** suggestion, we have evaluated additional open-source models on StreamBench and included their results in Tables 4 and 5.

**4. Data expansion plan.** According to the suggestions from reviewers **Bg29** and **yVyo**, we have provided our benchmak data expansion plan in the appendix, and we are continuing improve the quality of our benchmark from multiple aspects.

We believe these revisions significantly strengthen the manuscript. Please do not hesitate to leave comments with any further questions. We appreciate your continued feedback.

Best regards,

The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a new benchmark and a model for question-answering and multi-turn dialog in videos in a streaming setting. This is a relevant and timely problem, and it is a natural use-case for processing long videos.

Reviewers largely appreciated the dataset and benchmark contribution of this work. As it probes an underexplored area of video understanding research, it may have a significant contribution on the community. On the modelling side, reviewers had some concerns that the approach is not very well motivated, that other alternatives could have been considered and ablated, and that it involved too much systems engineering. Authors are also encouraged to discuss prior [streaming architectures](https://openaccess.thecvf.com/content/CVPR2024/papers/Zhou_Streaming_Dense_Video_Captioning_CVPR_2024_paper.pdf) in the revision.

Whilst Reviewer yVyo was overall negative on the paper, the AC believes that the authors have sufficiently addressed the reviewers concerns (ie comparisons with prior work, dataset size) in the rebuttal and in the revision of the paper.

Therefore, the final decision is to accept the paper.