Streaming Video Question-Answering with In-context Video KV-Cache Retrieval

Authors: Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, TaoZhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, Hao Jiang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Understanding, Multimodal Large Language Models, Streaming Video Question-answering

Abstract:
We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video analyzing and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730253574797

thread_type:
official_review

<thread_object>
creator: reviewer_3gu5

<content>
summary:
The paper introduces ReKV, a novel, training-free approach designed to enhance the efficiency of Video Large Language Models (Video-LLMs) for streaming video question-answering (StreamingVQA). Unlike traditional VideoQA systems that process entire videos before answering queries, ReKV processes video streams in real-time, allowing for prompt responses. The method employs a sliding-window attention mechanism to reduce computational overhead and uses a KV-Cache system to store and retrieve relevant video information efficiently. The approach separates video encoding and question-answering into distinct processes, enhancing efficiency. The paper demonstrates the efficacy of ReKV through comprehensive experiments, showing improvements in accuracy, latency, and memory usage over existing models.

strengths:
1.	Efficiency: The sliding-window attention mechanism and KV-Cache retrieval significantly reduce computational overhead and memory usage.
2.	Real-Time Processing: The method allows for real-time responses to queries, making it highly practical for applications like surveillance and live broadcasts.
3.	Comprehensive Evaluation: The paper provides extensive experimental results, demonstrating the effectiveness of ReKV across multiple benchmarks.
4.	Seamless Integration: ReKV integrates seamlessly with existing Video-LLMs without requiring additional training, making it easy to adopt.

weaknesses:
1.	Writing Quality: The organization of this paper could be improved. It is not appropriate to place ablation study before main experiments. Sec 2.1 Task definition and discussion should not be a part of Method. This part is too repetitive of the discussion in the introduction.
2.	Citation format: In Table 4 Line 391 there may be a misleading citation of Video-LLaVA-7B, pointing to the same reference of Video-ChatGPT-7B.
3.	Lack explanation: The term "oracle retrieval" from Table 2 and Line 305 is difficult for readers to understand. How is the “recall” metric calculated? How can it be 100?

questions:
1.	Generalizability of method: Since ReKV is a training-free method, can it be integrated with models other than LLaVA-OV? Are there any experimental results?
2.	Scalability: How does ReKV scale with increasing video length and complexity? Are there any observed limitations when dealing with very high-resolution videos or videos with a high frame rate?
3.	Implementation details: What is the hyperparameters of external retrieval?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730491331640

thread_type:
official_review

<thread_object>
creator: reviewer_g3ys

<content>
summary:
The authors present ReKV (Retrieve In-context Video KV-Cache) for streaming video question-answering. The authors incorporate a sliding-window attention mechanism on existing VideoLLMs, introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only queryrelevant KV-Caches. The authors evaluates the model on both long video QA and streaming videoqa.

strengths:
1. The number of evaluation benchmark for the proposed method is adequate. 

2. The improvements against standard VideoLLMs are substantial.

weaknesses:
1. Lack of fair comparison against existing memory-based models (including VideoStreaming and Flash-VStream). It would be better if the author could provide results for ReKV and previous memory-based models under the same VideoLLM backbone to show the effectiveness of the proposed method, for both long video benchmarks and streaming video benchmarks. 

2. Missing related works. The authors should discuss the novel contribution compared to the paper VideoLLM-online[1], MC-ViT[2]. 

3. The “External Video KV-Cache Retrieval” is confusing, do the authors mean selecting the keyframes using the query information via the CLIP-based models (like a cross-modal matching)? This is already investigated by a number of works, including ATP [3], SeViLA[4] and so on.  It would be if for the authors to clarify how "External Video KV-Cache Retrieval" differs from or improves upon the keyframe selection methods. 

[1] VideoLLM-online: Online Video Large Language Model for Streaming Video

[2] Memory Consolidation Enables Long-Context Video Understanding

[3] Revisiting the "Video" in Video-Language Understanding

[4] Self-Chained Image-Language Model for Video Localization and Question Answering

questions:
All weakness, and:

1. The citation format is inconsistent over the paper, the authors should unify this format. 

2. Since the model is claimed to integrate seamlessly with existing Video-LLMs, is it possible to apply to a bigger VideoLLM backbone, like around 70B scale? 

3. In Table 4,  for offline video question-answering, could the authors elaborate more on the baseline setting of the LLaVA-ov model, like frame numbers? Also, could the authors compare the efficiency of the proposed ReKV compared to the original LLaVA-ov model in the table?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634156743

thread_type:
official_review

<thread_object>
creator: reviewer_t8db

<content>
summary:
This paper introduces ReKV, a novel, training-free approach designed to enhance existing Video-LLMs for StreamingVQA. Traditional VideoQA systems struggle with long videos due to the need to process entire videos before responding and repeating this process for each new question. ReKV addresses these challenges by storing processed video key-value caches (KV-Caches) in RAM or disk to prevent information loss. ReKV introduces retrieval methods—both external (using models like CLIP) and internal (leveraging the Video-LLM's parameters)—to fetch only query-relevant KV-Caches, enhancing efficiency and accuracy in question-answering.
Experiments conducted on various benchmarks, including MLVU, QAEGO4DMC, EgoSchema, ActivityNet-QA, and StreamingVQA (RSV-Ego and RSV-Movie) datasets, demonstrate that ReKV improves VideoQA accuracy while maintaining stable inference latency and memory usage as the number of frames increases. The method enables real-time interaction and long-term context for StreamingVQA tasks.

strengths:
- The paper presents a novel and simple, training-free method that extends the capabilities of existing Video-LLMs for StreamingVQA. By integrating a sliding-window attention mechanism and efficient KV-Cache retrieval, ReKV addresses the challenges of processing long video streams in real-time.

- The methodology is well-motivated and thoroughly explained. The paper clearly defines the StreamingVQA task, differentiates it from traditional OfflineVQA, and outlines the specific challenges involved. The proposed solutions are detailed and logically sound.

-  The paper is well-organized and clearly written with figures to support the method explanation.

- ReKV significantly improves efficiency and accuracy over existing VideoQA models on multiple benchmarks. The ability to handle long video streams in a streaming fashion has practical importance for real-world applications. The training-free nature of ReKV can potentially enhance its applicability across different Video-LLMs.

weaknesses:
Currently, a major limitation of the method is that the method is that it is only evaluated on LLaVA-OV models (0.5B and 7B). Although these models are strong baselines, the applicability of ReKV to other Video-LLMs is not demonstrated. Evaluating ReKV on a broader set of models would strengthen the claim of its versatility and general applicability.
I’ll be happy to increase my score if that limitation is addressed.

questions:
- Have the authors tested ReKV with other Video-LLMs besides LLaVA-OV? Demonstrating the integration and performance of ReKV with different architectures (e.g., VideoChatGPT…) would confirm its general applicability and ease of integration.
- Table 5 shows that the internal KV-Cache retrieval reduces computational overhead compared to external retrieval. However the “internal retrieval” retrieves KV-Caches for each attention layer independently while it is only done once for the “external retrieval”. How do you explain that the internal is faster?


Minor: 
In practice, how does ReKV manage KV-Cache storage for extremely long video streams, such as surveillance footage that can run continuously for many hours or days? Are there mechanisms in place to prevent unsustainable increases in cache size, and how does this impact performance and resource requirements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653220419

thread_type:
official_review

<thread_object>
creator: reviewer_acrc

<content>
summary:
This paper presents video KV caches to make a streaming video question and answering video-LLMs in a training-free approach.
While it uses a sliding-attention mechanism to aggregate short-term temporal context, video KV caches and the proposed retrieval method are introduced to long-term temporal context.
This method shows efficiency with LLaVA-OV in several benchmarks.

strengths:
- This paper outperforms existing video-LLMs on long-form benchmarks.
- This paper is easy to follow.
- Ablation study shows the validity and impact of the retrieval methods.

weaknesses:
- My major concern is the novelty. Already, many LLM systems reduce the context-processing delay by using the KV cache of the context. This paper is also built on LLMs, while it is coupled with a video encoder. It's hard to find the specialty for the video streaming system. The causal attention and retrieval system with cosine similarity are also not new.

- Implementation with only one design (LLaVA-OV) with different sizes is limited to prove the generality of the proposed method.

- There are many recent methods to reduce the memory of KV caches such as adaptive KV cache (ICLR'24) and Keyformer (Muhammad Adnan et al., arxiv'24), compared to these methods, is the proposed attention and search method more effective?

questions:
- How about GFLOPs on streaming VQA?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732465295098

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their time and thoughtful feedback on our work. We are grateful for their recognition of the novelty (`t8DB`), clarity of writing (`AcRC`), and the demonstrated effectiveness (`g3YS`, `3gu5`) of our method. Below, we address the common concerns raised.

---

**Common Concern: ReKV + Different Video-LLMs**

Thank you for your interest in assessing the generalizability of our approach. To this end, we conducted experiments with additional Video-LLMs, including `Video-LLaVA-7B` [1], `LongVA-7B` [2], and `LLaVA-OV-72B` [3].

| Model                       | #frames       | MLVU dev | QaEgo4D test | EgoSchema |
|-----------------------------|---------------|----------|--------------|-----------|
| Video-LLAVA-7B             | 8             | 46.6     | 36.8         | 41.3      |
| &ensp;**+ReKV**      | 0.5 FPS&rarr;8  | **49.1 (+2.5)**     | **40.4 (+3.6)**        | **42.6 (+1.3)**     |
| LongVA-7B                  | 32            | 57.0     | 42.2         | 42.4      |
| &ensp;**+ReKV**           | 0.5 FPS&rarr;32 | **59.1 (+2.1)**    | **45.4 (+3.2)**        | **43.5 (+1.1)**     |
| LLAVA-OV-72B               | 32            | 69.7     | 53.6         | 59.6      |
| &ensp;**+ReKV**        | 0.1 FPS&rarr;32 | **73.7 (+4.0)**    | **58.4 (+4.8)**        | **62.3 (+2.7)**     |

- ReKV consistently improved performance across all models, demonstrating its robustness and adaptability.
- For LLaVA-OV-72B, the need for model sharding significantly slowed inference. To address this, we set the FPS to 0.1 to maintain efficiency during evaluation. 

In the final draft, we plan to include results on additional benchmarks such as ActivityNet-QA, RVS-Ego, and RVS Movie, which leverage ChatGPT for quantitative evaluation.

[1] Lin, Bin, et al. "Video-llava: Learning united visual representation by alignment before projection." In EMNLP, 2024.

[2] Zhang, Peiyuan, et al. "Long context transfer from language to vision." arXiv:2406.16852, 2024.

[3] Li, Bo, et al. "Llava-onevision: Easy visual task transfer." arXiv:2408.03326, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission explores a training-free approach for Visual Question Answering (VQA) in Video Large Language Models (Video LLMs). It introduces a novel technique based on sliding window attention and the KV-caches which can be later used to improve the efficiency in VQA tasks. All reviewers are in favor of accepting the submission.