Streaming Video Question-Answering with In-context Video KV-Cache Retrieval
Authors: Shangzhe Di
Zhelun Yu
Guanghao Zhang
Haoyuan Li
TaoZhong
Hao Cheng
Bolin Li
Wanggui He
Fangxun Shu
Hao Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: Video Understanding
Multimodal Large Language Models
Streaming Video Question-answering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a training-free approach that integrates with existing Video-LLMs for streaming video question-answering.

Abstract: We propose ReKV, a novel training-free approach that enables efficient streaming video question-answering (StreamingVQA), by seamlessly integrating with existing Video Large Language Models (Video-LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding to queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon as user queries are received. Building on a common Video-LLM, we first incorporate a sliding-window attention mechanism, ensuring that input frames attend to a limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory as needed. Additionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video analyzing and question-answering across different processes and GPUs, significantly enhancing the efficiency of StreamingVQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models.

Published as a conference paper at ICLR 2025
STREAMING VIDEO QUESTION-ANSWERING WITH IN-CONTEXT VIDEO KV-CACHE RETRIEVAL
Shangzhe Di! Zhelun Yu?
Guanghao Zhang
Haoyuan Li
Tao Zhong?
Hao Cheng²
Bolin Li
Wanggui He²
Fangxun Shu?
Hao Jiang"
1 Shanghai Jiao Tong University
AAlibaba Group
di thangzhe@sjtuu wtw.eeeutttuuuu....ommoom
ABSTRACT
We propose ReKV, novel training-free approach that enables efficient streaming video question-answerrngg
(eraminnVQQA) by seamlessly integrating with exist- ing Video Large Language Models LLMs). Traditional VideoQA systems struggle with long videos, as they must process entire videos before responding queries, and repeat this process for each new question. In contrast, our approach analyzes long videos in a streaming manner, allowing for prompt responses as soon user queries are received. Building on common Vidco-LLM, we first incorporate a Alddng-window attention mechanism, ensuring that input frames at- tend to limited number of preceding frames, thereby reducing computational overhead. To prevent information loss, we store processed video key-value caches (KV-Caches) in RAM and disk, reloading them into GPU memory needed. Ad- ditionally, we introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMM to retrieve only query-relevant KV-Caches, ensuring both efficiency and accuracy in question answering. ReKV enables the separation of video encoding and queestion-nsswert and GPUs, significantly enhancing the efficiency of Streaming VQA. Through comprehensive experimentation, we validate the efficacy and practicality of our approach, which significantly boosts efficiency and enhances applicability over existing VideoQA models.
across different processes
INTRODUCTION
In the literature, video understanding tasks, such as action recognition (Caba Heilbron et al., 2015; Goyal et al., 2017; Kay et al., 2017), visual object tracking (Huang et al., 2019; Muller et al., 2018), and video question-answant answering (Xu et al., 2017; Jang et al., 2017; Xiao et al., 2021; Li et al., 2024b), have primarily focused short clips lasting from few seconds to minutes. However, as vision models increasingly find applications in real-world scenarios like robotics, surveillance, and live broadcasts, the research in the vision community has gradually shifted towards understanding con- tinuous video streams, where long-tern tengteerm contexts and real-time interaction are crucial.
In this paper, we consider the problem of streaming video question-answering (Streaming VQA) As shown in Figure ((a), it involves continuously processing long video streams and promptly re- sponding queries about the visual content at any moment. It can be treated generalization of the standard offline VideoQA, where the model processes the entire video and all questions si- multaneously. By definition, such task of AtreaminVVQA presents three core challenges: (i) Effi cient Video Encoding: Unlike traditional offline VideoQA, where models have access to the entire video clip, Streaming VQA demands real-time analysis of continuous streams. Models must effi ciently process incoming frames without access to future frames frequent revisiting of distant past frames. (ii) Video Context Preservation: To accurately answer questions posed later in the stream, models must preserve relevant information from earlier frames, making long-term context retention a key challenge. (iii) Real-Time Response: The model must provide accurate answers with minimal delay, requiring efficient retrieval of video context and rapid question-answertin
Current Video-LLMs often struggle to encode long video streams due to the large volume of video tokens, forcing most models to process only sparse subset of frames (Maaz al., 2024; Zhang

---

Published as a conference paper at ICLR 2025
(I) Encode Video Stream and Preserve KV-Caches
CGBench
RVS-Movie (30ml))
(25mm)
cnu Werna
Input Stream
RVS Sap
(00mm))
20
QAESCAD
(8mi))
10-
101
(2) Retrieve KV-Cache
(3) VideoQA
20
mamme
00N
Sumple Framass
Baky
MIYU
Accivto
Where put lestuce!
fridge.
(12ms))
commn
What did put the rack?
A small
EgoSchema (3mik)
10'
15-
10'
Fromes
(a) Overview of the StreamingVQA task.
(b) VideoQA Accuracy, Latency, and Memory Usage.
Figure 1: Overview of the Streaming task and our proposed ReKV. (a) Streaming VQA requires model continuously process video streams and
questions about previously viewed content any moment. (h) We propose RcKV enhance efficiency and accuracy in StreamingVQA. Tested with 1ilavA-Ov-78 on an H800 (8GGB) GPU, ReKV maintains stable lateney and GPU memory usage, pre- venting out-of-memory (OOM) errors as frames increase. It also improves the accuracy on seven long-form VideoQA benchmarks compared to the uniform sampling baseline. Further details are provided in Section 4.
et al., 2024c; Li et al., 2024a). This results limited video lengths or a significant loss of fine- grained visual information. While techniques like average pooling (Li et al., 2024c) and memory compression (Wu et al., 2022; Wang et al., 2023; He et al., 2024; Zhang et al., 2024a; Qian et al., 2024) reduce token volume, they come the of losing details, particularly in temporal and lower-level visual features that are essential for complex question answering.
To address the challenges, we propose ReKV (Retrieve In-context Video KV-Cache), a framework that seamlessly integrates with existing Video-LLMs (Maaz et al.. 2024: Zhang et al., 2024c; Li al., 2024a) without additional training. Our method employs two strategies for aggregating both short- and long-term temporal information. For short-term temporal context, the model adopts causal attention with sliding- window mechanism (Han et al., 2023), where tokens attend only to a limited set of preceding tokens during encoding. For recalling long-term information, we enable dynamic access to any point within the video sequence via retrieval. Specifically, our method retains and reuses past computations (K (KV-Cache) to avoid redundant processing while enhancing long-term reasoning without sacrificing detail. For extremely long videos, KV-Caches can be offloaded to RAM or disk prevent memory overflow.
To ensure real-time time and accurate responses, we retrieve fixed number of KV-Caches relevant to the current question. This design strikes balance between efficiency and accuracy by avoiding the need to process all past frames, while still accessing the most critical information. We experimented with two retrieval methods: one using external CLIP-like models (Radford al., 2021; Zhai et al., 2023) for semantic matching, and another leveraging internal attention weights for faster, more integrated, and potentially stronger retrieval (Xiao al., 2024a; Li al., 2024d).
In summary, ReKV efficiently encodes long video streams, preserves and retrieves in-context KV- Caches to address complex video question-answeringg ing from question-answering into distinct processes, further enhancing efficiency. As shown in Figure (bb, ReKV improves VideoQA accuracy while maintaining stable inference latency and memory usage as frames increase. The remainder of the paper is organized as follows: Section provides an overview of the relevant literature. Section formulates the Streaming VQA task and describes our proposed method in detail. In Section 4, we present ablation studies and compar- isons to validate our approach. Consequently, our approach not only enhances accuracy on long VideoQA benchmarks, including MLVU (Zhou et al., 2024a), QAEgo4Dze (Di & Xie, 2024), EgoSchema (Mangalam et al., 2023), and ActivityNet-QA (Yu et al., 2019), as well as Stream- ing VQA benchmarks (Zhang et al., 2024a), but also reduces inference latency and memory usage.
addition, ReKV separates video encod-
2

---

Published as a conference paper at ICLR 2025
2 STREAMING TASK DEFINITION AND DISCUSSION
This paper considers the problem of streaming video questiion-nsseend StreamingVQAA where model continuously processes video stream and can respond to questions about past visual content any moment. In this section, we formally define the task and outline the design principles for our proposed solution.
Given a video stream y² = [11, U2,...; UT] consisting of 'T' frames and set of N questions Q {q), ................................ gN Streaming VQA aims to answer a question qi at any time step (1 S S T), using only the frames seen up to that point, V = [U1, 02..... vt).
Streaming VQriinngQQAAA VQA vs. Offline StreamingVO2 involves continuously analyzing an incoming video stream and answering questions based on the observed visual content any moment. In contrast, conventional video
Discussion-I:
models (Yang et al., 2022; Maaz et al., 2024; Zhang et al., 2024c: Li al., 2024a) operate in an offline mode, referred to as OfflineVQA. The two paradigms differ in that: 1) Streaming VQA processes continuous video stream, while OfflineVQA handles a predefined video input, and 2) Streaming VQA allows questions to be asked at any point during the stream, whereas Offline processes questions only after the entire video has been viewed. Notably, Offline VQA can be considered a special case of Streaming VQA, where all questions posed after the video is fully processed.
Conventional approaches typically employ a visual encoder (Radford et al., 2021; Zhai et al., 2023; Fang et al., 2023) and projection module (Zhang al., 2024c; Li al., 2023) to process video frames (V"). The output is concatenated with the tokenized question to form a sequence [V,, qi which is then passed to LLMs to predict an answer. However, this approach is impractical due to the high computational cost associated with processing a large number of frames (T).
A common workaround is sparse frame sampling (Maaz al., 2024; Zhang et al., 2024e; Li al., 2024a), but this introduces new problems: (i) loss of critical visual information, leading to incom- plete or inaccurate responses, and (ii) the need to reprocess frames for different questions, since questions asked at different time points require distinct frame samples. This becomes increasingly inefficient T and N grow.
Given these challenges, current Offline methods fall short when applied to Streaming VQA scenarios. Therefore, designing new approach optimized for StreamingVQA crucial to handling video streams more efficiently, enabling real-time question answering and unlocking more interac- tive video analysis applications.
Design Principles for Efficient StreamingVQA. To tackle the aforementioned chal- lenges, we can exploit the causal nature the LLM decoder to avoid redundant computations and strike a balance between accuracy and speed. During attention calculations, causal masking prevents the model from accessing future tokens, ensuring that video tokens are encoded independently of the questions. This allows to decouple video encoding from question-answeree
Discussion-II:
For video encoding, we leverage the KV-Cache optimization to accelerate inference. However, as number of frames grows large, handling the massive number of video tokens becomes increasingly inefficient and may exceed the model's capacity (Han et al., 2023; Xiao et al., 2024b). To address this, we adopt sliding-window window attention mechanism (Han et al., 2023), which limits the attention scope to only the most recent frames.
Regarding question-answerrngg Video KV-Caches stored and can be reused context answer different questions. However, long video sequences produce a substantial amount of KV-Caches, leading to excessive GPU memory consumption, computational overhead, and unnecessary distrac- tions if all are used. To address this, we introduce an efficient retrieval method that selects the most relevant video key-value vectors from the video KV-Caches. These selected vectors then serve as context, enabling efficient and scalable StreamingVQA.
We maintain the original notation for simplicity.
3

---

Published as a conference paper at ICLR 2025
(a) Video Stream Encoding w/ Sliding-window Attention
Video KVACache
Input Frames
Input Question
Layers
Retrieved Video KV
Generated Answer
Offload RAM/ Disk
Local Window
(b) In-context Video KV-Cache Retrieval
(c) VideoQA using Retrieved KV
Compress Calc. Cosine Similarity
Reload GPU
Figure 2: Overview ReKV. We modify the attention mechanism in Decoder-based Vidco-LLMs: (a) The vidco stream is encoded with dliding-windon attention (Equation 1), with out-of-window Vidco KV-Caches offloaded to RAM disk. (b) Upon receiving question, relevant key-value vectors are retrieved based on cosine similarity. with compressed vectors to accelerate retrieval (Equation 2). (c) The retrieved key-value vectors are reloaded onto the GPU and utilized for autoregressive answer generation (Equation 3).
REKV: RETRIEVE IN-CONTEXT VIDEO KV-CACHE
This section introduces ReKV, an approach that integrates seamlessly with Video-LLMM to enable efficient StreamingVQA without requiring additional training. Overall, ReKV efficiently encodes the video stream, maintains its KV-Caches, retrieves relevant caches based on the given question, and uses them for accurate guestion-answering.
We aim enable Video-LLMs, trained on limited frames, to perform Streaming VQA without ad- ditional training. As shown in Figure 2, the proposed ReKV has three components: video stream using the retrieved key-value vectors.
encoding, video KV-Cache retrieval, and question-answerts
Video Stream Encoding with Sliding- window Attention. We encode the vidco stream V in- crementally, processing it chunk by chunk. At each step, the inputs include past key-value vectors P (k), v and the current tokens X (tetir 1 where 1p denotes the lengths of past key-values, and 1x refers to the chunk size. The local key-value vectors within a window 4, can thus be derived as L P The attention calculation then formulated as:
(1) where WQ. Q. WK. and W are the attention layer parameters, La and L, correspond to the key and value vectors L. All video KV-Caches are stored for future retrieval. For extremely long videos, manage memory constraints by offloading KV-Caches to RAM disk, in (Xiao et al., 2024a).
O Attn (WQX, Lt, X. Le, WvX)),
External Video KV-Cache Retrieval. Here, we utilize an external CLIP-like model (Radford et al., 2021: Zhai et al., 2023) to retrieve question-relevant video KV-Cache, primarily assess whether retrieval can enhance VideoQA performance, as demonstrated in Section 4. Specif- ically, CLIP-like model transformers each video frame into vector = fe(v) E R where fe represents the visual encoder, D) denotes the vector dimension. Similarly, the question is encoded as q q fr(g) E R where is the text encoder. We then compute the cosine similarity between the embeddings of frame and question:
bascline to
V [|v| lq
Sim(v, q)
where 7 is learnable temperature parameter. This similarity is calculated at the frame level, rather than at the token level. Alternatively, we can group b consecutive frames into blocks by averaging their frame vectors and then compute block-level similarity scores. Finally, the most relevant video frames or [r/b] video blocks are retrieved. The corresponding video KV-Cache, denoted as R. is subsequently loaded onto the GPU for question-anssering.

---

Published as a conference paper at ICLR 2025
Internal Video KV-Cache Retrieval. Building on recent advancements in handling long sequences with LLMs (Xiao et al., 2024a; Li et al., 2025; Fountas et al., 2025), we further explore using self- attention layers within Video-LLMs for retrieval. Similar to external retrieval, internal retrieval is still performed at the level of video frames or blocks.
During video modeling. the average of the key vectors of frame is computed its representative frame vector: Ny Σ kj E ER'', where N, is the number of tokens per frame, and k, the j-th key vector. To reduce computational overhead, we do not differentiate between attention heads and instead concatenate them into single vector, with D' as the resultant dimension. Similarly, the question vector is computed as q = N₂ Ex 9k E R."", where Ng is the number of tokens in the question, and qu is its k-th query vector. The similarity computation and video KV-Cache retrieval are identical to that of external retrieval, except that is set to I.
Note that, internal retrieval offers several advantages over external retrieval. First, it operates in- dependently within each self-attention layer, allowing different layers to retrieve different video blocks. 3 This allows for broader capture of video context. Additionally, internal retrieval reuses already computed hidden representations and does not introduce extra parameters, which reduces the computational overhead compared to external retrieval.
Using Retrieved KV. The retrieved Video KV-Caches serve as the context for video pression--nsweerig Formally, the attention calculation is formulated as:
O Attn (WoX, RRg, WK RR.. WyX)),
(3)
where X represents either the question tokens or the current token being decoded, and R. and R. the key and value vectors from the context, which includes the retrieved video, question, and previously generated tokens.
Our baseline Video-ILLMs employ Rotary Position Embeddings (RoPE) (Su et al., 2024), a commonly used relative positional encoding method. Our video streaming encod- ing process follows LM-Infinite (Han et al., 2023), where RoPE operates normally within the local window but is constrained by a "distance ceiling" for more distant tokens. For guestion-answering, we do not account for the original positions of the retrieved KV-Caches, as handling unseen dis- tances among tokens presents significant challenges (Han et al., 2023). Instead, we treat these re- trieved tokens as regular consecutive tokens. We also experimented with a static variation from Inf-LLM (Xiao et al., 2024a), where all retrieved tokens are assigned the same position. Our results show that applying standard RoPE to retrieved video tokens leads to better performance, likely due to the importance of capturing temporal information in video comprehension.
Positional Encoding.
4 EXPERIMENTS
4.1 BENCHMARK AND METRICS
MLVU dev mc (Zhou et al., 2024a) is the multiple-choice subset of the MLVU-dev benchmark. It focuses on evaluating the long-form video understanding of MLLMs. The question-ance answer pairs are manually labeled and can be divided into 3 groups: single-detail, multi-detail, and holistic. The evaluation metric Accuracy.
QAEgo4D test-me (Di & Xie, 2024) is the multiple-choice subset of the QAEGo4D-test bench- mark, focusing on guestion-answertte answering in long egocentric videos. It includes annotations marking video segments relevant to each question. The evaluation metric is Accuracy.
EgoSchema (Mangalam et al., 2023) is a diagnostic benchmark for long VideoQA, featuring over 5000 multiple-choice questions and long temporal certificate length. It challenges Al models with long-term understanding, as current statc-of-the-an models achieve significantly lower accuracy compared to human performance.
ActivityNet- (Yu et al., 2019) encompasses human-annotated QA pairs on 5,800 videos derived from the ActivityNet (Caba Heilbron et al., 2015) dataset. This benchmark is designed to assess the
Foor simplicity, we omit the layer index in the above explanation.
5

---

Published as a conference paper at ICLR 2025
capabilities of VideoQA models in long-tern spatiotemporal reasoning. Our evaluation methodol- ogy aligns with that Ideo-ChatGPP) (Maaz et al., 2024), employing G 5-turbo-06113 to judge the accuracy of the open-ended VidcoQA responses.
RVS-Ego and RVS-Movie (Zhang et al., 2024a) Streaming VideoQA bench- marks, constructed using 10 long videos from the Ego4D dataset (Grauman al., 2022) and 22 long videos from the MovieNet dataset (Huang et al., 2020), respectively. These benchmarks fea- ture open-ended questions paired with timestamps, which are initially generated by GPT-4V (OpenAI, 2023b) and GPT- 4 (OpenAI, 2023a), and subsequently re- fined through manual filtering.
Table I:
Summary of the evaluation benchmarks. MC
stands for multiple-choice VideoQA, while OE refers open-ended VideoQA
Benchmark
Duration
#Vidcos
#QA
Тype
MLVU dev-no
12 min
1,242
2,175
MC
QAEGo4D testtme
8.3 min
148
500
MC
EgoSchema AetivityNet-QA RVS-Ego RVS-Movic
min
5,031
5,031
MC
min
800
8,000
60 min
10
1,465
OE
30 min
22
1,905
OE
CGBench
27 min
1,219
12,129
MC
CGBench (Chen et al., 2025a), the multiple-choice subset of CGBench, is designed for clue-
grounded question answering in long videos. It focuses on the ability to retrieve relevant clues for questions, making it an ideal testbed for ReKV.
4.2 IMPLEMENTATION DETAILS
We primarily evaluate our approach by integrating into LLLVA-OV-O.55 5B and LLaVA-OV-7B (Li al., 2024a), chosen for their simplicity and strong performance. In the Appendix, we conduct experiments with several other Video-LLMs as further validations.
All experiments conducted NVIDIA A100 (80GB) GPUs with FP16 precision. For video modeling, we process the video stream at 0.5 FPS, in line with GPT-4o's testing on MLVU (Zhou al., 2024a). The local window size set to 15K. For external video KV-Cache retrieval, we use S040 OM (Zhai et al., 2023) as the retriever. For internal KV-Cache retrieval, we set the block size (b) to and the number retrieved frames (r) 64 by default, with further hyper- parameter variations explored in Section 4.3.
Unless otherwise specified, ReKV refers to the use of internal video KV-Cache retrieval.
4.3 ABLATIONS
In this section, conduct ablation studies on the effectiveness of in-context retrieval, number of retrieved frames, and the block size.
Effectiveness of In-context Retrieval. The ex- as presented
Ablation study on QAEGo4D "Oracle Retrieval" refers to scenario where the an- notated, question-relevant video segments are used as input, with uniform sampling of up to 16 frames. This setup, by definition, has 100% recall and defines the upper- VideoQA performance.
Table 2:
periments on QAEGO4D Table 2, demonstrate the effects of various re- trieval methods on VideoQA accuracy and re- call. The recall metric, defined the percentage of question-relevant video frames retrieved, ex- hibits a strong positive correlation with VideoQA performance: higher recall consistently leads better accuracy. Uniform Sampling, which sparsely selects frames, achieves the lowest re- call and, consequently, the poorest VideoQA ac- curacy. In contrast, Oracle Retrieval, with per- feet recall, delivers the highest VideoQA accu- racy, significantly outperforming Uniform Sam- pling. While External and Internal Retrieval fall short of Oracle-level precision, both surpass Uni- form Sampling. with Internal Retrieval excelling due to its higher recall.
Retrieval Method
VideoQA Acc.
Recall
LLaVA-OV-O 00-0.55
Uniform Sampling
42.6
6.1
External Retrieval
48.0
58.1
Internal Retrieval
50.0
63.4
Oracle Retrieval
52.0
100
LLaVA-OV-
Uniform Sampling
53.0
6.1
External Retrieval
54.2
58.1
Internal Retrieval
56.0
70.5
Oracle Retrieval
64.4
100

---

Published as a conference paper at ICLR 2025
65
Accuracy (%)
Accuracy (%)
65
60
QAEgo4D
MLVU
55
60
50
8
16
32
64
1
2
8
Retrrieved Frames
Block Size
(a)
(b)
Figure 3: Ablation study of retrieval hyperparameters: (a) number of retrieved frames and (b) number of frames per retrieval block. Experiments conducted with LLVVB-OV-/B.
Table 3: A Ablation study MLV Uew The experiments are based LLaVA-OV-7B.
Single Detail
Multi Detail
Holistic
Task
Avg.
Needle
Ego
PlotQA
Order
Count
Topic
Anomaly
Uniform Sampling
74.1
59.7
69.8
45.9
32.0
87.9
72,0
64.7
External Retrieval
78.6
69.6
71.6
40.2
37.9
84.5
63.0
66.3
Internal Retrieval
75.8
66.6
76.3
45.2
36.9
90.1
74.5
68.5
The MLVU benchmark (Zhou al., 2024a) encompasses three types of VideoQA tasks: Single Detail requires identifying a single critical plot within a long video, Multi Detail necessitates the in- tegration of multiple plots, and Holistic demands comprehensive understanding of the entire video. This makes MLVU an ideal platform for evaluating our in-context retrieval method. As shown in Ta- ble 3, both External and Internal Retrieval enhance the overall VideoQA accuracy over the Uniform Sampling baseline. The enhancements are most pronounced in Single Detail tasks, demonstrating that ReKV effectively retrieves question-relevant video context. Furthermore, Internal Retrieval sig- nificantly outperforms External Retrieval in Holistic tasks, likely due to its ability to capture broader context and leverage the Video-LLM's video modeling capabilities, discussed in Section 3.
Number of Retrieved Frames. We fix the block size (b = 1) and evaluate the impact of varying the numbers retrieved frames (r {8, 16, 48, 64, 80)) on the QAEGo4D and MLVU bench- marks. As illustrated in Figure 3(a), increasing the number of retrieved frames generally improves VideoQA accuracy, as it implies capturing more relevant visual context. However, on MLVU, this improvement platcaus more frames retrieved since the additional irrelevant information hin- ders the subsequent anestion-answeriig process. Additionally, retrieving more frames increases the computational overhead of the question-answeriing stage, further slowing down inference.
Retrieval Block Size. When processing video streams, we group b consecutive frames into blocks for block-level retrieval. For this experiment, we fix the number of retrieved frames at 64 and evaluate different block sizes (b E 1, 4, 8, 16). With a fixed r, larger block sizes result in fewer, more concentrated retrieved blocks. Figure 3(b) shows that increasing block size negatively affects accuracy on MLVU, while performance on QAEGo4D remains relatively stable. This suggests that MLVU tasks benefit from retrieving more dispersed visual cues, aligning with its design of multi- detail and holistic tasks (Zhou et al., 2024a). In contrast, QAEGO4D primarily relies on a single relevant clip per question (Di & Xie, 2024).
4.4 OFFLINE VIDEO QUESTION-ANSWERING
Streaming video understanding is relatively under-explored area, with limited Streaming VQA benchmarks available (Zhang et al., 2024a). As discussed in Section 2, Offline can consid- ered special case of Streaming VQA. Thus, first evaluate method in the offline setting using four widely adopted form VideoQA benchmarks, comparing our results against state-of- the-art VideoQA methods. A summary of these benchmarks be found in Table 1.

---

Published as a conference paper at ICLR 2025
Table 4: Offline video question-answering four long-form benchmarks. "Acc." denotes accuracy, and
"Score" the open-ended answer rating by 1.tp-3-tturr-0065 on a scale from
MLVU
QAEGo4D
EgoSchema
ActivityNel-QA
Method
dev Acc.
test Acc.
Acc.
Acc.
Score
GPT-4V (OpenAI, 2023b)
49.2
57.0
GPT-4o (OpenAI, 2024)
64.6
Gemini-1.5-HHe (Team et al., 2023)
65.7
55.3
Gemini-1.5-Pro (Team et al., 2023)
72.2
57.5
Video-ChaiGPP-77B (Maaz al., 2024)
31.3
LLaMA-VID-7B et 2024c)
33.2
47.4
3.30
MiniGPP4--Vddeo777 (Ataallah et al., 2024)
44.5
44.3
3.35
Video-LLaWA--7B (Lin al., 2024)
47.3
LongVA-7B (Zhang et al., 2024b)
56.3
50.0
VideoStreaming (Qian et al., 2024)
44.1
Flash VSree-SSre (Zhang al., 2024a)
50.2
38.2
38.1
51.9
3.40
LLaVA-OV.0.5B et al., 2024a)
53.2
42.6
29.6
30.5
3..02
+ReKV (0.5 FPS - 64 Frames)
56.1 (+2.9)
50.0 (+7.4)
31.0 (+1.4)
52.1 (+1.6)
3.15 (+.13)
LLa (Li et al., 2024a)
64.7
52.8
59.8
56.6
3.29
+ReKV FPS FPS 64 Frames)
68.5 (+3.8)
56.0
(43.2)
60.7 (+0.9)
60.4 (+3.8)
3.52 (+.23)
Table 5: Streaming benchmark results, All methods are tested under identical conditions. "Vidco Enc." frames encoded per second. "Lateney" is measured from question input response completion. "GPU" indicates peak GPU memory usage, and "KV-Cache" refers the video KV-Cache size offloaded per hour.
RVS-Ego
RVS-Movie
Running Speed
Memory Usage
Retrieval Method
Acc.
Score
Acc.
Score
Video Enc.
Latency
GPU
KV-Cache
Flash-VStream- VStream-7B
57.3
4.0
53.1
3.3
14 FPS
2.4s
GB
Lia VX-OV-77
Uniform Sampling
56.2
3.7
43,0
3.3
2.9s
21 GB
External Retrieval
62.4
3,9
53.6
3.5
11 FPS
5.8s
S5 GB
18.8 GB/h 18.8 GB/h
Internal Retrieval
63.7
4.0
54.4
11 FPS
3.38
GB
TLaVA-5V- 07-6..5
Uniform Sampling
51.8
3.7
37.2
3.2
2.5s
GB
External Retrieval
54.1
3,8
44.7
3,4
17 FPS
4.18
37 GB
4.0 GB/h
Internal Retrieval
54.7
3.9
44.6
3.4
17 FPS
1.6s
19 GB
4.0 GB/h
As shown in Table 4, our proposed ReKV always enhances the performance L. L.LVVA-V-O,, 0. and LLaVA-OV-75 without additional training. Notably, two memory-based Streaming models (VideoStreaming (Qian et al., 2024) and Flash- VStream (Zhang et al., 2024a)) by a large margin. While the base model already demonstrates strong performance, and do not claim credit for this achievement, method integrate seamlessly with Video-LLMs, benefiting from their ongoing advancements.
outperforms
4.5 STREAMING VIDEO QUESTION-ANSWERING
We then evaluate our method on the streaming setting using the RVS-Ego and RVS-Movie bench- marks. During video stream modeling, questions are input immediately after their annotated end timestamps and answered based on preceding video content.
Question-answertt Performance. Table 5 presents the Streaming VQA performance. Both ex- ternal and internal retrieval methods significantly outperform the uniform sampling baseline. Ad- ditionally, approach enables -LaVA-OV-1 7B to surpass Flash-VStream (Zhang al., 2024a), demonstrating ReKV's effectiveness for the StreamingVQA..
Running Speed and Memory Usage. We also examine the running speed and memory usage under controlled conditions. Specifically, 1 -hhour, 1080P video from RVS-Ego with 100 seattered questions is used. Each question is padded 64 tokens, and the generated answers fixed 128 tokens in length. The video frames are pre-extracted at 0.5 FPS (1,800 frames in total) and streamed to the Video-II LLM frame by frame.
8

---

Published as a conference paper at ICLR 2025
NE-EA
NECElR
Question: What tool did open the carton with? Prediction: You used small knife.
Question: What did take out of the fridge? Prediction: You took out bottle of water
Figure 4: StreamingVQA qualitative examples. The example is drawn from the QAEGO4D benchmark. The video stream is processed frame by frame. and mark the timestamps which questions posed. D and indicate the relevant video contexts that support answering these questions.
As illustrated Table 5, both retrieval methods maintain high video encoding speeds, with LLaVA-OV-7B achieving 11 FPS and LLavA-Ov-O. 5B achieving 17 FPS. Moreover, KV- Cache offloading remains manageable, with LLaVA-OV-7B at 18.8GB/h and LLavA-OV-O, 5B 4.0GB/h (see appendix for more details). External retrieval, however, introduces higher latency and GPU memory usage due to additional computations in the external retriever, whereas internal retrieval significantly reduces both. Figure has also demonstrated that latency and GPU mem- ory usage remain stable as more frames are processed. Flash-VStream also shows good efficiency. However, only maintains a relatively small memory footprint (681 tokens) (Zhang et al., 2024a), leading to potential information loss when dealing with extremely long videos.
answering. Our approach continuously processes video streams while responding to questions posed at different timestamps. To improve efficiency, stores and retrieves relevant video K KV-Caches as contextual information for answering these questions.
Qualitative Examples. Figure 4 presents an example of streaming video
We provide additional implementation details and experimental results in the Appendix.
RELATED WORK
LLMs for Video Understanding. In recent years, there has been a surge of interest in leveraging Large Language Models (LLMs) for video understanding, leading to the development of several innovative approaches (Maaz et al., 2024; Zhang et al., 2024c; Li al., 2024a). These models typically use a Vision Encoder to extract video features, followed by mapping step with Linear Projection, MLP, or Q-Former (Li et al., 2023). The mapped features are combined with textual data and fed into large language models (LLMs) to generate text output. These models have relatively simple architectures, requiring less training data and computational resources, yet they achieve strong performance short video understanding benchmarks (Xu et al., 2017; Xiao al., 2021; Li et al., 2024b). However, they employ sparse sampling or token compression techniques reduce the number of tokens, which can result in significant information loss when dealing with longer or more content-rich videos. As a result, they are not well-suited for long video understanding or streaming video understanding.
Long Video Understanding. A central challenge in long video understanding is effectively com- pressing the information from lengthy videos. Many approaches use language bridge, condens- ing videos into dense captions (Zhang et al., 2023; Islam et al., 2024; Zhou et al., 2024b). While this achieves good results in some cases, compressing video content into text often leads to the loss of crucial visual details. Besides, as pioneering approach to streaming video understand- ing, VideoLLM-Online (Chen et al., 2024) employs a data-centric methodology by interleaving video and text during training, In contrast, our approach is training-free, allowing seamless integra- tion with various existing Video-LLMs to extend their StreamingVQA capabilities. Additionally, VideoLLM-Onling retains only single token per frame to handle long videos, which may result in visual information loss. Our method preserves complete visual information and leverages In- In-Context KV-Cache Retrieval enhance efficiency.

---

Published as a conference paper at ICLR 2025
Another line of research focuses compressing long videos into memory bank (Wu et al., 2019; 2022: Wang et al., 2023). MC-ViT (Balazevic et al., 2024) adapts existing pretrained video trans- formers by fine-tuning them to attend to condensed visual memories. It relates closely to the token- pruning, merging, and memory-based video understanding methods. In comparison, we propose a training-free method specifically tailored to the StreaminVQAa task. Incorporating MC-ViT into the Streaming VQA task could be an interesting avenue for future research, and we acknowledge its po- tential in this domain. This approach has been integrated into Video-LLMs for streaming video un- derstanding. as shown in works like VideoStreaming (Qian et al., 2024) and Flash- VStream (Zhang al., 2024a). These methods dynamically update the memory during video processing and utilize it for downstream tasks. Despite their innovation, major limitation of these methods is their failure to account for video length and information density, especially when using a fixed memory size. For example, Flash-VStrean compresses both 10-second clips and hour-long videos into the same 681 tokens. Furthermore, these methods lack interpretability. making it difficult to determine how much information is being compressed into the memory whether relevant video information is being accurately retrieved during downstream tasks.
In pursuit of greater interpretability in long video understanding, methods such GroundVQA (Di & Xie, 2024) and GeLM (Chen et al., 2025b) advocate for localizing relevant video clips while responding to user queries. Drawing inspiration from these, this work refrains from excessively condensing video information. By harnessing the causal capabilities of Video-LL.Ms, it preserves the entire Video KV-Cache, allowing for the retrieval of relevant information when required. This strategy effectively mitigates the substantial loss of video content while improving interpretability.
Long Context Handling for LLMs. Handling long text sequences in LLMS has been a major challenge due to high computational and memory costs, leading to training constraints on shorter sequences. Techniques like StreamingLLMM (Xiao et al., 2024b) and IM-Infintte (Han et al., 2023) use sliding window attention to process long sequences incrementally, but discard distant tokens, limiting the model's ability to capture long-range dependencies. Recent approaches (Xiao et al., 2024a; Li et al., 2025; Fountas et al., 2025) address this by storing and retrieving previously com- puted KV-Caches, enabling better recall of distant contexts.
Retrieval-Augmenned Generation. Retrieval-augmmente generation (RAG) combines retrieval mechanisms with generative models to enhance performance across various NLP tasks by incorpo- rating external knowledge (Guu et al., 2020; Lewis et al., 2020; Borgeaud et al., 2022) and improving performance in rision-languaa
tasks (Xu et al., 2024). In-context retrieval, recently proposed for handling long inputs (Ram et al., 2023), retrieves information from the input document itself rather than an external knowledge base. In-context KV-Cache retrieval further improves efficiency by pre- encoding long documents, avoiding redundant encodings, and leveraging the L.M''s own retrieval capabilities for faster, more effective performance.
6 CONCLUSION
In conclusion, this paper introduces training-free approach, ReKV, designed to enhance the ef- ficiency of Video Large Language Models (Video-LLMs) for streaming video question-answering (StreamingVQA), Unlike conventional video question-answent cess entire videos before answering, ReKV enables rapid, real-time responses. By employing a sliding-windon attention mechanism, it ensures that the model only considers subset of previous frames while encoding the video stream, significantly cutting down on computational demands. To retain key video context, we developed an in-context KV-Cache retrieval method that efficiently stores and reloads key-value vectors that relevant for each query. This targeted retrieval strategy, separate processes
(VideoQA) systems that must pro-
combined with the ability to perform video modeling and question-answering and GPUs, results in a highly efficient streaming VideoQA system. Extensive experiments show that ReKV not only surpasses existing VideoQA models in performance but also enhances their practicality for real-world streaming applications.
This work is supported by National Key R&D Program of China (No. 2022ZD0161400), We thank Yikun Liu for discussions and conducting experiments on CGBench.
Acknowledgements.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. In CVPR Workshop, 2024.
Ivana Balazevic, Yuge Shi, Pinclopi Papalampidi. Rahma Chaabouni, Skanda Koppula, and Olivier Henaff. Memory consolidation enables long-context video understanding. In ICML, 2024.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmaan,, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video under- standing. In ICLR, 2025a.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wa Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video. In CVPR, 2024.
Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. In AAAI, 2025b.
Shangzhe Di and Weidi Xie. Grounded question- answering in long egocentric videos. In CVPR, 2024.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Zafeirios Fountas, Martin Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lam- pouras. Haitham Bou- commAmmar, and Jun Wang. Human-liks episodic memory for infinite context llms. ICLR, 2025.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kristen Grauman, Andrew Westhury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. ICML,, 2020.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint crXVv:2308.1617, 2023.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao. Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: augmeneed large multimodal model for long-term video understanding. In CVPR, 2024.
Lianghua Huang. Xin Zhao, and Kaigi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. TPAMI, 2019.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In ECCV, 2020.
11

---

Published as a conference paper at ICLR 2025
Md Mohaiminul Islam, Ngan Ho, Xitong Yang. Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. CVPR, 2024.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR, 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv: 705.06950, 2017.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gener- ation for inowledge-intensse nlp tasks. In NeurlPS, 2020.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 03326, 2024a.
Jingyao Li, Han Shi, Xin Jiang. Zhenguo Li, Hong Xu, and Jiaya Jia. Quickllama: Query-aware inference acceleration for large language models. In COLING, 2025.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-trainin pre-training with frozen image encoders and large language models. In ICML, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: comprehensive multi-modal video understanding benchmark. In CVPR, 2024b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vidd An image is worth 2 tokens in large language models. In ECCV, 2024c.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. In NeurlPS, 2024d.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by aligmment before projection. In EMNLP, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Vidco-chatgpt:
Towards detailed video understanding via large vision and language models. In ACL, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. NeurIPS, 2023.
Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Trackingnet:
A large-scale dataset and benchmark for object tracking in the wild. In ECCV, 2018.
OpenAl. Gpt-4, March 2023a. gipt--systend card. pdd.
URL htttps:/cddn.or htt.anaaaccooopppps/s//aa
OpenAl. Gpt-4v, September 2023b.
URL httpp:///ppeaai.com/
gpt- 4v- sor-systemmaoe cardm/.ar/......
OpenAI. Gpt-4o, May 2024. URL
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang- Streaming long video understanding with large language models. In NeurlPS, 2024.
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. 2021.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Leyton-Brown, and Yoay Shoham. In-context retrrevallaugmmenedd language models. In ACI., 2023.
12

---

Published as a conference paper at ICLR 2025
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.
2023.
Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-andlanticiatton trans- former for online action understanding. In ICCV, 2023.
Chao-Yua Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019.
Yuao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit:
multiscale vision transformer for efficient
long-term video recognition. In CVPR, 2022.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training- free long-context extrapolation for Ilms with an efficient context memory. In ICML Workshop, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICIR, 2024b.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa:next phase of question- answering to explaining temporal actions. In CVPR, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangman He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval- augmented egocentric video captioning. CVPR, 2024.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurlPS, 2022.
X Ye. Calflops: A flops and params calculate tool for neural networks in pytorch framework, 2023.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-tinining. In ICCV, 2023.
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang. Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple I1m framework for long-range video question-answering. arXiv preprint 1rXi::2312.17233, 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.0 280058085, 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.. 16852, 2024b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024c. URL github. tatlb.io/bll hoobooog///2------
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tięjun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint arXiv:2406.0422 2024a.
Xingyi Zhou, Anurag Amab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Na- grani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, 2024b.
13


Reviews:
[{"review_id": "1EXNB1dNX0", "reviewer": "Reviewer_AcRC", "summary": "This paper presents video KV caches to make a streaming video question and answering video-LLMs in a training-free approach.\nWhile it uses a sliding-attention mechanism to aggregate short-term temporal context, video KV caches and the proposed retrieval method are introduced to long-term temporal context.\nThis method shows efficiency with LLaVA-OV in several benchmarks.", "questions": "- How about GFLOPs on streaming VQA?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- This paper outperforms existing video-LLMs on long-form benchmarks.\n- This paper is easy to follow.\n- Ablation study shows the validity and impact of the retrieval methods.", "weaknesses": "- My major concern is the novelty. Already, many LLM systems reduce the context-processing delay by using the KV cache of the context. This paper is also built on LLMs, while it is coupled with a video encoder. It's hard to find the specialty for the video streaming system. The causal attention and retrieval system with cosine similarity are also not new.\n\n- Implementation with only one design (LLaVA-OV) with different sizes is limited to prove the generality of the proposed method.\n\n- There are many recent methods to reduce the memory of KV caches such as adaptive KV cache (ICLR'24) and Keyformer (Muhammad Adnan et al., arxiv'24), compared to these methods, is the proposed attention and search method more effective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mbFbexLE6m", "reviewer": "Reviewer_t8DB", "summary": "This paper introduces ReKV, a novel, training-free approach designed to enhance existing Video-LLMs for StreamingVQA. Traditional VideoQA systems struggle with long videos due to the need to process entire videos before responding and repeating this process for each new question. ReKV addresses these challenges by storing processed video key-value caches (KV-Caches) in RAM or disk to prevent information loss. ReKV introduces retrieval methods—both external (using models like CLIP) and internal (leveraging the Video-LLM's parameters)—to fetch only query-relevant KV-Caches, enhancing efficiency and accuracy in question-answering.\nExperiments conducted on various benchmarks, including MLVU, QAEGO4DMC, EgoSchema, ActivityNet-QA, and StreamingVQA (RSV-Ego and RSV-Movie) datasets, demonstrate that ReKV improves VideoQA accuracy while maintaining stable inference latency and memory usage as the number of frames increases. The method enables real-time interaction and long-term context for StreamingVQA tasks.", "questions": "- Have the authors tested ReKV with other Video-LLMs besides LLaVA-OV? Demonstrating the integration and performance of ReKV with different architectures (e.g., VideoChatGPT…) would confirm its general applicability and ease of integration.\n- Table 5 shows that the internal KV-Cache retrieval reduces computational overhead compared to external retrieval. However the “internal retrieval” retrieves KV-Caches for each attention layer independently while it is only done once for the “external retrieval”. How do you explain that the internal is faster?\n\n\nMinor: \nIn practice, how does ReKV manage KV-Cache storage for extremely long video streams, such as surveillance footage that can run continuously for many hours or days? Are there mechanisms in place to prevent unsustainable increases in cache size, and how does this impact performance and resource requirements?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper presents a novel and simple, training-free method that extends the capabilities of existing Video-LLMs for StreamingVQA. By integrating a sliding-window attention mechanism and efficient KV-Cache retrieval, ReKV addresses the challenges of processing long video streams in real-time.\n\n- The methodology is well-motivated and thoroughly explained. The paper clearly defines the StreamingVQA task, differentiates it from traditional OfflineVQA, and outlines the specific challenges involved. The proposed solutions are detailed and logically sound.\n\n-  The paper is well-organized and clearly written with figures to support the method explanation.\n\n- ReKV significantly improves efficiency and accuracy over existing VideoQA models on multiple benchmarks. The ability to handle long video streams in a streaming fashion has practical importance for real-world applications. The training-free nature of ReKV can potentially enhance its applicability across different Video-LLMs.", "weaknesses": "Currently, a major limitation of the method is that the method is that it is only evaluated on LLaVA-OV models (0.5B and 7B). Although these models are strong baselines, the applicability of ReKV to other Video-LLMs is not demonstrated. Evaluating ReKV on a broader set of models would strengthen the claim of its versatility and general applicability.\nI’ll be happy to increase my score if that limitation is addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EABgkTuzVt", "reviewer": "Reviewer_g3YS", "summary": "The authors present ReKV (Retrieve In-context Video KV-Cache) for streaming video question-answering. The authors incorporate a sliding-window attention mechanism on existing VideoLLMs, introduce a retrieval method that leverages an external retriever or the parameters within Video-LLMs to retrieve only queryrelevant KV-Caches. The authors evaluates the model on both long video QA and streaming videoqa.", "questions": "All weakness, and:\n\n1. The citation format is inconsistent over the paper, the authors should unify this format. \n\n2. Since the model is claimed to integrate seamlessly with existing Video-LLMs, is it possible to apply to a bigger VideoLLM backbone, like around 70B scale? \n\n3. In Table 4,  for offline video question-answering, could the authors elaborate more on the baseline setting of the LLaVA-ov model, like frame numbers? Also, could the authors compare the efficiency of the proposed ReKV compared to the original LLaVA-ov model in the table?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The number of evaluation benchmark for the proposed method is adequate. \n\n2. The improvements against standard VideoLLMs are substantial.", "weaknesses": "1. Lack of fair comparison against existing memory-based models (including VideoStreaming and Flash-VStream). It would be better if the author could provide results for ReKV and previous memory-based models under the same VideoLLM backbone to show the effectiveness of the proposed method, for both long video benchmarks and streaming video benchmarks. \n\n2. Missing related works. The authors should discuss the novel contribution compared to the paper VideoLLM-online[1], MC-ViT[2]. \n\n3. The “External Video KV-Cache Retrieval” is confusing, do the authors mean selecting the keyframes using the query information via the CLIP-based models (like a cross-modal matching)? This is already investigated by a number of works, including ATP [3], SeViLA[4] and so on.  It would be if for the authors to clarify how \"External Video KV-Cache Retrieval\" differs from or improves upon the keyframe selection methods. \n\n[1] VideoLLM-online: Online Video Large Language Model for Streaming Video\n\n[2] Memory Consolidation Enables Long-Context Video Understanding\n\n[3] Revisiting the \"Video\" in Video-Language Understanding\n\n[4] Self-Chained Image-Language Model for Video Localization and Question Answering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZtlHcPBsX", "reviewer": "Reviewer_3gu5", "summary": "The paper introduces ReKV, a novel, training-free approach designed to enhance the efficiency of Video Large Language Models (Video-LLMs) for streaming video question-answering (StreamingVQA). Unlike traditional VideoQA systems that process entire videos before answering queries, ReKV processes video streams in real-time, allowing for prompt responses. The method employs a sliding-window attention mechanism to reduce computational overhead and uses a KV-Cache system to store and retrieve relevant video information efficiently. The approach separates video encoding and question-answering into distinct processes, enhancing efficiency. The paper demonstrates the efficacy of ReKV through comprehensive experiments, showing improvements in accuracy, latency, and memory usage over existing models.", "questions": "1.\tGeneralizability of method: Since ReKV is a training-free method, can it be integrated with models other than LLaVA-OV? Are there any experimental results?\n2.\tScalability: How does ReKV scale with increasing video length and complexity? Are there any observed limitations when dealing with very high-resolution videos or videos with a high frame rate?\n3.\tImplementation details: What is the hyperparameters of external retrieval?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1.\tEfficiency: The sliding-window attention mechanism and KV-Cache retrieval significantly reduce computational overhead and memory usage.\n2.\tReal-Time Processing: The method allows for real-time responses to queries, making it highly practical for applications like surveillance and live broadcasts.\n3.\tComprehensive Evaluation: The paper provides extensive experimental results, demonstrating the effectiveness of ReKV across multiple benchmarks.\n4.\tSeamless Integration: ReKV integrates seamlessly with existing Video-LLMs without requiring additional training, making it easy to adopt.", "weaknesses": "1.\tWriting Quality: The organization of this paper could be improved. It is not appropriate to place ablation study before main experiments. Sec 2.1 Task definition and discussion should not be a part of Method. This part is too repetitive of the discussion in the introduction.\n2.\tCitation format: In Table 4 Line 391 there may be a misleading citation of Video-LLaVA-7B, pointing to the same reference of Video-ChatGPT-7B.\n3.\tLack explanation: The term \"oracle retrieval\" from Table 2 and Line 305 is difficult for readers to understand. How is the “recall” metric calculated? How can it be 100?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)