Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
Authors: Vaggelis Dorovatas
Soroush Seifi
Gunshi Gupta
Rahaf Aljundi
Venue: NeurIPS 2025 poster ()
Keywords: streaming video understanding
vlms
training-free
memory
Primary area: applications

Abstract: Video Large Language Models (Video-LLMs) excel at understanding videos in-context, assuming full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95\% of unimportant visual tokens with minimal performance loss; 2) Hierarchical selection of tokens combined with natural language understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.

Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
Vaggelis Dorovatas 1.2
Soroush Seifi
Gunshi Gupta
Rahaf Aljundi'
Toyota Motor Europe
Archimedes RU, Athena RC 3 University of Oxford
Abstract
Video Large Language Models (Video-LLMs) excel at understanding videos in- context, provided they have full access the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free free approach compatible with standard Video-LLMss leveraging three key concepts: 1) LLM--iformee selection of visual tokens identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-
selection allows us to discard up to 95% unimportant visual tokens with minimal performance loss; 2) Recurrent processing past selected tokens to generate temporally coherent understanding of each processed clip: 3) Caption-baset question answering for lightweight and accurate responses. Our method achieves offte--f-hhearr performance on streaming video benchmarks, striking balance between efficiency and effectiivneess
Introduction
Efficient and effective video question answering and understanding are crucial for deploying Large Language Models (LLMMs) as intelligent assistants in domains requiring continuous visual input comprehension. This capability essential for applications such autonomous driving, surveil- lance, healthcare, and entertainment, where dynamic visual information must be understood in real-time. Current video understanding research with VLMs (3117 typically processes entire videos in-context, presenting densely sampled frames alongside text queries to generate responses, with common training approaches focusing on text generation tasks like video captioning or question answering. While effective for short clips, this brute-force approach faces critical limitations with longer videos: computational costs escalate as visual tokens multiply, and exceeding context length limits forces sparse sampling that risks information loss. These challenges become particularly pronounced in streaming scenarios where continuous visual input renders full-video processing unsus- tainable. This necessitates efficient compression techniques that selectively process frames, condense visual information, and maintain easily retrievable memory structures for effective query response.
Recent efforts have focused on compressing the information from short video clips, moving away from the brute-force approach processing the entire video single pass, and thus extending model capabilities handle long video understanding, These approaches include methods that either compress only the visual information (before the LLM) encoded by a vision encoder [38] 12 [0], store only textual descriptions of short clips M. and retrieve only those relevant to the input
*First two authors provide contracted services for Toyota. * *Correspondence: vdorovatas @ hotmail.gr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generated Descraption
Anmer: there three pocale the houre, the shild
sper
Layer
LLM
Sindlarity
Layer
Aotention scores
It may angle are
Layer
Intor space in depicced with
lisatery tokane S
Current visuml clsp takens
X;
Instructton tokens X,
Accumulated text descriptions
Figure 1: Pipeline of rLiVS: (Left): Long videos are broken down into short visual clips which are processed streaming fashion. A short clip pre-pended with selected visual tokens from past clips in the stream to generate textual description. History is accumulated by selecting subset of tokens from the short clip based on attention scores. (Right): Given query on the past video, the accumulated textual descriptions over long video are compared to the query embedding and the most similar subset is input to the LLM (within the videoLLMM for question-answeriin
query, Another line of work involves the LLM in video processing by generating compressed representations in the form of summarization tokens, like VideoStreaming [24], or storing K V-cache during inference, like ReKV (9) effectively combining the two modalities. While all these approaches mark important progress, they face amplified challenges in streaming settings, where past frames
cannot be revisited and inputs can span hours. Training-bar based approaches 22 38
face extrapolation
issues with arbitrarily lengthy videos, and scaling their training can be computationally prohibitive or suffer from data scarcity and annotation issues long videos. Training- approaches like ReK while avoiding retraining, rely storing large K KV-caches, which becomes memory-intensive and introduces potential redundancy in streaming, ultimately affecting overall performance. Conversely, clip-wise caption-only methods (e.g., Goldfish (I) offer efficiency but lack continuity, making difficult to track entities and reason across clips, thus hindering holistic understanding.
In this work, we propose an efficient, training- free, and video L.M--agnotic solution for streaming video question answering. Our approach processes long videos (offline or online) by splitting them into short clips and focuses on three key aspects, integrating visual perception with language under- standing for holistic video comprehension: 1) Compressing video information avoid redundancy and inefficiencies in memory usage by selecting small set of key visual tokens per short clip, 2) Accessing past selected visual tokens when processing new short clips, enabling visual information sharing across short clips and enhancing overall video comprehension, and 3) Performing text-based retrieval and answering, harnessing LLMs' proven strength in reasoning over extended contexts (such multi-clip captions).
Specifically, during the processing of each short clip (i.e., generating a caption), we draw inspiration from cognitive neuroscience, particularly the interplay between attention and memory (6]. Given memory's s limited capacity [23 M. attention is key selective encoding 6 Accordingly. instead of randomly sampling visual tokens, we retain those that received the highest attention from the LLM, treating them compressed memory trace of the clip. Furthermore, building on the idea that past experiences shape current attention [6, we reuse previously selected tokens when processing subsequent clips. These past tokens are passed forward with new inputs, allowing memory to guide attention and support context-aware understanding. Inspired by the brain's 's recurrent visual processing [15), we implement a simple FIFO memory: newly selected tokens are prepended, and the oldest are discarded once the context limit is reached.
Finally, when answering a question about past events in a video stream, we retrieve the top K captions based on the similarity of the question to the previously stored captions of short clips. This approach is motivated by the efficiency of storing captions alone and empirical evidence suggesting that long video reasoning is more effective with representative text. This approach enables us to answer questions based on the stored captions, rather than reprocessing the previous visual input, which is common practice in current works [38 [0]
Figure Tiaureeelllseee Iillsstaaees our method.

---

Our approach can be applied to any video-LLM pretrained on short clips, enabling long video understanding without additional training and maintaining a low memory footprint. We evaluate our method on three offline and streaming benchmarks, achieving state-of-the-art performance with significantly lower memory requirements. The simplicity and strong performance of our design set a robust baseline for online long-video question answering.
Our contributions are as follows:
We propose Recurrent LLM-informed Visual Selection (rLiVS), simple, training-free approach for long video understanding and question answering.
Our approach agnostic to the Video-LLM architecture and does not require any external modules.
We achieve state-of-the-an performance with significantly lower memory requirements.
In the following, we discuss the closely related work in Section and present our method in Section We evaluate our method and ablate the design choices in Section 4 and conclude in Section 5
2 Related Work
Recent video- language models have achieved strong performance on short- form video tasks by aligning visual features with text using pretrained vision-language transformers 30 [6 19]. These models typically sample small number of frames from each video and encode them into visual tokens using a vision backbone (c.g., VIT), which are then fed into a language model either directly or via cross-modal fusion. However, the number of visual tokens grows linearly with the number of frames and spatial patches- e.g. sampling 32 frames with a ViT-base backbone (16x16 patch size) yields over 1,000
Video-Languae Models:
making it prohibitively expensive process long videos end-to-end [34] 29]. As a result, these models are limited to short clips (typically under 30 seconds), lack persistent memory, and cannot capture dependencies across longer temporal horizons without substantial computational and memory overhead.
Offiine long-video understanding: To overcome the limitations of short-clip models, several offline approaches have been proposed for long video understanding by introducing mechanisms to compress or summarize information across time. Some methods focus on compressing only the visual modality, either by keyframe selection, feature pooling, or storing tokenized visual memory [32 34 28 [O].
Others rely solely textual abstraction, converting short clips into natural language summaries that are later retrieved and composed to answer queries ①. More recently, hybrid methods like VideoStreaming (4 train auxiliary models to generate trainable summarization tokens that condense the content of each clip into a compact representation, which is stored and retrieved during inference. While effective, these methods often require significant offline training to align the summarization with downstream tasks, are tied to specific pretraining pipelines, and typically process each clip independently limiting their ability to build hierarchical or persistent memory across long videos.
Streaming and streaming video understanding aims to process incoming video frames incrementally, enabling real- time or low-latency inference over long temporal horizons. -augmeetd methods maintain external memory stores of past embeddings and retrieve relevant context on-the-fly based on the current input, avoiding full- selleneqeece reprocessing 338. Similarly, MeMViT (35) incorporate visual memory modules they often rely on fixed-length context windows or soft attention mechanisms, which scale poorly or suffer from memory drift over time. In the same direction of visual memory, Video-XI. (27 introduces a custom-traines Video-I.IM architecture that learns to compress and retain visual information over time through supervised training. However, its performance tightly coupled to the specific model architecture and the distribution of video lengths and compression strategies seen during fine- tinettunigg Recently, ReKV [9] explored streaming video question answering by storing full decoder key-value caches and retrieving relevant context either through internal attention-basee mechanisms or external CLIP-based similarity, demonstrating that retrieval over cached embeddings can be both effective and efficient. However, many of these methods either treat memory flat buffer or require storing large volumes of raw activations, limiting scalability and longer-term abstraction. Our work builds on these insights by introducing memory consolidation mechanism that persistently retains and hierarchically abstracts multimodal context across a stream, enabling scalable and task addativee long-term understanding.
Video Understanding: In contrast to offline approaches,
pross-modal attention over cached tokens improve temporal coherence, though
3

---

3 Method
We aim to develop efficient method for streaming scenarios. Our approach uses short-clip video- LLM as the backbone, which is capable of multimodal processing of short video clips and generating textual responses. Specifically, for the general captioning task, the model, given sequence of visual tokens (representing the input frames, V) extracted from a visual encoder (eg. CLIP (25]) and potential instruction ("Describe what is happening in the video.") "), generates textual description C, representing the model's understanding of the short clip. Rather than storing all the information (as in [9] we aim to compress the visual information by leveraging the relation with the textual tokens of each short clip, producing compressed ssual-guided guided-by-tees by-text representation, S. We further target aagnos-a design differently from [T] and (24). The key aspect here is that we have the input to the LLM, V, and its corresponding caption, C, which can be interpreted as the model's s understanding expressed in text.
3.1 Attention based Visual Token Selection
These methods typically involve dropping tokens from early layers to enhance the efficiency of processing long contexts in later layers. Evidence from these studies suggests that a model's 's attention can serve as a strong indicator of the relevance of visual tokens.
Several works have explored token selection based on attention weights 37 13 26
Our goal to globally select tokens from short clip after has been processed by the VideoL.LM, providing context for subsequent clips. These tokens will guide the LLM's attention in the next short clip to focus on content that coherent with past events, thereby reducing attention to background tokens, noise, and redundant visual tokens.
Short Clip Processing. We employ a pre-trained video-LLM. which consists of visual encoder VE, an LLM and projector module P. Given a a T'-frame video clip V and an instruction X1. outputs a textual caption of the video, C:
Xv P(VE(V)), C-LLM(XV X,)
where Xv E TN X and Ny denotes the number of spatial tokens per frame, and D is the dimension of the LLM tokens. The target is to select sparse set of tokens S NyxD where Ns < Nv.
Attention Coefficients Calculation. Let denote X the embedding of the generated caption. By the last generated token, attention coefficient tensor is estimated between the generated text tokens and the input tokens. We define the attention matrix A E (N)
calculated on X [Xy, X,,X,) X in given layer and a given attention head h where Nι is the number of text tokens in the instruction prompt and No is the number of tokens in the number of generated caption tokens.
After transferring X into Key KI,,A Value V and Query Qhi,, the attention scores A are computed using the scaled dot-produc attention:
Q Q.hh x'w! K'hh x
= =
(2)
where With with with are the learnable weight matrices for queries, keys, and values, respec- tively!
Q
QUu(KKLTT
A Softmax
Vdl.
where dg is the dimension of attention heads.
First we extract the attention coefficients representing the cross attention between the generated caption and the input visual tokens;
L. = A L' + N, TNV + Nr + No, 0: TNy]
(4)
'Our method focuses on attention weights, making operations involving the values V unnecessary and thus discarded.

---

We then calculate an attention score for each input visual token ,v by averaging the attention scores from all generated caption tokens Nç. Finally, the global importance score for given visual token 3, s.: is determined by averaging the attention scores across different attention heads H and layers I,, representing the overall attention of the model.
aj =
1
1 NC
_ ==1
4 A
ij
(5)
In practice, to limit the compute cost, we consider only a subset of layers. Note that instead of averaging considering other operations such max pooling is straight forward.
Selecting Top Tokens: Finally, the top Ng tokens of Xy with the highest attention coefficients are selected:
S X v [=(=(1), +2), .... TT
(N))
(6)
where ग = argsort(a) are the indices that sort a in descending order
3.2 Long Video Recurrent Processing.
For the very first short clip X (0) we have selected a set of Ns tokens ( receiving the highest attention from the generated caption. Now for next shortclip x we provide as input to the LLM the previously selected tokens (()) as input along with the full set of tokens comprising the current video clip x Similarly, after generating a caption C(2) we select a sparse set of most relevant tokens receiving highest attention scores from x(": (
For the following video shortelips, we create FIFO queue of past selected tokens [S(0), (()), ",.",,,.,,. S(9) which provided to the LLM as context long with the next short clip raw tokens X (t+1) up to the limit of the context window length W or compute constraint.
3.3 Efficient Video Question Answering
Algorithm 1 Streaming Video Processing and Query Answering with rLiVS
We have illustrated how to process each video clip tokens x representing T' frames while having access to a long window of past selected
Streaming Video Processing
1: Mi []. M, < queue(. B+ []
2: MAX_MENT + 16, CLIP_SIZE + 16
tokens. During this process a caption C()) is generated, capturing the L.M thoughts and un- derstanding about the current clip events con-
3: while frames available do
4:
(get _ecett________e fettt__et_ff_rreee -faame ()) if length(BB) == CLIP _SIEE then
5:
ditioned on past clips provided in the context
6:
context <- M, + B
window. We store these generated captions em-
7:
B. B.clear ()
beddings (x(') } and given question q, 8:
S.C Attn_ Attn_Seeeeectttt Sew_seeeeeinnnnno (contteet)))
compute the average cosine similarity between 9:
if length (,,) MAX_MEM MEM then
the question tokens Xg and the tokens of each 10:
M. Mo.peeteef
stored caption (x) Instead of retrieving
end if
the top K captions based solely on average co-
12:
M, append (S)
sine similarity to the query we use Maximal 13:
M/.append(C)
Marginal Relevance (MMR) 4. common re- trieval technique where a score is calculated by balancing relevance to the query (cosine simi- larity between caption and query) with diversity among the selected captions (cosine similarity between candidate caption and already selected ones). Utilizing this technique, we effectively reduce potential redundancy that can emerge in recurrent caption generation, while promoting coverage of distinct information when answering. Algorithm summarizes approach.
14:
end if
end while
Query Answering
16: Q < embed
17: C' < Retrieve__ooppeeeeeeeeeeee Top.(Q M))
context + QQ
19: answer + LLM_ LAM__Generate__nserr (context)
Our simple design and reliance on generated captions for video question answering is stemmed from the efficiency of storing the caption tokens IDs and the utility of similarity estimation between textual
2 Basically, we select the top N's visual tokens (based on the attention scores), and store them in their original temporal order.
5

---

tokens without the reliance on any external embedding encoder such as CLIP 25 Although we can store and provide both visual tokens X / and captions _ to the LLM for question answering, our experiments show that video-LLMs perform better with only caption input.
4 Experiments
In this section, we evaluate our proposed method against state-of-the-arr approaches in streaming and long video understanding benchmarks, demonstrating strong performance and notable efficiency across experiments.
Evaluation Benchmarks. We evaluate our method's effectiveness in online scenarios using the Realtime VStream-QA benchmark (38) which includes RVS-Movie (emphasizing plot understanding) and RVS-Ego (focusing on visual comprehension) both featuring 40-minute videos with diverse open-ended questions. To demonstrate robustness, also report results on offline benchmarks: MovieChat [28] (170 videos averaging 576 seconds across various genres with 510 questions testing Long-range comprehension), offline VStream QA (VS-Movie and VS-Ego) and CG-Bench [5] (1,219 videos averaging 27 minutes with 12K multiple-choice questions). Additionally, we conduct an ablation NexrQA-valset [36] (570 shorter videos averaging 44 seconds with 5K multiple-choice questions) to validate our attention- visual token selection approach.
Baselines. We compare our method against established video LLMs and recent long video un- derstanding approaches. Training-requiree baselines include: VideoChatGPT 22 (employing temporal-spatial pooling), Chat-UniVi [14] (offering unified image/video representations through (using learning-based compression with con- text/content tokens per frame), VideoScan 20 (compressing frames into single learned semantic tokens preserving temporal context), and Flash- VStream-7B [38 (processing frame-by by-ne----frmee with hierarchical memory including FIFO spatial memory. weighted k-means temporal consolidation, and past-current information abstraction). All operate on non-overlapping sliding windows. Training-free baselines include: MovieChat [28 (using FIFO short-term memory and similarity- based long-term consolidation), Goldfish 1 (processing independent short clips with caption generation and re- trieval), and ReKV [9] (storing full K retrieved through aatssanttetionn to queries). We coin our method rLiVS as a short for recurrent LLM-informed Visual Selection.
set of dynamic visual tokens), LLaMA-VID 21
Evaluation Metrics. For benchmarks with open-ended questions, we report both Accuracy and Score based on GPT-3.5 evaluation, following prior works (we use 5-ttrrr sistent with the evaluation setup of ReKV (9). The model is given the question, ground truth, and prediction, and asked to assess whether the prediction matches the answer ("Yes/No") and to assign compatibility score. The prompt template is provided in the Appendix. For multiple-choice bench- marks, we compute accuracy by directly comparing predictions ground truths. Additionally, for the streaming benchmark, we report end-to-end latency, GPU usage (from query arrival to response), and K V-Cache utilization. Benchmarkings are conducted single A100 GPU.
con-
Implementation Details. We implement our method on LLaVA-One Vision 17]. strong VLM for image and video tasks, enabling direct comparison with ReKV (current state-of-the-art on RVS benchmarks). We demonstrate versatility by evaluating both 7B and 0.5B variants, with 7B used unless specified otherwise. Since LLaVA-OV is trained with 32 frames (196 visual tokens each), we allocate 16 frames for current short clips and 16 for short-term recurrent memory. We select only 196 visual tokens from 3,136 available (16x196) retaining just 6.25% of total visual information per short clip. Following previous works, we process RVS- at FPS, and CG-Bench and offline VS-Stream at 0.5 FPS, with 10K context tokens for retrieval and generation. We average attention scores from (of 28) backbone layers across experiments. To show generalization across video-LLM, backbones of our streaming video pipeline, we also incorporate rLIVS to Qwen2.5-Vi VLL 3 and test on RVS streaming benchmark. We provide details for this setup in Appendix A. Finally, in the Appendix we include ablations on layer selection (Appendix B as well as on attention aggregation methods (average vs. max), and analysis how context length affects performance and latency in Appendices C and D respectively.
and RVS-Ego 0.5 FPS [38] D. MovieChat

---

4.1 Results
We next present results across the discussed benchmarks: starting with an ablation on visual selection methods for short video understanding. followed by evaluations on offline long video QA, and concluding with streaming video understanding.
Table I: Comparison on offline long video benchmarks. Accuracy (Acc.) and Score (Sco.) are reported where applicable. Results for ReKV are taken from [9], for VS-Stream from 38 for Moviechat from [28] and I while for CG-Bench from (5).
Method
VS-Ego
VS-Movie
MovieChat
CG-Bench
Acc.
Sco.
Acc.
Sco.
Acc.
Sco.
Acc.
Video-ChatGPT 22
51.7
3.7
54.4
3.4
47.6
2.5
MovicChat 28
52.2
3.4
39.1
2.3
62.3
3.2
UniVi 14]
50.9
3.8
54.0
3.4
25.9
LLaMA-VID 21
54.8
3.9
51.4
3.4
53.2
3.8
Goldfish
67.6
4.2
38
59.0
3.9
56.1
3.4
rLIVS (Ours)
61.0
3.9
59.3
3.6
78.0
4.0
33.1
4.1.1 Evaluation of Visual Token Selection Strategies Short Video Datasets
Visual token selection is central to our method, as it brings past information into current short-clip processing via recurrency, enhancing captioning and understanding of the overall video stream. We argue that naive methods such as uniform sampling or mean pooling risk discarding im- portant information, while traditional clustering approaches like K K-Means can be slow and sub- optimal in high-dimensional spaces. Instead, by leveraging the L.M'ss attention-alreade com- puted during caption generation we gain powerful, low-overhead signal that enables ag- gressive compression with minimal performance loss. To empirically validate this, conduct ex- periments on the NextQA-valser a short video benchmark, assess the token's 's selection qual- ity independently from the factors of long video understanding. We compare the full model's performance while retaining only 6% or 12% of visual
Table 2: Accuracy (Ace.) of different selection methods evaluated on NextQA-valset.
Selection Method
Next-QA QA (valset) Acc.
Full Model
78.6
Uniform Sampling (6%)
75.5
Mean Pooling (6%)
70.7
K-Means (6%)
76.8
Attention (ours) (6%)
77.0
Uniform Sampling (12%)
76.7
Mean Pooling (12%)
75.5
K-Means (12%)
78.1
Attention (ours) (12%)
78.4
with different methods:
1) via uniform sampling, 2) via mean pooling, 3) via K-Means and 4) based on our attention selection
mechanism the generated caption, described Section 3.1 Table 2 contirms our intuition; while uniform sampling results in 2-3% performance loss, our attention-based selection outperforms uniform sampling even twice the compression rate-incurring only 1.5% performance drop with 6% of visual tokens, and nearly matching full-model performance at 12%. Moreover, mean pooling, naively combining visual tokens without considering their relevance, yields the worst per- formance, while K-Means performs comparably to attention-base selection but is significantly slower especially when operating the high-dimensional input space of the L.LM iintrduucinninggggg unnecessary computational overhead.
4.1.2 Offline Video Question Answering Datasets
In Table I we present results on offline long video understanding benchmarks. rLiVS incorporated in LLaVA-OV-7B achieves state-of-the-art performance in VS-Ego and VS-Movie benchmarks, outperforming the previous best by 2% and 3%, respectively. On MovieChat and CG-Bench, our method achieves strong results, outperforming prior approaches, demonstrating the effectiveness of heavily compressing visual tokens without harming performance. Overall, these results indicate that our method serves as strong baseline, efficiently handling long inputs, requiring no additional training fine-tuning- thile- while maintaining minimal memory storage requirements.
7

---

Table Evaluation on the Realtime VStream- QA streaming benchmark [3]. consisting of RvS-Ego
and RVS- subsets. Accuracy (Acc.), Score (Sco.), Latency, VRAM and KV-Cache usage are reported. Results for baselines are taken from 20].
Method
RVS-Ego
RVS-Movie
Latency
VRAM
KV-Cache
Acc.
Sco.
Acc.
Sco.
MovieChat 28
50.7
3.4
36.0
2.3
LLaMA-VID 21
53,4
3.9
48.6
3.3
Flash- Meash-VStream---i
38
57.3
4.0
53.1
3.3
2.1s
19GB
VideoScan 20
60.9
4.0
54.1
3.5
2.1s
18GB
LLaVĀ-ÖV 0.5B
+ ReKV [9]
54.7
3.7
44.6
3,4
1.6s
19GB
4.0 GB/h
+ rLiVS (Ours)
57.6
3.8
51.3
3,4
1.5s
11GB
LLaVĀ-VV 7B
+ ReKV [9]
63.7
4.0
54,4
3.6
2.7s
36GB
18.8 GB/h
LLiVS (Ours)
65.3
4.0
57.7
3.6
1.9s
25GB
Soven2.5-VL. 7B
rLiVS (Ours)
68.1
4.0
56.1
3.6
2.7s
19GB
4.1.3 Streaming Video Question Answering Datasets
Finally, we present our results on RVS-Ego and RVS-Movie, long video streaming benchmarks that represent the primary focus of this work and the main setting for evaluating our proposed method. rLiVS demonstrates strong suitability for this setting as shown Table outperforming the previous best, ReKV, with the same backbone model, by 2-3% and achieving new ofatt--o-ttheart results on the benchmark, while also being nearly second faster than ReKV. Importantly, our attention-based compression technique-combinee with recurrency and caption-guided retrieval and answering-provee robust across both subsets of the streaming benchmark. Unlike ReK V, rLiVS does not require external memory offloading and operates within 10K context window, enabling both efficient and effective performance, while resulting in 1 IIGB less peak VRAM usage with the same video-LLM backbone. Interestingly, when paired with the lightweight 0.5B model, our method outperforms the previous second-bes approach that uses 7B model, as well as RcKV with the same 0.5B model by 2.9% RVS-Ego and 6.7% on RVS-Movie, highlighting the strength of caption- based answering even in smaller model variants emphasizing the efficient aspect of our design choices. Finally, by coupling improve performance on RVS-Ego, reaching 68.1% accuracy. This highlights the model-agnostic and plug-and-play nature of our porposed rLiVS.
method with the recent strong multimodal LLM Qwen2.5-VL, further
key strength of our framework is ability to adapt token selection dynamically based on task instructions, which serve top-down signals. Through instruction attention, the model selects a sparse subset of semantically relevant visual tokens, cnabling efficient, context-aware understanding. As shown in Appendix H. token selection is highly sensitive to the instruction, with different tasks yielding distinct attention patterns. This adaptivity supports a broad range video understanding tasks without retraining or architectural changes, highlighting promising directions for scalable and generalizable long video models.
4.1.4
Ablation of rLivS Design Choices
Next, we ablate our method's core design choices through targeted ex- periments, demonstrating the follow- ing: (1) the importance of recurrency when processing long videos split in short clips: (2) the superiority of cap- tions over selected visual tokens-oo their combination-
Table 4: Ablation on the effect of accessing past visual infor- mation via recurrency and attention selection.
Method
RVS-Ego
RVS-Movie
MovieChat
Acc.
Sco.
Acc.
Sco.
Acc.
Sco.
LLvSS
65.3
4,0
57.7
3,6
78,0
4.0
retrieval and answering in long video streams, (3) the performance benefits of LLM-
w/o recurrency
62.5
3.9
53.7
3.5
74.1
3.9

---

informed attention-based selection compared to naive uniform sampling in streaming scenarios; and (4) the effect of compression rate on downstream performance.
The importance of recurrency in long streams. As previously discussed, I proposes processing long videos as independent short clips, storing only the textual outputs (captions) generated by a video-LLM and retrieving relevant information based cosine similarity to the input question. While this approach is simple and efficient, it suffers from a lack of continuity that becomes increasingly problematic as video length increases. Specifically, the absence of shared visual information across clips and sole reliance on textual representations can hinder the model's ability to consistently track entities- as people or objects objects-acrone across time, especially when captions lack distinguishing context. These limitations naturally impact performance. To empirically prove this, we present results (Table ④ on RVS-Ego, RVS-Movie, and MovieChat, showing that introducing recurrency consistently improves performance by 3-4%, under otherwise identical conditions. Importantly, recurrency in our method serves dual purpose: (1) it enhances continuity and coherence across short clips, boosting long video understanding, and (2) it guides the LLM's attention during visual token selection, reinforeing its own impact.
Captions vs Visual Tokens for retrieval and answering. A natural question is whether textual information (captions) or selected visual tokens provide a better signal for retrieving and answer- ing user queries-that is, which representation is the best choice for effectively preserving cru- cial information while enabling compression. For retrieval, the input query is embedded into the LLM's input space. Visual tokens, projected ad hoc from the visual encoder, lack learned semantic alignment with queries, leading to less meaningful similarity scores (as illustrated in Ap- pendix FF. Proper similarity between them would require projecting both queries and visual tokens into a common space (like CLIP encoders (25)), adding extra computational steps to our pipeline. In contrast, captions share the same modal- ity as queries and can be directly embedded to the LLM's input space, making them nat- urally better suited for retrieval via simple cosine similarity. For answering. Table presents results using visual tokens, cap- tions, or their combination as retrieved in- formation for each short clip. Among the approaches. using only captions achieves the best performance. Despite visual to- kens theoretically offering richer informa- tion, they under-perfone likely due to the mismatch between video-LL.M training data (predominantly short clips of few seconds) and evalua- tion contexts (minutes/hour steaming videos). Current video-LLMs struggle to comprehend visual information at larger timescales, even when computationally representable as input. This aligns with findings from 33), which demonstrated that despite the existence of architectural optimiza- tions that enable long video processing (through memory structures or compressed representations), video-LLMs still demonstrate significant performance degradation in these contexts. In contrast, LLM research has extensively explored long-context adaptation, developing effective modeling and training techniques that enable reasoning over extended sequences of textual inputs, such as multi-clip captions. Thus by utilizing only the captions as past information during retrieval and answering, we effectively transform the problem into text-based long context QA problem, harnessing L.MMs' proven strength in reasoning over these extended contexts.
Table 5: Comparison of vision vs. language modalities for information retrieval and query response on RVS- Ego/Movie benchmarks.
Retrieved Modality
RVS-Ego
RVS-Movie
Acc.
Sco.
Acc.
Sco.
Selected Visual Tokens
58.2
3.9
48.4
3.5
Captions Combination
65.1
4,0
57.7
3.6
63.0
4.0
54.3
3.5
Attention-based vs Uniform Visual Selec- tion on streaming scenarios. We previously demonstrated (Sec. [...I] that our attention- based selection outperforms uniform sampling in short video QA, even twice the compres- sion rate, enabling aggressive token reduction with minimal performance loss. We now extend this evaluation to the streaming setting, where selection drives captioning (through recurrency)
Table 6: Comparison of selection methods within pipeline on RVS Ego and RVS-
Selection Method
RVS-Fgo
RVS-Movie
Acc.
Sco.
Acc.
Sco.
Uniform Sampling
64.2
3.9
56.0
3.5
Attention (ours)
65.1
4.0
57.7
3.6
and thus plays critical role in overall performance. Our results in Table are consistent with those

---

in the short clip benchmark: attention- based selection, leveraging LLM's s ability to attend to crucial information, effectively filtering out noise and redundancy, is superior in both subsets by 1-2%. without introducing any additional overhead, since caption generation is part of our pipeline. Note that uniform sampling is integrated into our full pipeline, replacing only the selection strategy. This approach serves significantly stronger baseline compared to the uniform sampling baseline of ReKV [ as demonstrated in the Appendix.
Compression Rate and Downstream Performance.
Table 7: Acc. on NexiQA valset for dif- ferent % of visual tokens selected with our proposed attention-based method.
Our choice of aggressive visual token compression (e.g., retaining only 6%) may raise concerns about potential in- formation loss and downstream performance degradation. However, this reflects practical trade-off in real-time video understanding. Long-form videos often contain sub- stantial visual redundancy, allowing for effective token filtering without harming performance. To validate this, Table shows how accuracy varies with different compres- sion rates on the NextQA validation set. Retaining 12% of visual tokens nearly matches full performance, while 6% Our framework remains flexible, with the compression ratio tunable to task requirements. Looking ahead, adaptive
Selected Tokens
Acc. (%)
1%
68.0
6%
77.0
12%
78.4
19%
78.7
25%
79.0
100%
78.6
leads to negligible los Fl
dynamically adjusting retention based on visual content complexity-is promising direction. While may improve performance in visually rich scenes, it in- troduces computational overhead, which we currently avoid to maintain online efficiency. Qualitative results in Appendix [ further demonstrate the robustness of our method under extreme compression.
Diseussion
Streaming video question answering and understanding an emerging landscape that can enable the reliable deployment of intelligent agents in various domains requiring visual comprehension. A key design requirement for practical solutions is low memory and compute usage. In this work, we propose a simple approach based on two design choices: first, the instant selection of visual tokens based on the LLM's ' attention, resulting in ~ 95% compression rate of short clip content; second, the use of recurrent processing and caption based question answering to smoothly enable higher-level reasoning over longer time spans. With these components, we achieve state-of-the of-the--rt performance on streaming video benchmarks. Our approach is agnostic to the video-I.IM deployed and requires no training, setting a strong baseline for future work.
Beyond question answering, our framework shows potential as a general-purpose backbone for long-form video processing and understanding tasks such as summarization, retrieval, and object tracking. Preliminary results on long video summarization (Appendix I provide initial evidence of the method's generalizability. Additionally, our analysis in Appendix H demonstrates that the memory selection mechanism is both task sensitive and cntrructioo-condiiion flexibility and adaptability. We view this as a promising direction for future research, where natural language task descriptions can dynamically guide memory selection and reasoning.
underscoring its
Limitations. Despite strong performance, our method prioritizes efficiency over completeness by attending only to selected content during short clip processing. This may overlook fine-graines details and harm continuity, as the LLM lacks access to the full video history. While our FIFO memory buffer offers short-term context and captions serve as long-term memory, this temporal mechanism may not always capture semantically salient information. A promising direction is to explore more semantic, rather than purely temporal, memory selection strategies that remain efficient.
Our approach is also training-free and fully reliant on pre-trained backbones, inheriting their limita- tions in visual understanding and temporal reasoning. The recurrent captioning process, conditioned on prior frames and instructions, can introduce redundancy across captions, which may affect retrieval performance. We partially address this by considering both novelty and relevance during retrieval, Future work includes integrating the method into training pipelines and evaluating across wider range of models and benchmarks to better understand its strengths and limitations.
3 An additional benefit is that selecting only 6% of tokens allows more memory slots in the FIFO queue. enhancing long term understanding.
10

---

References
[1] Kirolos Ataallah, Xiaoqian Shen, Eslam AbdeIrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-language understanding of arbitrarily long videos. In European Conference on Computer Vision, pages 251-2677 Springer, 2024.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 1e:::9 2023.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint wrXiv:2502.1392: 2025.
[4] Kamil Bennani-Smires, Claudiu Musat, Andreea Hossmann, Michael Baeriswyl, and Mar- tin Jaggi. Simple unsupervised keyphrase extraction using sentence embeddings. In Anna Korhonen and Ivan Titov, editors, Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 221-229, Brussels, Belgium, October 2018. Association for Computational Linguistics.
[5] Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Slue-grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412. 2024.
[6] Marvin M Chun and Nicholas B Turk-Browne. Interactions between attention and memory. Current Opinion in Neurobiology, 17(2):177-844, 2007. Cognitive neuroscience.
[7] Nelson Cowan. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24:87 114, 2001.
[8] Tri Dao. Flashattention--: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
[9] Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, TaoZhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question question-answert answering with in- context video KV-cache retrieval. In The Thirteenth International Conference Learning Representations, 2025.
[10] Gueter Josmy Faure, Jia-Fong Yeh, Min-Hung Chen, Hung-Ting Su, Shang-Hong Lai, and Winston H Hsu. Hermes: temporal-coherent long-form understanding with episodes and semantics. arXiv preprint 1iti::2000.17444.3 2024.
[11] Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression large language model. arXiv preprint arXiv:2307.068 00940067755 2023.
[12] He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmenter large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13504-13514, 2024.
[13] Xiaohu Huang, Hao Zhou, and Kai Han. Llm-vtp: Llm-reasoned visual token pruning for efficient multi-modal video understanding.
[14] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700-13710, 2024.
[15] Victor A.F. Lamme and Pieter Roelísema. The distinct modes of vision offered by feedfor- ward and recurrent processing. Trends in Neurosciences, 23(11):571-579 2000.
[16] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara I. Berg, Mohit Bansal, and Jingjing Liu. Less more: Clipbert for ande--and-languaa learning via sparse sampling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7331-7341, 2021.
2021.
11

---

[17] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.033366 2024.
[18] Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: large-scale video question answering benchmark for complex and professional sports. arXiv preprint arXiv:2401. 100001..1000 505,0, 2024.
[19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-12900. PMLR, 2022.
[20] Ruanjun Li, Yuedong Tan, Yuanming Shi, and Jiawei Shao. Videoscan: Enabling ef- ficient streaming video understanding via frame-level semantic carriers. arXiv preprint arXiv:2503.0938% 2025.
[21] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[22] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.00 05424, 2023.
[23] George A. Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63(2)881-97, March 1956.
[24] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:1
2024.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning,
pages 8748-8766 PmL.R, 2021.
[26] Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. Discovering the gems in early layers: Accelerating long-context Ilms with 1000x input token reduction. arXiv preprint ar Xixi 17422, 2024.
[27] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Patlern Recognition Conference, pages 261699 2025.
[28] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision anfd Pattern Recognition, pages 18221-18233 2024.
[29] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for supervised video pre-training. Advances in neural information processing systems, 35: 15:10 100110000993,,,,,001111110111101 2022.
[30] Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-211 2220002222220022 2021.
[31] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00m: 006344 2024.
[32] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14549-14560,
2023.
12

---

[33] Weihan Wang, Zehai He. Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, al. Lvbench: An extreme long video understanding benchmark. arXiv preprini MXXi::2406.0803: 2024.
[34] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03199) 2022.
[35] Chao- Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition, 2022.
[36] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-97888 2021.
[37] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22126-2213,, 2025.
[38] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08008 2024.
[39] Junjie Zhou, Yan Shu, Bo Zhao Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong, Bo Zhang, al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13691-13701, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction explain the contributions shown in method and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
14

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: in the main paper and the Appendix list all details needed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will release the code after finalizing the patent.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwtc//ggui////eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: in Section.4 4 and in the Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: due to the computational cost for running each experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt:::///euiiip.eeeee httprrss...cppblllchhhhhhsseedeo
Answer:[Yes,
Justification: We conform to the NeurIPS Code Ethics. We include details to reproduce the method and will add model cards once released.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include broader impact section in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not train new models or release new data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: yes cite the data use and models and check the license.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are only used for checking text formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur/i: httppp///..///ooeeeeeee///////// for what should or should not be described.
20

---

Technical Appendices and Supplementary Material
A Experimental Details
In this section, we provide detailed description of our evaluation setup and methodological choices ensure the reproducibility of our experiments:
Visual Tokens Selection: As described in the main paper, our attention-based selection method selects k out of N visual tokens globally. The backbone video-LLM we use (LLaVA-OV) is trained on 32 video frames, each contributing 196 tokens, resulting in total of 6,272 visual tokens. We split the 6,272-token context window into two halves: the first is used encode past information (by injecting previously selected visual tokens), and the second is reserved for the raw visual tokens of the current short clip. We choose & such that evenly divides the 3, 3,136 tokens of the second half (current short clip) in order fully utilize the context. During selection, we consider only on the second half, corresponding to the current short clip (i.e.. 3,136 tokens). For both long-video offline and streaming benchmarks, we select 196 tokens from this set, which is Tв (or approximately 6. 6.25%) of the total. For simplicity, we report this as 6%s throughout the paper. In our uniform baseline, everything remains the same and we just sample 196 out of the 3,136 tokens of the current short clip.
Resources:
Across all experiments, we utilize one or two NVIDIA A100 GPUs with 40GB of memory. Latency and VRAM measurements are conducted on single GPU.
LLM Evaluation: As reported in the main paper. we follow prior work 2 [8] for LLM- based evaluation and use gpt-3. 5-tuuro.. The prompt templates employed are provided in Section
Qwen2.5-Vi VL setup: This model dynamically embeds frames based on input resolution, resulting in different number of tokens per frame. In experiments, number of tokens per short clip as LLaVA-OV to ensure fair comparison.
the same
B Layer Selection for Attention-Based Visual Token Selection
In method, visual tokens selected based on their attention scores to the generated captions. Rather than using attention from all layers, we compute scores from a small subset. Modern fast attention implementations (e.g., FlashAttention-2 (8) avoid explicitly materializing the full N attention matrix (with N as the sequence length) to reduce memory and computational cost. To maintain efficiency and avoid additional overhead, we use attention from only small l of L total number of layers (28 in LLaVA-OV). In the main paper experiments, we set / = 4 and we select these layers via uniform sampling across the depth of the network. As shown Table 8 this simple strategy provides robust performance comparable to using all layers, with minimal variance. In contrast, selecting layers from localized regions (e.g., early, middle, or late) yields less stable results, and using single layer results in significantly worse performance.
C Methods for Aggregating Attention Scores in Token Selection
In our method, we opted for average score over different attention heads for token selection; however, we noted that alternative score aggregation strategies could also be explored. Here present results on the NextQA-valser using max as the aggregation method and selecting 6% of the visual tokens (to directly compare with the results of the main paper). As shown in Table 9 both methods perform similarly, with negligible difference of only 0.1% in accuracy.
D Context Length for Retrieving & Answering
In the main paper, we use context length of 10k tokens for retrieving and answering user queries. This choice is motivated by two key factors: (1) Latency: In streaming video applications, where user
'These layers used across experiments in the main paper.
21

---

Table 8: Accuracy on NexiQA validation set for different layer selections, keeping 6% of visual tokens.
Layers
Acc. on NextQA-valset (%)
Full Model
78.6
all layers
76,8
[5. 9, 14, 20
77
[3, 12, 18, 24j
76.8
[1, 10, 15. 25]
77.2
[4, 16, 28)
76.7
[3. 4, 6]
75.2
[13, 15, 16]
76.6
[24, 25, 26, 27]
76.6
[3]
73.3
[15]
76.4
[26]
75.7
Table Accuracy on NexIQA validation set for different layer aggregation methods of attention heads, keeping 6% of visual tokens.
Aggregation Method
Acc. on NextQA- (%)
Full Model
78.6
Avg. Max
77.0
77.1
queries arrive alongside frames in real-time, low-latency responses critical. Increasing the context length substantially slows down both retrieval and LLM inference. (2) Noise Filtering: Long video streams are highly redundant, and larger contexts increase the risk of including irrelevant information, which can degrade performance. We empirically validate these considerations Table o0 RVS- Movie and RVS-Ego streaming benchmarks, showing that while increasing context from 6k 10k improves accuracy, a further increase to 20k offers no additional benefit and in some cases, even hurts performance-while incurring higher computational costs. Overall, a 10k context length offers a favorable trade-off between effectiveness and efficiency. Finally, we note that ReK V [9] retrieves 64 frames (each consisting of 196 tokens, since the model is used), resulting in larger context and, thus, in higher latency (2.7s) as reported in the main paper.
Table 10: Comparison of accuracy and score under different context lengths on RVS-Movie and RVS-Ego.
RVS-Ego
RVS-Movic
Context Length
Latency
Acc.
Score
Acc.
Score
20k
65.3
4.0
56.0
3.5
3.9s
10k
65.3
4.0
57.7
3.6
1.9s
6k
62.7
3.9
57.0
3.5
1.6s
E
Positional Information
A potential concern with our proposed method is the manipulation or loss of spatial positional information during visual token selection. rLiVS, we retain only the temporal order of selected tokens, potentially altering their original spatial layout when reinserting them into the LLM context. However, our initial experiments suggest that this design choice does not hinder performance. As shown in Table 7 selecting only 19% of the visual tokens- -aaedaaaeeeee on our relevance scoring and preserving temporal order-achieves comparable performance using full set visual tokens on the NexIQA validation set. Interestingly, selecting 25% of tokens even outperforms the full-token baseline, indicating that reducing redundancy and filtering out less informative content more beneficial than strictly preserving spatial structure. This finding aligns with prior observations (e.g..
22

---

[3] that video data contains substantial redundancy, which can degrade performance if not mitigated. Additionally, since uision-language models (VLMs) are primarily pre-trained for caption generation, we posit that leveraging this objective as an intermediate step acts an effective noise filter. During pretraining, these models learn to focus on the most informative visual tokens tokees-those that align with the semantics of captions. Our method builds directly on this property by selecting tokens that contribute most to caption generation, thereby enhancing relevance and reducing noise. Finally, we note that by choosing not to address the challenge of accurately injecting positional information, method remains orthogonal to any specific video-LLM architecture. Developing improved strategies to explicitly retain or reintroduce spatial information and thereby better model spatio-temporal dynamics during processing remains an open research direction, which we leave for future work.
Visual tokens vs Captions based Retrieval
In our ablation studies, argue that visual tokens projected ad hoc from the visual encoder lack learned semantic alignment with user questions. As a result, they produce uninformative similarity scores and lead to confusing retrieval behavior. In contrast, captions-
in the same modality as
naturally lie within the same LLM input space, making them more suitable for retrieval. user iillustrate the distribution of cosine similarity scores between visual tokens and queries. as well as between captions and queries, aggregated across videos and questions in RVS-Movic. We observe that the similarity distribution between visual tokens and queries is narrowly centered around zero, with values ranging from approximately -0.02 to 0.06. In contrast, the similarity scores between captions and queries span a much broader range from about 0.4 0.9 m..9-nearly an order of magnitude wider- potentillly indicating greater capacity to distinguish relevant from irrelevant content during retrieval.
Figure 2
To verify this point, we present performance results on both RVS-Ego and RVS-Movie using three similarity-based retrieval methods: visual tokens, captions, and random selection As shown in Table I1 retrieval using visual tokens performs on par with random selection, reinforcing the claim that visual tokens provide limited value. In contrast, caption-based retrieval consistently yields improvements of approximately 2-3%, underscoring its effectiveness.
Histogram Querylatt token Simliarities
Histogram of Query/Caption Similarities
3000
2500
2000
2000
2500
2500
3000
500
500
-0.02
0.00
0.62
0.04
0.06
(a) Visual Tokens
(b) Captions
Figure 2: Distribution of cosine similarities between visual tokens and queries (left), and between captions and queries (right), averaged over videos and questions in RVS-Movie.
G Our uniform baseline ReKV's 9
In the main paper, we compared attention-based and uniform visual token selection on the streaming benchmarks RVS-Ego and RVS-Movie, showing that our attention-basee method consistently outper- forms uniform sampling by approximately 1-2%. Notably, the uniform baseline in that comparison was strengthened with two key components of our approach: recurrency and short-clip level token selection both of which significantly improve performance. Here, we present simpler uniform
Across methods, captions retrieved for answering.
23

---

Table 11: Comparison of aceuracy and score under different retrieval strategies (based on captions.
visual tokens or random) on RVS-Movie and RVS- RVS-Ego.
RVS-Ego
RVS-Movie
Retrieval
Acc.
Score
Acc.
Score
Random
64.6
3.9
55.2
3.5
Visual Tokens
63.7
3.9
55.0
3.5
Captions 65.3
4.0
57.7
3.6
sampling baseline more aligned with the setting of [9], where tokens are sampled uniformly across the entire video. To adapt this to framework, uniformly select 32 frames globally to fill the context, consistent with the frame length that our video-LLM backbone was trained. As shown in our strengthened
Table [2] this naive baseline performs significantly worse-abour 10% uniform variant, highlighting the importance of our proposed design choices.
Table 12:
Comparison of accuracy and score of global uniform sampling (sampling frames) vs our recurrent short-clip caption-base based baseline on RVS-Movie and RVS-Ego
RVS RVS-Ego
-goo
RVS-Movis
Baseline
Acc.
Score
Acc.
Score
Global Uniform
56.2
3.7
44.1
3.1
Recurrent short-clip caption-base Uniform
64.2
3.9
56.0
3.5
H
Attention-based Selection based on Input Instruction
H.1
Core Mechanism
We propose a flexible top-down attention mechanism in which input instructions guide the selection of visual tokens through a caption- or response- follows:
disponse-generation generation step. The overall pipeline is structured as
Instruction - Generated Caption/Response - Intention-Based Token Selection
Tokens are selected based on their attention scores during text generation, aligning with task-specific semantic relevance. As discussed, this mechanism draws inspiration from human memory systems, wherein attention modulated by task-driven goals rather than uniform processing across all inputs.
While prior work has leveraged attention scores for token reduction 13 typically operate within individual layers, pruning tokens based localized attention during early processing stages. In contrast, our method introduces several key innovations. First, we compute cross-layer attention aggregation, identifying tokens that receive consistently high attention across multiple layers. Second, we apply sparse token retention, keeping only small subset of visual tokens (approximately 6%) based on the aggregated scores. Third, through recurrent token reuse, the selected tokens are propagated across successive short video segments, enabling temporal continuity without any model fine-tuning tuning. This design not only reduces computational cost but also preserves semantic coherence across clips, facilitating efficient and context-awan video reasoning.
26 37
these approaches
H.2 Task Adaptability
Although our main experiments focus on caption- guided token selection for visual question answering (VQA), the proposed framework is inherently instruction-adaptive and capable of generalizing across a variety of video understanding tasks through modification the input instruction. For example:
Person Tracking: "Track the movement of the person in the red shirt throughout this video sequence."
24

---

Action Recognition: "Identify and describe the specific athletic movements performed."
Object Detection: "Locate and describe all vehicles appearing in this scene."
Each task-specific instruction elicits distinct model outputs, leading to unique attention patterns and token selections that reflect the semantic priorities of the task. To qualitatively assess this behavior, we conducted comparative analysis of token selection overlaps under varying instructions.
Instruction Sensitivity We selected two representative videos and evaluated the overlap of selected visual tokens using generie captioning prompt versus task-specific instructions:
Case Study:
Video 1: A girl with sunglasses the main foreground object.
Instruction:
"Track the locations of the girl wearing the sunglasses in the video."
Token Overlap with Generic Prompt: 44%
Video 2: Two people playing cards, with interest focused on the background.
Instruction: "Locate and describe the objects appearing in the background of the video."
Token Overlap with Generic Prompt: 8%
The generic prompt used in both cases was: "Describe what's happening in the video," consistent with the one used during main captioning experiments. The results demonstrate that token selection is highly sensitive to the given instruction, validating the dynamic adaptability of our attention-based mechanism.
I Long Video Summarization Task
In this section, we conduct a preliminary evaluation of our method's task generalizability through long video summarization. Using the MLVU validation split (39
compute the holistic video summarization score and compare our approach to Video-XL (27). a recent training-based streaming
video understanding model that also employs recurrent mechanisms. As shown in Table [3 rLiVS outperforms Video-XL in holistic summarization quality.
Table 13: Holistic video summarization scores MLVU (validation).
Method
Model Size
MLVU Summarization Score
Video-XL
7B
3.40
rLiVS (Ours)
7B
3.65
These results demonstrate that rLiVS not only matches but outperforms specialized, trained architec- tures like Video-XL -L while remaining training- fuining-free and easily extensible, We include more detailed breakdown of this comparison the revised version of the paper.
J Qualitative Analysis
To assess the effectiveness of our aggressive token compression complex and dynamic scenarios, we conducted qualitative evaluation on a subset of videos from the SportsQA dataset (18). Although full quantitative results are pending due dataset access constraints, we compare caption outputs generated using method (6% token retention) against those generated using all visual tokens.
Despite the significant reduction in token count, the model retains its ability to produce semantically rich, fine- mme-graieed descriptions, closely matching the full-token baseline. Below are selected examples from the aerobic gymnastics category:
"The video showcases synchronized gymnastics routine performed by group of athletes at the 10th European Championships for Aerobic Gymnastics." 6%: "The video is about gymnastics routine performed by the Italian team the 10th European Championships."
Full:
25

---

Full: "The video showcases rhythmic gymnastics performance by three athletes."
6 6%: "The video showcases a rhythmic gymnastics routine performed by three athletes."
Full: "The video showcases synchronized gymnastics routine performed by two athletes at the 5th Aerobic Gymnastics Asian Championships."
6%: "The video is about two gymnasts performing synchronized routine at the 5th Aerobic Gymnastics Asian Championships."
Full: "The video is about rhythmic gymnastics performance by two athletes at the 7th FIG Aerobics World Age Group Competitions Incheon, Korea."
6%: "The video showcases rhythmic gymnasties routine performed by two athletes."
These examples indicate that our model maintains high semantic fidelity, even under aggressive compression, and generalizes well to fast-changing scenes with multiple actors and fine-grained actions. This suggests the viability of our fixed but efficient token selection approach high- complexity domains such as sports.
K
Prompt Templates
We show here the prompts we use for caption generation and for the LLM evaluation:
Caption Generation. "Describe what's 's happening in the video."
LLM evaluation. For this, follow previous works 38 and utilize the following commonly
used template:
messages=[
{
"role": "system",
"content":
"You are intelligent chatbot designed for evaluating the correctness of generative outputs for question-answer pairs.
"Your task to compare the predicted answer the correct answer and determine if they match meaningfully. Here's s how you can accomplish the task:"
"#INNSRUCTIONS: "
Focus the meaningful match between the predicted answer and the correct answer. An"
Consider synonyms paraphrases valid matches. \n"
" Evaluate the correctness the prediction compared to the answer."
"role": "user",
"content":
"Please evaluate the following viseo-based question-ance -nsweer pair: \n f "'Question: {question]\as
" Correct Answer: (answer>\n"
f "Preddicedd Answer: (pred>\n'n"
"Provide your evaluation only a yes/no and where the is integer value between and 5, with 5 indicating the highest meaningful match.
"Please generate the response in the form of Python dictionary string with keys 'pred' and 'score', where value of 'pred' is string of 'yes' or 'no' and value of 'score' is in INTEGER, not STRING."
"DO NOT PROVIDE ANY OTHER QUTPUT TEXT EXPLANATION. Only provide the Python dictionary string.
"For example, your response should look like this: {'pred': 'yes', Pscore: 4.8)."
]
26

---

L Broader Impact
The broader impact of our approach can be discussed from three main aspects: the environmental impact, accessibility, and ethical consideration. Our method significantly reduces computational costs and energy consumption, subsequently lowering the carbon footprint associated with long video understanding. Along with the efficiency, our approach makes streaming video question answering readily accessible to practitioners and researchers with limited resources. However, as the case of most Al research, our approach can potentially be misused, and it is crucial to ensure a responsible application. As such, preventing its use unauthorized surveillance or privacy violations is important. In general, developing comprehensive guidelines and regulations on the ethical use of large video language models is essential to mitigate these risks.
27


Reviews:
[{"review_id": "mhqK1GqAW5", "reviewer": "Reviewer_WDNs", "summary": "This paper aims to address the task of understanding long videos in a streaming manner. Specifically, it introduces rLiVS, a training-free and Video-LLM-agnostic method for efficient streaming video question answering on long videos. rLiVS leverages a LLM-informed attention-based token selection to retain only the most relevant visual tokens in short clips, followed by recurrent processing for temporal coherence and performing caption-based retrieval question-answering. The authors use the pretrained LLaVA-OneVision model for the experiments on the streaming video benchmarks. The paper demonstrates the effectiveness of rLiVS by outperforming state-of-the-art models across streaming and offline long video understanding benchmarks such as RVS-Ego and  MovieChat.", "questions": "Please look at the abovementioned weaknesses. It will be much more insightful to provide more detailed analysis of the empirical results.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength 1 - In terms of clarity and quality of the paper, the paper is relatively well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the proposed rLiVS approach.\n\nStrength 2 - The contributions of this work may be significant since it addresses an important research problem that can be useful for downstream tasks such as robotics. In particular, rLiVS is a training-free method that can be integrated easily with a pretrained video-LLM that is trained on short video clips to understand much longer videos. \n\nWeakness 1 - It will be much more insightful to analyze if the proposed rLiVS generalizes to different video-LLMs, both proprietary and open-sourced models.  While the former may be costly, it will be helpful for the readers to gain a better understanding of how robust the approach is to models that are trained on different kinds of data and methods.", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "We thank the reviewer for their constructive feedback and are glad that you found our paper well-written with a significant contribution. Below, we address your request in detail.\n\n---\n## **Experiments with Additional Video-LLM**\n\nTo support our claim that rLiVS is Video-LLM-agnostic, we have integrated our method with an additional video-language model, namely **Qwen2.5 VL**. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the RVS-Movie streaming long video understanding benchmark and we achieve **53%** whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and **compatibility across architectures**.\n\n\n---\n\n### **Applicability to Proprietary Models**\n\nWhile rLiVS is designed to be **model-agnostic**, its current implementation relies on access to **internal hidden states**, namely the cross layers attention scores. This requirement limits its direct applicability to **proprietary models**, which typically restrict access to such internals and do not allow manipulation of visual tokens.\n\n---\n\n## **Additional Analysis**\n\nWe would like to highlight several ablation studies and analyses that support the design choices in our method:\n\n- **Token Selection Strategy (Table.2):**  \n  We compare our attention-based selection with several alternatives, including **Uniform Sampling**, **Mean Pooling**, **K-Means**, and **Full Token Usage**. Our method achieves a **significant improvement** over these baselines, with only a **~1% drop in performance** compared to using all tokens, while retaining just **6% of the visual tokens**.\n\n- **Effect of Recurrency (Table.4):**  \n  We ablate the impact of our recurrent mechanism and observe a **3–4% performance gain**, demonstrating its importance for maintaining temporal consistency across short clips.\n\n- **Retrieved Modality (Table.5):**  \n  We analyze the effect of the retrieval modality and find that **textual retrieval outperforms** visual alternatives, supporting our choice of caption-based retrieval.\n\n- **Layer Selection Strategy (Appendix.2, Table.1):**  \n  We evaluate the impact of selecting attention scores from different layers. Our **cross-layer aggregation strategy** provides **robust and stable performance**, comparable to using all layers. In contrast, selecting from localized regions (e.g., early, middle, or late layers) results in less stable outcomes, and using a single layer leads to **significantly degraded performance**.\n\n- **Token Retention Rate (Appendix.5 Table.4):**  \n  We show that retaining only **12% of the visual tokens** maintains performance **without degradation**, highlighting the efficiency of our selection mechanism.\n\n- **Additional Ablations:**  \n  - **Retrieval strategies and selection mechanisms** are discussed in **Appendix.6**.  \n  - **Context length** is ablated in **Appendix.4**.  \n  - **Attention aggregation strategies** are analyzed in **Appendix.3**.\n\nWe are happy to provide any additional analysis upon the reviewer’s request."}, {"review_id": "4T8puP7mjY", "reviewer": "Reviewer_Hid4", "summary": "This paper tackles the problem of long streaming video understanding with large language models (LLMs). The authors claim that existing VideoLLMs face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. This is mainly due to the large number of visual tokens to be processed. To address this problem, authors proposed a training-free approach that is compatible with multiple standard VideoLLMs. The proposed approach includes three aspects, namely 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip, 2)  recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip, and 3) caption-based question answering for lightweight and accurate responses. Experiments have been conducted on public benchmarks to verify the effectiveness of the proposed scheme.", "questions": "Please refer to the weakness part. My major concerns are about the novelty, motivation and lack of in-depth analysis.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation is reasonable. Processing short clips instead of the entire long videos is a more suitable strategy for long video understanding. And for most questions, a large amount of the visual tokens are indeed redundant.\n2. Experimental comparisons demonstrate the significance of the proposed method compared with previous ones.\n\nWeaknesses:\n\n1. The authors claim that the proposed method is LLM-agnostic, but further experiments using different LLMs seem to be missing. Such claims should be supported by adequate experimental results. Besides, attention-based token selection seem to be common strategy for token reduction.\n2. During the token selection for each short clip, authors mentioned the proposed attention-based strategy is inspired from human memory, while in-depth discussions/experiments/analysis on whether this strategy can indeed select the most task-relevant tokens is missing. I believe a more in-depth analysis would significantly strengthen the theoretical completeness of the paper.\n3. The reason for using captions to help retrieve short clips is unclear. Authors mentioned and proved that using captions can lead to better semantic alignments between the query and clips (compared to visual tokens which are not well-aligned with text), thus lead to better results. However, the generated captions (text) can only convey limited semantics (abstracted from semantic-rich frames). How could the model ensure that the generated captions are optimal for retrieval?", "was_revised": true, "final_justification": "After reading the reviews from other reviewers and the rebuttal/responses from the authors. My evaluation is that the current version of the paper might not be ready for publication, and should be carefully revised to clarify the claimed underlying biological basis and demonstrate why using text for retrieval is the optimal choice compared with using visual information. I'm keeping my original rating from my perspective, and will leave it to AC for the final decision.", "rebuttal": "We thank the reviewer for their constructive feedback and insightful suggestions. Below, we address the raised concerns in detail. We are happy to elaborate further during the discussion period.\n\n---\n\n### **Generalization Across Different Video-LLMs**\n\nTo support our claim that rLiVS is Video-LLM-agnostic, we have integrated our method with an additional video-language model, namely **Qwen2.5 VL**. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the  RVS-Movie streaming long video understanding benchmark and we achieve **53%** whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and **compatibility across architectures**.\n\n---\n\n### **Significance and Novelty of the Token Selection Strategy**\n\nWe appreciate the reviewer's feedback regarding the foundation of our attention-based token selection strategy. We'd like to provide additional clarification on the mechanism and its task-adaptability.\n\n#### **Core Mechanism**\n\nOur method employs a flexible *top-down attention mechanism*, where the input instruction guides the token selection process. The pipeline operates as follows:\n\n> **Instruction → Generated Caption/Response → Attention-Based Token Selection**\n\nVisual tokens are selected based on their attention scores during text generation, ensuring semantic relevance to the task (see below, *Task Adaptability*,  for further analysis). This mechanism aligns with human memory principles, where attention is directed by task-specific goals.\n\n#### **Prior Work Explored Text-Image Cross Attention Scores**\n\nWhile attention scores have been previously used for token reduction [9, 19, 29], that was only intended to reduce tokens processed in later layers following the attention score of previous layers. Our approach is distinct in several ways:\n\n- We extract **cross-layer attention scores**, identifying tokens most attended to across the model’s depth, with only a **sparse subset (~6%)** of tokens is retained.\n- These selected tokens are **recurrently reused** to guide the processing of subsequent short clips, enabling continuity in reasoning with **zero training overhead**.\n\nThis strategy not only reduces computational cost but also maintains semantic coherence across video segments.\n\n\n\n#### **Task Adaptability**\n\nWhile our current experiments focus on caption-based selection for visual QA (since these tasks require comprehensive “all-around” video understanding), our framework is designed to be **instruction-adaptive**, allowing it to generalize across tasks by modifying the input instruction. For example:\n\n\n- **Person Tracking**: *\"Track the movement of the person in the red shirt throughout this video sequence.\"*\n- **Action Recognition**: *\"Identify and describe the specific athletic movements performed.\"*\n- **Object Detection**: *\"Locate and describe all vehicles appearing in this scene.\"*\n\n\nEach instruction type produces distinct textual responses and attention patterns, guiding the selection of task-relevant tokens.\n\n\nTo qualitatively assess this, we compared token selection overlaps under different instructions (one focused on main object and the other on background) for two videos:\n\n\n- **Video 1**: A girl with sunglasses is the main foreground.  \n  - Instruction: *\"Track the locations of the girl wearing the sunglasses in the video.\"*  \n  - Overlap with generic prompt: **44%**\n\n\n- **Video 2**: Two people playing cards.  \n  - Instruction: *\"Locate and describe the objects appearing in the background of the video.\"*  \n  - Overlap with generic prompt: **8%**\n\n- Generic Prompt used:  \"Describe what’s happening in the video.\" (consistent with what we used in the main experiments for captioning)\n\nThese results suggest that our token selection reflects the video-LLM’s attention under different task instructions. We will include these findings along with qualitative visualizations in the updated version to further support this analysis.\n\n\n---\n\n## **Caption-Based Retrieval Strategy**\n\nWe thank the reviewer for this insightful question regarding our retrieval design choices. Below, we clarify both the rationale and limitations of our approach.\n\n\n **Adaptive Caption Generation for Retrieval**\n\nAs discussed previously (in the context of token selection), the caption—or more generally, text—generation is **adaptive** based on the input instruction. This allows the model to emphasize semantic aspects that are most relevant for retrieval, depending on the task.\nWe view captions as the **LLM’s internal thoughts** on what it has observed in a given short clip. When answering a query, we access these thoughts without needing to reprocess the corresponding visual tokens.\n\n **Limitations and Justification**\n\nWe agree with the reviewer that captions inherently **compress semantic-rich visual information** into textual form. While this abstraction may omit certain details, our design is motivated by the following considerations:\n\n- **Current VLM Architecture Constraints**  \n  Existing video-LLMs struggle with effective visual-textual alignment for retrieval tasks (as shown in *Figure.1*  Appendix.6).\n\n- **Computational Efficiency**  \n  Alternative approaches—such as using CLIP features for visual-text alignment—introduce significant computational overhead, which conflicts with our goal of **efficient online video understanding**.\n\n\nWe acknowledge that incorporating visual information directly into retrieval could offer additional benefits. However, our current **text-based approach** achieves strong performance within existing architectural and efficiency constraints, setting a solid baseline for future research. We consider this an important direction for future work and appreciate the reviewer’s suggestion.\n\n---\n## **Limited In-Depth Analysis**\n\nWe would like to highlight several ablation studies and analyses that support the design choices in our method:\n\n- **Token Selection Strategy (Table.2):**  \n  We compare our attention-based selection with several alternatives, including **Uniform Sampling**, **Mean Pooling**, **K-Means**, and **Full Token Usage**. Our method achieves a **significant improvement** over these baselines, with only a **~1% drop in performance** compared to using all tokens, while retaining just **6% of the visual tokens**.\n\n- **Effect of Recurrency (Table.4):**  \n  We ablate the impact of our recurrent mechanism and observe a **3–4% performance gain**, demonstrating its importance for maintaining temporal consistency across short clips.\n\n- **Retrieved Modality (Table.5):**  \n  We analyze the effect of the retrieval modality and find that **textual retrieval outperforms** visual alternatives, supporting our choice of caption-based retrieval.\n\n- **Layer Selection Strategy (Appendix.2, Table.1):**  \n  We evaluate the impact of selecting attention scores from different layers. Our **cross-layer aggregation strategy** provides **robust and stable performance**, comparable to using all layers. In contrast, selecting from localized regions (e.g., early, middle, or late layers) results in less stable outcomes, and using a single layer leads to **significantly degraded performance**.\n\n- **Token Retention Rate (Appendix.5 Table.4):**  \n  We show that retaining only **12% of the visual tokens** maintains performance **without degradation**, highlighting the efficiency of our selection mechanism.\n\n- **Additional Ablations:**  \n  - **Retrieval strategies and selection mechanisms** are discussed in **Appendix.6**.  \n  - **Context length** is ablated in **Appendix.4**.  \n  - **Attention aggregation strategies** are analyzed in **Appendix.3**.\n\nWe are happy to provide any additional in-depth analysis upon the reviewer’s request."}, {"review_id": "KuCysCKJIg", "reviewer": "Reviewer_grhS", "summary": "The paper proposes rLiVS --- a training-free approach for efficient streaming video understanding with Video-LLMs, which aims to tackle the challenges of processing hour-long videos online and enabling timely responses. It leverages three key concepts: LLM-informed selection of visual tokens to discard about 95% of unimportant ones with minimal performance loss, recurrent processing of past selected tokens for temporally coherent understanding, and caption-based question answering for lightweight and accurate responses. The method achieves state-of-the-art performance on streaming video benchmarks, balancing efficiency and effectiveness, and is compatible with standard Video-LLMs without requiring additional training, while significantly reducing memory requirements.", "questions": "1. About the model generalizability: The paper states in line 77 that \"the approach is agnostic to the Video-LLM architecture\", but the experiments are only conducted on the LLaVA-OV architecture. Has rLiVS been tested on other Video-LLMs?\n\n2. About the detail capture capability: The 95% compression rate may lead to the loss of rare or background information. It is recommended to analyze the information loss rate of rLiVS for rare scenarios.\n\n3. For the model's dynamic scene robustness: Can attention-based selection handle rapid motion (e.g., SportsQA)? How does rLiVS handle rapid scene changes or concurrent multi-object actions? Experiments in such scenarios would strengthen the claims.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "+ The method looks a comprehensive solution: The paper demonstrates notable strengths in proposing rLiVS, a training-free approach that innovatively combines LLM-informed visual token selection, recurrent memory processing, and caption-based QA to address streaming video challenges. \n\n+ The model performance is strong: rLiVS efficiently compresses visual tokens by ~95% using LLM attention weights, maintaining performance while reducing computational overhead, and employs a FIFO memory mechanism to enhance temporal coherence across video clips. \n\n- The FIFO memory mechanism may suffer from information drift in long videos, potentially degrading performance.\n\n- The aggressive 95% token compression risks missing rare or context-critical details, limiting applicability in high-precision domains.\n\n- There are some clarification issues. For example, Figure 1 does not clearly demonstrate how historical information is accumulated through attention scores.", "was_revised": true, "final_justification": "Thanks for the rebuttal. I have no further questions, and will keep my positive score.", "rebuttal": "We thank the reviewer for their constructive feedback and we are glad to read that the reviewer found our solution comprehensive. Below we answer the reviewer questions and we are happy to elaborate further in the discussion period.\n\n\n## **Generalization Across Different Video-LLMs**\nTo support our claim that rLiVS is LLM-agnostic, we have integrated our method with an additional video-language model, namely **Qwen2.5 VL**. Within the time and compute constraints of this rebuttal, we evaluated rLiVS with Qwen2.5 VL on the   RVS-Movie long video understanding benchmark and we achieve **53%** whereas the uniform baseline achieves 51% on Qwen2.5 VL. This confirms rLiVS effectiveness and **compatibility across architectures**.\n\n---\n\n## **FIFO Memory May suffer Information Drift**\n\nOur design intentionally uses FIFO memory as a **short-term context buffer**, allowing the Video-LLM to process the current clip with awareness of recent visual content. In parallel, we treat the **captions of stored short clips** as a form of **long-term memory**, enabling the model to answer questions about earlier events.\n\nWe agree that this approach may not fully capture the complete historical context. However, processing the full video history would introduce significant computational and latency overhead, and would require models trained to handle such long visual sequences. Our method offers a **training-free, efficient alternative** that leverages the LLM’s pretraining on long text to support question answering about past events, based on its understanding at the time each short clip was processed.\n\n---\n\n## **Possible Information Loss from High Compression Rate**\n\nWe thank the reviewer for raising this important point regarding potential information loss due to our high compression rate. While a 95% reduction may seem aggressive, our design is motivated by the nature of long videos and the demands of online processing.\n\n\n\n### **Rationale Behind High Compression**\n\n- **Visual Redundancy in Long Videos:**  \n  Long videos often contain significant redundancy. Our goal is not lossless compression, but rather **intelligent filtering** of less informative tokens.\n\n- **Trade-off in Online Processing:**  \n  In real-time video understanding, there is an inherent trade-off between **efficiency and information preservation**. Some degree of information loss is expected—and acceptable—when optimizing for **speed and responsiveness**.\n\n\n### **Flexibility of the Framework**\n\n- Our method is **tunable**: the compression ratio can be adjusted based on task requirements.  \n- For tasks requiring higher accuracy, one can retain more tokens at the cost of increased computation.\n\n\n\n### **Empirical Support**\n\n- **Appendix.6 Table 4:**  \n  On a dataset with rich temporal and object interactions:\n  - Retaining **12% of tokens** yields **no significant performance drop**.\n  - Our **6% token selection** results in only a **~1% performance drop**, despite the aggressive compression.\n\n###  **SportsQA Evaluation**\n\nWe encountered delays in obtaining the **SportsQA** dataset, which prevented a full quantitative evaluation within the available time frame. However, we conducted a **qualitative comparison** between the responses of **LLaVA-OneVision** using **6% selected visual tokens** and the same model using **all visual tokens**.\n\nThe comparison reveals that even with only 6% of selected tokens, the model retains the ability to capture fine-grained details comparable to the full-token baseline. Below are a few illustrative examples of different videos - from the subset of 'aerobic gymnastics\":\n\n- **Full Prediction**: \"The video showcases a synchronized gymnastics routine performed by a group of athletes at the 10th European Championships for Aerobic Gymnastics\". \n\n  **Selected Token Prediction**: \"The video is about a gymnastics routine performed by the Italian team at the 10th European Championships\".\n\n- **Full Prediction**: \"The video showcases a rhythmic gymnastics performance by three athletes\".\n\n  **Selected Token Prediction**: \"The video showcases a rhythmic gymnastics routine performed by three athletes\".\n\n- **Full Prediction**: \"The video showcases a synchronized gymnastics routine performed by two athletes at the 5th Aerobic Gymnastics Asian Championships\". \n\n  **Selected Token Prediction**: \"The video is about two gymnasts performing a synchronized routine at the 5th Aerobic Gymnastics Asian Championships.\".\n\n- **Full Prediction**: \"The video is about a rhythmic gymnastics performance by two athletes at the 7th FIG Aerobics World Age Group Competitions in Incheon, Korea\". \n\n  **Selected Token Prediction**: \"The video showcases a rhythmic gymnastics routine performed by two athletes.\n\n\n\nThese examples demonstrate that our method maintains video understanding **close to the full-token baseline**, even in scenarios characterized by **rapid scene changes** and **multi-object interactions**. This suggests that rLViS exhibits strong robustness in dynamic scenes, supporting its generalization capabilities across complex video understanding tasks.\n\n\n\n\n### **Remarks**\n\nAn interesting future direction could be **adaptive compression**, where the ratio is dynamically adjusted based on the **visual richness** of each short clip. However, this would require additional computation for richness estimation, which may conflict with our goal of **efficiency in online settings**.\n\n\nOur fixed but tunable compression strategy strikes a **practical balance** between performance and efficiency, making it well-suited for real-time video understanding. We appreciate the reviewer’s suggestion and are happy to explore further improvements in future work.\n\n---\n\n## **Clarification on Figure 1:**\n\nWe apologize for any lack of clarity in Figure 1. The figure illustrates the processing of the **second short clip**. It shows how the **historical tokens** from the first short clip $S^1$ are included in the context, alongside the **full token sequence** of the second short clip $X^2_V$ and the instruction $X^I $.\n\nWe also depict the **cross-attention** between the generated caption and the tokens of $X^2_V$, which are used to select the top-K relevant tokens (e.g., $X^2_1$, $X^2_5$). These selected tokens are then **accumulated into the historical context** and will be provided as input for processing the **third short clip**.\n\nWe will revise the figure and its caption to make these steps clearer."}, {"review_id": "HI5pqNgJLK", "reviewer": "Reviewer_hCyV", "summary": "The paper proposes the Recurrent Attention-based Token Selection, which reduces the computational cost and improve the video LLM's performance. The strategies (LLM-informed selection of visual tokens, Hierarchical selection of tokens, Caption-based question answering) are used in the method. Experimental results demonstrate the method's effectiveness.", "questions": "Please refer to the main text", "limitations": "I haven't find the limitation section in the main text.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The method demonstrates the promising results.\n\nWeaknesses: \n1. The novelty is somewhat limited. I think the method is similar to Video-XL[1]. The difference is very minor (Both the two methods use the recurrent form to process the long video. The only difference is that the Video-XL uses the accumulated tokens, while the rLiVS uses selected tokens.). I think the authors should further demonstrate rLiVS's novelty and compare it with video-XL\n\n2. The method section is somewhat hard to understand. It has too many equations and without any illustration. Although I understand the section, I think the authors can add a figure to easily illustrate the method.\n\n3. The efficiency is not reported in the paper. I think the authors should compare rLiVS's inference efficiency with its counterparts.\n\n4. The application of rLiVS is somewhat narrow. Can rLiVS be used in more complex tasks such as video grouding?\n\n[1]Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding. Arxiv 2409.14485", "was_revised": true, "final_justification": "Thanks for the rebuttal. My concerns have been partly solved. rLiVS is different from Video-XL as a training free model. And the efficiency has been reported. Despite the author has not shown the model's performance on grounding, I tend to increase my score.", "rebuttal": "We thank the reviewer for their constructive feedback. Below, we clarify the raised points and address the specific questions. We are happy to further discuss any remaining concerns during the discussion phase.\n\n\n### **Novelty w.r.t. Video-XL**\n\nWe thank the reviewer for bringing up Video-XL [1]. While both approaches aim to address long video understanding via recurrent mechanisms, there are **key differences** in design and applicability:\n\n- **Video-XL** introduces a custom-trained Video-LLM that learns to compress visual tokens, which are then attended to by future tokens. This approach **relies on model-specific training**, and its effectiveness is constrained by the **compression rates and video lengths** seen during training. For example, on **Long Video Bench**, our base model (LLaVA-OneVision 7B) achieves **56.4%**, outperforming Video-XL (7B) at **48.8%**. This highlights the limitations of architecture-specific designs, which can quickly become outdated and fail to generalize to longer videos beyond those seen during fine-tuning.\n\n- In contrast, **rLiVS is training-free** and model-agnostic. It leverages:\n  1. **LLM-informed token selection** via cross-attention,\n  2. **Recurrent accumulation** of selected tokens, and\n  3. **Text-based retrieval** for long-term memory.\n\nThis modular design enables **plug-and-play scalability** to new models without retraining, while maintaining high efficiency.\n\nTo directly compare with **Video-XL**, we have included an evaluation on the task of **video summarization**  using the **MLVU validation split**, we assessed the **holistic video summarization score**. Our model, **rLiVS**, achieved a score of **3.65**, outperforming **Video-XL** (3.4). We will include a more detailed comparison with Video-XL in the revised version.\n\n---\n\n### **Illustration in Method Section**\n\nWe kindly refer to **Figure 1**, which illustrates key components of our method:\n\n- The **left side** of the figure shows the processing of the **second short clip**. It includes the **historical tokens** from the first short clip $S^1 $, the **full token sequence** of the second short clip $X^2_V$, and the instruction $X^I $.  \n- It also visualizes the **cross-attention** between the generated caption and the tokens of $X^2_V$, which is used to select the top-K relevant tokens (e.g., $X^2_1$, $X^2_5$). These selected tokens are then **accumulated into the historical context** and passed forward for processing the **third short clip**.\n\nThe **right side** of the figure illustrates the **question-answering phase**, where the similarity between the query and the stored descriptions of each short clip is computed. The most relevant subset is then provided as input to the LLM (within the Video-LLM) for answering the question.\n\nWe also refer the reviewer to **Algorithm 1**, which outlines the main steps of our approach.\n\nIf the reviewer finds that Figure 1 and Algorithm 1 are insufficient, we would be happy to include an additional illustration in the revised version to further clarify the method.\n\n---\n\n### **Efficiency Not Reported?**\n\nWe thank the reviewer for raising this point. We refer to **Table 3**, where we report **latency**, **VRAM usage**, and **KV-cache requirements** on both **RVS-Ego** and **RVS-Movie**, in line with prior work.\n\nOur method achieves the **lowest latency**, requires **no KV-caching**, and uses significantly **less VRAM** compared to the previous state-of-the-art, **ReKV [5]**, while also delivering **superior performance**.\n\n---\n\n### **Clarifying the Broader Applicability of rLiVS**\n\nWe appreciate the reviewer’s observation regarding the scope of rLiVS. While our current evaluation focuses on long-term streaming video understanding and question answering in a training-free setting, we believe this does not imply a narrow application.\n\nrLiVS is designed to enable **streaming, never-ending video understanding**, a key capability for real-world deployment of **visual assistants** and **robotic systems**. These systems often operate in dynamic environments where continuous perception and reasoning over historical visual context are essential.\n\nWe follow established benchmarks and protocols from prior work (e.g., [5]) to ensure fair and meaningful comparisons.\n\nRegarding temporal video grounding, rLiVS can naturally identify the short clip most relevant to a given query. However, pinpointing exact timestamps is beyond the scope of streaming video understanding and not the focus of our current work.\n\nAs stated above we included the task of **video summarization** as an additional benchmark alongside **video question answering**. Using the **MLVU validation split**, we assessed the **holistic video summarization score**. Our model, **rLViS**, achieved a score of **3.65** compared to the base **LLaVA-OneVision** score of 3.57.\nThese results demonstrate rLViS **generalization capability across tasks**.\n\n\nIn this light, we argue that rLiVS addresses a **significant and practical challenge** in video understanding, with design choices motivated by real-world deployment scenarios.\n\n\n[1] Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding. Arxiv 2409.14485"}]

Author rebuttal:

Meta-review: The paper introduces a recurrent attention-based token selection mechanism that significantly improves the efficiency of streaming Video-LLMs while retaining strong accuracy across multiple video understanding tasks. The approach is technically sound, addresses a pressing scalability issue, and is supported by solid experimental evidence.

For the camera-ready, I encourage the authors to: (a) integrate the constructive suggestions raised by Reviewer Hid4, particularly clarifying ablation details and expanding analysis on selection stability and more relevant supporting evidence from neuroscience if available, (ii) refine claims on generalization by situating the method more carefully against recent efficient video modeling works, (iii) provide more detailed discussion on trade-offs between latency and accuracy, and (iv) explicitly acknowledge limitations and possible extensions to longer-horizon or multi-modal streaming scenarios.

This decision was made after carefully considering all mixed reviews. While some reviewers raised concerns regarding clarity and scope, the consensus is that the technical contributions are both useful and timely.
Decision: Accept (poster)