RIVER: A Real-Time Interaction Benchmark for Video LLMs
Authors: Yansong Shi
Qingsong Zhao
Tianxiang Jiang
Xiangyu Zeng
Yi Wang
Limin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Online Video Interaction
Multimodal
Video Understanding
Primary area: datasets and benchmarks

Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering their real-time interactivity. To address this gap, we introduce the Real-tIme intERaction Benchmark for Video LLMs (RIVER Bench), designed for evaluating their real-time interaction ability with humans through perceiving the streaming videos. RIVER Bench introduces a novel evaluation framework comprising Retrospective Memory, Live-Perception, and Proactive Response tasks, closely mimicking interactive dialogues with humans rather than understanding the entire videos at once. We conduct detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online interaction paradigm, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enhances models’ flexibility in real-time interaction. We believe this
work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. The code and data will be released.

Published as a conference paper at ICLR 2026
RIVER: A REAL-TIME INTERACTION BENCHMARK FOR VIDEO LLMS
Yansong Shi Qingsong 3 Tianxiang Jiang 1.2+ Xiangyu Zeng 4.2 Yi Wang²
Limin Wang4.2.f
1 'School of Information Science And Technology, University of Science and Technology of China
Shanghai Artificial Intelligence Laboratory
"College of Computer Science and Artificial Intelligence, Fudan University
*State Key Lab of Novel Software Technology, Nanjing University
sh mniyysssnn@@plllcc org.c.r.comm Imwangênju. jee.ennuuudduu... cn
ABSTRACT
Multimodal large language models (MLLMs) have demonstrated impressive pabilitties, yet nearly all operate in an offline paradigm, hindering their real-time interactivity. To address this gap, introduce the Real-tIme intERaction Bench- mark for Video LLMs (RIVER Bench), designed for evaluating their real-time interaction ability with humans through perceiving the streaming videos. RIVER Bench introduces novel evaluation framework comprising Retrospective Mem- ory, Live--iieeeeeeee Percepeerc,,,n,, and Proactive Response tasks, closely mimicking interactive dialogues with humans rather than understanding the entire videos at once. We conduct detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question- answering tasks, they struggle with real-time processing. Addressing the limitations existing models in online interaction paradigm, especially their deficiencies proposed a general improvement
long-termi memory and future perception, method that enhances models' flexibility in real-time interaction. We believe this work will significantly advance the development of real-time interactive video un- derstanding models and inspire future research in this emerging field. Datasets and code are publicly available
INTRODUCTION
The emergence of online multimodal interaction, where Al systems process streaming visual inputs while maintaining temporal-aware dialogues, presents new challenges for evaluating multimodal large language models (MLLMs). Although models like GPT-4o (Achiam et al., (023) and Gemini (Team al. 2024) demonstrate impressive capabilities in controlled settings, existing benchmarks inadequately address the dynamic requirements of online applications such augmented reality
navigation (Cheliotis et al. (023) or robotic task supervision (Brohan et al. 2022 Jaegle et al.) 2021
creating bottlenecks for systematic progress in online interaction research.
Existing studies (Weichbroth) 2025 Li et al. 2023a) summarize this task requires a reasonable
and timely response to human requests regarding historical / current future status based video content. Previous works such as VideoLLMMonllin
(Chen et al. 2024a). OV-Bench (Huang et al.) 2024), and OVO-Bench (Li al. 2025) have established preliminary frameworks for these aspects. We posit that an effective online MLLMM (oMLLM) must exhibit three core competencies: (1) long-term memory retention track evolving visual narratives, (2) precise temporal grounding for timely responses to dynamic queries, and (3) proactive reasoning to anticipate future states. Current benchmarks inadequately quantify the temporal degradation of memory (e.g.. forgetting curves) or the joint optimization of response accuracy and timeliness, critical factors for real-world deployment.
Equual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Timeline
Cue
Answer
Retro-Memory
Live-Pereeptio
How many birds there flying at the beginning of the video?
What color is gross around Honess lying plone in the grass?
Yellow
Pro-Response Instant
Pro-Response Streaming
When scrub the cooking pon, what else did do?
You scrubbed the whisk
take
remove the nut from the crocker
video real time.
nun.
crocken
Figure I: Illustration of different online interaction tasks. The question (Query), reference events (Cue), and answers timings are represented by > and respectively. Based on the frequency and timing of reference events, questions, and answers, we further categorize online interaction tasks into four distinct subclasses, as visually depicted in the figure. For the Retro-Memory. the clue is drawn from the past; for the Live-Perception, response. For the Pro-Response task, Video LLMSs need to wait until the corresponding clue appears and then respond quickly possible.
comes from the present-bott demand immediate
Therefore, our work focuses on quantifying the retrospective and anticipatory capabilities of oMLLMs by precisely evaluating their temporal feedback. The former computes the memorization variations of oMLLMM over time, and the latter reflects the timing of a prompt to a proactive plan. In addition, we also measure the response the live multimodal understanding for formal integrity.
To measure the temporal awareness of oMLLM regarding live queries, especially in retrospection and anticipation, we design online interaction benchmark RIVER Bench. It inherits the evaluations on the key aspects from seminal works featuring Retrospective Memory (Retro-Memory Live- Perception and Proactive Response .Prooeespnss). Specifically, in atro-memory, we measure memory persistence by analyzing performance decay over increasing temporal intervals, modeled by forgetting curve analysis. For live- pee-perception, we quantify real-time multimodal understanding through time-sensitive question answering with latency-accuracy tradeoff optimization. Regarding pro-response, we evaluate future-state prediction via joint modeling of event forecasting confidence and temporal localization precision.
Based on a large number of existing video evaluation datasets (Vript-RR (Yang et al., 2025). LVBench (Wang et al. 2024ah. Long VongWiieeBench (Wu et al., 2025), Ego4D (Grauman et al. 2022 and HVHihlighhs (Lei ct al. ()2T)), we have rigorously selected and validated the data through both manual and automated methods, reconstructing the original annotations into multiple forms of online interaction. Our benchmark meticulously defines the timing, content, and reference points of interac- tions, enabling comprehensive and accurate evaluation of models' online interactive capabilities.
We conduct extensive evaluations across four model categories: vanilla offline MLLMs, sliding- window adapted models, existing online MLLMs, and models fine-tuned with our proposed training paradigm. Key findings reveal that while offline models excel at single tasks by leveraging full video context, their ability to process and interpret videos in strict real-time scenarios remains severely limited. Traditional approaches adopt sliding techniques, trad- ing comprehensive temporal comprehension for partial real-time responsiveness. However, recent advances in online video understanding frameworks have demonstrated marked improvements in multiple dimensions of real-time processing, including latency, accuracy, and temporal reasoning, showcasing significant advantages over conventional offline methods. To further bridge this gap, we introduce a novel training dataset specifically designed to enhance online interaction with video content. Experimental results validate that our dataset substantially improves the real-time compre- hension capabilities of existing of-e-of-theeart models, enabling robust performance in complex, dynamic, streaming video environments.
(QA)
2

---

Published as a conference paper at ICLR 2026
In summary, the contributions of this paper as follows:
We define the interactive form of online video comprehension. Furthermore, we propose the RIVER Bench, which provides precise annotations and questions for events at different positions in videos, enabling quantitative evaluation and comparison of models' perceptual capabilities regarding past, present, and future events, as well as their real-time interactive abilities.
Towards enabling robust long-term memory in multimodal models, we incorporate long-short term memory module to dynamically preserve visual information. Validated across multiple architectures, this general approach represents a pivotal step towards substantially boosting temporal understanding.
We construct specialized interactive training dataset to meet future interactive demands. Through fine-tuning, the model's capability for future interactions is markedly improved.
2 RELATED WORKS
Online Interaction. Online interaction with video content is characterized by two capabilities memory and anticipation, which are not adequately addressed in previous studies. Some models have explored offline long video understanding, e.g., LongVA (Zhang et al. 2024c) and VideoChat- Flash (Li et al. 2024b), achieving impressive results by extending the input frame capacity of offline models to tens of thousands of frames. However, without effective memory mechanisms, GPU memory will eventually overflow as time progresses. To address this, several models, including MovieChat (Song et al., (224)), VideoL (Wang et al., 2024d). VideoStreaming QQin et al. (2025), FStrh-V (Zhang et al.) (224a), and VideoChat- Online (Huang et al.) 2024), have incorporated memory cache modules. When new video information is input, these models store relevant information the memory module and clear redundant data to free up valuable storage space. StreamForest (Zeng et al., (025) uses a Persistent Event Memory Forest to adaptively merge event-level trees based on temporal, similarity, and frequency penalties, maintaining constant memory and enabling efficient long-term retention with real-time reasoning even at extreme compression.
In terms of real-time capabilities, other models such as MMDuet (Wang et al. 2024c) and VideoLLM- Online (Chen al. 2024a have enabled models to acquire the ability for proactive online interaction through the construction of specialized training data, allowing them to understand user intent and provide responses or guidance. Specifically, VideoLLM-
(Chen et al.. 2024a) achieves this by using the LM Head to predict special tokens, which determine when the model should respond. MMDuet (Wang al.) 2024c). on the other hand, calculates the information and relevance scores for each sampled video frame and decides whether the model (i.c., the assistant role) should interrupt the video and initiate its turn. StreamChat (Liu al. 2024b) updates the visual context information in real-time during the decoding process, ensuring that the latest visual information is used during decoding, which crucial for the real-time operation of online models.
Furthermore, some models, such as GPT-4o (Achiam et al.) (023). support voice input along with other modalities. Examples include IXC2.5-OL (Zhang et al. [024b)), VITA-1.5 (Fu et al., (025), and MiniCPM-o-2.6 (ao et al. 2024). IXC2.5-OL (Zhang et al. 2024b) also incorporates memory module that selectively retains relevant information. However, its voice module integrates multiple components, resulting a less streamlined design. VITA-1.5 (Fu et al. 2025 draws inspiration from the Freeze-Ommi model (Wang et al. 2024b), enabling a more natural integration of voice and video understanding capabilities. Since voice interaction aligns more closely with everyday communication, it holds significant potential for deployment on edge devices.
Online Video Benchmarks. Traditional offline video understanding focuses on holistic compre- hension of video content, and numerous excellent benchmarks (Li et al., [202at Fu et al.) 2024) have emerged and gained widespread adoption. However, the task types and evaluation methods for online video understanding have not been precisely defined. Some models have attempted to enhance online capabilities and proposed their own benchmarks, but the evaluation formats remain largely similar to traditional offline benchmarks, failing to adequately address the requirements of online video understanding. For instance, "Stream-QA (Zhang et al., (024a),, composed of VStreamQA-Egg and VStream- VStream-QA-Movvee QA-Movie, incorporates reference timestamps in its question-answe pairs, which are crucial for online video understanding tasks. It uses GPT-4V (Achiam et al. (023) to generate frame descriptions and GPT-4 (Achiam et al. 2023) to create multiple-choice questions, covering
3

---

Published as a conference paper at ICLR 2026
Proquency
very
sucund
400
lung
Short
man
Woman
2006
P520
10000
12500 35020 37500
blue
short
Liee Weteedoon
Retra- Mamoory
Duration(s)
appear
F
RIVER
stage
Presency
han
Fed
AVR sittin
WEB
very
video
Stoearing
Question time (Ratio)
Figure 2: The pie chart on the left illustrates the quantitative distribution of various tasks within the benchmark. The two bar charts in the middle depict the statistics of video duration and the proportion of question timestamps, respectively. On the right, word cloud which is constructed from the annotated textual data within the dataset, visually emphasizing the most frequently occurring terms.
scene and event descriptions, temporal existence, and sequence judgment. However, its evaluation format does not fully capture the interactive and real-time nature of online video understanding. Similarly, VideoLLLM-Omline (Chen et al.) 2024a) emphasizes the fluency and timeliness of responses but neglects the accuracy of question answering, which is critical for interactive systems.
MovieChat-1K K (Song et al. 2024) introduces breakpoint mode, which involves asking questions specified timestamps in the video stream, aligning with the immediate response type in online video understanding. However, lacks comprehensive framework for other interaction types, such as recalling awaiting responses. StreamingBench (Lin et al. 2024) eeelicipliiiii timestamps and incorporates multi-modes information and complex contextual details from the video stream, addressing tasks like real-time visual understanding, omni-source understanding, and contextual understanding. Despite its rich task design, it does not fully formalize the dynamic nature of online interactions. OV-Bench (Huang et al. 2024) defines three temporal categories- past, and future-based based on the relationship between the question timestamp and the video timeline, transforming tasks like grounding and STAL
explicitly links questions to specific
Action Localization) into an online understanding format. While it introduces temporal categorization, it does not explicitly address the continuous and interactive aspects of online interaction tasks. OVO-Bench eLi (025) categorizes online video understanding tasks into three types: Backward Tracing, Real-Time Visual Perception, and Forward Active Responding, featuring flexible format for questions and answers. It represents the most relevant existing work for defining online video understanding tasks but it lacks fine-grained temporal segmentation of the response or clue intervals, which is crucial for analyzing the online interaction capability of Video L.LMs. In contrast, our proposed RIVER Bench provides a more comprehensive and precise definition of online interaction with video, encompassing retro-memory, live-perception, and pro-response. This approach addresses the limitations of existing benchmarks and offering a robust evaluation standard for future research.
3 RIVER BENCH
We design RIVER Bench for assessing the temporal awareness of MLLMs on online interactions. It measures the retrospective memory, live perception, and proactive anticipation of oMLLMs by evaluating the accuracy, relevance, and timeliness of their generated responses. Based on the temporal relationships between the moments of clues, questions, and answers, categorize the questions in The statistic results of RIVER Bench are
the benchmark into three main types, as shown in Figure]] | shown Table and Figure 2 We first present the formulation, followed by a detailed description of the benchmark construction and its characteristics.
and
Formulation. We formulate this online interaction into window-based video-text-to-text task, as:
f -- log Po(ra) Vea, q, heee
4

---

Published as a conference paper at ICLR 2026
Table Comparison of RIVER and other video online benchmarks.
Memory & Perception
Anticipation
Benchmark
Videos Questions
General Short Medium Long
Very Long
Instant Stream
VStream-QA QA (hang et al. 2024a
3,500
X
x
x
X
StreumingBench [Lan ct al. 2024
2022
900
4,500
X
X
x
x
X
OVBench (Huang et al. 2024
1,463
4,874
X
OVO-Bench (Li et al RIVER (ours)
3,100
1,067
4,278
/
V
V
V
v
V
where P denotes the oMLLM parametrized by 9. Ve is the streaming video started at time until now while q. r. her stand for users' queries (in words), desired responses, and the historical modeling of P. Note for each in learning. it can contain more than one EOS (end-ofssequence) tokens at arbitrary locations for simulating silence or pauses in live conversation.
3.1 INTERACTIVE TASK TYPES
We summarize three main task types of RIVER Bench retro iitoommmmrry dive-perception, and pro- response, according to the happening time (denoted as ty) of the queried event or target.
Retro-Memory; oMLLM answers users based on the history (herr, ). as users' question concerning past events (where tv < t'), such as "Where did just put my bag?".
Live-Perception: oMLLLM responds immediately as the question relates to the current or short-term visual inputs (where /' < S t).
Pro-Res Response: For this task, the oMLLM must continuously monitor the video stream and execute
response precisely when a user-specified condition met (where lv > 1). For example, when the user is searching for wrench that is temporarily not visible, once the wrench appears, oMLLM will inform the user. special case involves the model continuously narrating its visual inputs upon request (e.g.. keep describing what you see'), task analogous dense video captioning (Johnson et al. 2016 Krishna et al., 2017).
Note previous works have already defined these types and evaluated them in forms. Compared with seminal researches (Li et al. [2025] Huang et al. 2024). we intend to study MLLM's understanding curve of historical and future events according users' queries. Specifically, how the time interval A l|tv t|l (A 0 when t' < tv I f) impacts oMLLM's performance on retro-ment memory and pro-response.
3.2 DATA CONSTRUCTION
Unlike conventional QA benchmarks, online video comprehension benchmarks emphasize inter- activity, requiring precise definitions of query time, cue time, and response time. To this end, we meticulously curate datasets from diverse sources, ensuring each question is grounded in accurate temporal annotations that explicitly specify these time points. Specifically, we utilized datasets from Vript-RR (Yang al. 2025). LVBench (Wang al. 2024a). Long VideoBench (Wu al.) 2025). Ego4D (Grauman et al. 2022) and QVHighlights (Lei et al. (2021). which underwent rigorous filtering, restructuring and verification to form the final version the benchmark. The process of data processing and generation is presented in Figure
Retro-Memory. For retro-memory questions, we evaluate the model's ability to recall events across different time spans by categorizing queries into four distinct temporal intervals: short (15-30s), medium (30-60s), long (300-900s), and very long (1800-3600s). To ensure precise evaluation, each question is carefully designed so that its correct answer can only be derived from one specific moment in the video. constructed this benchmark by systematically filtering and restructuring datasets from multiple sources, including Vript-RR (Yang et al.) 2025). LVBench (Wang al. 2024a), and Long VideoBench (Wu et al. 2025). inserting questions at strategically selected temporal positions. The resulting evaluation set consists of 1.5k high-quality multiple-choic QA pairs, each with precisely annotated timestamps for the queried events to eliminate any temporal ambiguity in both questions and answers.
5

---

Published as a conference paper at ICLR 2026
basic: video pulh, duration
PS
acnol
BIEA2
Q&A
Jnn03
LLM
question:
choices answer: time, ground truth
clue: time
LLM
videos
(dense caption nomaal QA)
THe ameelation
rules
collection
filtering
generating annotation
verification
Figure 3: Illustration of data processing process.
Percepeecn Live-Perception questions assess a model's ability to comprehend visual infor- mation within the current temporal window (typically a few seconds), which is also fundamental capability for video multimodal models. The queries encompass environmental context, actions, object attributes (e.g., quantity, color), with particular focus on both dynamic sequences of changes and rich static visual semantics. The live- peeeepppptiioiuutttonn remains similar to conventional video understanding tasks, but emphasises real-time interaction. We derived these questions from the same data sources as the past-oriented questions, totaling 0.4k samples.
Pro-Response. For pro-response questions, we categorize them into instant and stream types, representing anglee and multi-answer scenarios, respectively. Stream-type questions require the model to continuously describe the scene or guide the user, generating textual outputs at dif- ferent timestamps. While similar to dense captioning tasks, this process operates in real-time conversational manner rather than pist-video analysis. Following the data construction approach of VideoLLM-onlinee curated evaluation data for continuous responses from the validation set, totaling 1.2k samples. Instant-type questions arise when current or past video in- formation is insufficient for accurate recall retrieval, requiring the model to observe further and respond at the appropriate moment (e.g., predicting the next event). In our benchmark, the model must not only determine the optimal timing but also provide coherent answers. We filtered raw annotations from Fgo4ID-Narration and QVHighlights, which cover both egocentric and third-person videos, using sentence embeddings (Devlin al. 2019). then synthesized anticipatory questions and distractor options via LLMs (Yang et al. (2024). Based on temporal intervals, instant-type questions further classified into short, medium, long, and very long, totaling 1.4k samples.
Narration
3.3 QUALITY CONTROL
Removing low-quality question-answen pairs. To ensure high-quality question-answer pairs in our benchmark, we employ a multi-stage filtering process combining sourcs large language models (LLMs) (Yang et al.) 22244] AI (224) and rigorous human evaluation. First, use LLMs to identify and remove questions that can be answered correctly without visual input, thereby mitigating the influence of language priors and model biases. Questions that are overly broad, span excessively long time durations, or rely on ambiguous cues are also filtered out, as they may be answerable with minimal visual context and do not adequately test a model's recall or temporal reasoning abilities. Detailed guidelines and examples of our review criteria are provided in Appendix A
Removing ordinary and meaningless event descriptions. To reduce reliance purely temporal queries and instead emphasize behavior prediction and event sequencing grounded contextual localization, use semantic-similarity metrics curate annotations. This process isolates distinctive events that serve as unambiguous reference anchors for questions. Each anchor is paired with a precise timestamp. furnishing clear yet concise temporal context. The resulting dataset minimizes ambiguity and provides rigorous test of a model's 's capacity to recall video content accurately.
3.4 METRICS
Retro-Memory & Live-Perception. After obtaining model responses, we first use regular expres- sion matching extract answers for multiple-choice (MC) questions. For failed matches non-MC questions, we adopt an open-ended (OE) evaluation approach following (Song et al. 2024). We leverage the open-source Qwen2.5-72B (Yang et al. (024) to assess response consistency with reference answers. To ensure fair comparison, we have presented the accuracy rates of both methods (MC & OE) as comprehensively possible

---

Published as a conference paper at ICLR 2026
Qulput
Slient
V/N
Large Langauge Model
Long- Memory
Short-Torm Memory
Query
Querry You're helpful assisantt
Are you sure that..
Query'e
my hands just now? Remind me use resictent gloves.
Vision
Encoder
Figure 4: Pipeline enable MILMS support online inference capabilities. The Long Short-Term Memory module continuously receives new visual features and selects the most important parts. After query is posed at to, the model is queried each time window; if it decides answer, the final response is output.
Pro-Response. To evaluate proactive response capabilities, we introduce a Response Accuracy Metric that quantifies performance based temporal aligmment with ground-trute timestamps lg- Defined within a tolerance window of length w centered at lg. the metric assigns a full score responses falling inside this interval, reflecting acceptable anticipation. To align with human subjective tolerance, the scoring function strictly penalizes early responses with a score of zero to discourage false alarms, while applying a linear decay late responses. Specifically, scores for delayed predictions gradually decrease to zero as the latency increases beyond the window, acknowledging their diminishing utility without immediately discarding them total failures.
EXPERIMENTS
Utilizing the proposed RIVER Bench, we evaluate four categories rideo-processing multimodal large language models: (1) commercial closed-source models, (2) open-source models with native online inference support, (3) open-source video multimodal models, and (4) our proposed video multimodal model adapted for online inference.
To enable these models to handle ultra-long videos (up to 120 minutes), we applied their respective recommended frame sampling strategies. For HPPT-4o/chiann et al. (202)) and Gemini-1.5-pre (Team) et al. 2024), , we extracted 50 frames; for VideoChat2 (Li et al. 2024a), LLa Video (Zhang et al. 2024dh, and Intern VL2..5 (Chen al.) (024b), extracted 16 frames each. For the two models with native online inference support, VideoLLMM-Onlnne (Chen et al. 2024a) and Flash-VStream VStream (Zhang
al. 2024a), processed videos at frames per second (fps) before feeding them into the models.
For each retro-memory and live-perception question, we used the following prompt as model input: System Prompt >< Video >< V Videollere >< /Video Based on your observations, select the best option that accurately addresses the question. Question. A) Optionl; B) Option2; C) Option3; D) Option4. Only give the best option.
For pro-response question, we provided the models with the following prompt as input: < System Prompt >< Video >< VideoHere >< /V Video > Based on your observations, follow the instructions carefully and explain your answers.
4.1 MAKING OFFLINE MODELS WORK ONLINE
Typical multimodal large language models process the entire video input before answering user questions. This inference paradigm inherently limits their ability handle online interactive tasks.
7

---

Published as a conference paper at ICLR 2026
Table RIVER Bench evaluation results regarding the core online video understanding capability. "#F" indicates the total number of frames processed by the model during inference. "Loc" measures how precisely the model's response falls within the correct time window Pro-Response task. "ø" means that some models do not have the corresponding capabilities. All numbers in the table are presented in percentage (%). The gray background indicates methods that either natively support online inference have been adapted to enable online inference capabilities.
Retro-Memory Sive-Pereeption
Pro-Response
Models
LLMs
Instant
Streaming
OE
MC
OE
MC
Loc
MC
OE
Closed-Souree Models
GPT-4o
50
39.09
59.56
40.08
61.05
Ø
Ø
1.63
Gemini- 1omini-1..5prr 1.5-pro
50
24.24
36,35
34.53
52.19
1.51
Open-Souree Models
VidcoChat2
Mistral-7B
Jiang et
2023
19.41
31.97
18.76
29.28
Ø
2.21
VL25
IntemLM-881
et
2024
25.07
42.68
29.34
43.65
Llava- Video
Qwen2-7B
Yang ct al.
2024
24.94
46.00
27.57
41.00
VideoChat- Flash
Qwen2-7B
Yang et
2024
21.91
41.51
27.91
41.44
3.68
VideoChat2
Mistral-7B
Jiang ct al. 2023
1fps
19.65
35.52
23.66
41.16
4.51
28.98
4.58
1L2e
InternL--81
[Cai al.
2024
1fps
21.13
39.72
34.68
58.84
17.95
30.86
4.09
Llava- Video
Qwen2-78 7B
Yang et al. 2024
ffps
24.51
42.71
33.87
51.38
19.50
27.55
6.21
VidcoChat-Flash
Qwen2-7B
Yang ct al. 2024
1fps
25.68
45.75
33.60
56.35
20.24
35.90
Vicuna-7B
(Chiang ct al.,
2023
ffp
10.43
27.28
12.43
29.28
1.31
As illustrated in Figure 4 we propose framework that integrates sliding window sampling strategy with long-short term memory module, enabling the models to support online inference capabilities, while simultancously maintaining coherent temporal awareness across extended video sequences.
We employ a sliding window approach with a sampling rate of 1 frame per second (fps) for processing long video inputs. The window length is set to the recommended optimal number of frames for the models. The short-term memory consists of video frame tokens from the current window. The long-term memory module comprises compressed tokens from video frames prior to the current window, with a fixed length of M memory slots. Each memory slot contains the same number of tokens as the short-term memory. In our implementation, configurations for all models and kept the number of added long-term memory tokens consistent to ensure fair comparison. Since VideoChat2-HH 2023b) as the connector between visual features and the LLM, which generates a fixed number of tokens per slot, we adjusted the number of slots instead to align the total count of memory tokens. Humans tend to integrate events occurring within adjacent time intervals and abstract them into higher-level semantic representations (Nader & Einarsson] 2010). Inspired by this, we maintain long-term memory slots using nearest-neighbor averaging strategy.
maintained the original optimal
(Li al., 2024b) employs Q-Former (Li et al.
During user query inference, we explicitly equip the model with timeline information by syn- thesizing context from both short- and long- term memory directly into the system prompts, which includes precise timestamps and key vi- sual features. Specifically,
Table 3: Online evaluation results regarding the Pro-innicinttion capability.
Pro-Response
the follow- ing prompts: The following video tokens con- tain long memory of 0.0 to [:] seconds. < Video >< Long-T erm >< /Video >> The following video tokens contains a short memory sampled from [:] to {:) seconds. < Video ><
Models
#F
Instant
Strcaming
Loc
MC
OE
Sars--Streaam
1fps
1.31
VileoLLMMOOnine
25ps
23.88
6.67
4.41
MiteoLLL-Onliee..rrr
21ps 33.28
9.84
5.03
ValeoLLM-Onlinc..qrr
4tps
35.16
10.53
5.47
Chort-Terme ></ /Video >.
4.2 TRAINING THE ONLINE MODELS.
We adopt an architecture aligned with MideoLLM-Onlinn (Chen et al. 2024a), comprising three core components: visual encoder, MI.P projection layer, and a large language model. Specifically,
8

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation results of Retro-Memory type questions.
Short
Medium
Long
Very Long
Models
LLMs
MC
OE
MC
OE
MC
OE
MC
OE
Closed- Source Models
GPT-4o
63.
41.99
63.26
41.99
58.01
38.12
52.21
34.25
Gemini- 1.5-pro
50
41.77
28:45
36.98
24.03
34.73
23.20
31.94
20.99
Open-Source Models
VideoChat2
Mistral-7B
[Jiag al.
2023
16
32.04
19.34
32.32
20.44
31.49
19.34
32.04
18.51
V1.255
InternLM-885
(Cai
2024
16
46.13
29.28
47.51
26.52
39.78
24.03
37.29
20.44
Llava- Llava-Video
Qwen2-7B
Yang
2024
16
49.17
29.01
48.07
25.97
44.20
24.59
42.54
20.17
VideoChat-Flash
Qwen2-7B Yang al.
2024
16
44.20
22.93
43.92
22.93
40.06
22.38
37.85
19.38
VideoChat2
Mistral-7B (Jiang
2023
1fps
34.25
20.44
36.19
21.27
33.43
17.13
32.60
15.75
Intem
InterLLM-88 (Can al.
2024
1fps
37.02
22.10
39.50
21.27
32.60
15.47
30.66
12.16
Llava Video
Qwen2-7B Yäng al.
2024
1fps
44.20
24.31
43.65
24.59
37.02
19.89
37.29
VideoChat- Flash
Qwen2-7B Yang et al.
2024
1fps
43.92
24.86
48.90
29.28
41.44
22.10
38.12
18.55
Video
LLaMA3-SB
AA1]2024
2fps
3.87
3.87
3.32
Flash VStream
Vicuna-7B
Chuang al.
2023
1fps
28.73
10.50
25.97
9.39
27.90
9.12
26.52
12.70
encoder (Zhai et al.) connector to extract video frame representations at rate of 4 frames per second (fps), following the sampling strategy of LLaVA-1.5 (Liu et al. 2024a). Each video frame encoded as a tensor of shape ++3 3) d where d denotes the hidden dimension, the 3 term corresponds spatially sverage-pooles patch tokens, and the represents the global CLS token. These visual embeddings are interleaved with language tokens and jointly processed by the LLM. To enable efficient adaptation, we integrate Low-Rank Adaptation (LoRA) (Hu et al. 2022) into all linear layers of the LLaMA3-8B backbone (Al 2024).
we employ the
2023) coupled with a two-layer MLP
Consistent with VideoLLM-Onling our optimization objective combines standard language modeling (LM) loss with a streaming-speciin loss to enhance temporal responsiveness. However, distinct from their setup, we train for single epoch with learning rate of 3 x 10-5 utilizing DeepSpeed (Rasley ct al. 2020) with ZeRO-2 optimization to maximize memory efficiency and training throughput. Notably, our training data incorporates randomized timestamps for user queries rather than anchoring them at the default O-second mark (see Section 3.2). a design choice that significantly bolsters generalization across diverse real-world interaction scenarios.
4.3 EVALUATION RESULTS AND ANALYSIS.
We evaluate various multimodal large language models (MLLMs) the RIVER Bench, assessing their core online interaction capabilities. Table 2 compares native online inference models and enhanced non-native MILMM.. GPT-4o achieves the best performance, excelling in live-perception, retro-memory, and pro-response tasks.
Non-native models augmented with our online inference pipeline deliver highly competitive results. For live-perception questions, our approach outperforms native multimodal inference. In retro- memory questions, VideoChat-Flash, fine-tuned long videos, and Intern VL2.5, trained primarily on image-text data, exhibit divergent trends. We attribute this to VideoChat-Flass 's long-video compression design, which reduces sensitivity memory token compression, while Intern VL2.5 suffers performance drops without this mechanism.
Intriguingly.xxisting models that claim support for streaming video QA underperform substantially on our benchmark. We attribute this underperformance to two primary factors. First, optimized for long-video comprehension rather than interactive QA. Second, VideoLLM-Onling training relies an offline methodology, similar to LLaVA-1.5, which makes it susceptible to overfitting on specific scenes and predefined QA formats, limiting its adaptability to dynamic streams. RIVER Bench's 's pro-response training set improves results (Table 3) demonstrating our data's 's effectiveness for enhancing online interaction, particularly in proactive response tasks (1 28% accuracy improvement over baseline).
Fine-tuning VideoLLM-Online

---

Published as a conference paper at ICLR 2026
45
w memory
Table 5; The evaluation results for different types of visual cues. FC: fine-gran era-gga visual cue. CC: causal cue. BC: background cue.
Malti Multi-choike choice accurncy(%)
40
memory
video-agent
PC
CC
BC
Models
LLMs
30
MC
MC
MC
VideoChat2
Mistral-7B
30.23
29.69
I1.emV11.2.5
Intern.M- 8B
16
48.74
37.36
46.23
Llava-Video
Qwen2-7B
16
53.16
38.25
47.04
VideoChat Flash
Qwen2:
47.55
36.44
08
30s
60s
300s
600s
1800s
3600s
Question
VidcoChat2
Mistral-7B
lfps
36.06
31.15
46.89
InterVV1.2.5
Intern .M
1fps
35.29
Minva-Video
Qwen2-7B
lfps
50.08
34.25
51.48
Figure 5: Memory curve of MLLM Li al. 2024a) and video agent (Fan et al., 2024 under different query time conditions.
VideoChat- Flash
Qwen2-78 7B
lfps
50.39
40.92
54.10
4.3.1 MODEL MEMORY CAPABILITY
Table shows performance on retro- memory questions across different durations. As recall du- ration increases, most models exhibit declining visual memory retrieval and reasoning abilities. is an exception. While its overall performance remains modest, it maintains con- sistent accuracy across durations. This result underscores the criticality of memory in MI.LMs, demonstrating that the choice of memory mechanism significantly impacts long-term information retrieval performance. Notably, models modified for online inference show improved performance on medium-to-long-tern memory questions.
4.3.2 MODEL. MEMORY CURVE
Figure D shows accuracy curves for recall-type questions across durations. Adding memory modules significantly boosts retrieval, cutting the performance drop-off (decay slope) by 12% compared to models without memory. Notably, in contrast to the classic Ebbinghaus forgetting curve (Hu ct al. 2013), MLLMs equipped with these modules exhibit superior retention stability within 1-hour timeframes, suggesting fundamentally different memory mechanisms from human cognition.
4.3.3 PERFORMANCE ACROSS DIFFERENT CLUE CATEGORIES
5 performance on retro-memory questions categorized by the type of visual cues: Fine- grained Cues (FC), which target specific object attributes or detailed appearances; Causal Cues (CC), which require reasoning about event dynamics and temporal dependencies; and Background Cues (BC), which focus on static scene context environmental settings. All methods perform poorly on CC questions, revealing their greater difficulty and highlighting the need for future work on visual perception integrated with event attribution. The modified VideoChat- shows improvements across all cue types, achieving best performance context-dependent questions.
CONCLUSION
This paper addresses the challenges in evaluating multimodal large language models for online multimodal interaction. We propose RIVER Bench, which quantifies key aspects like retrospective memory, pee-perception, and proactive response. Through comprehensive evaluation of different model categories, our benchmark effectively identifies their key limitations real-time interaction. Moreover, specialized fine-tunine dataset is introduced. After training this dataset, existing models show significant improvements in online video understanding, enabling better performance in dynamic, real-time scenarios.
Limitations and Future Work. Currently, our dataset does not include audio data. Given that sound is one of the most readily available modalities for real-time time interaction, integrating audio into the evaluation of online video content is crucial. In the future, we plan to update our video annotation data to incorporate audio.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by the Basic Research Program of Jiangsu (No. BK20250009), the National Key R&D Program of China (No. (022ZD0160000) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
This work was completed the Shanghai Artificial Intelligence Laboratory. We gratefully acknowl- edge the computational resources provided by the Shanghai Artificial Intelligence Lab, which made this research possible.
ETHICS STATEMENT
Our work involves the collection and annotation of videos sourced from publicly available platforms and existing open datasets. All data are selected in accordance with their respective licenses and terms of use, and we do not include any private, sensitive, or personally identifiable information. The benchmark focuses on general video understanding and interaction scenarios rather than sensitive human activities, and no clinical, offensive, or invasive content included. Annotation was conducted by trained annotators with clear guidelines to ensure accuracy, consistency, and fairness. Potential risks such as misuse for generating mislcading real-time interactions or unauthorized surveillance are mitigated by relcasing the dataset strictly for academic research purposes. We adhere to the ACM Code of Ethics in all aspects of this work, including dataset documentation, annotation transparency, and clear reporting of model limitations.
REPRODUCIBILITY STATEMENT
We have taken several steps ensure the reproducibility of our benchmark and experimental findings. Sec. 3 and Appendix A describe the video sources, selection criteria, and dataset composition, including duration and source distribution. Sec. detaills the construction of the real-time interaction tasks, namely Retrospective Memory, Live Percepeecn,, and Proactive Anticipation. Evaluation metrics and detailed experimental setups for all compared models are specified in Sec. All code for data processing, benchmark simulation, and evaluation, along with the benchmark data (under appropriate licenses), will be released in an anonymized repository to facilitate reproduction and extension by the research community.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
Meta Al. Introducing meta Ilama 3: The most capable openly available I1m date. https 1lama-a2 3/, 2024. Accessed: 2025-03-05.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et Rt-1: Robotics 2022.
transformer for real-world control at scale. arXiv preprint arXiv:2212.0683
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiy preprint arXiv:2403.17297 2024.
Kostas Cheliotis, Fotis Liarokapis, Margarita Kokla, Eleni Tomai, Katerina Pastra, Niki Anastopoulou, Maria Bezerianou, Athanasia Darra, and Marinos Kavouras. A systematic review of application development in augmented reality navigation research. Cartography and Geographic Information Science, 50(3):249-22 271, 2023.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-onlinee Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418, 2024a.

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv arXiv:2412.05277 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https?
// ///mays..rgg/bba 1ttpy//..rg/bloo//220000000000 20030g//222000000 30 30-
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171 4186, 2019.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: memory- augmented multimodal agent for video understanding. In European Conference on Computer Vision, pp. 75-92. Springer, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Iims in video analysis. arXiv preprint arXiv:2405.21075. 2024.
Chaoyou Fu, Haojia Lin, Xiong Wang. Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501 101957, 2025.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18995-19012, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)33 2022.
SG Hu, Y Liu, TP Chen, Zengcai Liu, Q Yu, LJ Deng, Y Yin, and Sumio Hosaka. Emulating the ebbinghaus forgetting curve of the human brain with a nio-based memristor. Applied Physics Letters, 103(13), 2013.
Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: A comprehensive benchmark and memory-augmented method. arXiv preprint Atl::2501.005844 2024.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651-4664. PMLR, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anat Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:://axxtt org/ 2310. 06825
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4565-4574, 2016.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706-715, 2017.
Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 1846-1 11588, 2021.
12

---

Published as a conference paper at ICLR 2026
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306005225 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195 22206, 2024a.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.0055 000744, 2024b.
Yifei Li, Junbo NIu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-Ilms from real-world online video understanding? arXiv preprint arXiv:2501 105510, 2025.
Jumming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streaminghench: Assessing the gap for milms to achieve streaming video understanding. arXiv preprint arXiv:241 1.03628, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
pp.
Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M Alvare. Streamchat: Chatting with streaming video. arXiv preprint arXiv:2412.08646, 2024b.
Karim Nader and Einar Ör Einarsson. Memory reconsolidation: an update. Annals of the New York Academy of Sciences, 1191(1):27-1, 2010.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 17:119336-11988 2025.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2020.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long vidco understanding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18221-18232. IEEE, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL http::/aaxxi. org/abs/24033005530, 2024.
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang. Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024a.
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-ommi: A smart and low latency ppeech-to-spece dialogue model with frozen Ilm. arXiv preprint arxiv:241 1. .00074, 2024b.
13

---

Published as a conference paper at ICLR 2026
Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Videollm knows when to speak: Enhancing teme-sensitive video comprehension with video-text duet interaction format. arXiv preprint arXiv:2411.1799) 2024c.
Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video under-
Standing with recurrent memory bridges. arXiv preprint arXiv:2409.0107) 2024d.
Pawel Weichbroth. Usability issues with mobile applications: Insights from practitioners and future research directions. arXiv preprint XXiv:2502.05120, 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context interleaved langu-an understanding. Advances in Neural Information Processing Systems, 01188888-285577 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 1rXiv:2412.15115, 2024.
Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video worth thousands of words. Advances in Neural Information Processing Systems, 37:57240-57266 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:2408.01 000000 2024.
Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang. Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, et al. Streamforest: Efficient online video understanding with persistent event memory. arXiv preprint arXiv:2509.24877 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-inain In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint drXiv:2406.08085, 2024a.
Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. internim-xcomposee 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.09556, 2024b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 168.22666.68... 16822, 2024c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024d.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
A.1 BENCHMARK DETAILS
RIVER Bench is primarily composed of two types of video data: one consisting of question-answer (QA) pairs with visual information, and the other containing only dense action annotations. The construction process of RIVER Bench is illustrated F
For the first category of data, first apply rule-based filtering process based the existing annotations. This involves removing questions that contain specific personal names, correspond to excessively long video segments, are overly general in nature. Subsequently, language model (based filtering step: if unimodal language model can correctly answer a question without access to the visual modality, the corresponding QA pair considered invalid.
employ large
The second category of data consists of videos annotated with dense, timestamped event descriptions. Redundant, trivial, repetitive annotations are removed using custom-designed rules. Next, compute sentence embeddings for each event and select the most distinctive one as the key event based on the similarity matrix. Finally, an LLM generates QA pairs using the annotation information along with predefined question template bank. These generated QA pairs then undergo the aforementioned LL-M-based filtering procedure ensure quality.
LLM
rules
Normal QA dalaset
Normal QA dalaset.
Filter and remove the following:
Questions contai ning special names
Filter and remove following:
Questions corresponding video
Questions solved by single-moda model
segments that are too long
Quastion solved using few frumes
Questions that too general
Dense annotation dataset Filter and remove the following:
Dense annotation dataset
video & annos
Select special activity Gemerate QA
Actions are loo genoric, such skanding
prodict, previous or fulure activity
up, walking looking around
Filter and remove the following:
Actions are repeated tno many times
Questions solved by single-modi model
The activity definition is broad
Question solved using few frames
Figure 6: Illustration of data processing.
To clarify the selection process for Retro-Memory and from Vript-RR Yang et al. (2025), L.V L./Bench Wang et al. (2024a), and Vongeieeenn Wu et al. (2025). which were filtered and reconstructed as follows:
questions, we used annotations
Vript-RR: A subset of the Vript-Hard benchmark, designed to address ambiguity in long-vides question answering. It contains 152 high-quality entries with detailed hints for locating relevant scenes and concise questions. These were directly incorporated for their quality despite the limited quantity.
LVBench: benchmark for evaluating video-language models (VLMs) on extremely long videos, assessing capabilities like temporal localization, summarization, reasoning, entity recognition, event understanding, and information retrieval. We excluded questions involving specific names or those spanning entire videos focus temporally relevant tasks.
Long VideoBench: A benchmark for fine-grate lne-grained multimodal retrieval and reasoning in long videos. We selected perception-related questions and removed those relying complete dialogue hints.
After this initial filtering, we obtain a set of valid hints and questions. We further apply multiple large language models (LLMs) to identify and remove low-quality ambiguous instances. Finally, for
15

---

Published as a conference paper at ICLR 2026
each event, randomly sample query timestamp after the event occurrence (for Retro-Memory) or synchronized with the event time (for Pev-Peercepion) forming the final annotations.
During the construction of Proactive Anticipation Instant questions, the following prompt is employed. The filtered, timestamped dense event annotations are provided as imput to a large language model
(LLM). Based on this input, the model generates semantically relevant questions together with correct answer and several plausible distractor options.
Prompt used for generating Proactive Anticipation Instant questions
You are a very intelligent multimodal assistant helps the user to do the following tasks:
Ploase complete the conversation between user and assistant. Note that the assistant will actively provides clear, concise, real-time language assistance. The assistant does not know the absclute time, so don't mention timestamp in response. Sometimes the user may ask irrelevant questions, the assistant is very helpful and will alsc answer that. Also, generate 3-5 wrong choices that are easy to confuse the assistant. Firstly, you may need to refine the question which is asking about things that will happen in the future. The only current key event. The wrong choices must contain the objects mentioned in the key event. All the pronouns questien and choices should be the same and correct.
correct choice must be closely arcund the
Annotations:
<timestamp 1>: <annotation
<timestamp 2>: <annotation 2>
<timestamp <annotation n>
Key event:
<timestamp <annotation
Output format:
Person: When <generated question>
Correct choice:
Wrong cheices:
The following is the template library used for synthesizing Proactive Anticipation Instant questions. Templates from this library are randomly selected generate the questions.
16

---

Published as a conference paper at ICLR 2026
Questions used for Proactive Anticipation Instant
Next
["What's the next step?", "What should I do after this?", "what action should take next?", "What's the next thing do?", "What am supposed to do next?", "What's the next procedure?", "What's the following procedure?", "What to be done next?", "What needs to be done next?", "What comes after this?", "What do I need to do next?", "What's s the following action?", "what are the next actions should take?", "what action is required after this?", "what is the subsequent step?", "What's s my next move?", "What should come next in this process?", "Mhat's the following step?", "What's the next task I should complete?", "What do I have to do after this?", "What will do next?", "What the "What should my next action be?", "What is my next step geing to bo?", "What do intend to do next?"
thing should de?",
Previous
"What happened?", "What is done?", "What has been done?", "What was done?", "what just happened?", "what did just do?", "what steps did just take?", "What was the process I followed?" "what occurred?", "What just took place?", "What actions led to this?", "What did just experience?", "What has just transpired?", "What decisions did I make?", "What just went down?", "What sequence of events led this?", "What happened right before this?", "What took place?", "What happened previcusly?", "What occurred in the past?", "What happened back then?", "what actions were taken?", "what happened earlier?", "What went on?", "what has occurred?"
A.2 PROACTIVE ANTICIPATION ABILITY
The hyperparameters and model configurations of our trained online interaction model are summarized in Table 6 The training data originates from the training set corresponding to the benchmark usage data, constructed using the same methodology. Given the sparsity and uniqueness of special event annotations, there is risk of data leakage.
Parameter
Value
attn implementation learning_ran rate optim Ir scheduler type
flash_attene 2athhhattenion 0.0002
adamw_tone torch cosine
model. (rojllllllpookkkkk.. aate:pojiii.pmnjlooooomoooiinnnn
lora aaraueoseess
lora_r Jora_alpha
128
256
frame fps
houne_resolution max max_nuu frames frame trame_tokeen__e token_okeeie cls frame_token token pooeedd embed_ mark
384
1024
True
[3.3]
143x3
Table 6: Training parameters of online model.
17

---

Published as a conference paper at ICLR 2026
A.3 LONG-SHORT TERM DETAILS
In the long-shore songsshor-ttem memory mechanism, as some MLLMs generate a large number of visual tokens when encoding images/video clips, excessive redundant information accumulates in the long-term memory window. This not only consumes significant memory space but also affects the similarity calculation between adjacent clips. Therefore, in the long-term memory module, we employ average pooling to downsample the visual tokens to a specified range. The number of frame features stored in the long-term memory is set to 16. Some details are shown in Figure 7 and Algorithm [
Prompt used for MILM inference using long-short term memory
Carefully watch the video and pay attention to the cause and sequence of events, the detail and movement of objects, and the acticn and pose of persons. Based on your observat inss, ons, select the best option that accurately addresses the question. Only give the best opticn.
This contains a long memory of 0.0 to 1:] seconds.
<Lngg-term Memory>
This contains short clip sampled from (:) to (:) seconds. <Short-term Memory>
Question:
<Question>
avg
0.26
0.78
0.44
Cosine Similarity
Vision Encoder
Window
Figure Design of long-term memory.
A.4 DIFFERENT TYPES OF VISUAL CUES
To investigate model performance across different question types, we classify each question using large language model (LLM). The classification prompt is designed to ensure balanced distribution across categories. A representative version the prompt as follows.
18

---

Published as a conference paper at ICLR 2026
Algorithm Pseudocode of long-short term memory.
video pre-processing memory bank initialization
video reshape and move_to,_a__ooooooooooee gpu (video) cong_embeddings []
short_embeddinge
batch processing
for batch in videc_batches:
with ()_ggrad()
embeddings Long_embeddings apwppeddddnns..ppddddddd.. (embeddings)
(batch)
short_embeddiings append (embeddings)
memory updat
if len (iong__embbedingg > max_memory;
similarities calculate similarities (1ong longg_eebediinges
calculate_simiiariiee
while 1en wew.enneeeediinig max_memory;
werge_most_simiaa___e htt.://l......aooooooooooooooaao
(Long_ebbeddingg,, similarities)
get final embeddings
einal_embeddinge = combine_embeddignn (Long_emeeddings, short_embeddings))
get response
question format_quest lquestion_texs options)
response get_model [r__model_response _esponse (model, questions, Final_embeddings)
Prompt used for classification process of different types of visual cues
You are an intelligent chatbot designed to categorize video-related questions into three predefined types based on their content. Your task is to analyze each question and determine which categery (1, 2, or 3) belongs to. Here's s how you can accomplish the task:
INSTRUCTIONS:
Carefully read the question and determine its main focus. Assign the question to one of the following categories: Category 1: Video content and details (specific scenes, objects, actions, colors, words, etc.)
Category 2: Characters and events (interactions, motivations, event summaries, reasons, etc.)
Category Scenes and ronientt (locations, weather, background, video type, etc.)
Your response should directly indicate the matching category without any additional explanation or text. You must choose one the three categories ('1', '2' or (3') for each question."
Please categorize the following videc-based question: Question: question
Provide your evaluation by directly outputting the corresponding category ('1', '2', '))). DO NoT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the matching category as a single string. For example, your response should look like this: "I"."
B DATA EXAMPLE
"video wvideo_source"; "LVBench", "video id" "Cm?3na6Ibes", "duration_ser 3665.5,
"Eps": 24,
19

---

Published as a conference paper at ICLR 2026
"question id"; "Cm73ma6Iocss"1",
"question": "what year appears in the opening caption of the
video?",
"choices":
(A) 1636",
(B) 1366",
(C) 1363",
(D) 1633"
"correct answer"; "D", "time reference"; [
15,
19
qqestton type": "Inmediate", "question time:: 19, "video ;edeo_ppth"; "00000000.mg mp4
"video _rideo_source": "LVBench", "video id" "Cm73ma6Ibes" "duration_sed" :ecoon_sscc: 3665.5,
"fps": 24,
"question_id"; "Cm73mm61bcs81"
"question"; "Khat year appears in the opening caption of the video?"
"choices":
(A) 1636",
(B) 1366",
(C) 1363",
(D) 1633"
1,
"correct_ "correcttanswer"" "D", "time_ remeeeereeeee" [
15,
19
},
"question : "Recalling@short?
"question_te ; 6.. 7467886009942
"video path": "00000000.mo mmp"
"video_ssurce" "aa "video id"
"duration_see sec": 74.5,
"fps": 4,
"question id"; "5Sa002a--tae--ceehhtttttSSSeee3
"choices":
(A) You tap on the table with your fingers.",
(B) You push something aside.",
(C) You operate the mouse",
(D) You pull something the table"
,,
"question": "Khen I rub the table with my hand, what should come next in this process?",
"correct answer"; "A" "time reference"; 43.3896682, "question_type" "Awaiting@short", ieaastionntime"" 10. 10.612453361 655376, "video_pattie
mp4
20

---

Published as a conference paper at ICLR 2026
"video_source" "EgosD-Narration--aa","" "video_ide ;id";
"duration_sed secioonssee:: 312.5, "fps": 4.0,
"question_type type": "Ongoing", "question_dd;; @yjooco-baa-/aahttoeeelllllllll1
],
"question_tiine time": [
270.8421305546634
305.95987055466338
"question:
"What can you tell me about? Be concise.", "simlly interpret the scene for mo.
"choices: [],
correct answer";
:eoereettannwer": {
"You file piece of furniture with hand file.", "You wipe the furniture with your right hand.",
"You remove the sellotape from the furniture."
"You pick up some sellotapes from the furniture with your right hand.", "
"You walk to front of the furniture.", "You touch the camera."
"time reference";
oppbe50-.haee/55--hbbeeeeel11111 270.8421305546634
275 8754550245554466314, 277 977.9583555446634
I,
"8 "ebeccoo-bbeettasabbeeeeelllll2"
305. 305.95987055466344
309 309.00634055466344,
311.0571705546634
]
Here are some rejected examples.
"uid": "83", "question": "what part of the cook is different from other people?",
"answer": "D",
@eiie_rrefeencee "03:00-03:03",
"choices":
(A) Leg",
(B) Foot",
(C) Eye",
(D) Hand"
"note": "easy guess"
21

---

Published as a conference paper at ICLR 2026
"uid": "3790", "question": "what is the stoppage time for the first half?", "answer": "B",
"time reference"; "47:10-47:15", "choices":
(A) 1 minute",
(B) 0 minute",
(C) 2 minute",
(D) 3 minute"
}, "note": "the closest one"
"uid": "4167",
"question": "What the content the entire video about?", "answer": "D", "00:00-118:28",
"time ses_me_reference"" "choices":
(A) This video narrates the entire process of the 2016 men's s volleyball final, in which the Brazilian team defeated China 3-0 to win the championship",
(B) This video narrates the entire process of the 2016
men's volleyball final, in which the Brazilian team defeated Italy 3-2 to win the championship",
(C) This video narrates the entire process of the 2020 men's s volleybal final, in which the Brazilian team defeated Italy 3-0 to win the championship",
(D) This video narrates the entire process of the 2016
men's volleyball final, in which the Brazilian team defeated Italy 3-0 to win the championship"
"note": "history event"
b
"uid": "1418", "question": "what is the intentien behind the resi denta showi the photos the video?", "answer": "A",
"time reference"; "05:13-05:59*, "choices":
(A) To inform the audience that some areas here were not submerged by water in the past",
(B) To remonst their photography hobby",
(C) To share memories of their childhood",
(D) To showcase their cooking skills"
},
"note": "preference to longer cheice"
"uid": "3415",
"question"; "what might a passenger think about vlogger from a third person perspective?", "answer": "C",
"time reference"; "00:00-37:34", "choices":
(A) Strange. Because talks to his phone",
(B) Normal, Because he talks to himself",
(C) Normal, Because he talks to his phone",
(D) Strange. Because he talks to himself"
},
"note": "easy to guess; the time reference span too long"
22

---

Published as a conference paper at ICLR 2026
Wvideo_source". "Ego-Egooh-Karaaicc-TTaaaa id": :
"video_ "duration_sec", 100.75, "fps": 4,
"question_id". id"; +230006-0222---aa-------00-00000
"choices"; II,
"question": "If you happen to netice 'I scroll through my phone' action, would greatly appreciate your immediate - alert.",
"You scroll through your phone.", "time reference": 98. 08.3071471999999,
"correct_answer
"question typee: "kwaiting@shortt,
"question_tines time": 67.47545180739994,
"qa": "Ferson: when I scroll through my phono, what do I intend to do ncxt?\nCorrect choice: Check messages AnWrong choices: Answer calls, Clean the floor, Turn the lights, Fut down the vacuum cleaner",
"qa question"; "When I scroll through my phone, what do intend
+ to do next?",
"qa correct answers"; "Check messages",
"qa wrong answers":
"Answer calls", "Clean the floor",
"Turn on the lights",
"Put down the vacuum cleaner"
},
"al1 choices abe":
(A) Clean the floor",
(B) Turn on the lights",
(C) Check messages",
(D) Answer calls",
(E) Put down the vacuum cleaner"
},
Ccorrect_choice___ee_ ___aaabb. "C",
"llm qwen": "ç", "lim pred 1lama": "C"
"Eg--BgoDD-Marattio--taai
"video_ id"; : "a33511166771--66--22222233888 "duration_sec"; 299.25,
"fps": 4,
"question_ide "8c336ba7 1elesbaaa-444---0 2ag_
"choices": [],
"question"; "Please send me an immediate alert when you notice A man interacts with woman' occurring.", "correct answer"; "A man interacts with woman.", "time reference"; 52. 52.741182399999999
"qqusstion_type": "Awaiting@short", "question_time" 0,
"qa": "Person: When a man interacts with woman, what did they do at that nenentPAninCorrect choice: They started conversation. Anurong choices: An- They sprinkled seasoning into the food, An- They placed the salt container on the sholf. \n- They mixed French frics in a pan, \n- They adjusted the cookwaro on the cooker.",
"qa question": "When man interacts with woman, what did they do that moment?",
correctt "atrrectaasswers"" "They started a conversation.". "qa wrong answers":
"They sprinkled seasoning into the food.", "They placed the salt container the shelf.".
23

---

Published as a conference paper at ICLR 2026
"They mixed French fries in pan.", "They adjusted the cookware en the cooker."
},
(A) They started conversation.",
(B) They mixed French fries in pan.",
(C) They adjusted the cookware on the cooker.",
(D) They sprinkled seasoning into the food.",
(E) They placed the salt container on the shelt."
1,
"correct chcpceehhhhhhhhheeeeeeeeeeeeeeee _beeaaab "A"
"11m Balm__pee___wee qwen": "A", 1alm_pred_1lamm "A"
"video source"; "Bg--lgoo--laaaationnnil
"wideo_id". "9c23c7aa- "duration_sec": 581.5,
"fps": 4,
"question_id" id": "9c23e7aa-226c (at2aaaa-2226---55---------0---0
"choices": [],
"question": "Should you see 'I clean bicycle' happening, don|u20193 hesitate to notity me immediately.",
"correct answer": "You clean bicycle.",
"time reference"; 564.3109939999, "question_type". "swaiting@short", "question_time": 581.6720716807865,
"qa": "Person: when I clean a bicycle, what happened
before?\nCorrect choice: tock tissue. AnWrong choices: I fixed the tire., I cleaned the place., I put the used tissue in the dustbin.",
"qa_ question"; "When I clean bicyele, what happened before?", "qa Sborrect_answerree "I took tissue.",
"qa_wwe anoog_anssers"
"I fixed the tire.",
" cleaned the place.",
"I put the used tissue in the dustbin."
1,
"al1 choices abe":
(()) I took a tissue.",
(B) I put the used tissue in the dustbin.",
(C) I fixed the tire.",
(D) I cleaned the place."
},
"correct Cnorrect_choice___ choicecatheleee abca: "A",
"llm pred qwen": "A" "1im pred 1lama": "A"
"video source"; TBo--gooD-Naerratiic-Taaaa
"video_id"" id"; (cabb88--351-22--a13--888--- "duration_sec"; 708.25,
"fps": 4,
id": "cacb6d38-
"choices": [],
promptly alerted if you identiff 'He washes his hands any situation.", "He washes his hands.", "time reference"" 610. 3921992,
"question": "I would like
"correct answer"; "correct_answer"
"question_type" type": "Avaiting@short",
"qquetiiontiim time": 569.730563347959,
24

---

Published as a conference paper at ICLR 2026
"qa": "Person: When he washes his hands, what's the next step?\nCorrect choice: He puts the soap back on soap dish. AnWreng choices: \n- нe puts the bowl in the sink. An-
He wraps the aluminum foil arcund the bowl. An- He puts the meat in the fridge.",
"qa question": "ühen he washes his hands, what 's the next step?",
cra_coorect_aoe sesreett_nsswrss. "He puts the soap back on the soap dish.",
"qa wrong answers";
"He puts the bowl in the sink.",
"Hee wraps the aluminum foil arcund the bowl.",
"iee puts the meat in the fridge."
},
"al1 choices abe":
" (A) He puts the meat in the fridge.",
(B) He puts the soap back on the soap dish.",
(C) He wraps the aluminum feil around the bowl.",
(D) He puts the bowl in the sink."
"correct wwwichceheheccccccccc _abe:: "B",
"11m Laam_pree______ :red_qqen": "B", "11m pilm 1lama": "B"
C DATASET OPEN SOURCE PROTOCOL
We adopt index-only release strategy respect original licenses and avoid video redistribution, which is shown in Table 7 We provide metadata, annotations, and download links, enabling reproducible research while ensuring legal and ethical compliance.
Table Dataset licensing and release strategy comply with legal restrictions.
Dataset
License / Restriction
Our Action for Release
Long VideoBench
CC-BY-NC-SA
4.0
(non-
Index-only release we provide JSON/CSV metadata and download links; videos are not re-hosted.
commercial only)
Vript-RR
Academic-use-only, redistri- bution
Index-only release questions, and timestamps only; users must download videos themselves under the orig- inal license terms.
release scripts,
LVBench
MIT (code) original video copyrights
Index-only release README explicitly states: "Video copyrights remain with origi- must comply with their
nal platforms; respective terms."
Ego4D
Ego4D Terms non-commercial, required)
(academic, registration
Index-only release our README directs users to the official Ego41) portal and li- cense page for data access. No videos or clips are included.
QVHighlights
Original repo forbids video re- distribution
Index-only release - we release only the an- notations and evaluation code; README provides direct link to the official reposi- tory for video access.
25


Reviews:
[{"review_id": "aPuLf0TVrS", "reviewer": "Reviewer_L8cs", "summary": "The paper targets online video interaction for MLLMs and proposes RIVER Bench, a benchmark and protocol to evaluate three competencies under streaming conditions: Retrospective Memory (recall past events; forgetting curves), Live-Perception (time-sensitive understanding with latency–accuracy trade-offs), and Proactive Anticipation (detect/forecast future states with correct timing). RIVER reformulates and precisely timestamps items from multiple sources (Vript-RR, LVBench, LongVideoBench, Ego4D, QVHighlights), yielding: Retro-Memory (≈1.5k MCQs across four recall intervals), Live-Perception (≈0.4k items), Pro-Anticipation (≈1.2k “stream” continuous narration + ≈1.4k “instant” trigger items). Quality control uses LLM + human filtering and distinctive time-anchored events. Experiments cover commercial models, “native online” models, offline models adapted via a sliding window + long/short-term memory wrapper (16 slots with NN-averaged compression), and a fine-tuned online model (SiGLIP encoder + LLaMA3-8B with LoRA; LM + streaming losses). Metrics include MC exact match, open-ended judging via Qwen2.5-72B, and a response accuracy (Res Acc) that credits answers emitted within a ground-truth time window. Findings: offline models that excel at single-turn QA struggle online; the proposed online adaptation narrows the gap and improves pro-anticipation (+11.28% over baseline); memory modules flatten forgetting by ~12%; causal-cue retro-memory is hardest.", "questions": "* Please detail the compression, slot update, and NN averaging mechanics, and report ablations on #slots, compression ratio, prompt format, and context tokens to separate memory design from context size.\n\n* How stable are results under alternative judges (e.g., Claude, Gemini) and prompts? For Pro-Anticipation, can you report latency error distributions (mean absolute time error), early/late penalties, and false-trigger rates in addition to Res Acc?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tClear, timely problem: shifts evaluation from offline video QA to interactive, streaming settings with explicit timing of query/cue/response.\n\n•\tThree-facet design: jointly measures recall, live perception, and anticipation, and ties performance to the temporal gap \\Delta (forgetting/anticipation curves).\n\n•\tProtocol precision: items carry exact timestamps; “instant” vs “stream” anticipation reflects real interaction patterns (trigger vs continuous narration).\n\n•\tMethodological baselines: shows how to wrap offline models for online inference (sliding window + memory), a practical recipe many will try.\n\n•\tAnalyses: memory-duration breakdowns; cue-type breakdown (fine-grained / causal / background); evidence that more frames/time help under this protocol (unlike some offline MCQ benches).", "weaknesses": "•\tData novelty & provenance: benchmark reuses existing datasets; novelty is primarily the reconstruction into an online protocol. That’s valuable, but the paper should be transparent about how much is new annotation vs relabeling/retiming, and quantify human effort & agreement.\n\n•\tMixed evaluation formats: retro-memory remains MCQ, while other parts use open-ended judged by Qwen2.5-72B. This mixture complicates cross-task comparability and may inherit LLM-judge biases (version drift, style sensitivity).\n\n•\tRes Acc metric under-specified: scoring “within a window” ignores latency error magnitude, early/late asymmetry, and false positives (spurious triggers). For stream narration, quality/coverage metrics (recall@IoU, redundancy, hallucination) are not detailed.\n\n•\tSampling & budget fairness: per-model frame budgets differ (e.g., 50 frames for GPT-4o vs 16 for several open-source; 4 fps for “native online”), and the adapted wrapper uses 1 fps windows + 16 memory slots. Without token/pixel budget normalization or policy ablations (uniform vs shot/motion-aware), comparisons may be confounded.\n\n•\tMemory module specificity: “nearest-neighbor averaging” over 16 slots is only lightly described. How are keys built, how is compression performed, what is the retention/refresh policy, and what’s the context token cost? Gains could stem from more tokens, not better memory.\n\n•\tQuality control dependence on LLMs: LLMs filter “answerable-without-video” items and help synthesize anticipation questions/distractors, which can inject generator artifacts and style priors. Human audit/IAA and leakage checks are not quantified.\n\n•\tForgetting-curve confounds: longer recall intervals may correlate with different content types/difficulties. Matching controls (same target type at different lags) are not described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o4HpBqvSFS", "reviewer": "Reviewer_9Ctw", "summary": "This paper proposes RIVER Bench, a benchmark for online video interaction evaluating multimodal LLMs (MLLMs) on three temporally grounded competencies: live-perception, retrospective memory, and proactive anticipation. The authors formalize an online video-text-to-text task with explicit timestamps for query, clue, and response. They offer task taxonomies, dataset statistics, and metrics that jointly assess accuracy, timeliness, and latency–accuracy tradeoffs. RIVER aggregates and restructures items from Vript-RR, LongVideoBench, LVBench, Ego4D, and QVHighlights into precisely timed QA pairs and streaming tasks. The evaluations make a comparison between public and closed-sourced MLLMs, and demonstrate sizable gaps between offline QA success and real-time performance.", "questions": "1. Can you add a study equating tokens/frames/FLOPs across models (e.g., standardizing to a fixed #frames or fps) to isolate modeling differences from budget differences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper creates an accurate online task formalization with retro/live/pro-anticipation split and timing semantics. In addition, the windowed formulation ties the accuracy to when the answer is generated.\n2. Curated and broad construction across long videos, equipped with explicit filtering to mitigate language-only priors and ambiguous items.\n3. Operational online protocol that makes many offline models evaluable in real time, which enables informative cross-family comparisons.\n4. Comparatively useful metrics and analyses, which include duration-stratified forgetting curves, cue-type breakdowns and proactive Res Acc.", "weaknesses": "1. OE scoring depends on Qwen2.5-72B; prompts, thresholds, and sensitivity analyses are not deeply reported. The Res Acc window width and tolerance are not exhaustively justified; in addition, user-centric latency–utility tradeoffs are not evaluated as well.\n2. From my perspective, even though the pipeline filters items that are language-answerable, deep LLM participation risks unintended stylistic mimicry and inherent bias.. More transparent human IAA and QA metrics would be quite helpful.\n3. The non-public models use the determined 50 frames the meanwhile, others utilize 16 frames or 1 frame per second streams; the differences in budgets are able to confound absolute rankings. More apples-to-apples ablations would strengthen the claims to a great extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "39xxC1N0T5", "reviewer": "Reviewer_HLZ6", "summary": "This paper presents RIVER Bench (Real-tIme Video intERaction Bench), a new benchmark designed to evaluate online and real-time video understanding in multimodal large language models (MLLMs). Unlike traditional offline settings, RIVER Bench defines three interactive task types — Retrospective Memory, Live Perception, and Proactive Anticipation — to simulate dynamic, conversational interactions with ongoing video streams. Comprehensive annotations and evaluations reveal that while existing offline MLLMs perform well on static QA tasks, they struggle with real-time comprehension and long-term consistency. To address this, the authors propose a general enhancement method that improves model adaptability and responsiveness in real-time scenarios, setting a foundation for future interactive video models.", "questions": "1. How is the response time window for the Proactive Anticipation task defined (e.g., fixed ±1s, or dynamically adjusted based on event duration)? Will the authors consider adding an uncertainty prediction metric (e.g., model confidence in its predictions) to assess decision-making in ambiguous scenarios?\n\n2. What is the proportion of samples in the Retrospective Memory task for the \"extremely long duration\" (>3600s) category?\n\n3. The experiments show that all models perform poorly on causal clue (CC) questions. What specific technical directions do the authors suggest for improving the integration of visual perception and event attribution in such tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper fills a clear gap by introducing RIVER Bench for real-time video interaction, moving beyond the traditional offline paradigm. Its design with Retrospective Memory, Live Perception, and Proactive Anticipation tasks realistically mimics dynamic, interactive scenarios.\n\n2. The dataset is diverse and well-annotated, combining multiple video sources with fine-grained temporal labeling and strong quality control, ensuring high reliability.\n\n3. The experiments are comprehensive, covering various model types and introducing well-designed metrics (e.g., response localization, memory decay) that align with real-time understanding needs.\n\n4. The method is simple yet effective, using a sliding-window and long-short memory mechanism to adapt offline models for online use, showing clear performance gains (+11.28% on RIVER).", "weaknesses": "1. RIVER Bench only supports video-text interaction, not including audio, which is crucial for real-time tasks like voice navigation or human-robot interaction. While this is mentioned as a limitation, it would be helpful to test and report ASR performance, which would increase the benchmark’s practical value.\n\n2. The data primarily comes from Ego4D-Narration, focusing on simple, static tasks like desk operations and furniture organization, and lacks more complex dynamic scenarios such as traffic flow prediction or industrial fault detection. This could limit the benchmark’s generalizability to real-world applications.\n\n3. Many experimental settings lack ablation studies, such as the impact of memory compression strategies or the number of memory slots. Additionally, inference latency (e.g., per-frame processing time, response delay) is not reported, which is crucial for real-time interaction. Key details like frame sampling rates and the computational cost of memory modules are missing, making it hard to assess the feasibility of this method in actual deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2DlkvPsllF", "reviewer": "Reviewer_ya2u", "summary": "RIVER Bench proposes a benchmark for real-time video interaction, shifting evaluation from offline QA to online, temporally grounded tasks. It formalizes three competencies—Retrospective Memory, Live-Perception, and Proactive Anticipation—with precise query, cue, and response timings. Built from curated long-video sources, it spans 1,067 videos and 4,278 questions, enabling forgetting-curve and latency–accuracy analyses. The authors also adapt offline MLLMs via sliding windows plus a long–short memory module and release training data that improves proactive responses. Experiments across closed- and open-source models expose gaps in online processing and show gains from RIVER-based fine-tuning.", "questions": "**1.** How would adding synchronized audio (speech and environmental sounds) alter forgetting curves and response timing? Which fusion (early vs. late; streaming ASR alignment) best supports Pro-Anticipation? \n\n**2.** Can the fixed 16-slot long-term memory evolve into content-adaptive eviction/merge policies or learned key-value compression? What are the latency/recall impacts over 1-hour horizons? \n\n**3.** Do memory and anticipation degrade differently across egocentric vs. third-person sources, scene types, or demographics? Could per-subset breakdowns and bias diagnostics guide responsible deployment? \n\n**4.** What is agreement between different LLM judges and humans for open-ended scoring, and could pairwise preference tests or reference-free metrics capture timing quality without a single judge model? \n\n**5.** Beyond coarse time bins, can item-response curves and psychometric difficulty by cue type (fine-grained, causal, background) support curriculum design and targeted fine-tuning? \n\n**6.** How do results transfer to embodied/assistive agents where interruptions, safety stops, or missed cues have costs? Can RIVER simulate intervention thresholds and recovery from misfires? \n\n**7.** Will future releases broaden domains (AR, robotics, meetings, sports) and ship licenses, prompt templates, and rejection sets so the community can audit and reliably reproduce results?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**1. Precise task formalization and timing.** Clear definitions of Retro-Memory, Live-Perception, and Pro-Anticipation with explicit query/cue/answer timestamps yield faithful temporal grounding and enable memory-decay and response-latency analyses. \n\n**2. Broad coverage with actionable baselines.** 1,067 videos and 4,278 questions assembled from multiple long-video sources; results reported across closed/open models with consistent prompts and a streaming setting support fairer comparisons. \n\n**3. Practical online adaptation.** A sliding-window plus long–short memory design and a targeted training set improve proactive response metrics and repurpose offline MLLMs for streaming use.", "weaknesses": "**1. Novelty of method.** The “make-offline-models-online” recipe (sliding windows + long–short memory with nearest-neighbor averaging) feels incremental amid prior memory-cache approaches for streaming video.\n\n**2. Comprehensiveness of the benchmark.** Despite 1,067/4,278 scale, sources cluster around a few public datasets (e.g., LVBench, LongVideoBench, Ego4D, QVHighlights), leaving underexplored domains like AR navigation, robotics, multi-party meetings, or broadcast sports—potentially limiting ecological validity. \n\n**3. Audio omission.** The current release excludes audio, a core signal for live interaction (speech, alarms, ambience).\n\n**4. Evaluator reliance and metric scope.** Open-ended answers are judged by a single LLM (Qwen2.5-72B), and Pro-Anticipation timing is reduced to response-within-window (“Res Acc”). Broader, judge-agnostic metrics (timing error, hesitation penalties, continuous narration quality) could reduce bias. \n\n**5. Closed-source comparability.** GPT-4o/Gemini are limited to 50 frames while “online-ized” open models run at 1–4 fps; heterogeneous context budgets and sampling policies risk conflating interface limits with modeling capacity. \n\n**6. Data generation and leakage risk.** LLM-synthesized anticipatory questions/distractors, though filtered, may import language priors; publishing templates and rejection sets would help audit artifacts and overfitting. \n\n**7. Robustness to timestamp noise.** The benchmark assumes precise cue/query/answer times; stress-tests under timestamp jitter, dropped frames, or (future) ASR lag would better mirror deployment. \n\n**8. Scalability and latency costs.** The long-term memory keeps 16 slots mirroring short-term token size. Growth behavior, pruning policy, and on-device latency/compute trade-offs remain under-explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)