StreamForest: Efficient Online Video Understanding with Persistent Event Memory
Authors: Xiangyu Zeng
Kefan Qiu
Qingyu Zhang
Xinhao Li
Jing Wang
Jiaxin Li
Ziang Yan
Kun Tian
Meng Tian
Xinhai Zhao
Yi Wang
Limin Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: MLLM
Video Understanding
Primary area: deep_learning

Abstract: Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in video understanding. However, their effectiveness in real-time streaming scenarios remains limited due to storage constraints of historical visual features and insufficient real-time spatiotemporal reasoning. To address these challenges, we propose StreamForest, a novel architecture specifically designed for streaming video understanding. Central to StreamForest is the Persistent Event Memory Forest, a memory mechanism that adaptively organizes video frames into multiple event-level tree structures. This process is guided by penalty functions based on temporal distance, content similarity, and merge frequency, enabling efficient long-term memory retention under limited computational resources. To enhance real-time perception, we introduce a Fine-grained Spatiotemporal Window, which captures detailed short-term visual cues to improve current scene perception. Additionally, we present OnlineIT, an instruction-tuning dataset tailored for streaming video tasks. OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction. To evaluate generalization in practical applications, we introduce ODV-Bench, a new benchmark focused on real-time streaming video understanding in autonomous driving scenarios. Experimental results demonstrate that StreamForest achieves the state-of-the-art performance, with accuracies of 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench. In particular, even under extreme visual token compression (limited to 1024 tokens), the model retains 96.8% of its average accuracy in eight benchmarks relative to the default setting. These results underscore the robustness, efficiency, and generalizability of StreamForest for streaming video understanding.

StreamForest: Efficient Online Video Understanding with Persistent Event Memory
Xiangyu 1.2 Kefan Qiu"!, Qingyu Zhang"), Xinhao Li', ,Jing Jaagg,Jiaxiin Li'
Ziang Yan Kun Tian', Meng Tian Xinhai Zhao', Yi Wang?, Limin Wangf
Nanjing University SShanghai Al Laboratory
*Zhejiang University
Noah's Ark Lab, Huawei
YYiwaang Intelligent Tech.
Abstract
Multimodal Large Language Models (MLLMs) have recently achieved remark- able progress in video understanding. However, their effectiveness in real-time streaming scenarios remains limited due to storage constraints of historical visual features and insufficient real-time spatiotemporal reasoning. To address these challenges, propose StreamForest, novel architecture specifically designed for streaming video understanding. Central to StreamForest is the Persistent Event Memory Forest, memory mechanism that adaptively organizes video frames into multiple event-level tree structures. This process is guided by penalty functions based temporal distance, content similarity, and merge frequency, enabling efficient long-term memory retention under limited computational resources. To enhance real-time perception, we introduce Fine- Fine-grained Spatiotemporal Window, which captures detailed short-term visual cues to improve current scene percep- tion. Additionally, we present OnlinelT, an instruction-tuning dataset tailored for streaming video tasks. OnlinelT significantly boosts MLLM performance in both real-time perception and future prediction. To evaluate generalization practical applications, we introduce ODV-Bench, new benchmark focused on real-time streaming video understanding in autonomous driving scenarios. Experimental results demonstrate that StreamForest achieves the state-of-the-art performance, with accuracies of 77.3% on StreamingBenchh 60.5% on OV Bench, and 55.6% on OVO-Bench In particular, even under extreme visual token compression (limited to 1024 tokens), the model retains 96.8% of its average accuracy in eight benchmarks relative to the default setting. These results underscore the robustness, efficiency, and generalizability of StreamForest for streaming video understanding.
Introduction
In recent years, multimodal large language models have made significant progress in video under- standing tasks, demonstrating strong semantic comprehension and reasoning capabilities across videos of varying durations and scenarios 30 52 35). Benefiting from large-scale pretraining and enhanced tross-modal modeling capabilities, these models have been widely adopted in various
domains 228] 42 55 59]. However, with the growing demand for real-time intelligent processing
in online applications such autonomous driving [49], live video streaming (6]. and robotics [79], researchers have increasingly shifted their focus from conventional offline video understanding to the
more challenging task of streaming video processing [5] 23 46
Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Previous sata (OHline video MLLM)
Streamilorest Visual Token Limit)
StreamForeet (8k Visual Taken Limi)
Aduntry stulay
77.3
70.0
70.2
$9.9
60.5
61.4
01.7
62.
55.5
55.6
$7.2 S0.5
Bench
BeramingBanch
CVO
VideoMME
MLVU
Parceptin
Online Video QA
Long Video QA
Short Video QA
Neew could
un seed
sedeert
aoutt spter link daring driing whin maiinnrameet on crtmwalls
Streamiooeet
StreamForest
User
StreamFForest
Real-time perception
Future prediction
Post memory
StreamForest achieves strong performance across various evaluation benchmarks while using significantly fewer visual tokens. effectively handles key tasks in streaming video scenarios, including past memory, real-time perception, and future prediction.
Figure I:
In the field of streaming video understanding, efficiently caching continuously arriving video frame features remains a long-standing and challenging problem. To mitigate the storage and computational overhead associated with past frames, prior work has primarily employed two strategies for visual
feature reduction: compression during sampling S 46 and compression during storage [52]
72 23
Compression during sampling reduces large portion of incoming visual features, which severely limits the model's capacity for fine-grained spatiotemporal reasoning. As a result, it can only perform coarse semantic summarization of the current scene. Conversely, compression during storage typically involves merging or discarding adjacent frames based on inter-frame similarity. While more memory-eeficient, this strategy is susceptible missing critical foreground actions due background noise. It may also result in excessive local merging, introducing spatiotemporal irregularities that degrade the model's ability retain and reason about key events over time.
To address the challenges of streaming video understanding, we propose a novel architecture called StreamForest. At its core is the Persistent Event Memory Forest, mechanism designed to efficiently store and manage long-term visual information. This memory system enables a MILM to process ultra-long streaming video at constant rate of 1 fps by dynamically organizing video segments into a tree structure based on event boundaries. The merging of segments is guided by three penalty functions that consider temporal distance, content similarity, and merge frequency, ensuring an adaptive and meaningful memory hierarchy. To enhance real-time perception, we introduce a Fine-grained Spatiotemporal Window, which extracts rich local spatiotemporal features from nearby frames. This module enables the MLLM to better understand the current scene by focusing on temporally relevant visual context. We also present OnlinelT. a fine-tuning dataset specifically designed for streaming video understanding. OnlinelT improves the MLLM's ability to perceive the present moment and anticipate future events by leveraging both recent observations and long-term historical cues. It addresses the problem of hallucinations caused by spatiotemporal distribution shifts between past and current frames. In addition, we introduce ODV-Bench, new benchmark for evaluating streaming video understanding in autonomous driving scenarios. ODV-Bench emphasizes real-time perception and future prediction, providing systematic framework for assessing the generalization and real-world effectiveness of streaming video MI.MMs in downstream tasks.
We conducted extensive experiments on both online and offline video understanding benchmarks to validate the effectiveness of StreamForest. Under the default setting with visual token limit of 8192. StreamForest significantly outperforms previous state-of-the-art streaming video understanding MLLMs. It achieves an average accuracy of 77.3% on StreamingBench, 60.5% on OV Bench, and 55.6% on OVO-Bench. StreamForest also matches or surpasses the performance of leading offline video understanding MLLMs on both long and short video benchmarks, despite operating in a streaming video input setting. Moreover, StreamForest demonstrates strong resilience under extreme compression. With reduced visual token limit of just 1024, it retains 96.8% of its average performance across eight benchmarks compared to the default setting. These results highlight the robustness and efficiency of our approach in continuously processing streaming video input.

---

2
Related work
Multimodal Large Language Model. Extending multimodal capabilities from static images to dynamic video sequences introduces additional complexity, requiring models possess stronger abilities in modeling long-range dependencies and understanding events [30 40 81
41 71 58).
Recent advances MLLMs for video have introduced a variety of innovative strategies to tackle the challenges of efficiently processing and reasoning over long video inputs 37
29 51 33
LongVILA 7 proposes a Multimodal Sequence Parallelism system for long- congeconextt modeling. enabling efficient parallel training and inference on extended video content. However, most current research on video understanding remains focused on offline settings [57 69 62 34 where the model has full access to the complete video sequence before inference. Although this setting facilitates global semantic modeling, falls short streaming scenarios, where real-time understanding of continuously evolving scenes is required. Therefore, the development of models specifically designed for online video understanding is of critical importance.
Streaming Video Understanding. In real-world applications, users increasingly expect MILMS to support online processing and real-time interaction. This demand has prompted growing interest in the task of streaming video understanding. Recently, several works have explored this emerging
area [5] 72 63
23 However, most existing streaming video understanding approaches
12], focusing solely on summarizing semantic content from visual frames. As result, they struggle to handle essential tasks such as memory recall and real-time perception, which are critical for comprehensive streaming video understanding. Moreover, in pursuit of computational efficiency, many methods apply aggressive compression to video frame sequences [46] 74, 64), making them unsuitable for complex and dynamic tasks that require fine-grained and real-time spatiotemporal understanding, such as autonomous driving. To address these limitations, our goal is to develop a more generalizable and practical approach for online video understanding. It emphasizes fine-grad -ee-gra spatiotemporal features the moment of query and supports persistent memory storage based on events.
are primarily designed for streaming dense video captioning [5] 60 32
3 Methodology
3.1 Streaming Video Understanding Architecture: StreamForest
In this section, we detail our proposed StreamForest. Specifically, the core design of StreamForest lies in Fine- aine-grained Spatiotemporal Window and Persistent Memory Forest, which work in tandem to enable the model to retain long-term memories of past events while supporting real-time perception.
3.1.1 Fine-grained Spatiotemporal Window
To meet the real-time spatiotemporal perception requirements of streaming video understanding, we introduce the Fine-graines Spatiotemporal Window (FSTW). We observe that in practical applications, most of the clues requiring fine -rained spatiotemporal reasoning are concentrated near the time of the question. Therefore, we retain only second-level short-term fine-grained spatiotemporal features.
Specifically, the FSTW consists of two components: real-time perception and short-term spatiotem- poral memory. Real-time perception directly samples high-resolution visual features from the current frame, which are encoded with spatiotemporal positional information. As new frames arrive, older frames are compressed along the spatial dimension and transferred into short-term spatiotemporal memory. At the same time, the model computes inter-frame similarity between new and old frames to enable subsequent event-level segmentation. The short-term spatiotemporal memory maintains a frame sequence with a duration of t, seconds. When its capacity exceeded, overflowing visual features are offloaded into the Persistent Event Memory Forest. We segment continuous visual features into meta-events by identifying the position with the local minimum inter-frame similarity in the frame sequence. This ensures that each meta-event captures a coherent spatiotemporal transition. meta-event treated as an independent node, which consists of a collection of visual tokens from similar consecutive frames. These nodes form the foundation of the MLLM's long-term memory.
3

---

LLM
Answer
Projector
Question at 06:10
Persistent Memory
Fine-grainod Spatiotemporal Window
Seviiderm sjafttenpeed mettory
Viston Encodar
Nifory JEETT So
06:10
Video
preferenccal uurginn
preferenceat meryting
pro Jereenntioo manginng
stattarity
Ieter
FeBer merged
staclartty Stmilarit Penalty
More merged
Temporal Pesalty
Margo Count Penalty
Figure 2: Overview of our proposed StreamForest. The Fine-grained Spatiotemporal Window captures instance-level spatiotemporal features, while the Persistent Event Memory Forest adaptively organizes event-level representations into a set tree structures. Dashed arrows and feature tokens illustrate potential operations performed during each memory update iteration.
3.1.2 Persistent Event Memory Forest
To efficiently process continuously arriving video frame features in streaming scenarios, propose the Persistent Event Memory Forest (PEMF), a memory architecture specifically designed to support long-term memory in the context of streaming video. Unlike prior methods that rely direct inter- frame similarity compression [22 or static memory hierarchies [23). PEMF adaptively compresses and organizes video information at the event level. It constructs a hierarchical, tree-structured memory guided by three penalty functions, enabling the model to retain semantically rich and non-red non-redundant content while managing memory efficiently as it evolves over time. To control memory growth, we impose an upper limit Lg on the number of long-term memory tokens stored in PEMF. When this limit is exceeded, PEMF performs hierarchical memory consolidation by adaptively merging adjacent event nodes into single nodes within the tree structure. The selection of nodes for merging is guided by three penalty functions that account for temporal distance, content similarity, and merge frequency. ensuring that the memory remains both informative and compact.
Similarity Penalty. In long videos, adjacent video segments often exhibit high visual similarity, result- ing in substantial feature redundancy. Therefore, we introduce a similarity penalty that encourages the merging of event nodes with highly similar visual content. Due differences in event durations, two candidate event nodes (denoted as Ii, Ti+-1 may contain different numbers of visual tokens. To handle this discrepancy, inspired by ToMe B we adopt a bipartite graph matching approach. Specifically, we treat the visual features of the two event nodes as sets in bipartite graph and compute pairwise similarities between tokens across these sets. Let X, E R., denote the visual token features of the event node I,, where n, is the number of tokens in Z4. We compute the pairwise cosine similarity matrix S, = sim(X; X E R and select the top k; highest similarity scores, corresponding to the most similar token pairs between two event nodes. The similarity penalty P, is defined as one minus the average of these k; highest similarity scores:
Ps (211
=
s(P.2) (p-9)ET,
Σ
Te = arg (p.9) (p.))))))s))))) (S!P.)) ks).
(I)
Merge Count Penalty. When event nodes repeatedly participate in rre-structured hierarchical memory merging, their visual details may gradually degrade due to accumulated information loss. This degradation can lead to local spatiotemporal inconsistencies, ultimately impairing the accuracy of long-term video understanding. To mitigate this issue, we introduce a merge count penalty as regularization term. It penalizes overly merged nodes and encourages a more balanced memory integration process, thereby preserving the fidelity of each event representation. Let 6; denote the historical merge count of the event node 2,, with its maximum value at the query time denoted as

---

Cmaz. We define the merge count penalty Pm as follows:
Ci + c++1
Pm (141)
2Cman
Temporal Distance Penalty. In
streaming video understanding scenarios, frames that are temporally closer to the current query time often carry more relevant information. This observation suggests that recent visual features should be preserved with higher fidelity, while historical features can be compressed more aggressively. To implement this intuition, we introduce temporal distance penalty, which encourages the model to retain more detailed representations of temporally proximate events while promoting the forgetting of details from distant past events. Let t denote the current query or interaction time, tę denote the time of event i. The calculation of tę is detailed in the Appendix A We define the time penalty Pa follows:
Pz(,,, D+41) = d: + di+1
bg 4
d,
(3)
2
Overall penalty. We incorporate the above three penalties to guide the adaptive merging process of event nodes in the PEME, where the combination of these three factors determines the merge priority of event node pairs.
P(41) = w. Pr,P (xi,, + Wmm (1;,
wy Pr T1+1)-
The penalty weights Was Wm. and collectively determine the behavior of PEMF. When only the similarity penalty is applied, the strategy degenerates into similarity- based compression. Using the merge count penalty alone leads to behavior similar to uniform downsampling. When the temporal distance penalty is used in isolation, the method approximates FIFO. By adjusting these penalty weights, our method enables flexible trade-off among these strategies. allowing it to adapt effectively to various streaming tasks, enabling balance between efficient storage saving and the retention of task-relevant information across diverse real-world scenarios.
The nodes selected for merging are determined by identifying the pair with the lowest overall penalty score. We employ ToMe [3] for the merging process, compressing the number of visual tokens to half the total tokens of the selected node pair. Upon receiving a user query, the visual features of all root nodes in PEMF, along with all visual features stored in FSTW, are fed into the LLM to support real-time, streaming interaction.
3.2 Instruction-tuning Dataset: OnlineIT
Existing offline long video datasets often exhibit distributional bias, where the key evidence for answering questions is typically concentrated in the middle of the video. As result, MLLMs fine-tuned on such data tend to overemphasize historical content, potentially leading to hallucinations in accurately interpreting the current moment. Although some datasets for streaming video under- standing have been released [5] 60 63 47). they remain limited in terms of data volume, quality, and task diversity. To address these limitations, we construct OnlinelT, training dataset specifically designed for streaming video understanding. OnlinelT focuses on fine- mee-grained event comprehension and real-time spatiotemporal understanding in streaming settings, and it significantly enhances the performance of MLLMs on streaming video understanding tasks.
OnlinelT-general. Based on criteria of diversity, length, and difficulty, we curated and refined several existing high-quality fine-tuning datasets of streaming video understanding 23 60 47).
Building upon these, we further developed two new datasets comprising 32K high-quality streaming training instances. This dataset features a larger scale, broader distribution, and greater task diversity, facilitating the learning of more generalizable streaming video representations.
OnlinelT-drive. It includes 89K streaming QA training instances from autonomous driving scenar- ios. This dataset is designed to enhance MLLMs' performance on complex, real-time downstream tasks. Specifically. by integrating scene semantics, traffic regulations, and common driving events, we extract key elements from driving scenes and video clips to generate question-answer dataset grounded in autonomous driving contexts. OnlinelT--rrve primarily covers four areas: (1) real-time localization and semantic awareness, (2) understanding of static traffic entities, (3) understanding of dynamic traffic entities, and (4) risk event and accident assessment.
5

---

Gesumbtie
Serral Seraats
Wa
Jidan
R 281
Bey
Seyal
Filmer
bourd lighh The paral dnargphe heige woll cinscten
4-Quily Contrnl
tha dele foret deverten desanee 20
Driving Soctaroo
Adivoet sul optiee
Surpling Besagg
The Acodiiet
LA
Optin llaallig
fgen nlinelly
RAA Prodictie Aeciiot
(al Caagare Diariaaiion Bbxd
(c) Ocuenatonn pipdinc BoW-
Teak oumppec ODK-Baxd
Figure 3: (a) The distribution of task types and the number of QA pairs. (b) The detailed pipeline for constructing the ODV-Bench. (c) Typical task examples in
4
ODV-Bench
Many existing benchmarks for streaming scenarios are derived from offline video evaluation datasets 39 23 36 [1], and may not adequately reflect real-world applications of streaming video under- standing. Although some of them already incorporate Ego4D videos of daily activities [36 61], these evaluation samples primarily evaluate MLLMs' ability to perceive static scenes and narrate human-enviromment interactions in stepwise manner. In contrast, autonomous driving presents dynamic, high-stakes environments with rapidly changing scenes, complex multi-agent interactions (vehicles. pedestrians, and traffic signals), and demanding prediction tasks (such as risk assessment and motion planning). These scenarios require models to balance long-term event memory with fine-grained short-term perception to avoid accidents and make timely decisions. To address this gap, we introduce ODV-Bench, a benchmark specifically designed for online video understanding in autonomous driving scenarios.
4.1 Task Formulation
As shown in Figure 3 (a), we first explore the key traffic elements in autonomous driving scenarios and summarize them into three categories of task scenarios: (1) Static-target-orinneel tasks, which involve the recognition and retrieval of stationary traffic elements such as traffic signs. lights, and road indicators: (2) Dy maamii-targe--orrentt
tasks, which focus on behavior and trajectory prediction of dynamic road participants such as vehicles and pedestrians; and (3) Event-oriented tasks for multi-agent interaction, which capture complex interactions, risk scenarios, and accidents involving multiple agents. Next, guided by temporal cues and the practical needs of driving, we further define fine-grained task types based on these categories to comprehensively assess model understanding in realistic online driving video scenarios. For more details on task formulation, please refer to Appendix C.I.
4.2 Benchmark Construction
The construction process of ODV-Bench is illustrated in Figure (b). We adopt four-stage approach to ensure the quality of each generated question, and then present some typical task examples across different driving scenarios in Figure (c).
Data Collection. (1) Video Selection. To align with real-world driving scenarios, we first curated datasets [4 82 66 73, 4 from different task scenarios within the autonomous driving domain, from normal driving to unexpected events. Then, we designed a semi-automatic pipeline that primarily relies on annotation filtering and YOLO- detection (25). supplemented by manual inspection, to select task-relevant videos from the collected dataset. (2) Meta-Annotation Generation. To obtain meta- veta-annntattinss with detailed spatiotemporal and semantic information, we developed tailored methods based on existing dataset annotations. For well-annotated datasets, we effectively convert existing labels into task-specific meta-annotations. For others, we design semi-automatic pipeline that begins with coarse annotations generated by VLLMM and YOLO 25), followed by structured human verification to ensure quality.

---

Table Evaluation results on ODV-Bench. Our model significantly outperforms state-of the-art of- fline and online video MLLMs under zero-shot testing conditions, and achieves further improvements after fine-tuning on triving-domain data.
Static Target
Dynamic Target
Event Orlented
Method
Size Fraames
Overall
RTP
HD KIE TCD DDM PTM Ave.
DP Avg.
RP
RA
ARA Avg.
Human
Human Agents
96.8
97.6
98.2
95.7
95.9
94.4
95.9
83.7
87.9
90.4
88.2
91.9
94.9
93.0
92.5
91.4
Open-source Offine Video MILMs
MiniCMM-V26
65
7B
64
20.0
87.8
15.1
49.1
26.4
20.6
27.3
71.2
73,4
47.2
60.0
73,4
33.3
16.7
53.6
49.8
LongVA 2S
29.9
7.3
37.7
47.3
38.0
33.6
31.8
50.9
56.6
57.5
58.1
46.
56.7
50.2
LLaVA-Onevision
36.0
4,9
22.6
60.0
34.2
47.4
55.1
57.9
72.2
47.4
51.6
LL2
40.1
16.3
62.5
54.0
60.6
50.6
54.2
VideoChat
29.6
76.4
61.0
67.1
Qwen2.5.VL.
1fps
51.8
8.1
79.3
36.0
46.9
57.5
52.6
Open-sowrce Online Video MLLM;
Flach-VStrean
22
7B
1pps
25.4
1.6
11.3
50.9
36.0
22.1
24.8
25.5
39.8
47.2
40.2
32.4
48.6
30.1
38.1
35.7
Dispider 47
7B
fps
31.1
7.3
34.0
63.6
34.0
35.4
32.5
43.2
73.1
45.8
52.7
38.2
55.4
36.5
44.3
45.2
VideoChut- Online
33
4B
lfps
36.9
0.8
62.3
49.1
21.5
47.0
36
70.2
86.7
46.4
62.9
51.2
69.4
45.5
57.4
54.5
StreumForest
7B
lfps
51.4
15.5
54.7
56.4
38.6
65.3
51.5
72.6
83.2
46.0
62.3
60.2
73.3
47.4
63.8
59.9
StreamForest (F T-drive)
7B
1fps
70.1
17.1
100.0
60.9
32.7
83.6
64.6
64.0
96.6
39.6
70.7
71.8
93.4
58.3
78.5
71.2
QA Construction. (1) MCQ Generation. To enable efficient automatic QA generation, first design accurate and diverse templates tailored to each defined task. These templates are then populated with fine-grained and precise annotations to generate high-quality QA pairs. Next, we develop multiple-choice generation pipeline based on an option pool, introducing plausible yet misleading distractors alongside the correct answer to ensure the realism and effectiveness of choices. (2) Quality Control. To ensure benchmark quality. we first conduct multiple rounds of manual review to verify the clarity and accuracy of QA pairs and the plausibility of distractor options. Besides, to enhance scene diversity and task-type balance, we apply sampling strategy that allocates questions proportionally to video length, maximizing coverage across scenarios.
5 Experiments
Implementation Details. We adopt SigLiP-so400M[70 as the visual encoder, use an MLP as the projection head, and we employ Qwen2-7B as the LLM. By default, the number of visual tokens is capped at 8192. Among these, 729 tokens are allocated to real-time perception, while short-term spatiotemporal memory consists of 18 frames, each represented by 128 visual tokens. We set the penalty weights for similarity. merge count, and temporal distance to 0.4, 0.4, and 0.2, respectively. The model trained on 32 A100 GPUs using our proposed OnlineIT dataset, supplemented with offline video data from VideoChat-Flash
3 and LLaVA- Video (76). well as image data from LLa Vision 28). We adopt a five-stage training strategy to train StreamForest from scratch.
The first three stages follow the training paradigm of offline long video MILMs (3 The fourth stage performs streaming video fine-tuning to yield the base StreamForest. fifth stage can be incorporated by training with the OnlinelT-Drive, which yields the StreamForest(FTT drive). During the evaluation phase, we constrain the model to process streaming frames at 1 FPS. For more detailed implementation specifics, please refer to the Appendix D
addition, an optional
5.1 Online Benchmark Results
We evaluate the performance of our model on four benchmarks for online video question answering: ODV-Bench, StreamingBench (39), OVBench (23, and OVO-Bench (36). These benchmarks follow streaming video QA scenarios, where the VideoLLMs must process only the video content available before the current timestamp.
ODV-Bench. closely integrates spatiotemporal information to comprehensively evaluate MILMs' ability to understand fine-grained details in online videos and to make future predictions based on both historical and current context in autonomous driving scenarios. The benchmark includes tasks such as identifying subtle objects actions, describing object positions and spatial relations, and
7

---

Table 2: Comparison of our method with existing approaches on video question answering tasks across various scenarios. Our approach significantly outperforms previous methods on streaming video understanding benchmarks, while maintaining strong and competitive performance on both long and short video understanding.
Online Video
Long Video
Short Video
Method
Size
Berce Real-Time All
OV Bench Avg
OVO-Bench Overall
VideoMME
MLVU
MVBench PerceptionTest
w/o sub.
M-Avg
Avg
Open-source Offinee Video MLLMs
TntemVLL2
63.7
48.7
64.0
LongVA
75
LLaNa-OneVision
Qwen2-VL 55
7B
69.0
49.7
52.7
63.3
67.0
66.9
LongVU 50
7B
48.5
60.6
65.4
66.9
LLaVA. Video
76
7B
53.1
63.3
70.8
58.6
67.9
Open-sowrge Online Video MLLMs
M-deoli 5
8B
36.0
9.6
12.8
MovieChat
30.9
38.2
55.1
Flasb-VStream
72
23.2
31.2
33.2
VideoChat-Onlia
Dispider +
67.6
StreumFores
77.3
60.5
55.6
61.4
70.0
70.2
73.1
StreamForest (PT-drive)
7B
76.8
61.6
$3.6
61.9
69.6
68.6
71.6
forecasting object trajectories. These tasks require strong real-time spatiotemporal perception and contextual understanding. As shown in Table StreamForest achieves an average accuracy of 59.9% ODV-Bench without being trained OnlinelT--rive and further improves to 71.2% after training on it. This significantly outperforms generalization capability of our method downstream streaming video understanding tasks. These results highlight its potential for real-world applications.
existing online and offline MILLMs, demonstrating the strong
StreamingBench OVBench & OVO-Bench. shown Table StreamForest demonstrates
strong performance across existing open-source streaming video understanding benchmarks. It achieves an accuracy of 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench. These impressive results highlight the robustness of StreamForest in wide range of online video understanding scenarios. The superior performance of our model can be attributed to two key architectural innovations. First, the Fine- Fine-grained Spatiotemporal Window enables precise spatial perception and responsive short-term temporal modeling, which are critical for real-time perception and forward responding tasks. Second, the Persistent Event Memory Forest adaptively organizes long-term visual content into a structured and efficient memory forest, significantly enhancing the MLLM's ability to retain and reason over past events. Together, these two modules offer complementary capabilities that allow our model to handle dynamic, long-horizon streaming video inputs effectively, while maintaining high contextual coherence.
5.2 Offline Benchmark Results
We further evaluate method two long video understanding benchmarks (VideoMME and MLVU(77) and two short video datasets (MVBenchf) and PerceptionTTst(455 setting, the entire video is provided input to the MILM. We sample video frames at 1 FPS, with maximum limit of 2048 frames. For videos exceeding this limit, frames are uniformly sampled across the entire duration. As shown in Table our method demonstrates superior performance both long and short video understanding tasks compared to recent addition, it outperforms leading offline models in most benchmarks. achieving 61.4% VideoMME, 70.0% on MLVU, 70.2% on MVBench, and 73. 1 % on Perception This strong performance in offline scenarios highlights the robust generalization capability of our proposed method.
the offline
the-art online Video MLLMs. In
5.3 Ablations
Effectiveness of the Persistent Event Memory Forest: We replace the proposed PEMF with several methods used in previous work. To ensure fair comparison, we keep the visual token budgets consistent across all methods and fine-tune each model accordingly. The ablation results

---

Table 3: Comparison between our proposed PEMF and other commonly used memory strategies.
Table 4: Ablation study on the key components of StreamForest
OVBench OVO-Bench
MLVU
OVBench
OVO-Bench
MLVU
Memory Policy
Model
Avg
Overall
M-Avg
Avg
Overall
M-Avg
Unitorm Sampling
58.2
52.7
69.4
ESTW PEME
58.0
52.5
51.8
First First Out
58.7
52.9
56.7
FSTW
59.1
53.7
69.4
Similarity Merge
60.3
53.4
68.0
PEMF
58.9
53.5
56.6
Pyrumid Memory Bank
23
60.3
53.9
68.2
Event
52.6
69.1
PEMF (Ours)
60,5
55.6
70.0
Ours
60.5
70.0
are shown in Table The FIFO strategy shows the worst performance. This is especially evident on the long-video benchmark MLVU (56.7% vs. 70.0%), where the method fails due to unfiltered discarding of historical visual features. OV OVBench primarily emphasizes short-term, fine-grained spatiotemporal perception. Uniform sampling reduces the resolution of recent visual information, which is crucial for real-time understanding (58.2% vs. 60.5% on OVBench). Similarity Merge achieves performance comparable to our PEMF on OVBench (60.3% vs. 60.5%). However, its limitations become clear in tasks that require persistent memory and long- long-horizon reasoning, On OVO-Bench, PEMF outperforms Similarity Merge by +2.2%, and on MLVU by +2.0%. This is because Smilarity-basee merging may over-merge frames within local video segments, potentially leading spatiotemporal irregularities and the loss of local event-level representttionn The pyramidal memory bank maintains memory through frame replacement. However, fixed capacity limits its ability capture long-range spatiotemporal features (53.9% vs. 55.6% on -VO-Bench and 68.2% vs. 70.0 MLVU). In contrast, our method evaluates each visual event based event-level similarity, merge count, and temporal distance. Then it performs memory consolidation at the event level. This strategy supports efficient and persistent maintenance of historical visual features.
Effectiveness of the Overall Architecture: We conduct ablation studies on three key architectural Spatiotemporal Window and the Persistent
components. Specifically, we ablate the Event Memory Forest, while ensuring that the total number of visual tokens remains consistent with the original configuration. In addition, we replace event-based node construction with frame-based approach. As shown Table removing both modules leads to the most significant performance drop. Using either FSTW or PEMF alone improves performance compared to the baseline, but the best results are achieved when both components are integrated (+2.5% on OVBench, +3.1% on DVO-Beneh, and +18.2% on MLVU). This joint ablation confirms that FSTW and PEMF provide complementary benefits. FSTW enhances real-time spatiotemporal perception near the query timestamp, while PEMF supports efficient and persistent long-term memory, together yielding the strongest overall performance. Moreover, event-level node construction effectively prevents over-merging within events, enabling the compression of visual features the level of complete visual events rather than individual frames.
1403
1409
27
32
110.0
Figure 4: Performance under varying visual to- ken budgets.
Figure 5: Average inference time under varying
visual token budgets (on single A100 GPU).
Robustness to Different Visual Token Budgets: We evaluate the robustness of our method under varying budget constraints for visual tokens, ranging from 1K to 10K. Figure illustrates the performance variation on the benchmarks MLVU, VideoMME,, and OVO-Bench under these settings. Notably, under the strict constraint of only IK visual tokens, StreamForest achieves average

---

Table 5: Quantitative analysis of runtime and memory usage of StreamForest.
Table 6: Runtime comparison of PEMF with other memory mechanisms for 500 frames.
Input Frames
Memory (GB)
FLOPS (1)
I Lateney (5)
Viethd
Vis. Eneode (s) Mem. Update (8)
LEAE infer (8)
Total (6)
64
15.8
93.1
0.776
Simularity Merge 5
5.198
0.183
1,388
6,769
256
17.1
134.1
1.126
PMB 23
5.201
0.451
1.381
7003
1024
17.2
137.3
1.497
(Ours)
5.218
0.172
visual compression ratio of up to 99.8% on long video benchmarks. Despite this extreme level of compression, the model still maintains competitive performance, which strongly demonstrates the robustness of our approach in persistently preserving long-term, event-level visual memories. We also conduct direct comparison between our PEMF and the Similarity Merge. The results clearly demonstrate two core benefits of our approach. First, PEMF exhibits superior absolute performance, consistently outperforming Similarity Merge across all token budgets with an average accuracy improvement of 2-3%. Second, our method shows stronger resilience to extreme compression. Under the most severe 1K token budget, PEMF retains a higher fraction of its full-budget performance, achieving a notable +1.8% relative retention advantage on VideoMME. These experimental results confirm that the performance gains stem from the intrinsic design of PEMF, which adaptively consolidates event-level memory to preserve semantically salient information, thereby ensuring both high accuracy and token efficiency under stringent resource constraints. Figure J presents the average inference time of StreamForest under different budgets of visual tokens. The fast and stable inference speed highlights the practical applicability of StreamForest.
Computational cost: We provide quantitative analysis of the runtime and memory usage of StreamForest. To isolate the effect of the memory mechanism, we assume that frame-level visual features are already extracted by the vision encoder in real time. As shown in Table 5 PEMF enforces strict upper bound of visual tokens (8K here), ensuring that memory usage remains stable (I7 GB) regardless of the number of processed frames. Consequently, the FLOPs and inference latency do not grow significantly even with longer streaming inputs. In addition, we compare PEMF with other memory mechanisms, including the Pyramid Memory Bank (23) and the Similarity Merge strategy [52] As summarized in Tāble the overall runtime is dominated by vision encoding (5.2s for 500 frames, ~9 FPS). The memory update of PEMF is extremely lightweight 172s for 500 frames), which is negligible compared to the efficiency gains achieved by significantly reducing the total number of visual tokens.
Impact of Training Data: Table 7 presents the impact of our training strategy that integrates both online and offline datasets. The results clearly demonstrate that com- bining OnlineIT with existing offline VideoQA datasets significantly improves performance on streaming video un- derstanding benchmarks. OnlinelT is specifically designed for real-tima perception and future prediction streaming scenarios, effectively mitigating hallucinations caused by inconsistencies between historical spatiotemporal context and the current moment.
Table 7: The contribution of our train- ing data to performance the stream- ing video understanding benchmark. O.general refers to OnlinelT-gente generaa,, while O.drive refers to OndinelT--drvee
ODV-Bench OV Bence OVO-Bench
Data
Aug
Aug
Overall
base
56.3
53.9
53.5
goncral
59.9
60.5
55.6
base O generall d.drive
71.2
61.6
55.6
6 Conclusions
In this work, we have proposed StreamForest, a novel architecture for streaming video understanding that addresses the limitations in long-term memory and fine-grained perception. By introducing the Persistent Event Memory Forest, our method effectively manages historical visual information through adaptive merging guided by temporal distance, content similarity, and merge count penalties. Coupled with the Fine-grained Spatiotemporal Window, the model maintains precise understanding of the current scene. We also present OnlinelT, streaming video understanding fine-tuning dataset that mitigates spatiotemporal shift issues and enhances real-time perception and reasoning. As well as ODV-Bench, a new benchmark tailored for real-time autonomous driving scenarios. Extensive experiments demonstrate that Stream Forest not only outperforms oftt--of-the-art streaming video ML..Ms but also rivals top offline video MILMSs under strict streaming input settings, showcasing its robustness and practical value in real-time streaming video understanding applications.
10

---

Acknowledgement
This work is supported by the National Key R&D Program of China (No. 20222D0160900) the Basic Research Program of Jiangsu (No. BK20250009), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1]
Anthropic.
Claude 3.5 sonnet, 2024.
URL https:/www.anthrrpp..coo///
claude- 3- 5-sonett
[2] Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 139233, 2025.
[3] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
[4] Zhengping Che, Guangyu Li, Tracy Li, Bo Jiang, Xuefeng Shi, Xinsheng Zhang, Ying Lu, Guobin Wu, Yan Liu, and Jieping Yc. DP-city: large-scale dashcam video datasct of diverse traffic scenarios. arXiv preprint wwxi::9944.0 01975, 2019.
[5] Joya Chen, Zhaoyang Lv. Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao. Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-onlinee Online video large language model for streaming video. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407-18418, 2024.
[6] Joya Chen, Ziyun Zeng, Yiqi Lin. Wei Li, Zejun Ma. and Mike Zheng Shou. Livecc: Learning video Ilm with streaming speech transcription at scale. arXiv preprint arXiv:2504. 160020004..00.... 2025.
[7] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang. Shang Yang, Zhijian Liu, al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[8] Zhe Chen, Weiyun Wang, Yuc Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries open- source multimodal models with model, data, and test time scaling. arXiv preprint arXiv:2412.05271, 2024.
[9] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye. Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.
[10] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo. Deli Zhao. al. Videollama Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint urXiv:2406.07476, 2024.
[11] Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint 2003000500,0 2025.
[12] Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai. Zhibo Chen, and Ting Cao. Streammind: Unlocking full frame rate streaming video dialogue through event-gated cognition. arXiv preprint arXiv:2503.0622 2025.
[13] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Deng. Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In CVPR, pages 5374-5383, 2019.
[14] Jianwu Fang, Lei-lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv, Jianru Xue, and Tat-Seng Chua. Abductive ego-view accident video understanding for safe driving perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 22030-22040, 2024.
[15] Jiajun Fei, Dian L., Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-atiention masks for short and long videos. arXiv preprint 2408. 14023, 2024.
[16] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-min modal Imm in video analysis. arXiv preprint 2rXiv:2405.21075. 2024.

---

[17] Jiyang Gao, Chen Sun. Zbenheng Yang. and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, pages 5267-5275, 2017.
[18] Chunhui Gu, Chen Sun, Sudheendra Vijayanarasimhan, Caroline Pantofaru, David A. Ross, George Toderici, Yeqing Li, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik, Ava: video dataset of spatio-temponee temporally localized atomic visual actions. CVPR, 2017.
[19] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower 1lm to grasp video moments. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271-14280, 2024.
[20] De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localizeiee assistant, European Conference on Computer Vision, pages 202 218. Springer, 2024.
[21] Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video captioning. In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, P Proceedings of the 1st Conference the Asia-Pacific Chapter the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, December 2020.
[22] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-1Ok: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):
2021. doi: 10.1100/PA...2199.00 2957444..
[23] Zhenpeng, Huang, Xinhao Li, Jiaqi Li, Jing Wang. Xiangyu Zeng. Cheng Liang, Tao Wu, Xi Chen, Liang Li. and Limin Wang. Online video understanding: Ovbench and videochat-online. In CVPR, pages 3328-3338, 2025.
[24] Salman Khan, Izzeddin Teeti, Reza Javanmard Alitappeh, Mihaela C. Stoian, Eleonora Giunchiglia, Gurkirt Singh, Andrew Bradley, and Fabio Cuzzolin. Road-waymo: Action awareness at scale for autonomous driving, arXiv preprini arXiv:2411.066. 211.0063,, 2024.
[25] Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements. arXiv preprint 2:Xiv:2410.17722 2024.
[26] Ranjay Krishna, Kenji Hata, Frederie Ren, Li Fei-Fei, and Juan Carlos Nichles. Dense-captioning events in videos. ICCV, 2017.
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 23:32-733, 2017.
[28] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Fasy visual task transfer. arXiv preprint 2408:2200003322 2024.
[29] Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofci Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. stava-st: A multimodal large language model for fine-grained spatial-temporal understanding. arXiv preprint 2xXi:25001.82855 2025.
[30] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videnchat: Chat-centric video understanding. arXiv preprint 2335:2305.06355, 2023.
[31] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: comprehensive mouti--odal video understanding benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 22195 222922220,, 2024.
[32] Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. arXiv preprint arXiv:2503.088: 036663 2025.
[33] Xinhao Li. Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang. Jianfei Gao. Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-fias flash: Hierarchical compression for long-context vidco modeling, arXiv preprint ar Xrvvv:2001. 05540444 2024.
[34] Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng. Yinan He, Yali Wang. Yu Qiao, Yi Wang. and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforcement tine-tuning arXiv preprint 2rXiv:2504.06950 2025.
12

---

[35] Yanwei Li, Chengyao Wang. and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference Computer Vision, pages 323-340. Springer. 2024.
[36] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-Ilms from real-world online video understanding? arXiv preprint arXiv:250.005500 2025.
[37] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint urXiv22311.00122 2023.
[38] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 266999, 2024.
[39] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for milms achieve streaming video understanding. arXiv preprint 2xXiv22411.33266 2024.
[40] Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma. Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint ar 240: 15542, 2024.
[41] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468 2024.
[42] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12966 2024.
[43] Andreca-Maria Oncescu, Joao Henriques, Yang Liu, Andrew Zisserman, and Samuel Alhanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, pages 2265-2269. IEFE, 2021.
[44] OpenAL. Hello gpt-4o.
http:/ppww....ooooee////lllo--aa 2024.
2024.
[45] Viorica Păträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang. Carl Docrsch, Tatiana Mate- jovicova, Yury Sulsky, Antoine Micch, Alex Frechette, Hanna Klimezak, Raphacl Koster, Junlin Zhang, Stephanic Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Penception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, 2023. httl:tttts://:pettttttt
net forum? d-
[46] Rui Qian, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:1 19336- 113336-119360 19360, 2024.
[47] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Fiabling video lmss with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint arXiv:2501.031 083218, 2025.
[48] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14313-14322 2024.
[49] Hao Shao, Yuxuan Hu, Letian Wang. Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120-15130, 2024.
[50] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu. Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arxiv preprint arXiv:2410.17434 2024.
[51] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Viden-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409. 14485, 2024.
[52] Enxin Song. Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221-18233 18232, 2024.
13

---

[53] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: large-scale dataset for comprehensive instructional video analysis. In CVPR, pages 1207-1216. 2019.
[54] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[55] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- aisionllangugg model's perception the world any resolution. arXiv preprint arXiv:2409. 121119 12191, 2024.
[56] Weiyun Wang. Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-secing project v2: Towards general relation comprehension of the open world. arXiv preprint arXiv:2402. 19474, 2024.
[57] Xidong Wang, Dingjie Song, Shunian Chen. Chen Zhang, and Benyou Wang. Longllava: Scaling multi- modal Ilms to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409.02888 2024.
[58] Yi Wang, Kunchang Li. Xinhao Li, Jiashuo Yu. Yinan He, Guo Chen, Baoqi Pei. Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and I Limin Wang. Intervideo2:: Scaling foundation models for multimodal vidco understanding. In ECCV, pages 396-416, 2024.
[59] Yi Wang. Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvide22. 5: Empowering video milms with long and rich context modeling, arXiv preprint arXiv:2501. 12386, 2025.
[60] Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Vidcollm knows when to speak: Enhancing time-sensitive video comprchension with video-text duet interaction format, 2024. URL //trx/aaavvvv///////a//aa org/ abs 2411. 17991
[61] Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge. Lu Zhang, Jiawen Zhu, and Huchuan Lu. Streaming video understanding and multi-round interaction with memory-enhanced knowledge. arXiv preprint X122:25000113666 2025.
[62] Ziang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu 7eng. Zilei Wang. Yali Wang, Yu Qiao, et al. Task preference optimization: Improving multimodal large language models with vision task alignment. arXiv preprint orXiv:2412. 19326, 2024.
[63] Zhenyu Yang. Yuhang Hu, Zemin Du, Dizhan Xue, Shengsheng Qian, Jiahong Wu, Fan Yang. Weiming Dong, and Changsheng Xu. Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding. arXiv preprint arXiv:2502. 10810, 2025.
[64] Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. arXiv preprint arXiv:2504. 2025.
[65] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.01880 2024.
[66] Fisher Yu, Hanfeng Chen, Xin Wang, Wenqi Xian, Vingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogencous multitask learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[67] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, pages 69 85. Springer, 2016.
[68] Abhay Zala, Jacmin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mchdad, and Mohit Bansal. Hicrarchical video-moment retrieval and step-captioning. In CVPR, pages 23056-23065, 2023.
[69] Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li,, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving milms for long video understanding via grounded tuning. arXiv preprint 2rXiv:2410. 19702, 2024.
[70] Xiaohua Zhai Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- traininiig. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975- 1 1986. 2023.
14

---

[71] Boqiang Zhang, Kehan Li. Zesen Cheng. Zhiqiang Hu, Yuqian Yuan. Guanzheng Chen, Sicong Leng. Yuming Jiang. Hang Zhang, Xin Li, al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 2025.
[72] Haoji Zhang, Yiqin Wang, Yansong Tang. Yong Liu, Jiashi Feng, Jifeng Dai. and Xiaojie Jin. Flash-vstream: Memory-based
understanding for long video streams. arXiv preprint arXiv:2400.08055 2024.
[73] Jun-Bo Zhang, Wei Feng. Meng-Biat Biag Zhao, Fei Yin, Xu-Yao Zhang, and Cheng-Lin Liu. Video text detection with robust feature representation. IEEE Transactions on Circuits and Systems for Video Technology, 34(6):4407 4420, 2023.
[74] Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei. Chen, Yifei Li, Junbo Niu, Shuangrui Ding, et al. Internln-xxomposer2. 5-omnilive: A comprehensive multimodal system for long-term streaming video and audio interactions. arXiv preprint arXiv:2412.099 2024.
[75] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:24066 16852, 2024.
[76] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[77] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprchensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406. 04264, 2024.
[78] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. AAAI Conference on Artificial Intelligence, 2017. URL https://api semant cachol r.
[79] Haoyi Zhu, Honghui Yang, Yating Wang. Jiange Yang, Limin Wang. and Tong He Spa: 3d spatial- awareness enables effective embodied representation. arXiv preprint arXiv:2410.08200 2024.
[80] Zhe Zhu. Dun Liang, Songhai Zhang, Xiaolei Huang, Baoli and Shimin Hu. Traffic-sign detection and classification the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 10182,18, 2016.
[81] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe -aassnn-sstuch... Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apolla: An exploration of video understanding in large multimodal models. arXiv preprint arXiv:2412. 10360, 2024.
[82] Jannik Züm, Paul Gladkov, Sofía Dudas, Fergal Cotter, Sofi Toteva, Jamie Shotton, Vasiliki Simaiaki, and Nikhil Mohan. Wayvescenes101: A dataset and benchmark for novel view synthesis in autonomous driving. arXiv preprint arXiv: 08280, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of approach in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our paper does not include theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the implementation details for our experiments our paper and all results be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Our models, data and code have been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
public cellc//uiiiee tww.g//idee/eegg CodeeSub ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. w/npps.cppubbll wttpccccppppppptteeecccccccccccp cetii///uie///eeeeee/eoe.sscssee ies CodeeSumm ss cioooooliio ooooicyyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We list the core experimental details in the main text, and more detailed experimental details are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
Our experiments did not conduet statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computing resources needed for the experiments the Experi- ments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpppps//euuiippcccc//abataaaaa
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss potential societal impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license of each asset we have used and cite them properly.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetttett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We deseribe the usage of LLMs in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ht /tpp::://euippss /tt.arrrpppccccooooeoeenneeee//e 2025/ LLM) for what should or should not be described.
22

---

A Details of node's timestamp
Each event node's s timestamp is initialized as the average time of the frames it represents (e.g.. an event node spans frames from 10s to 14s, its timestamp is initialized as 12s). When two event nodes are merged, the timestamp of the new node is computed as a seeen--ouut-weigheee average of the original nodes:
4 . n4 + 12
new
74 n2
where 4, i are the timestamps of the original event nodes, and nis n, are the numbers of visual tokens contained in each node, respectively. This weighted scheme prevents timestamp drift during multiple rounds merging, especially when the merged nodes contain significantly different amounts of visual tokens.
B Details of OnlinelT
In this section, we provide a comprehensive description of the task categorization and data distribution of the OnlinelT dataset. It is specifically designed to enhance the streaming video understanding capabilities MLLMs terms of real-time perception, future prediction, and event continuity. As shown in Table 8 the dataset is divided into two major components: OnlinelTT-genrre which targets general streaming video understanding, and OnlinelT-drive, which focuses on autonomous driving scenarios. Each subset is carefully designed to cover diverse range of fine-grained perception and reasoning tasks with high-quality annotations.
Table 8: Task types and data volumes of OnlinelT.
Dataset
Categories
Task
Sourse
Instance Num
RefCOCO 57
43k
Spatial Grounding
Allsecing V2 56
455k
Spatial Perception
Multi-round Spatial Understanding
Genome
<.33
Spatial Grounded VQA
Allsecing- V2 56
<:3k
Relative Spatial Localization
LaSOT [3
1919
Charades 17
~11
Temporal Grounding
HiREST 68
-0.4k
gomennl general
Temporal Perception
QuerYD 43
~13
Reasoning Temporal Localization
ActivityNetRILL
10kk
Omlinelf
Multi-format Temporal Grounding
Intern Vid-VTiee
19
20k
Spatiotemporal Action Localization
AVA (18]
66
LaSOT [13
511k
Spatiotemporal Perception
Object Backward Tracking
GOT 22
558
Spatiotemporal Detection
~14
ActivityNet SetivityNet-Caation
10k
Dense Video Captioning
VITT 78
~5
Event Perception
Youcook2
lkk
COIN 53
99
Step Localization and Captioning
HiREST 68
00.5
Past Memory
Di-City
Statie Target
TT100k 80
<466
Onineef-drive
Real-time Perception
DC-City 4
~13
Localization Prediction
Road-waymo
Dynumic Target
Move Distance Prediction
Road-waymo
~7k
Accident Reasoning
MM- AU 14
~6k
Event Oriented
Risk Analysis
MM- AU
14
~7k
B.1 OnlinelT-general
OnlinelT-ggenrall encompasses a broad scope of tasks designed foster a comprehensive under- standing of spatiotemporal video content in streaming settings. As shown in Table the dataset is categorized into four primary task types: spatial perception, temporal perception, spatiotemporal perception, and event perception. To ensure diversity, robustness, and fine-grained task coverage,
23

---

compiled and refined data from a wide array of sources. In total, OnlinelT-genent comprises over 400k instances spanning various difficulty levels and video durations.
Spatial Perception. This task type includes four subtasks. Spatial Grounding requires the model to output the bounding box indicating the location of a queried object. Multi-round Spatial Under- standing involves identifying the object's spatial location through multi-turn dialogue or generating a caption for the object within a specified spatial region. Spatial Grounded VQA combines visual question answering with spatial localization, requiring the model to provide the bounding box of the relevant area while answering the question. Relative Spatial Localization challenges the model to determine the position of specified object relative to the overall scene. These tasks emphasize spatial grounding and reasoning, which are crucial for enhancing model's ' fine- fine-grained spatial perception real-time streaming video scenarios.
Temporal Perception. This category consists of three subtasks. Temporal Grounding involves interpreting a natural language query and identifying the start and end timestamps of the corresponding video segment. In streaming scenarios, the model must also assess whether the described event is currently ongoing. Reasoning Temporal Localization requires identifying the relevant time span of an event while answering a reasoning-basee question. Multi-format Temporal Localization incorporates both single-turn and multi-turn dialogues, covering a diverse range of question formats. These tasks focus on strengthening the MLLM's 's ability to track and reason about temporal dependencies, improving its understanding of both current and past moments in video stream.
Spatiotemporal Perception. This task type integrates spatial and temporal reasoning and includes three subtasks. Spatiotemporal Action Localization requires the model to predict both the spatial location and the action being performed by target at a specific query time. Object Backward Tracking tasks the model with identifying the current location of an object and tracing its position previous time points, such as one or two seconds earlier. Spatiotemporal Detection operates over broader temporal windows, asking whether an object visible in the current frame existed several seconds ago or requiring the model to locate an object at specified historical moment and determine its duration of existence. These tasks combine spatiotemporal cues capture actions, motion, and transitions, allowing the model to track object trajectories and anticipate future states based on past and present context.
Event Perception. This category includes two subtasks. Dense Video Captioning involves detecting a sequence of events in a video and generating corresponding timestamps along with high-level descriptions. Step Localization and Captioning differs by focusing on segmenting and narrating key procedural steps within long-form videos. These tasks are aimed at improving the model's understanding of complex, multi-step events, enabling structured interpretation of dynamic sequences in streaming video understanding.
B.2 OnlinelT-drive
OnlinelT-drive is designed specifically for the domain of streaming video understanding in au- tonomous driving. The dataset includes 89k instances, which organized into three major task categories. Collectively, these tasks aim to strengthen not only real-time perception capabilities but also the temporal reasoning and decision-making abilities of MILMs in high-stakes and rapidly evolving environments.
Static Target Understanding. To improve the model's capacity for static scene understanding, two task types are introduced. Real-time Perception requires the model to accurately perceive and interpret the semantics and spatial attributes of traffic-related targets as they appear in real time. Past Memory assesses the model's o ability retain and retrieve the semantics and spatiotemporal characteristics of traffic targets that were observed at prior point in time. These tasks collectively enhance the model's capability perceive, understand, and remember static traffic elements and environmental context, such as road infrastructure and regulatory signage.
Dynamic Target Understanding. It includes two task types that aim to enhance predictive un- derstanding of dynamic traffic participants. Location Prediction requires the model to estimate
24

---

the future position of a moving target based on its historical motion trajectory. Move Distance Prediction focuses on predicting the distance traveled between the ego vehicle and other moving agents, given motion-related observations. These tasks are designed to improve the model's ability to track continuously moving objects and to anticipate future trajectories.
Event Oriented Reasoning. It is intended to foster the development of reasoning abilities necessary for risk assessment and accident interpretation. Risk Analysis requires the model to detect potential sources of danger in the current traffic scene and to assess the likelihood of accident occurrence. Accident Reasoning involves post hoc analysis, where the model must infer the causes of an observed accident and articulate plausible preventive strategies. These tasks are designed to enhance the model's ' ability to reason about causal relationships and to anticipate or reflect on traffic risks with contextual awareness.
C Details of ODV-Bench
In this section, we detail the task taxonomy and formulation of the ODV-Bench, as well as the dataset statistics. We categorize task scenarios based on target entities and derive key perception and reasoning task types for each scenario in Table 9
Table 9: Overview of task categories, their subcategories, and question templates.
Talk Objective Secrumio
Sab-tack
Query Exumples
the mezaing of the caffic
the 1061,031..660000 0.33)
picture?
and position cxcdiiaato the traflic sign indkaning "Pokktriaa Chocing" is the cumani piclunc?
3) Whet is the meaning of the red boond at the (1.77.00.00 o. (.800.2) the cormett pictue?
Real time Truttic
Aussnling the road board (0.09.04660. 1900333 camently. if going
will go.
Perception
how for is it?
5) What is the oolor the tuitee lishs at the (0.420010045.0 in the curent picture? And what is its indication?
According to the road board at the (0.65 2.3 7..00 comently how far
Foogle?
What were the posttion conrdinates of the tafli sign indocting "No Lef Tuni" the scene 3 seconds ogor
2) What was the meaving the taflic sign the [0. 99,0 13,041 (.17) in the wames ecconds ago?
Past Traffic Mensey
The tratte sgnn is cumennl locased at (0.92.0.. U9g. were cocedinares seconds ago?
4) Whet was the ooler of the millii light the (035,)) 1 986622) is the some 2 socordh ago
According the road board the (4:47.0014
image
ago,
Henhee
which direction ahould zo
Staik Target
Driving Duxisico-Makinn
According to the road board at the 11.1,,018260001101 corrently. If we wasts
Libal Avenne, which direction
Acxomling to the road board at the 00,0040,,600000 cumonily if we warks torn left, which lane whould we should
it?
Key Information Extrastin
If we wants go Sudhon, which targe ahould the most atientios cuaronaly?
type aroedinatos.
Acconding to the road board at the 11..0031 how far currenty from Quagoo Town?
Hallucinazin Detection
What is the mesaing of the caffic sigs at the 0033,0.990.38.00030 in the cunera picture?
the tittle
at the 10990
corrert picture?
1) the cummnn momont. has the sallic sigual light indicating "uum right" shead lurned compleall grees?
Truffe Change Desection
the curmat
has the trafiic sipsal light shoal turnell compleily mil
Actiun Prufistarn
Wher will be the subeequee motion state or the car comently the JUMS, 0.615, 1.0. (.6)) lncotion)
Dyramic Torger
Locatin Predetion
What will the position box of the pelessiiin in the [0.544 0,561, 0.613, 0,895] locaiion be like in tse nos second?
Dustnce Prediction
distance between our car
in the (0.485, 0.485, 0,501, 0,694 getting forther closer?
is there high probability of tattic accidents eocerinng within cortain peried in the futme?
Risk Prodiction
Will signiiicun
risks within certani period tuture
There is high rak of caflic accidenta si prcsent. Biacd the cnviooment, who types aczidenta Bally to oosur,
Muli-agani
Inveaction livent
Risk Andyyss
and what is the bais for this predecion?
significast inffic risks al prescat, Beaad the envinument,
the of thes
and what apes of worderrs night they canet!
Ausislent Risous Amsscciing
What the canee of the assideat the vilco? Whet meseuns can be takes to avuid ic?
C.1 Task Taxonomy and Formulation
We first identify the primary categories of traffic entities relevant to autonomous driving and organize task scenarios into three groups: (1) Tasks for Static Targets, which involve the recognition and retrieval of stationary traffic elements such as traffic signs, lights, and road indicators; (2) Tasks for dynamic targets, which focus on behavior prediction and localization of moving entities such as vehicles and pedestrians; and (3) Tasks for multitarget interaction events, which capture complex interactions, risk scenarios, and accidents involving multiple agents. Based on these categories and guided by temporal cues and the practical needs of driving, we further define fine- mm--grai task types to comprehensively assess model understanding in realistic online driving video scenarios.
25

---

Static target oriented tasks
Q: What are the position coordinates the traffic
Reak-iime Traffic
sign indirating Pedestrian Croesing" currestl??
[RTP]
1AI Ib450.33.. 2.523.5500000 0.53)
4...7.2...888 (CI (0 (...0.34,,.77 0..,5.3...8 83) ID (0.57,0.000 0..311]
Porception
Arswva: (D)
traffic sign at 00,50.9,,005...3333 scene secunds ago?
CB0
[PTM]
Past Traefiic
Spead
1-7hm
Memory
(21 Height Bescieito
Q: Accoeding the traftic sign the
[0.77: 0.7,,,858,, I1] currontly, wands
[DDM]
Driving
Zhuhai, which direction should
Deciston Making
(A] Eall (8) Frozl lel.
(C) Stuiih (D) Right
Answer (P)
Q: wants to V eojiang, which target should
the mast attention curmenly?
[KIE]
(A) Car, 1.380.13,0.52.000
(3) Road 0.8 0.6.0.0.......2
(C) Road beard, (0.78.0
Acswar: (B)
Q: What the color the traffic light the
2300
(6.,0.66,,....8 in the current picture? [A] Yellow Grees
[HD]
Hallucination Detection
(C) Unable
Red
kaswen (C)
Q: the current moment, has the traffic light indisating turn right" aboud turned Time: 0,00
11.13
2.0
[TCD)
Trafficc Change Detection
Aresvr No 434 No (18443) (10
+ Yex (20143s)
Dynamic target oriented tasks
W het will be the subsequent mution state of the bus currently (0.35, 0.30, 0.57, location!
[AP]
cossiteg
cight
(C) Mosing thwrigh
Mexing the willow
Arseser: (b)
Q: What will the podition box of the cyclict in the j0.2
S, 0.53, 4636, 0.74] location be like in the nost second?
[LP]
[A] 20, 6.51. 8.49) (B) (0.5T, 8.52, 1:73] [C: 23.88 2.95, 1.20, 0.60] (DI (0.90 0.04, 0.41, D.83]
Anowen (6)
0: the distance betbween our car and the car the (0.37, 0.60, 0.61, 0.80) getting farther or closer? (AI Gettin chopr
Prediction
(BI Getting farian:
Aaswan (B)
Event oriented tasks for multi-agent interaction
Q: Will there be significnt traffic risks wthin certata pertod the future?
[RP]
Risk Prediction
Query Time 0.00
Arswve: No (2 (3606) + No (20.00) + Yas (27.50g
Q:
al prementt
Risk Analysis
miight they we go straight and our visuo is blocked. This may lesd lhst we lit crossng coe
Answen types
cressss
Q: hat is the cause of the accident to the video?
Accident Reason
hit messires can be taken to aveid it? Atsivee We na rod lagit We souule stricly abude by the
[ARA]
Anvwering
ratie rales dustag drving When the rod Light on,
Figure Examples of each task in ODV-Bench. The 12 tasks are divided into three different perception modes for online video understanding for autonomous driving.
C.1.1 Tasks for Static Targets
Static traffic elements, such as traffic signs and road indicators, play crucial role in driving decisions and hazard avoidance under normal driving conditions. evaluate the model's ability retrieve and recognize these elements in online video streams, we design a dedicated set of tasks. Specifically, we distinguish between basie perception tasks and more advanced reasoning tasks, and further refine them based on temporal cues and practical driving needs: (1) Real-time Traffic Perception:
Perceive
26

---

and interpret the semantics and spatial locations of static traffic elements in real time; (2) Past Traffic Memory: Recall and track the semantics and spatiotemporal states of previously observed static elements; (3) Driving Decision-Making: make driving decisions based on the perceived information; (4) Key Information Extraction: Identify and locate key traffic elements critical driving decisions; (5) Hallucination Detection: identify questions irrelevant to the existing video input; and (6) Traffic Change Detection: detect timestamps for changes in traffic elements, such as traffic lights.
C.1.2 Tasks for Dynamic Targets
The position and behavior of other road participants, such vehicles and pedestrians, crucial reference factors influencing autonomous driving decisions and safety. The ability to predict the position and behavior of dynamic traffic objects is essential ensure the safery of autonomous driving. Therefore, we focus on the following three tasks to effectively evaluate this capability: (1) Action predicting the next action of vehicles and pedestrians based on continuous spatiotemporal cues; (2) Distance Prediction: predicting the relative distance change between the ego-vehicle and other vehicles based on motion information; and (3) Location Prediction: predicting the future spatial position of dynamic traffic targets based on their movement trajectories.
Prediction:
C.1.3 Tasks for Multi Target Interaction Events
To achieve safe and reliable autonomous driving, the system must be able to identify risks and analyze accidents in complex road interaction scenarios. In the context of online video streams, this ability involves the dynamic recognition and analysis of multi-agent interactions, as well as the reasonable prediction of traffic risks. To evaluate this capability, design the following three task categories: predicting the occurrence of significant traffic risks and responding proactively; (2) Risk Analysis: detecting the sources of current traffic risks and analyzing the potential causes of accidents; and (3) Accident Reason Answering: post-accident analysis, providing potential causes for the incident and summarizing actionable lessons learned.
(1) Risk Prediction:
C.2 Dataset Statisties
BoV-Bench comprises 1,190 unique first-person driving video clips, encompassing a diverse range of driving scenarios across different countries from routine driving conditions to potential hazards and accidents. The length of videos ranges from 5 seconds to 90 seconds, effectively capturing the diversity of real-world streaming driving experiences. The benchmark includes 6,348 question- answer pairs, with an average query timestamp of 18.9 seconds. Specifically, the static-object-orientde category comprises 247 videos with total of 1,639 questions; the cynnamc-object-oriieee includes 162 videos and 2,999 questions; and the event-oriented category consists of 781 videos with 1,710 questions. All questions are multiple-choice format, with the number of options varying between 2 and 4 depending on the question type.
category
D More Implementation Details
Table 10: Parameter settings for three-stage offline pre-training.
Stage-1
Stage-2
Stage-3
Viston
Resolution x Num. frames
384
384x8
Max 384x512
#Tokens
64
64x8
Max 16x512
Dota
Dataset
Image Short Video
Image Short Video
Image Short Long Video
#Samples
0.6M 0.5M
3.8M 3.4M
0.5M & 2.8M
Madel
Trainable
Projector 16.98MB
Full Model
Full Model 8030.35MB
#parameters
8030.35MB
Batch Size
512
256
256 2 x
Traising
LR of vision encoder
1x
x10
LRR of connector & L.M
1x10-3
x10-5
1 x1005
Epoch
27

---

Table 11: Parameter settings for the fourth stage online fine-tuning and fifth dirve fine-tuning.
Stage
Stage5
Image & Comoor//ong/Onlinnee 0.4M & 1.3M
Image (Short/_ong/Online)
Dato
Dataset
#Samples
0.2M 0.5M
Vislom Model
Trainable
Projector LLM
Projector LLM
*parameters
7632.60MB
7632.60MB
Resolution
384x384
384x384
Frames
2--512
2--512
FPS
Real-time Perception Qouta
729
729
Spatioterporal Memory Quota
128 x 18
128 x 18
Menory
Total Visual Token Limits
8192
8192
Similarity Penalty Weight
0.4
0.4
Merge Count Penalty Weight
0.4
Temporal Distance Penalty Weight
Batch
LR
x10-5
x10-5
Epoch
Training
Optimizer
Adam
Adam
Weight Decay
0
Warmup Ratio
0.03
0.03
LR Schedule
cosine
cosine
Vision Select Layer
GPU Nums
We adopt five-stage training strategy to systematically train the proposed StreamForest model, aiming to fully exploit its potential for streaming video understanding tasks. In the first three stages, we follow and extend the training paradigm of VideoChat-Flash [33], employing offline training to endow the model with strong capabilities in long-form video comprehension and cross-modal alignment. These stages are designed progressively, covering diverse data scales and task objectives, enabling the model to gradually acquire core competencies such as basic laiion-laguage alignment, long-term temporal modeling, and complex scene reasoning. Detailed training procedures and hyperparameter configurations for these stages are provided Table I0
In the fourth and fifth stages, we perform online fine-tuning to enhance the model's ability to process streaming inputs realistic scenarios. By continuously feeding frame sequences during training. the model learns to retain a fine-graines perception of the current moment while maintaining long- term memory past events, even under high compression constraints. The full configuration and parameter settings for the online fine-tuning phase are listed in Table 1T These stages are critical for transitioning the model from offline understanding to real-time reasoning, significantly improving its robustness and practical effectiveness in real-world applications.
E
Full Performances
In the following parts, we present the full results and compare StreamForest with leading proprietary and open-source models. To comprehensively evaluate the effectiveness StreamForest, we conduct experiments on three online video understanding benchmarks: StreamingBench, OVBench, and OVO-Bench.
E.1 StreamingBench
Table 12 presents the full evaluation results on StreamingBench, covering 12 real-time video un- derstanding tasks. StreamForest achieves the highest average score (77.26%) among evaluated models, both open-source and proprietary, while operating efficiently at 1 fps. Notably, StreamForest outperforms leading proprietary MILMS such as GPT-4o (73. 73.28%) and Gemini 1.5 Pro (75.69%). also significantly surpasses top open-source offline models such as I.LaVA-OneVision (71.12%) and Qwen2.5-VL. (73.68%). underscoring its robust multimodal representation and reasoning capabili- ties. In the online video MLLM category, StreamForest sets new sate-of-the-as outperforming
28

---

Table 12: Full evaluation results of real-time understanding tasks on StreamingBench.
Method
Size Abrames
OP
CR
CS
ATP
BL
TR
PR
SU
ACP
CT
|ALL
Husta
89,47
92,00
93.60
91.47
95.65
92.52
X8.00
88.75
89.74
91.30
91.46
Propristary MLLMs
pco
lfps
GPT-4o [
64
77.11
80.47
K3.91
76.47
70.19
M3.80
66.67
62.19
69.12
49.22
73.28
Clande 3.5 Socnet
20
73.33
80.47
$4.09
K2.02
75.39
79.53
61.11
61.79
69.32
43.09
72.44
Open
Video MLLMs
Vidas11
7R
42.68
VILA-1.3
8B
14
53.68
69.22
70.98
56.86
53.42
53.89
59,63
48.78
50.14
17.62
52.32
Vide--CCAM
[5
[4]
96
56.40
57.81
65. 30
62.75
64.60
51.40
42.59
47.97
49.58
31.61
$3.96
LA 75
7R
128
70.03
63.28
61.20
70.92
62.33
59.50
61.11
53.60
54.67
34.73
50
8R
Kanganoa
7B
71,12
84.38
70.65
73.20
67.08
61
56.48
$5.89
62.04
38.86
VL.VA-NoXT-VVile
[6
92B
64
78.20
70.31
73.82
76.80
63.35
69.78
57.41
56. 10
64.31
38.86
66.96
MinLPMM-V26
[+
8B
32
71.93
71.09
77.92
75882
64.
65.79
70.37
56.
62.32
53.57
67:44
Qua12.5V1.
7B
10p
78.86
80445
76.73
78.50
79.63
63.41
66.19
53.19
73,68
Openssonv Onlone Video MTT
Plash-VStram VStream [2]
7B
25.89
43.57
24.91
23.87
27.33
13.08
18.52
25.20
23,87
48.70
23:25
Vidaol M-eoline
8B
25ps
Dispuder
7B
18ps
74.92
73.08
74.44
02.16
45.80
(uraas)Fresst(urs)
7B
18ps
83.11
82.1
82.65
84.26
77.50
78.39
76.85
09.11
75.64
54,20
77:26
open-source counterparts Dispider (67.63%) by wide margin. consistent accuracy and real-time efficiency demonstrate a strong potential for practical deployment in streaming applications.
E.2
OVBench
Table 13: Full evaluation results on OVBench.
TP
AVG
Subset Name
AA
GSP
MP
AP
SV
OP
AR
TR
AL
OP
AT
OT
AS
SL
OES
Proprictory MLLMs
Gemini-1 S-Plish
54
714
53.6
21.9
56.5
60.8
40.6
36.7
47.9
62.5
32.3
37.5
87.0
50.0
83.3
22.3
46.9)
50.77
Open-snurv Offirr Vileo MIL.Ms
IntemV11.2
57.5
$3.0
58.5
75.4
31.4
48.7
InterVL2
4B
57.7
57.0
14.4
59.2
49.4
60.0
30.3
61.8
46.3
30.9
20.1
83.0
32.3
70.7
29.4
3.4
44.1
LLMMA-VID
K5
78
43.6
50.9
19.6
64.0
47.5
46.8
29.4
48.9
51.2
31.9
11.2
75.7
24.8
59.1
26.0
40.0
41.9
11
ILaVA-Onevisin
7B
68.0
62.7
35.9
58.4
50.3
46.5
29.4
60.7
58.0
43.1
14.2
86.5
49.7
70.7
28,
50.2
49.5
LongVA
64.1
56.5
$4.9
$1.9
34.8
3.4
80.0
26.7
4.0
43.6
MiniCPMMV2.66
30.8
26.6
26.4
Qwen2-VV.
60.3
66.1
22.1
54.9
$1.5
51.1
37.8
69.3
35.3
28.5
97.0
49.4
65.1
30.8
11.7
49.7
LITA 20
78
19.2
24.5
19.9
40.8
48.9
24.9
3.1
27.3
6.4
6.9
14.6
35.2
23.9
27.4
0.5
3.4
20.4
TimeChar
45
7B
7.7
15.3
18.7
20.5
15.7
11.7
9.1
14.7
9.8
7.5
19.5
13.9
10.3
9.3
10.1
10.8
12.8
VTimeLLM
7B
37.2
23.4
64.8
43.8
53.2
25.9
38.8
32.5
25.9
20.4
40.9
6.8
48.4
43.5
8.5
33.1
Open source
VikooLLMMQunin
7B
1.8
20.9
5.2
5.9
32.6
2.3
26.7
0.6
26.6
0.9
19.9
0399
1.7
8.3
9.6
MovieChar 53
7B
23.1
27.5
23.6
58.4
41.9
40.3
25.6
31.1
23.9
26.9
30.6
24.4
28.9
29.3
25.5
21.9
30.9
Plash. Vstream
72
7B
26.9
37.6
23.9
60.1
41.9
40.0
23 4
35.3
26.1
24.7
28.8
27.0
21.4
29.8
25.6
26.8
31.2
Videochat-Onlnee
23
4B
64.1
63.1
58.3
62.8
24.8
88.7
48.5
73.0
25.9
54.9
StreumForest
54.0
61.3
64.2
Table 13 shows the comprehensive results on OVBench, encompassing six diverse task categories (FP, THV, PM, SP, STP, TP). StreamForest achieves the top average score of 60.5%, surpassing all open-source online and offline Video MLLMs. It significantly outperforms other open-source online models, e.g., Videochat-Online (54.9%) and Flash-VStreal (31.2%) well as offline models such as Qwen2-Ve VL (4)7%% and LLa LLaVA-One Vision (49.5%). Compared to Demini-1.5-FFla (50.7%), StreamForest delivers nearly 10 points higher accuracy average, affirming its capability to balance real-time efficiency with high performance.
E.3 OVO-Bench
Table 14 details performance on OVO-Bench, where StreamForest again leads among open-source online video MI..Ms with an overall average of 55.57% outperforming Dispidier-7BB (41.78%) and Plash-VStreant (33.15%) It excels in critical areas such as real-time visual perception (61.20%) backward tracing (52.02%), and forward active responding (53.49%), showcasing robust temporal
29

---

Table 14: Detailed evaluation results OVO-Bench.
Pereptton
Tracing
Reppoaling
Modd
Francca
AC3R
STU
191
ASI
HED
Overal Aug.
Humas
93,96
92.57
94.83
92.70
91.09
94.02
93.20
92.59
93.02
91.37
90.35
95.48
89.67
93.56
92.90
92.31
Proprittrr MILMs
Genini 1.5
34
Upp
85.91
66.97
79.31
59.43
63.37
61.95
69.32
58.59
76.35
$2.64
02.54
15.53
74.24
61.77
57.15
63.00
GPT-4o (4)
64
69,50
64.22
71.55
51.12
70.30
59.75
64.40
$7.91
75.68
48.06
00.75
21.58
73.21
59.40
53.40
29.54
Open-uun Oflir Video 677.4
Vileo-1
76
64
69:50
59.63
56.3 38
50.56
12.28
$1.41
63.34
51.15
64.19
9.68
41.68
34.10
57.55
60.53
54.17
$3,06
64
67:11
73
80 81
00.44
11.29
00.33
67.19
52.53
56.18
23.66
64.99
36.70
56.91
60.85
40.85
$7.45
Qwen2-V1-78 [S
b4
09.15
55.21
58.30 74
90:36
66.
50.87
60.55
44.04
65.89
34.41
68.56
10.00
65.06
50.55
48 8m
52.70
IntevVL2 Xg
64
68.46
58.72
66.17
64.54
67.33
55.95
60.73
43.10
61.49
27.41
64.00
25.79
57.55
52,92
15.42
50,05
LongVi-78
XX
Tpp
55.70
49.54
59 08
68.31
08.32
03.04
57.40
03.10
06.22
9.14
39.49
10.02
181000
00.00
08.54
48.48
Open-aaee Ontorr Video M/2.8r
ViteoLLMM onlar 8B 5
Zip
5.05
23.85
12:07
14,04
45.54
21.20
20.79
22.21
18,50
12:18
17.73
12,54
18 22
Uge
25.30
32.11
20.3
33.71
29.37
28.80
28.86
16.35
33.78
5.01
25.35
5.44
60.25
60000
44.23
33.15
Diepila-78 41
Upe
57.72
49.54
52.07
44.86
61.39
51.63
34.55
48.48
53.41
4.30
36.06
18.03
57.36
45.75
34.72
41.78
StvomFForst-70 78 (Oars)
Ups
68.46
53.21
71.55
47.35
65.35
00.57
63.20
58.92
64.86
32.26
50.02
32.11
70.59
57.08
53.44
53.57
reasoning across both past and future events. These results position StreamForest as practical and powerful solution for real-time langugla understanding.
F Qualitative Comparison
08:15
Video Stream
User
What
oggu as rurbent
Strem
pense toblet
100K0 capleve cating lodte or dees
crienal wathed cntities
Dispider
people can uas becaae thy specainng
The oinaaal the vdea snnched earaw sags
01:10
00:20
01:30
Video Stream
User
whlte dia meneral
When
capy deakmata'
StreamForest
The subottdis 20. 078 O...M
quer
Dispider
peron wertigg the wele vee el diteett keather
Mi. Rean stated dener offer
69:82
69: 85
09:23
Vićeo Stream
User
codition striihh1
StreamForest
Dispider
No: should rop becruss
The vidoo soows acdddrts hagaer sucden cafk spacc keep salk ctanc fore other car..
Figure 7: Qualitative comparison between StreamForest and other method.
Figure presents a qualitative comparison between our model and other method. In the top example, StreamForest demonstrates superior ability capture fine- tinegrained visual details and maintain persistent memory over time, enabling more coherent and informed inferences. The middle example highlights StreamForest's strong spatiotemporal grounding capabilities, accurately localizing objects and events across space and time. The bottom example illustrates the model's potential in intelligen driving scenarios, where it delivers precise real-time perception and supports future predictions based on both historical and current observations.
30

---

G More Ablations
2 41.0
4.
965
865-
565
660
Senturay Pendtyy Serott
Merge Penolry Welee
Resood Ditaaee heady Steget
Figure 8: Ablation experiments of three penalty weights.
To evaluate the effect of each penalty the effectiveness of long-term memory, conducted ablation studies by varying the weights of the similarity penalty, merge count penalty, and temporal distance penalty. The results presented in Figure 8 which reports the accuracy of both Vi and OVBench under different settings. demonstrates that balanced combination of penalty weights is more effective. Specifically, 0.4 for both similarity and merge count penalties, and 0.2 for the temporal distance penalty yield the most effective memory construction. This configuration achieves favorable trade-off between preserving semantic coherence, maintaining diversity in memory representation, and ensuring reasonable temporal continuity.
H Efficiency of Multi-round Inference
We evaluated StreamForest's multi-round response efficiency by streaming 600-second video to the model at a constant rate of FPS. To isolate processing throughput from text generation latency, the model constrained to produce single-token response for each frame. Under this rigorous setting, StreamForest achieved an average processing speed of 9.9 FPS, which is competitive with VideoLLM- Online (12.3 FPS), model renowned for its real-time capabilities. Crucially, StreamForest delivers this high efficiency without compromising its substantial superiority in reasoning accuracy over Lideoi.LM-OOninee In stark contrast, Qwen2-VL, another model that also prioritizes reasoning accuracy, demonstrated severe performance bottlenecks. Its processing speed dropped below FPS on video merely two minutes long, and it encountered out-of-memory (OOM) errors on a single A100-80G GPU after processing only 79 frames.
Table 15: Comparison of mound inference speed.
Method
Resolution
FPS
Qwen2.5-VL.
384
00M
VideoLLM-Online
384
12.3
StreamForest (1k)
384
9.9
I Discussions
I.1 Limitations
Despite the effectiveness of our proposed method, several limitations remain that warrant further investigation. Our approach can only rely computing inter-frame similarity to determine moments when the model should proactively produce outputs. Specifically, the method identifies local minima in similarity scores to detect transitions. However, this technique primarily captures coarse scene changes and often fails to accurately detect true semantic event boundaries. To address this limitation, one possible solution is to incorporate lightweight MILM as an auxiliary reminder module. This module could provide semantic-level guidance to support more precise and context-aware output decisions. These limitations suggest promising directions for future work.
31

---

1.2 Broader Impacts
Our proposed method shows strong potential for real-world streaming video understanding, especially in critical applications like autonomous driving. With domain fomain-specific fine-tuning, it can be adapted various downstream tasks that require continuous visual processing. As shown the main text, the model performs well in autonomous driving scenarios, where accurate and timely perception is crucial for safety and decision-makinn It can efficiently process live video streams while preserving fine- grained perception and long-term contextual memory. This capability is particularly valuable under limited computational resources, helping improve the reliability and responsiveness of intelligent systems in dynamic environments.
However, with many vision-language models, potential negative social impacts must also be considered. If deployed without proper safeguards, models may inherit or amplify biases present in training data, leading to unreliable behavior. For instance, performance disparities across different environments or conditions (e.g.. weather, lighting, geographic location) could affect the robustness of StreamForest. To mitigate such risks, we should explore techniques for enhancing interpretability and controllability of streaming video models in safety contexts.
32


Reviews:
[{"review_id": "yNrmScAcuo", "reviewer": "Reviewer_nvw1", "summary": "This paper introduces StreamForest, a novel architecture for streaming video understanding that incorporates a Persistent Event Memory Forest to efficiently manage historical visual information. Memories are merged based on temporal distance, content similarity, and merge count penalties, allowing the model to retain long-term context under constrained resources. In addition, the authors present OnlineIT, a new instruction-tuning dataset tailored for streaming video tasks, and ODV-Bench, a benchmark focused on real-time autonomous driving scenarios. The proposed method shows substantial improvements over state-of-the-art MLLMs, especially in constrained-token regimes.", "questions": "The ODV-Bench could be described in more detail. For example, \"task examples\" are mentioned in Figure 3(c), the paper does not clearly define a taxonomy of question types (e.g., object recognition, temporal reasoning, causal inference, prediction). I think such a taxonomy would be useful for interpreting performance in this benchmark.\n\nQuality control in the construction of the benchmark seems under-reported. \"multiple rounds of manual review,\" could be clarified with quantitative information regarding inter-annotator agreement, human accuracy baselines, variability as a function of taxonomy.\n\nJoint ablations across e.g. FSTW and PEMF could be included to identify synergistic benefits.", "limitations": "The limitations are only discussed in the SM, I think limitations should also be summarized in the main ms.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This work tackles a challenging and impactful problem that is of current interest, namely the efficient long-term understanding of streaming video. While the use of memory structures for summarization is not entirely novel, the proposed integration into MLLMs and the accompanying benchmarks demonstrate a meaningful advance. The results suggest real practical value, especially under resource constraints.\n\nThe paper is technically solid, and the proposed components are clearly motivated. However, several issues should be addressed to strengthen the empirical claims:\n\nThe paper is generally well-written and logically organized. The motivation and contributions are easy to follow, and the architectural components are clearly described. Some figures could benefit from better labeling (e.g., Figure 1), but the text is otherwise clear. In Figure 1, the meaning of the bars is unclear due to missing axis labels and legends. Proper labeling is essential for interpreting results.\n\nComputational Overhead: The paper does not report the computational cost of the Persistent Event Memory Forest, in terms of runtime or memory usage. This is particularly important for streaming applications, and should be quantitatively evaluated.", "was_revised": true, "final_justification": "I've reflected on the authors' proposed changes which will improve the quality and clarity of this paper imo thus Ive increased these scores.", "rebuttal": "We are very grateful for your constructive comments, which have been instrumental in improving our paper. The following is a point-by-point response to the reviewer's comments.\n\n\n>**W1: In Figure 1, the meaning of the bars is unclear due to missing axis labels and legends. Proper labeling is essential for interpreting results.**\n\nThank you for your valuable suggestion. We will revise Figure 1 in the updated version of the manuscript to ensure the results are clearly interpretable.  Specifically, we will enhance the clarity of figure by: (1) adding a clear legend indicating that the blue bars represent Previous sota, the gray bars denote StreamForest (1k visual token limit), and the green bars denote StreamForest (8k visual token limit); and (2) explicitly labeling the vertical axis to indicate that the bar heights correspond to the average accuracy on each evaluation benchmark.\n\n\n>**W2: Computational Overhead: The paper does not report the computational cost of the Persistent Event Memory Forest, in terms of runtime or memory usage. This is particularly important for streaming applications, and should be quantitatively evaluated.**\n\n\nThank you for raising this point. We have supplemented a quantitative analysis of the runtime and memory usage associated with StreamForest. In order to better observe the impact of memory mechanism, we assume that frame-level visual features have already been extracted by the vision encoder in real time. Thus, we report the the computational cost only for the combined of LLM and memory system (PEMF + FSTW). PEMF enforces a strict upper bound on the number of visual tokens (8K in the experiments here), ensuring that memory consumption remains stable regardless of the number of processed frames. Consequently, the computational cost does not grow significantly with longer streaming inputs. The following table presents the computational resource consumption for a single effective response:\n\n|Input Frames|Memory (GB)|FLOPs (T)|Inference Latency (s)|\n|:-|:-:|:-:|:-:|\n|64|15.8|93.1|0.776|\n|256|17.1|134.1|1.126|\n|1024|17.2|137.3|1.497|\n\n\nThe computational cost of updating PEMF is negligible compared to that of the vision encoder and LLM. We have added a comparison of PEMF with other memory mechanisms, including the Pyramid Memory Bank [13] and the Similarity Merge strategy [36]. The following table reports the time consumption of each method when processing a 500-second video stream (1 FPS) and producing one effective response at the 500th second.\n\n|Method|Vision Encoding (s)|Memory Update (s)|LLM Inference (s)|**Total (s)**|\n|:-|:-:|:-:|:-:|:-:|\n|Similarity Merge [36]|5.198|0.183|1.388|6.769|\n|Pyramid Memory Bank [13]|5.203|0.451|1.381|7.035|\n|**PEMF (Ours)**|5.218|**0.172**|1.394|6.784|\n\nThe results show that vision encoding dominates the overall runtime, processing at 95 FPS, while PEMF’s event consolidation accounts for only 0.172 seconds for 500 frames (2900 FPS). With PEMF integrated, the preprocessing speed for the LLM’s visual inputs decreases by only 3 FPS (5.218+0.172=5.39s, from 95 to 92 FPS), which is negligible compared to the efficiency gains achieved by significantly reducing the total number of visual tokens.\n\n\n\n>**Q1: The ODV-Bench could be described in more detail. For example, \"task examples\" are mentioned in Figure 3(c), the paper does not clearly define a taxonomy of question types (e.g., object recognition, temporal reasoning, causal inference, prediction). I think such a taxonomy would be useful for interpreting performance in this benchmark.**\n\n\nThank you for your valuable suggestions. In response to your concern about the *clarity of task types and subtask categorization*, we clarify that the classification criteria for the tasks and subtasks in Figure 3(a) are systematically explained in **Appendix B** (in the Supplementary Materials). Due to space constraints, we placed the detailed methodology there to preserve the coherence of the main paper.\n\nThe \"Task Examples\" in Figure 3(c) illustrate representative scenarios from real-world driving, covering both normal and anomalous cases. The mapping between these examples and subtasks is also detailed in **Appendix B**.\n\nBriefly, we organize the benchmark tasks using a two-level hierarchical taxonomy, based on key considerations in autonomous driving. At the top level, we categorize the tasks into three broad scenarios according to the nature of the target objects:\n\n1. Static traffic element perception\n2. Dynamic object behavior prediction\n3. Multi-agent interaction and risk reasoning\n\nBased on this, we further define specific subtasks using the following criteria:\n\n- Temporal cues involved (e.g., real-time perception vs. past memory vs. future prediction);\n- Practical requirements in real autonomous driving systems;\n- The cognitive skills required (perception, memory, reasoning, etc.).\n\nWe believe this structured taxonomy provides researchers with a clear and multi-faceted view of model performance in video understanding tasks. We will also further strengthen the cross-referencing between the main paper and the appendix in future versions to enhance clarity and readability.\n\n\n\n\n>**Q2: Quality control in the construction of the benchmark seems under-reported. \"multiple rounds of manual review,\" could be clarified with quantitative information regarding inter-annotator agreement, human accuracy baselines, variability as a function of taxonomy.**\n\nThank you for emphasizing the importance of quality control in benchmark construction. To improve transparency and reproducibility, we provide the following details about our multi-stage manual review process:\n\n**1. Meta-Annotation Review**\n\nOur meta-annotations include semantic labels and spatiotemporal trajectories. Quality control strategies vary by data source:\n\n- Dynamic Target and Event-Oriented datasets: These datasets come with relatively complete raw annotations. We performed manual verification, error correction, and format standardization to generate high-quality meta-annotations.\n- Static Target datasets: For in-house or partially annotated public data, we first applied YOLO (for detection) and VLLM (for semantics) to generate initial labels, followed by multi-round manual refinement. Then, samples with major disagreement (e.g., ambiguous classes or trajectories) underwent additional review.\n- Inter-Annotator Agreement: For Static Target tasks, Cohen’s Kappa averaged 0.825, reflecting strong annotator consistency.\n\n**2. QA Pair Review Mechanism**\n\nAll QA pairs went through multiple rounds of manual review, focusing on:\n\n- Relevance to the video and semantic clarity;\n- Correctness of the answer;\n- Quality and plausibility of distractors.\n\nEach QA item was independently reviewed by at least two annotators, with low-quality samples flagged and re-reviewed.\n\n**3. Human Accuracy Baselines and Task Variability**\n\nTo establish human baselines, we sampled QA pairs across subtasks and conducted blind testing with trained annotators. Results are as follows:\n\n|Method|Static Target|RTP|HD|KIE|TCD|DDM|PTM|Avg.|Dynamic Target|AP|LP|DP|Avg.|Event Oriented| RP|RA|ARA|Avg.||Overall|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|-:|\n|Human|-|96.80|97.56|98.17|95.68|95.85|94.41|95.85|-|83.73|87.85|90.44|88.18|-|91.92|94.89|92.97|92.54||91.36|\n\nThese results reveal varying task difficulties:\n\n- Dynamic Target tasks are most challenging (88.18%), due to their reliance on temporal cues, causal sequences, and future trend prediction\n- Event-Oriented tasks involve complex reasoning and multi-agent interactions but offer clearer context, yielding higher performance (92.54%).\n- Static Target tasks are visually grounded and relatively simple, resulting in the highest accuracy (95.85%).\n\nThese findings illustrate the cognitive diversity across tasks and provide a solid reference for evaluating model performance. We will include more statistical plots and sampling details in future versions to further improve clarity.\n\n>**Q3: Joint ablations across e.g. FSTW and PEMF could be included to identify synergistic benefits.**\n\nThank you for your valuable suggestion. We have added a joint ablation study to evaluate the synergistic benefits of the Fine-grained Spatiotemporal Window (FSTW) and Persistent Event Memory Forest (PEMF). The results are presented in the table below.\n\n|Model Variant|OVBench (Avg)|OVO-Bench (Overall)| MLVU (M-Avg)|\n|:-|:-:|:-:|:-:|\n|w/o FSTW & PEMF|58.0|52.5|51.8|\n|w/o FSTW|59.1|53.7|69.4|\n|w/o PEMF|58.9|53.5|56.6|\n|**Full Model (Ours)**|**60.5**|**55.6**|**70.0**|\n\n\nThe results show that removing both modules leads to the most significant performance drop. Using either FSTW or PEMF alone improves performance compared to the baseline, but the best results are achieved when both components are integrated. This joint ablation confirms that FSTW and PEMF provide complementary benefits. FSTW enhances real-time spatiotemporal perception near the query timestamp, while PEMF supports efficient and persistent long-term memory, together yielding the strongest overall performance.\n\n>**L1: The limitations are only discussed in the SM, I think limitations should also be summarized in the main ms.**\n\nThank you for your valuable suggestion. Due to the page limitations of the initial submission, we did not include a discussion of the method’s limitations in the main manuscript. In the revised version, we will summarize and incorporate the key limitations into the conclusion section of the main paper."}, {"review_id": "3YJVSm67qZ", "reviewer": "Reviewer_WUtt", "summary": "The paper proposes a streaming VideoLLM consisting of a persistent event memory forest (PEMF), which introduces three penalty functions in content similarity, merge frequency, and temporal distance. PEMF enables storing very long past context at reduced token count and hence reduced memory. In contrast, short-term memory is maintained at high resolution for capturing fine-grained details. A new instruction-tuning dataset, OnlineIT, is also proposed along with a new streaming video benchmark, ODV-Bench. The proposed model, along with OnlineIT, achieves SOTA on multiple benchmarks.", "questions": "I will consider the authors' responses to both questions while deciding my final rating.\n- **Q1.** I am not fully convinced by Fig. 4, as it should ideally include comparisons with other merging methods like [36]. Otherwise, the claim in L317–L318 and L20–L21 remains unsubstantiated. Is this a general trend across video benchmarks and different models, or is the proposed model genuinely helping with token efficiency? \n- **Q2.** While the paper is framed around streaming video understanding, a key aspect in real-time, low-latency performance is not discussed. In fact, PEMF is likely to increase latency due to its multiple online operations, which may hinder techniques like prefilling and KV cache usage. From Fig. 5, it appears the model runs at around 1 fps on average, which is significantly slower than existing streaming models like VideoLLM-online [4] (~13.5 fps). Can the authors clarify their motivation for presenting this as a streaming approach despite the latency constraints?\n\nThe following question will not affect my rating, but a response would be appreciated.\n- **Q3.** Why is the proposed benchmark focused on autonomous driving scenarios rather than something else like daily activities (e.g., Ego4D), where videos are naturally captured online and streaming understanding is equally or likely more important? Is there a specific motivation behind this choice?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- **S1.** The proposed long-short-term memory approach is a generalization of existing efforts under one end-to-end trainable architecture. The simple and elegant design is appealing.\n- **S2.** StreamForest achieves state-of-the-art performance on multiple benchmarks including streaming, long-term, and short-term ones.\n- **S3.** The model ablation is well-conducted, and the supplementary material is comprehensive with dataset details and model hyperparameters for reproducibility.\n\n**Weaknesses:**\n- **W1.** Upon closer inspection, it seems that a significant portion of the performance improvement may stem from the larger/better instruction-tuning dataset, rather than the proposed model itself. In Table 2, comparable online VideoLLMs such as [4, 46, 50], especially MovieChat [36], are trained on significantly less data. In the Table 3 ablation, it appears that using the same instruction-tuning data, the performance difference between [36] and the proposed method is relatively small. Improvements primarily driven by a better instruction-tuning dataset are less appealing and not the main focus of the paper.\n- **W2.** The discussion section in the experiments is underdeveloped and does not adequately reflect the observed performance. It is written in a rather generic manner, and connections to specific results are missing. For example, in L294–L297, the authors dismiss similarity-based merging due to “severe spatiotemporal irregularities,” yet Table 3 shows that the accuracy of Similarity Merge is quite comparable to PEMF. Similarly, L307–L309 is also broadly written and not well-grounded in the performance observed in Table 4. This defeats the purpose of ablations.", "was_revised": true, "final_justification": "I thank the authors for their detailed responses to my queries and concerns in the rebuttal.\n\nThe rebuttal has improved my understanding of the work and my overall appreciation of the paper, and I am recommending an Accept. I strongly urge the authors to revise the writing throughout the manuscript. For example, the response to Q3 in the rebuttal is clear and well-written and would be very helpful if included in the main paper. Also, I encourage the authors to include these efficiency statistics in the main paper, which will further strengthen the work.", "rebuttal": "We would like to express our sincere gratitude for your review. Your suggestions will greatly help us improve our paper.\n\n>**W1: Upon closer inspection, it seems that a significant portion of the performance improvement may stem from the larger/better instruction-tuning dataset, rather than the proposed model itself. In Table 2, comparable online VideoLLMs such as [4, 46, 50], especially MovieChat [36], are trained on significantly less data. In the Table 3 ablation, it appears that using the same instruction-tuning data, the performance difference between [36] and the proposed method is relatively small. Improvements primarily driven by a better instruction-tuning dataset are less appealing and not the main focus of the paper.**\n\nWhile the OnlineIT dataset contributes positively to the overall performance of StreamForest, the primary source of improvement does not solely lie in the dataset’s scale but rather in our architectural innovations. In particular, the Persistent Event Memory Forest (PEMF) enables the model to maintain long-term, event-level persistent memory, which is a capability that cannot be achieved by instruction-tuning data alone.\n\nAs shown in Table 3 in the paper, even when trained on the same instruction-tuning dataset, StreamForest still achieves notable performance gains compared with Similarity Merge [36], including a +2.2% improvement on OVO-Bench and +2.0% on MLVU. These results confirm that persistent memory architecture, rather than data scale alone, is crucial for effectively handling long-horizon streaming video inputs. Furthermore, the additional experiments we present in response to Q1 (comparing PEMF with other merging strategies under varying token budgets) reinforce this conclusion by demonstrating PEMF’s robustness and superior token efficiency.\n\n>**W2: The discussion section in the experiments is underdeveloped and does not adequately reflect the observed performance. It is written in a rather generic manner, and connections to specific results are missing. For example, in L294–L297, the authors dismiss similarity-based merging due to “severe spatiotemporal irregularities,” yet Table 3 shows that the accuracy of Similarity Merge is quite comparable to PEMF. Similarly, L307–L309 is also broadly written and not well-grounded in the performance observed in Table 4. This defeats the purpose of ablations.**\n\nThank you for pointing this out. We agree that the discussion in the ablation section should be more explicitly tied to the quantitative results, so that task-specific performance gaps are clearly reflected. In the revised manuscript, we will refine this section to ground the analysis in the observed data.\n\nFor example, the modifications to lines L294–L297 are as follows: OVBench primarily emphasizes short-term, fine-grained spatiotemporal perception. Similarity Merge achieves performance comparable to our PEMF on OVBench (60.3% vs. 60.5%). However, its limitations become clear in tasks that require persistent memory and long-horizon reasoning. On OVO-Bench, PEMF outperforms Similarity Merge by +2.2%, and on MLVU by +2.0%. This is because similarity-based merging may over-merge frames within local video segments, potentially leading to spatiotemporal irregularities and the loss of local event-level representations.\n\n>**Q1: I am not fully convinced by Fig. 4, as it should ideally include comparisons with other merging methods like [36]. Otherwise, the claim in L317–L318 and L20–L21 remains unsubstantiated. Is this a general trend across video benchmarks and different models, or is the proposed model genuinely helping with token efficiency?**\n\nThank you for the detailed feedback. To further substantiate our claim regarding token efficiency and validate the robustness of our proposed Persistent Event Memory Forest (PEMF), we extend Figure 4 by including a direct comparison with the Similarity Merge [36] under varying visual token budgets. The following table summarizes the results at different visual token budgets. For each setting, we report both the absolute accuracy and the percentage of performance retained relative to default(8k).\n\n|Visual Token Budget|Method|OVO-Bench|VideoMME|MLVU|\n|:-|:-|:-:|:-:|:-:|\n|**1k**|Similarity Merge [36]|52.7 (98.7%)|56.2 (93.5%)|60.4 (88.8%)|\n| |**PEMF (Ours)**|**55.3 (99.5%)**|**58.5 (95.3%)**|**62.9 (89.9%)**|\n|**2k**|Similarity Merge [36]|52.8 (98.9%)|58.1 (96.7%)|63.5 (93.3%)|\n| |**PEMF (Ours)**|**55.7 (100.2%)**|**60.6 (98.7%)**|**65.9 (94.1%)**|\n|**4k**|Similarity Merge [36]|53.1 (99.4%)|59.8 (99.5%)|65.2 (95.8%)|\n| |**PEMF (Ours)**|**55.8 (100.4%)**|**61.4 (100%)**|**67.6 (96.5%)**|\n|**8k (default)**|Similarity Merge [36]|53.4 (100%)|60.1 (100%)|68.0 (100%)|\n| |**PEMF (Ours)**|**55.6 (100%)**|**61.4 (100%)**|**70.0 (100%)**|\n\n**The results demonstrate that PEMF has two core benefits:**\n- **Superior absolute performance:** PEMF outperforms Similarity Merge [36] across all token budgets, with an average absolute performance improvement of 2%-3%.\n- **Stronger resilience to extreme compression:** PEMF retains a higher fraction of its full-budget performance under severe compression (at 1k tokens, +1.8% retention on VideoMME).\n\nThe above experimental results confirm that the performance gains are not merely a general trend across benchmarks but stem from the intrinsic design of PEMF, which adaptively consolidates event-level memory and preserves semantically salient information. This mechanism enables StreamForest to achieve both competitive accuracy and high token efficiency under stringent resource constraints.\n\n>**Q2: While the paper is framed around streaming video understanding, a key aspect in real-time, low-latency performance is not discussed. In fact, PEMF is likely to increase latency due to its multiple online operations, which may hinder techniques like prefilling and KV cache usage. From Fig. 5, it appears the model runs at around 1 fps on average, which is significantly slower than existing streaming models like VideoLLM-online [4] (~13.5 fps). Can the authors clarify their motivation for presenting this as a streaming approach despite the latency constraints?**\n\nIt is not accurate to conclude from Figure 5 that our method operates at only 1 FPS based on the reported inference latency (~1s). The reported latency reflects the average time required to generate a complete response to a user query (i.e., from receiving a question to producing a full answer). In contrast, VideoLLM-Online [4] counts the non-informative silent tokens as part of its FPS rate. Its actual generation of meaningful responses is much sparser, as the model frequently outputs only [EOS] tokens to indicate silence when no answer is needed.\n\nIn practical applications, the effective FPS of a streaming model is heavily influenced by the frequency of questions and responses. Therefore, we report the latency per valid response as a more reliable measure of system speed. As shown in the table below, our approach does not suffer from a computational efficiency disadvantage compared to existing methods.\n\n|Method|Inference Latency (s)|\n|:-|:-:|\n|VideoLLM-Online [4]|0.846|\n|Dispider [32]|1.539|\n|StreamForest (1k)|0.867|\n|StreamForest (8k)|1.489|\n\nThe computational cost of updating PEMF is negligible compared to that of the vision encoder and LLM. We have added a comparison of PEMF with other memory mechanisms, including the Pyramid Memory Bank [13] and the Similarity Merge strategy [36]. The following table reports the time consumption of each method when processing a 500-second video stream (1 FPS) and producing one effective response at the 500th second.\n\n|Method|Vision Encoding (s)|Memory Update (s)|LLM Inference (s)|**Total (s)**|\n|:-|:-:|:-:|:-:|:-:|\n|Similarity Merge [36]|5.198|0.183|1.388|6.769|\n|Pyramid Memory Bank [13]|5.203|0.451|1.381|7.035|\n|**PEMF (Ours)**|5.218|**0.172**|1.394|6.784|\n\nThe results show that vision encoding dominates the overall runtime, processing at 95 FPS, while PEMF’s event consolidation accounts for only 0.172 seconds for 500 frames (2900 FPS). With PEMF integrated, the preprocessing speed for the LLM’s visual inputs decreases by only 3 FPS (5.218+0.172=5.39s, from 95 to 92 FPS), which is negligible compared to the efficiency gains achieved by significantly reducing the total number of visual tokens.\n\nTheoretically, PEMF is also compatible with KV-Cache. Each event-level merge modifies only a small subset of visual tokens, so KV-Cache recomputation and prefilling are required only for those regions (There is no need to recompute the KV-Cache for all subsequent tokens beyond the modified region, since they were originally computed with a more complete set of preceding visual information). With stable frame inputs, each event consolidation compresses visual tokens in the affected interval by ~50%, meaning the number of recomputed KV-Cache tokens is roughly equal to the number of new tokens added to PEMF. As a result, KV-Cache update cost is roughly doubled compared to the no-memory case.\n\n>**Q3: Why is the proposed benchmark focused on autonomous driving scenarios rather than something else like daily activities (e.g., Ego4D), where videos are naturally captured online and streaming understanding is equally or likely more important? Is there a specific motivation behind this choice?**\n\nPrevious streaming benchmarks, such as OVO-Bench, already incorporate Ego4D videos of daily activities. These benchmarks primarily evaluate MLLMs’ ability to perceive static scenes and narrate human–environment interactions in a stepwise manner.\n\nIn contrast, autonomous driving presents dynamic, high-stakes environments with rapidly changing scenes, complex multi-agent interactions (vehicles, pedestrians, and traffic signals), and demanding prediction tasks such as risk assessment and motion planning. These scenarios require models to balance long-term event memory with fine-grained short-term perception to avoid accidents and make timely decisions."}, {"review_id": "sMtKQ2oF9y", "reviewer": "Reviewer_gQt1", "summary": "The paper proposes StreamForest for streaming video understanding. The key contribution in the modeling part is Persistent Event Memory Forest (PEMF), a memory mechanism that adaptively merges video frames into event-level tree structures. Additionally, the authors propose a Fine-grained Spatiotemporal Window to enhance real-time perception by capturing detailed short-term visual cues. The paper also introduces an instruction-tuning dataset OnlineIT, and ODV-Bench, a new benchmark for evaluating streaming video understanding in autonomous driving scenarios. Through extensive experiments, the authors demonstrate that StreamForest achieves state-of-the-art performance on multiple benchmarks.", "questions": "My critical concern lies in the technical contribution and organization of the paper. The author could highlight the main contribution and reformat the paper. If the dataset/benchmark is a critical contribution, it should be verfied across different mllm archs.", "limitations": "The author has discussed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "For strengths: \n\n1. The paper proposes a new method along with a new instruction tuning dataset along a new benchmark, which can benefit the whole community.\n\n2. The proposed method has some ablation studies to verify its effectiveness.\n\nHowever, I have the following concerns for the paper:\n\n1. The organization of the paper is somewhat chaotic. Since the author proposes a new model, a new instruction tuning dataset, and a new benchmark. It sounds like those three components are independent, so why should they be composed in one paper? When I read the paper, it is more like an A+B+C paper without a main topic. The author should reconsider the motivation and reformat the paper. \n2. Does the comparison fair? Since the author proposes a new model, a new instruction tuning dataset, and a new benchmark, as well as results on those benchmarks. Though the author provides some ablation studies in Table 3~5. But I noticed that with the baseline dataset, the proposed method only has comparable performance or even lower (like 54.9 vs 53.5), so I think the performance gain is mainly from the dataset part. So maybe the paper is more suitable for the dataset and benchmark track? I think for the main track, the technical novelty may not be sufficient.", "was_revised": true, "final_justification": "After rebuttal, I think most of my concerns are addressed. Thus I will give a borderline accept score.", "rebuttal": "We sincerely appreciate your careful evaluation of our work and present our detailed responses as follows.\n\n>**W1: The organization of the paper is somewhat chaotic. Since the author proposes a new model, a new instruction tuning dataset, and a new benchmark. It sounds like those three components are independent, so why should they be composed in one paper? When I read the paper, it is more like an A+B+C paper without a main topic. The author should reconsider the motivation and reformat the paper.**\n\nThe three contributions of our work are not independent components but complementary elements designed to address three fundamental gaps in streaming video understanding: model architecture, training data, and evaluation. **All three share a unified motivation: To enhance the performance of StreamForest on streaming video understanding tasks and to provide a rigorous, application-driven evaluation.**\n\n**At the core of our work is the StreamForest architecture, which directly addresses two key shortcomings of existing online video MLLMs:** The absence of scalable memory and the limitation in fine-grained spatiotemporal reasoning. StreamForest introduces (1) the Persistent Event Memory Forest (PEMF) for adaptive long-term event storage, and (2) the Fine-grained Spatiotemporal Window (FSTW) for precise real-time perception, enabling efficient handling of long-horizon, dynamic video streams.\n\n**The OnlineIT dataset is designed to unlock the full potential of StreamForest.** Its construction explicitly targets three capabilities: (1) persistent memory of past events (aligned with PEMF), (2) fine-grained perception of the current spatiotemporal context (aligned with FSTW), and (3) prediction of future states (where PEMF and FSTW jointly contribute). The dataset and model architecture are thus tightly coupled, and their synergy leads to substantial gains in streaming video tasks.\n\n**The ODV-Bench evaluates StreamForest in real-world downstream scenarios that existing benchmarks fail to capture.** Unlike prior streaming benchmarks, ODV-Bench emphasizes rapidly evolving, dynamic environments with complex object motion, multi-agent interactions, and high-stakes prediction tasks. Testing StreamForest on ODV-Bench highlights how our approach outperforms existing methods under the very conditions it was designed for, thereby showcasing its practical value.\n\n**Together, these three components form a cohesive framework rather than isolated contributions:** StreamForest (the architecture) delivers the core capability, OnlineIT equips it with aligned training data, and ODV-Bench validates its advantages in realistic streaming video applications.\n\n\n>**W2: Does the comparison fair? Since the author proposes a new model, a new instruction tuning dataset, and a new benchmark, as well as results on those benchmarks. Though the author provides some ablation studies in Table 3~5. But I noticed that with the baseline dataset, the proposed method only has comparable performance or even lower (like 54.9 vs 53.5), so I think the performance gain is mainly from the dataset part. So maybe the paper is more suitable for the dataset and benchmark track? I think for the main track, the technical novelty may not be sufficient.**\n\n\n**On the fairness of the comparison:** In addition to evaluating StreamForest on our proposed ODV-Bench, we also validate its effectiveness on seven widely recognized, high-quality open-source benchmarks (Table 2 in the paper). All evaluations strictly follow each benchmark’s official protocols, ensuring that our comparisons fairly reflect the capabilities of our method relative to existing approaches.\n\n**On the validity of the ablations:** Table 3 shows that, under the same instruction-tuning data and visual token budgets, replacing PEMF with existing methods results in a 1.6–3.4% drop in average accuracy on OVBench, OVO-Bench, and MLVU. The drop is most severe on MLVU, which requires robust long-term memory retention. Similarly, Table 4 demonstrates that removing either PEMF or FSTW leads to significant performance degradation, confirming the unique contributions of both components in StreamForest’s architecture.\n\nTo further validate PEMF’s contribution, we extend Figure 4 by comparing it with the Similarity Merge [36] under different token budgets. The results are summarized below (accuracy with retention percentage relative to each method’s 8k-token setting):\n\n|Visual Token Budget|Method|OVO-Bench|VideoMME|MLVU|\n|:-|:-|:-:|:-:|:-:|\n|**1k**|Similarity Merge [36]|52.7 (98.7%)|56.2 (93.5%)|60.4 (88.8%)|\n| |**PEMF (Ours)**|**55.3 (99.5%)**|**58.5 (95.3%)**|**62.9 (89.9%)**|\n|**2k**|Similarity Merge [36]|52.8 (98.9%)|58.1 (96.7%)|63.5 (93.3%)|\n| |**PEMF (Ours)**|**55.7 (100.2%)**|**60.6 (98.7%)**|**65.9 (94.1%)**|\n|**4k**|Similarity Merge [36]|53.1 (99.4%)|59.8 (99.5%)|65.2 (95.8%)|\n| |**PEMF (Ours)**|**55.8 (100.4%)**|**61.4 (100%)**|**67.6 (96.5%)**|\n|**8k (default)**|Similarity Merge [36]|53.4 (100%)|60.1 (100%)|68.0 (100%)|\n| |**PEMF (Ours)**|**55.6 (100%)**|**61.4 (100%)**|**70.0 (100%)**|\n\nPEMF consistently outperforms Similarity Merge not only in absolute accuracy but also in performance retention under extreme compression, most notably on VideoMME (e.g., at 1k tokens: +2.3% absolute accuracy, +1.8% retention). These results confirm that StreamForest’s performance improvements arise primarily from its architecture’s robust event-level memory design, not merely from additional data.\n\n**On the role of instruction-tuning data:** **Even when fine-tuned solely on the base dataset, StreamForest surpasses existing state-of-the-art streaming models on most benchmarks.** For example, it achieves 53.5% vs. 41.8% on OVO-Bench (vs. Dispider, +11.7%) and 56.3% vs. 55.7% on ODV-Bench (vs. VideoChat-Online, +0.6%). Only on OVBench does it slightly trail VideoChat-Online (53.9% vs. 54.9%, -1.0%).\n\n**It is also important to clarify that the “base” dataset used in Table 5 is not directly comparable to the instruction-tuning datasets used by other models.** Specifically, VideoChat-Online’s online instruction-tuning data (which OVBench relies on) was not fully included in the base set. To isolate the effect of instruction data, we fine-tuned StreamForest on the base dataset plus the full VideoChat-Online instructions, and report the results below:\n\n|Training Data|ODV-Bench (Avg)|OVBench (Avg)|OVO-Bench (Overall)|\n|:-|:-:|:-:|:-:|\n|base|56.3|53.9|53.5|\n|base + VideoChat-Online (full)|57.4|56.1|54.0|\n|base + O.general|60.5|60.5|55.6|\n|base + O.general + O.drive|67.3|60.5|54.5|\n\nAfter incorporating the same instruction-tuning data used by VideoChat-Online, StreamForest surpasses it on OVBench (56.1% vs. 54.9%), confirming that the performance gains of StreamForest cannot be attributed to OnlineIT data alone.\n\n**In summary, StreamForest’s improvements stem from its novel architecture, particularly PEMF’s event-level memory consolidation and FSTW’s localized perception, with instruction-tuning data serving as a complementary factor rather than the primary driver. Therefore, the method fits the main technical track, as it introduces substantive architectural innovations validated by extensive ablations and cross-benchmark testing.**\n\n\n>**Q1: My critical concern lies in the technical contribution and organization of the paper. The author could highlight the main contribution and reformat the paper. If the dataset/benchmark is a critical contribution, it should be verfied across different mllm archs.**\n\nThank you for the feedback. **The primary contribution of our work is StreamForest, a novel architecture that integrates the PEMF and FSTW to jointly address persistent memory and real-time perception in streaming video understanding.** These challenges cannot be solved by scaling data alone, as existing MLLMs still suffer from uncontrolled memory growth, inefficient long-term reasoning, and degraded temporal coherence when handling long-horizon, dynamic inputs. The inclusion of the OnlineIT dataset and ODV-Bench benchmark is intended to unlock the full potential of StreamForest and to provide rigorous, application-driven evaluation, rather than to serve as independent contributions.\n\n**The organization of the paper already reflects this prioritization.** The StreamForest architecture is clearly positioned as the main contribution in the title, abstract, and throughout the manuscript. In the methodology section, two pages (L98–L173) are dedicated to the architecture’s technical innovations, while the OnlineIT dataset and ODV-Bench benchmark together account for only ~1.5 pages (L174–L238). Similarly, the experiments (Tables 1–4, Figures 3–4) are designed primarily to analyze and validate StreamForest, with only limited space (Table 1, Table 5) devoted to the benchmark and dataset.\n\n**In terms of technical novelty, StreamForest introduces two innovations that distinguish it from prior Video MLLMs:**\n\n- **Event-level persistent memory with adaptive token control:** PEMF consolidates historical visual tokens into a structured forest of event nodes, guided by multi-criteria penalties on similarity, merge counts, and temporal distance. This mechanism prevents uncontrolled token growth, preserves semantically salient events, and maintains coherent memory over thousands of frames, all of which are absent in existing approaches.\n\n- **Localized, fine-grained temporal perception:** FSTW dynamically allocates high-resolution windows near the current query timestamp, ensuring the model retains rich local context without overwhelming the LLM with redundant tokens.\n\nIn the revised manuscript, we will further refine the organization to highlight StreamForest as the core technical contribution, make this hierarchy explicit in the introduction, and incorporate the new ablation results presented during rebuttal to more comprehensively substantiate the effectiveness of the proposed architecture."}, {"review_id": "1oQeeJIHEE", "reviewer": "Reviewer_AFDL", "summary": "This paper introduces a novel architecture tailored for real-time video understanding with multimodal large language models (MLLMs). The core of the model consists of two modules: the Fine-grained Spatiotemporal Window, which captures high-resolution visual features for accurate real-time perception, and the Persistent Event Memory Forest, which organizes long-term visual information into adaptive tree structures using penalties based on content similarity, temporal distance, and merge frequency. To support training and evaluation, the authors present OnlineIT, an instruction-tuning dataset for streaming video tasks, and ODV-Bench, a benchmark designed for autonomous driving scenarios. Extensive experiments show that StreamForest outperforms existing online and offline models on multiple online video benchmarks and performs highly competitive with current online and offline models on widely used offline long and short video benchmarks. Moreover, StreamForest maintains high performance under extreme visual token compression, and demonstrates strong robustness, efficiency, and generalization for streaming video understanding.", "questions": "* In Table 1, StreamForest performs worse than VideoChat-Online on the Dynamic Target subset. Could you provide any insights into this phenomenon?\n* The performance gap between StreamForest and other models on offline short video benchmarks may not be as large as what is reported in Table 2. Recent models, such as Qwen2.5-Omni and InternVL3, have demonstrated strong performance on these datasets.\n* How was the base model in Table 5 trained? Was it trained without any data related to online video understanding?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths:\n    * The two core modules proposed in this work—the Fine-grained Spatiotemporal Window and the Persistent Event Memory Forest—are innovative in design, with ideas that are straightforward yet effective.\n    * The experiments in this paper are comprehensive, comparing StreamForest with existing models on both the proposed datasets and established benchmarks, demonstrating the effectiveness of the approach.\n    * This paper introduces both a training dataset (OnlineIT) and a testing benchmark (ODV-Bench) for online video understanding. Although these resources have not yet been open-sourced, the authors have committed to releasing them in the future, which is expected to benefit subsequent research in this area.\n\n* Weaknesses:\n    * Some details of the proposed algorithm need further clarification. Including a flowchart or block diagram could help present the algorithm more clearly. For example, how should the time of event $t_i$ be computed after two nodes are merged?\n    * It would be much better if an experiment comparing the time complexity of different algorithms (e.g. with Pyramid Memory Bank) is included, as maintaining the PEMF involves computations between each pair of root nodes. This potentially costly computation may not meet the efficiency requirements of streaming audio understanding.\n\nAlthough there is still room for improvement, I believe the overall quality of the paper is good, and I am inclined to recommend its acceptance.", "was_revised": true, "final_justification": "Through the rebuttal, my concerns regarding certain algorithmic implementation details, model training procedures, and the algorithm’s complexity have been addressed. Therefore, I will maintain my initial score.", "rebuttal": "We would like to extend our heartfelt gratitude to the reviewer for your insightful comments and valuable feedback. Your questions have significantly contributed to the improvement of our work, and we appreciate the opportunity to address them.\n\n> **W1: Some details of the proposed algorithm need further clarification. Including a flowchart or block diagram could help present the algorithm more clearly. For example, how should the time of event  be computed after two nodes are merged?**\n\n\nThank you for your valuable suggestion. In the revised version of the manuscript, we will enhance the schematic illustration of StreamForest by adding a flowchart or block diagram to present the algorithm more clearly.\n\nRegarding the computation of event node timestamps: each event node's timestamp is initialized as the average time of the frames it represents (e.g., if an event node spans frames from 10s to 14s, its timestamp is initialized as 12s). When two event nodes are merged, the timestamp of the new node is computed as a token-count-weighted average of the original nodes:\n\n$$t_{\\text{new}} = \\frac{t_i \\cdot n_i + t_j \\cdot n_j}{n_i + n_j}$$\n\nwhere $t_i$, $t_j$ are the timestamps of the original event nodes, and $n_i$, $n_j$ are the numbers of visual tokens contained in each node, respectively.\n\nThis weighted scheme prevents timestamp drift during multiple rounds of merging, especially when the merged nodes contain significantly different amounts of visual tokens.\n\n\n> **W2: It would be much better if an experiment comparing the time complexity of different algorithms (e.g. with Pyramid Memory Bank) is included, as maintaining the PEMF involves computations between each pair of root nodes. This potentially costly computation may not meet the efficiency requirements of streaming audio understanding.**\n\nThank you for the insightful comment. To address this concern, we have added a comparison of PEMF with other memory mechanisms, including the Pyramid Memory Bank [13] and the Similarity Merge strategy [36]. The following table reports the time consumption of each method when processing a 500-second video stream (1 FPS) and producing one effective response at the 500th second.\n\n|Method|Vision Encoding (s)|Memory Update (s)|LLM Inference (s)|**Total (s)**|\n|:-|:-:|:-:|:-:|:-:|\n|Similarity Merge [36]|5.198|0.183|1.388|6.769|\n|Pyramid Memory Bank [13]|5.203|0.451|1.381|7.035|\n|**PEMF (Ours)**|5.218|**0.172**|1.394|6.784|\n\nThe results show that vision encoding dominates the overall runtime, processing at 95 FPS, while PEMF’s event consolidation accounts for only 0.172 seconds for 500 frames (2900 FPS). With PEMF integrated, the preprocessing speed for the LLM’s visual inputs decreases by only 3 FPS (5.218+0.172=5.39s, from 95 to 92 FPS), which is negligible compared to the efficiency gains achieved by significantly reducing the total number of visual tokens.\n\nPEMF achieves this efficiency through caching mechanisms: similarity scores, merge counts, and temporal distance states between root nodes are cached in three lists, initialized only once when nodes are created. For each incoming frame, if the visual token budget is exceeded, PEMF performs a single-step merge update. The total penalty for each node pair is directly retrieved from the caches, and after merging, only the similarities with the immediate neighbors need to be updated. Consequently, the overall time complexity grows linearly with the number of frames.\n\nFor each individual merge, the dominant cost is computing the similarity matrix between two merged event nodes, which scales as:\n\n$$ T(N,C) \\approx O(N^2 C) + O(N \\log N) $$\n\nwhere $N$ is the number of visual tokens contained in the event nodes being merged, $C$ is the feature channel dimension. The $O(N^2 C)$ term arises from the pairwise similarity computations, while subsequent operations such as `max`, `argsort`, and indexing on the matrix add at most $O(N \\log N)$ complexity.\n\n\nIn contrast, the Pyramid Memory Bank [13] maintains three hierarchical caches with progressively coarser spatial granularity but finer temporal sampling. When the upper-level cache reaches capacity, frames are downsampled and pushed to lower levels, triggering similarity computations with all frames in the lower cache at each replacement step. This additional processing leads to slower memory updates compared to PEMF (0.451s vs. 0.172s for 500 frames).\n\nOverall, PEMF introduces minimal computational burden, scales linearly with stream length, and is demonstrably more efficient than prior memory mechanisms while still enabling persistent, event-level memory retention for streaming video understanding.\n\n\n> **Q1: In Table 1, StreamForest performs worse than VideoChat-Online on the Dynamic Target subset. Could you provide any insights into this phenomenon?**\n\nThe Dynamic Target subset primarily focuses on future prediction tasks involving dynamic traffic participants, such as vehicles and pedestrians. These tasks require the model to accurately predict an agent’s next movement or location. For example, determining whether a vehicle will change lanes or continue straight, or estimating spatial relations such as approaching or receding distance.\n\nTo better simulate real-world traffic, the benchmark includes challenging cases where agents behave unpredictably or violate traffic rules. For instance, as shown in the [AP] example in Figure 1 (in the supplementary material), a bus is driving in a right-turn-only lane. While the correct future action is (D) Moving to the left lane (as the bus illegally crosses a solid line), our model, which tends to rely on explicit cues such as road signs and lane markings, was misled by the right-turn marking and incorrectly predicted (B) Turning right. This is reflected in the performance drop on the AP task from 73.4 to 57.0 after fine-tuning with OnlineIT.drive, which does not yet include sufficient data for such counter-intuitive behaviors.\n\nMoreover, VideoChat-Online benefits slightly on this subset because its original training data is biased toward short-term dynamic tasks, which overlap with the demands of these prediction-oriented evaluations. In contrast, StreamForest’s training strategy was designed to balance performance across eight diverse benchmarks while controlling training costs, leading to some trade-offs in short-term dynamic prediction coverage. To validate this explanation, we conducted a supplementary experiment by post-training StreamForest with the full VideoChat-Online dataset. As shown below, StreamForest then outperformed VideoChat-Online on the Dynamic Target subset:\n\n|Method|AP|LP|DP|Dynamic Target (Avg)|\n|:-|:-:|:-:|:-:|:-:|\n|VideoChat-Online|72.6|89.3|48.1|65.6|\n|StreamForest|73.4|81.2|47.9|63.3|\n|StreamForest (FT-drive)|57.0|96.8|46.2|63.3|\n|StreamForest (post-trained on full VideoChat-Online data)|74.3|90.1|48.6|**67.8**|\n\nIn future work, we plan to expand our dataset with more high-quality, behavior-diverse, future-prediction instruction-tuning data to further strengthen the model’s capability on dynamic target tasks.\n\n\n> **Q2: The performance gap between StreamForest and other models on offline short video benchmarks may not be as large as what is reported in Table 2. Recent models, such as Qwen2.5-Omni and InternVL3, have demonstrated strong performance on these datasets.**\n\nThank you for the suggestion. In the revised version of the manuscript, we will include comparisons with more recent models such as Qwen2.5-Omni and InternVL3, which have reported competitive or superior results on short-video evaluation benchmarks. These results were part of contemporaneous work that had not yet undergone peer review at the time of our submission. Therefore, Table 2 in the initial manuscript focused on widely used open-source baselines from prior studies.\n\nIt is also important to note that our primary contribution is not to surpass all offline short-video MLLMs, which is particularly challenging under streaming video settings, but rather to enable robust streaming video understanding through our Persistent Event Memory Forest and Fine-grained Spatiotemporal Window . These innovations specifically target the unique challenges of streaming scenarios, such as long-term memory retention and real-time perception, which are not directly addressed by most offline models.\n\n\n> **Q3: How was the base model in Table 5 trained? Was it trained without any data related to online video understanding?**\n\nThe base model in Table 5 was primarily trained on offline video data sampled from VideoChat-Flash and LLaVA-Video, along with image data from LLaVA-OneVision. Additionally, to ensure the model could adapt to the response patterns required for online video understanding, we incorporated a small subset of raw data drawn from VideoChat-Online (including Charades-STA, AVA, LaSOT, COIN, and ActivityNet-Captions, totaling approximately 32K samples). To further enhance the model’s fine-grained perception capability, we also included a portion of samples  from open-source image-based spatial grounding datasets RefCOCO (approximately 30K).\n\nAlthough the base dataset shares some video data sources with OnlineIT.general, all data used for the base model was uncleaned, original data. In contrast, all cleaned and re-annotated data are fully contained within OnlineIT.general."}]

Author rebuttal:

Meta-review: This paper introduces StreamForest, a streaming video understanding framework for MLLMs with two key modules: (1) Persistent Event Memory Forest (PEMF) for long-horizon, event-level memory under tight token budgets; and (2) Fine-grained Spatiotemporal Window (FSTW) for high-resolution short-term perception. Empirical results are provided across multiple benchmarks to support its strong efficacy.

Overall, all reviewers view the problem as important and found the proposed approach simple and effective. Meanwhile, they flagged some major concerns: 1) the computational overhead of PEMF should be thoroughly analyzed; 2) more ablations should be provided to quantify the gains from architecture vs. data; and 3) certain parts lack clarity in writing and presentation, and need revision

The rebuttal is considered, which successfully address these concerns. As a result, all reviewers unanimously updated their scores to positively support the paper.

For the final version, please ensure that all promised revisions from the rebuttal stage are fully integrated to ensure the clarity and quality of this NeurIPS publication.
Decision: Accept (spotlight)