Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
Authors: Yulin Zhang
Cheng Shi
Yang Wang
Sibei Yang
Venue: NeurIPS 2025 poster ()
Keywords: Egocentric Video Understanding
Multimodel Large Language Model
Streaming Understanding
Primary area: applications

Abstract: Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond to unfolding events. Towards this vision, we focus on the innovative task where, given ego-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning. This task embodies three key properties: (1) Proactive Coherence, (2) Just-in-Time Responsiveness, and (3) Synchronized Efficiency.
To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric—a novel framework designed for their rigorous assessment. Secondly, we propose a comprehensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises: (1) a data engine, (2) a multi-stage training strategy, and (3) a proactive dynamic compression technique. Our proposed model effectively addresses these critical properties while achieving state-of-the-art (SOTA) performance on the standard COIN benchmark.

Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
Yulin Zhang'
Cheng Shi
Yang Wang' Sibei Yangat
ShanggaaTTech University "School of Computer Science and Engineering, Sun Yat-sen University
Schooo of Computing and Data Science, The University of Hong Kong
Project Page: https: tttps://2hangyl-4 gi thub. htw.b..o/pbblcctiiio//eeeeeee wowpo///////ewwwddoooo///e////e/
Abstract
Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond unfolding events. Towards this vision, we focus on the innovative task where, given ago-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning. This task embodies three key properties: (1) Proactive Coherence, (2) Just-in-Time Responsiveness, and (3) Synchronized Efficiency. To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-FI metric-a novel framework designed for their rigorous assessment. Secondly, we propose a comprchensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises: (I) data engine, (2) a multi-stage training strategy, and (3) proactive dynamic compression technique. Our proposed model effectively addresses these critical properties while outperforming multiple baselines across diverse online and offline benchmarks.
Introduction
Imagine Al assistant that follows you through your day- assembling furniture, searching for misplaced keys [3, 15], or preparing a meal [46, 44]-not just watching, but understanding, anticipat- ing [54], and responding proactively when needed as events unfold. To function in such human-like settings, where visual input is egocentric and continuously streaming, and user needs shift from moment to moment, the assistant must go beyond passive observation. It should be able interpret the present, anticipate what comes next, and respond at exactly the right moment, all in real time.
As first step toward this vision, narrow focus to perception and understanding in egocentric streaming video, with a particular emphasis on the following innovative task: Given ego-streaming video input, the assistant proactively answers diverse and evolving questions at the right moment, while seeing and thinking in sync, as shown in Fig. 1. This task relies on three key properties:
Proactive Coherence: handling diverse question types, responding even when answers depend on future visual streams (proactivity), and maintaining contextual consistency across related questions. In ego-streaming scenarios, questions often go beyond the current frame, referencing future events or past observations. As shown in Fig. 1, the segment of the conversation highlighted in green is contextually dependent on the content within the segment highlighted in purple. Such queries require temporal integration of past and present information, followed by proactive answering as relevant visual evidence emerges.
Lust-in-Time Responsiveness: determining when to answer based visual readiness, neither soon nor too late, and only when necessary. Responding before enough evidence is available can
Corresppndiigg author is Sibei Yang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Synchronized Efficiency: Model Seeing and Thinkingin Sync
Just-inT Responsiveness
Rovih Screo
Coo 190
[2:088
siltenc
Commesstion
want sotractsome liquld froma flakk, should do7
Ganyyou sell me where the parbagg whe drop them?
[05:40]
partaagls dropped
Eiu
SiLence- 941 58]
(99:38
Silence- 26)
01:26)
[62) 68
SE Silence- 02:23]
(62:25)
[04:50)
There are some nsedies
gartage dropped
You can throw them from the courner and the leb bench into the yellow bucket near the shell.
font of that con use to eetaet fluld
Proactive Coherence: Diverse, Evolving Questions and Response
Start Point
User Query
X
Model Action Slay Silent
v
Model Action :Response and Reply
An illustrative example of the ESTP task. The figure is structured in three layers: the top layer depicts the model's s continuous visual processing and decision-making (See and Think), the middle layer shows the real-world egocentric scene with the human's 's trajectory, and the bottom layer presents the human-model conversation.
Figure 1:
lead to mistakes, while answering too late may miss the opportunity to help. Equally important is staying silent when uncertain and avoiding unnecessary repetition. As shown in the blue-highlightesr segment of Fig. 1, it is necessary to remain silent until the "face to counter". The assistant must continuously track the evolving visual context and respond at the earliest reliable moment.
Synchronized Efficiency: ensuring that answering and visual perception proceed syne without delay. Responses should not come at the cost of missing new visual input; perception and reasoning must remain temporally aligned. Regarding the purple segment depicted in Fig. 1, maintaining synchronization crucial to prevent missed answers. This requires answering while continuously observing, with zero latency, while also ensuring time and memory efficiency as the number of incoming frames grows over time.
Unfortunately, existing evaluation frameworks [51. 27, 26, 61, 13] and streaming models [5, 52] fall short in supporting or measuring the unified capabilities of proactive, just-in-time, and synchronized reasoning- and often struggle even with some individual aspects. Offline video benchmarks [14, 66, 30, 50, 1] evaluate video LLMs across diverse question types and scenarios, but their offline nature limits the assessment of the three core capabilities essential for online deployment. Recent efforts toward online and streaming benchmarks address this gap by introducing proactive tasks. Nevertheless, shown in Tab 1, they often offer limited question diversity, lack contextual continuity across queries, and-more importantly-rared evaluate just-in-time responsiveness or synchronized efficiency. As result, current online video LLMss remain confined to narrow tasks such as narration or simple question answering, lacking the capacity for continuous, multi-turn understanding. More critically, illustrated in Fig. 6, these models exhibit poor just-in-time behavior- often generating under--esponsive or over-extended answers. Similarly, although recent efforts [51, 33] have begun to address efficiency, they tend to focus solely accelerating response -otettoiiotttttalllllylyly the cost of answer accuracy while overlooking the need to balance perception and answering under synchronized constraints.
As first step toward addressing these challenges, we introduce new Ego STreaming Proactive (ESTP) benchmark and evaluation framework, specifically designed to capture the demands of the three key properties in streaming video. For proactive coherence, benchmark are proactive in nature: each question can only be answered based future video streams
tasks the

---

offire MLLNS
VLSS
Dataset
Online Benchmark
Ques. Type
Esg. Imp.
Prouctive Type
Con.
Ans. Furn
JFL Responsivnnes Eval
15 Pre.. Timeliness
Ques.
Cura
VStream [61]
StremmngBenchh [2T]
MC
DE
x
x
S
5
X
X
X
4,500 x
3,500
just-In-Time
Sireaming Rench [37]
(WVD-Bench [26]
Q-Matchh
MC
x
5
5
X
X
x
2,514
50
Colieee ence
Proactive
Emicennyy
OVO Bench (FAR) (25)
MMDDuutt
C&
M
M
X
X
2000
1618
Benokwark
LigoPlan
EgoPPlu22 [35]
bgoSDQEES ESTP (Ours)
Q-Mitch
3,771
O8
M
2264
Figure 2: ESTP Triangle of Table I: Comparison of datasets based on proactive and streaming
Impossibility shows trade- criteria. This table summarizes datasets by Question Types (Open- offs among the three dimen- Ended (OE); Multiple Choice (MC); Query Matching (Q-Matcl & sions: Proactive Coherence, Q); and Count (C)), Proactive Types (Explicit (Exp.); Implicit (Imp.); Just-in-Time responsiveness, and Contextual (Cont.)), and Just-in-Time (JIT) Responsiveness. Key and Efficiency, which are JIT Responsiveness aspects include Answer Turn (Ans. Turn) (op- quantified by contextual per- tions: Single (S), Multi (M)), Precision (Is Prec.), and Timeliness. formance, recall, and FPS. The notation '# Ques.' denotes the number of questions.
within one or more specific time intervals. To reflect different levels realistic scenarios, we group them into three types: (1) explicit, grounded in clear visual cues; (2) implicit, requiring reasoning beyond surface observations; and (3) contextual, involving temporally linked questions that demand consistent multi-turn answers. We collect 2,264 questions spanning 14 task types- such object localization, state change understanding, and intention prediction-acron over 100 types of distinct scenarios, including kitchen activities, social interactions, and daily object manipulation. For just-in- time responsiveness, we emphasize the importance of response timing: each question are annotated an average of 3.96 valid answer intervals, and prediction is considered valid only if it falls within the designated window. To assess this, we introduce ESTP-F1, a metric that integrates answer quality, response timing, and temporal precision. Additionally, 46% of questions are contextually linked, the need to continuously track the evolving stream from past to future and respond the right moment. For synchronized efficiency, we not only evaluate time and memory efficiency and answering accuracy independently, but also assess accuracy under tightly synchronized perception and response -oapein-offfering a comprehensive perspective streaming video LLM evaluation.
requiring coherent responses based on prior questions-highlightee
To address this novel task, we propose a comprehensive and novel technical pipeline-includiin data engine, multi-stage training strategies, and a proactive dynamic compression technique-to enhance the streaming video LLMs. Specifically,
The data engine automatically generates diverse, multi-turn questions and their corresponding answers to support the demands of continuous and proactive question answering. This involves three-stage generation pipeline covering (1) one-to-one; using LVLMs to generate captions and extract initial question-answer pairs with a single temporal answer interval; (2) one-to-many: apply- ing RAG to expand each answer into multiple valid intervals; and (3) many-to-many. composing coherent multi-turn questions from related QA pairs.
The multi-stage training strategy is employed to progressively learn: (1) passive interval re- sponsiveness, which provides basic ability trigger responses by distinguishing visually similar frames with different response labels, but often results in responsesnssss even when the correct response interval: (2) Proactive just-in- in-time responsiveness and accurate answering, which trains the model to actively request high high-resolution frames during uncertain timestamps, allowing it to use fine-grained visual details to pinpoint both the correct response moment and the accurate answer; (3) Coherence across multi-turn QA, which enables the model to maintain consistency by reasoning over prior QA history and current context, supporting contextual consistency answering.
The proactive dynamic compression technique fully leverages the streaming nature by applying two levels of token compression based on response likelihood, including: (1) when the model anticipates potential response, it proactively requests high-resolution inputs to improve the accuracy of perception and answering; (2) Otherwise, it applies a higher compression rate to past content to reduce token usage and improve efficiency; (3) Additionally, once a response is completed, the content preceding its timestamp is further compressed to free up resources without affecting future perception or answering.
3

---

In summary, our contributions include: the novel Streaminn Proactive (ESTP) task, distinguished by its three key properties; the ESTP-Benee Ben-hBBchh benchmark and the ESTP-F1 metric for robust evaluation of this task; and a comprehensive and novel technical pipeline, incorporating three key techniques, designed to address the ESTP task. Our results demonstrate that the proposed model effectively overcomes the key challenges posed by this task. Moreover, it demonstrates superior performance by substantially exceeding multiple baselines in diverse online and offline benchmarks.
2 Ego Streaming Proactivate Dataset & Benchharr
2.1 Data Source and Annotation
Data Source is validation set of Ego4D [17, 44] that includes raw annotations such event narrations and steps for completing consistent goals. Following [28, 5], we filtered out video with missing or uncertain annotations and converted annotations into natural language format. This process yielded 890 videos, encompassing over 100 distinct scenes and a wide array of human activities, including indoor home environments (e.g., cooking, cleaning), workspaces (e.g., working desk, labwork, baker), and public areas (e.g., grocery shopping). Furthermore, the videos exhibit rich dynamic diversity, ranging from periods of relative stillness (e.g., observing a static scene) highly dynamic moments involving rapid manipulation tasks or active locomotion (e.g., cooking, walking).
Annotation process follows two-step procedure. First, initial QA pairs automatically generated with the assistance of MLLMs [56, 49] and LLMs [10]. Second, these automatically generated questions provided inspiration for annotators, aiding them in identifying valuable instances or formulating question ideas. To ensure diversity of questions, we annotate three proactive types:
(1). Explicit Proactive Tasks are defined those re- quired to identify and respond to queries by directly leveraging and interpreting visual information present in the input. This category encompasses tasks where the relevant visual cues are explicitly referenced or are central to formulating a correct response. This category comprised of eight distinct task types: Object Recog- nition (OR), Attribute Perception (AP). Text-Rich Un- derstanding (TRU), Object Localization (OL), Object State Change (OSC), Ego Object Localization (EOL), Ego Object State Change (EOSC), and Action Recogni- tion (AR). (2). Implicit Proactive Tasks are defined as those requiring inference and deeper scene understand- ing that goes beyond immediate, direct observation. This category is comprised of four distinct task types: Object Function Reasoning (OFR), Information Func- tion Reasoning (IFR), Next Action Reasoning (NAR), and Task Understanding (TU). (3). Contextual Proac- tive Tasks are defined as those requiring the model to maintain awareness of dialogue history and visual co- herence across temporally extended interactions. This category is comprised of two distinet task types: Ob- ject Relative Context (ORC) and Task Relative Context (TRC). Fig. 3 illustrate dataset distribution.
Explili
Imprica
CBCA
(b)
the
sachine class. weights
Sttniig to mai
Keading books
Hosting
the Idog / pet
Zike
Craft
Saharg with Farely messoss
Baker
Talk
Walking street
Working desk
160
Iestbeer Aesartgee bealkiin))
Frequency
Figure 3: (a) Frequency of scenes or ac- tivities from which tasks and questions are derived. (b) Proportion of different proactive and question task types.
To enable the evaluation of fust-in-Time Responsiveness and eliminate ambiguity in answer intervals, human annotators are required to mark clear time interval boundaries based on the completeness of objects within frames or the start/end of events. Simultaneously, questions with ambiguous references are filtered out (e.g., "Remind me the location of the ceramic bowl." where multiple ceramic bowls might be present in different locations). Each sample's 's question, answer, and corresponding answer interval are verified by two annotators. This rigorous verification process resulted in a dataset of 2264 verified question-answer instances. Notably, every answer the dataset associated with precise temporal annotations. Statistical information regarding the annotated data is presented in Fig. 3.

---

2.2 Evaluation Metric in ESTP
To comprehensively measure performance along three key evaluation aspects answer quality, response timing, and temporal precision ground truth item It with content og, and prediction as iI with content oj and time &. Evaluation components are defined for matched pairs (ft, Is), where Pi a prediction that temporally matches Ox- For answer quality, LLM [10] is used measure correctness, defined for the predicted content oj relative to the ground truth content Oę- For evaluating response timing, we go beyond simply considering recall (which inherently accounts for False Negatives (FN)) and employ score Sime (f), 9k) to more precisely measure timeliness. Furthermore, for temporal precision, introduce precision, utilizing False Positives (FP) as a penalty term. These components contribute to the aggregated ground truth score S(gk). which replaces the traditional binary TP count. The ESTP-F1 score is computed as:
introduce the ESTP-F1 score. Here, denote
score Sanswer (ô1, o&)
2X Σ 1= S(g)) ESTP-FI 2 _Σ ((g)) + FP + FN
where M is number of GT. High answer quality (reflected by a high Sanswe score), effective response timeliness (characterized by high Süme for on-time responses and low False Negative (FN) rate), and high precision (indicated by a low False Positive (FP) rate) collectively contribute to a high ESTP-FI score. More details are provided in the Appendix.
3 Methodology: Videol.LM-EyeWW
In this section, introduce technical pipeline designed for the ESTP task. For the data engine, utilizing the Ego4D [17] training set and a three-stage generation pipeline as introduced in Sec. 1, we generate 60K single-turn and 20K multi-turn questions, shown in Fig. 4. Each generated instance includes questions, answers, and their corresponding valid answer intervals (named as ESTP-IT). See Appendix for data engine details. Subsequently, we detail the problem definition and preliminary, the multi-stage training strategies, and the proactive dynamic compression technique in respective subsections.
3.1 Problem Definition and Preliminary
Problem Definition. Given a streaming video input and a sequence of emerging queries Q (ge, fg.)). where qi is the query content and ty is the query timestamp. At each timestep t following query (i..., t > ). the model must leverage its historical memory H, (including visual input history and past query-response interactions). while concurrent observation O,, to decide whether to perform response action and generate corresponding content. The model's dlaision-making process at time & can be formulated as selecting the optimal action A, from a predefined set A:
A argmax Po = a qis Oz, He).
Here, & represent model parameter, A₁ is the model's action at time t, and A is the set of possible actions. Notably, while previous work typically considers an action space that only includes asilence (staying silent) and Gresponse (executing a response and generating reply), we expands this by including the action dask high (requesting a high-resolution frame), as introduced in Sec. 3.2 Stage-
Preliminary. LIVE [5] utilizes ground truth containing timestamps and applies cross-entropy supervision [48] to the model's action output each timestep. Specifically, if the current time / falls within a ground truth response region (denoted as E Tumestamp)) the model is supervised to execute the response action (dresponse) and generate reply, incorporating a language modeling loss LIM [59, 11, 48]. Otherwise, it is supervised to remain silent (Acontinne). ). This is formulated as:
C(1) log Po (Crespoose
9t. Ot, H:) + WLLM(t) if E T Timesstmpp otherwise,
log Po (Ogontinue q: O,, H₂)
where, w is balancing coefficient weighting the language modeling objective.
5

---

EGO
Stag2
Stage3
Cache Turn
There someenele neeiie in Infrrnter you thu Cae tornnt
Compression
One
BEBB
Low.Besolation
Framie
High Resolution Tolknns
LaBA
Comprene
lane Re Form
D DPO
W | DA D0
D
Maryy
System Prompt and Query
Low-ltavclution Frame Toleens
Resoluttttt Tokens
many
(a) Data Engine
(b) Multi-Stage Training Strategy
(c) Preactive Dynamic Compression
Antwer interval Count
Questiin Count
Sty Slient
Askt High Resolution Frame
Response
High Resolution Frame
f Not in IVV Cache
Figure 4: Overview of the proposed pipeline. The figure illustrates the three main components: (a) Data Engine (ESTP-Gen), which automatically generates diverse, multi-turn QA data through a three-stage pipeline. (b) Multi-Stage Training Strategy incrementally builds the model from basic responsiveness to proactive just-in-time accuracy, and ultimately to achieving multi-turn coherence, detailed in Section 3.2. (c) Proactive Dynamic Compression detailed in Section 3.3.
3.2 Multi-Stage Training Strategy
Following [5], LM-eoLLM-EyeWW utilizes the same network architecture and trained using LoRA [20]. However, the single-stage training and simple binary supervision strategy employed in [5] can lead to training conflict due to the high similarity of adjacent frames streaming inputs. This conflict necessitates difficult trade-off between over-extended and under-responsive. To address these limitations, we employ a multi-stage training strategy designed to progressively endow the model with response capabilities. The following subsections detail each stage of this training strategy.
Stage-1: : Passive Interval Responsivenes. To provide the basic ability for autonomous response triggering, we leverage the valid answer intervals within the ESTP-IT to achieve progressive transition from silence to response. Specifically, if current time / falling within a valid answer interval (where Tinerval is defined the set of all such intervals e,), we apply a weighted degree of response supervision, rather than direct binary classification, using the following loss function:
log
[t-e| 2--
Po (Cresponee 9t, Ot, H,)
96 Or, Hi) wLI((() if E(s, e] Timerval E
[s,e]
C(t)
log Po (Ccomtie
otherwise
(4)
The function f is a linear decrease map used as a weighting factor applied to the response probability loss. The highlight in Equ. 4 is used to distinguish the components specific to this stage.
Stage-2 :
Proactive just-in-time responsiveness and accurate answering. To use fine-graine mreigrained visual details to pinpoint both the correct responne moment and the accurate answer, we train the model to actively request high-resolution frames during uncertain timestamps in this stage. Specifically, we first introduce third predefined action 0ask high- When the model executes this action at time it triggers the acquisition of the high-resolution frame O corresponding the current observation O using the following loss function for training: Lask high
high (t) log (f
[
[[-e|
Po Qask_high 9z Ot, H1)
Gi. O,, 11:)) if t T Tuneertain
(5)
log Po (Acontinue
otherwise,
where Tuncertain denotes the of the model's uncertain (see more detail in Appendix D Stage-2 Input). We use this loss to enable the model to acquire the ability to request high-resolution frames, and then based the more detailed information, determine whether it is the correct time to respond and provide a more accurate answer, using the following loss:
log Po (dresponse qi O,, H4, O + WLM if E T Tumestamp
Cdeternine (t) =
(6)
log Po (Ccontimue yi, O,, H1, 0%:
otherwise

---

where O. represents the high-resolution frame acquired at time t. The overall loss function at timestep is the sum the two components:
[(t) = Lask high (igh + destermine (t)
See appendix for detailed uncertain timestamps Tuncertain identified.
Stage-3:
Coherence across multi-turn QA. Building upon the model's acquired proactive and timely response capabilities, we introduce separate training stage. Specifically, this stage involves training solely on multi-turn question, with the aim of further improving its contextual understanding while preserving its timely responsiveness.
3.3 Proactive Dynamic Compression Mechanism
In order ensure memory efficiency the number of incoming frames grows time, we propose the Proactive Dynamic Compression Mechanism, which applies two levels of token compression and employs uniform compression method, detailed respectively in the following two subsections.
Two-Level Compression. In contrast to fixed compression rates [29, 34, 4] and steps [43, 39, 58, 63), our mechanism leverages the streaming nature to allow the model to proactively determine both when to compress and which compression level to apply. Regarding the timing of compression, after the model generates response, the preceding visual input and the response content itself form natural segment or processing unit. Simultancously. lower compression rates are applied to question- relevant content such high-resolution frames, while higher rates are applied to other content, with these decisions proactively made by the model. Specifically, after a response, fixed number of compression tokens (e.g.. 1) are used compress the preceding content, absorbing information from potentially many now-resolution frames or a single high-reson nigh-resolution frame. This approach naturally achieves high compression rate for redundant parts of the past content, resulting in average token usage of only about one-tenth of the original sequence.
Uniform Compression Method. For achieving two-level compression, employ Uniform Compression Method. Specifically, unlike methods using additional compression modules [34, 33], insert special compression token ((ct)) after segments of original input, namely after single high- resolution frames, after multiple low-resolution frames, and after answer. This token is initialized using the text embedding of the "<EOS>" token. Leveraging the properties of the causal self-attention mechanism, this token prompts the LLM to compress the information from the preceding segment into compact representation stored in the KV cache.
During training, inspired by [43], LLM trained to process response turns sequentially. A response turn refers to turn of interaction, typically comprising visual input and a model's response. Training for the Proactive Dynamic Compression Mechanism, including the integration of high-resolution frame requests, commences in Stage of our multi-stage training strategy to ensure manageable training memory overhead.
4 Experiment
4.1 Baseline and Evaluation Settings
We evaluate three categories of models in this study: Offline MLLMs, VLMs, and Online MLLMs.
For Offline MILMS we selected representative models from different open-source MILM families, including LLLVA--On Vision [23], Qwen2-VI. [49], MiniCPM-V [56], LLLVA-NeXT-Videe [24], and VnternVL-V2 [7]. As offline MLLMs lack inherent proactive response capability, following previous studies [27, 26, 51, 5], employed two evaluation settings: (1) Response-in-Last The model processes the complete video and is tasked with generating textual reply with timestamps. (2) Polling Strategy: The model is periodically queried at fixed time intervals. If the model indicates readiness, it is then prompted to generate the answer. Specific details regarding the prompts and hyperparameter used in these settings provided in Appendix.
Regarding VLMs, following the approach of SDQES [13], we selected CLIP [36], La [64], and Ego [28] for evaluation. These models were evaluated by computing the similarity between each frame and the query, using 0.5 as the threshold to determine responsiveness. Notably, as these models cannot generate open-ended replies, their reply score is set to 0.
7

---

Model
Expheit Pronctive lask
Impheit Pronctive Iask
Contextual Q
Overall
OR
AP
TRU
OL OSC BOL EOSC
AR
All
OFR
IFR
NAR
TU
All
ORC
TRC
All
Offiine MLLMs Response-in--aau
I..VVA-One ViViion
7.2
11.5
4.9
4.9
3.8
11.6
10.8
8.2
8.7
Qwen2-VL
11.7
8.1
14.9
10.5
1.7
8.9
10.6
6.0
9.0
10.2
4,4
265
49.5
22.6
13.3
94
11.3
13.3
MiniCPM-V
12.3
12.6
10.7
13.7
8.6
7.5
11.9
10.4
11.8
36.0
55.3
28.1
32.6
25.4
29.0
18.1
LL.VA-NeTTVVdde
8.3
9.4
7.4
10.2
7.8
7.4
10.3
5.6
8.3
6.4
6.7
21.1
45.9
20.0
10.1
9.8
9.9
11.9
VL-
9.3
14.6
9.5
10.6
1.7
6.3
3.0
3.6
7.3
3.3
9.2
15.5
28.2
14.0
16.9
15.6
16.2
10.5
VLMs for Streammng Detection
CLIP
7,3
9.5
7.4
8.5
1.8
4.7
2.2
5.5
2.8
5.2
51.3
29.3
22.2
4.6
3.8
4,2
10.1
LaViLa
8.4
10.7
9.0
9.1
3.1
5.4
3.6
4.3
6.7
7.8
10.0
56.2
34,4
27.1
9.4
28.9
19.2
14.3
EgoVLP
11.0
8.7
6.2
58.4
48.3
8.0
16.6
Offinee MLLM Polling Strotegy
LLavA, LLaVA-OneVision OneVisson
8.3
8.8
22.8
25.4
13.5
9.8
9.6
10.3
13.6
20.3
20.9
35.9
49.9
31.8
14.6
1.9
8.2
18.0
Qwen2-VL
15.4
17.8
56.4
13.0
10.4
MiniCPM-Y
14.9
16.8
17.1
26.8
7.7
12.9
12.5
13.1
15.2
15.9
21.0
46.8
62.2
36.5
24.3
28.9
26.6
22.9
LLaVA-NeXT-Video
15.6
14.6
21.9
12.8
13.5
12.3
16.5
18.6
23.2
44.9
34.6
19.9
13.8
21.3
InternVL.-V2
11.3
5.9
7.0
10.1
0.7
2.7
5.2
2.2
5.6
8.3
2.9
4.3
11.2
6.7
6.1
5.1
5.7
5.9
Ouline MFJ.s
LIVE(ibrebolld-0.5)
9.7
11.0
7.4
10.8
19
6.0
5.6
7.0
4.2
7.4
12.9
12.8
9.3
19.6
13.8
11.8
9.1
LLVE(heesoll:d0.9)
11.2
13.9
7.9
13.2
5.6
9.4
6.0
8.9
9.5
5.8
8.9
41.0
46.7
25.6
11.3
26.5
18.9
15.5
MMDuet
7.2
10.3
17.6
10.2
4.2
6.1
8.8
8.5
9.1
10.0
7.7
50.1
69.1
34.2
17.4
23.1
20.3
17.8
26.6
26.6
25.1
19.8
20.8
20.7
23.6
24.8
75.3
78.7
39.5
43.6
Table 2: Experimental results of various models evaluated
the ESTP-Benche We present perfor-
mance across Explicit Proactive, Implicit Proactive, and Contextual Question task types, as well as the Overall score, for Offline MILMS (Resoonse-inLaan -nst and Polling Strategy), VLMs for streaming detection, and Online MLLMs. Deep blue highlights the best overall performance, while blue indi- cates the best performance within each model category and evaluation setting group.
For Online MLLMs, we selected VideoLLM--nling [5] and MMDuet [51], which provide open- source weights and streaming inference code, for evaluation. For VideoeLL--Onlnee, we experimented with different thresholds to assess its performance variations.
4.2 Benchmarking in ESTP-Bench
Comparative Analysis of Baseline Models. Tab. shows the performance of different models across three proactive types and fourteen task types under various evaluation settings, the experimental results consistently demonstrate that ESTP tasks pose significant challenges for current types of models. Analysis revealed variations across model categories, with certain models exhibiting stronger capabilities within their respective groups (e-g.. MiniCPM-V [56] and Qwen VI.-2 [49] among offline MILMMs aligning with previous work [8]. and temporal VL.Ms like LaVil La [64] and VLP [28] outperforming spatially- strategy significantly impacts performance. Specifically, offline MLLMs showed notable disparity, performing on average better under the Polling Strategy compared to the Response-in-
models like CLIP [36]). Furthermore, the evaluation
strategy,
with improvements up to 5.4%. This highlights the effectiveness of -enchh in evaluating models from a timeliness perspective and underscores the limitations in temporal grounding of existing offline models.
Performance of VideoL. L.deoL.LM-EyeWO Against Baselines. As presented in Tab. 2, our proposed model achieved significant performance improvements across all proactive tasks. Compared to the baseline videol. LM-M-l [5], our model demonstrated an improvement +19.2% Furthermore, outperformed the best- ressoperforring model, MiniCPM-V [56](using the Polling strategy), by +11.8%
4.3 In-Depth Analyses in ESTP-Bench
Challenges with Coherent and Contextual Questions: Fig. 5 illustrates the average performance of different models across 14 tasks. (NAR) and (TU) exhibit significantly higher performance compared to other tasks. Upon visualizing the proportion of valid answer intervals relative to the input video duration for these two tasks, we observe that this proportion is substantially higher than for other tasks. This is attributed to these annotations originating from the raw GoalStep [44] labels, which involve segmenting continuous actions towards consistent goal, thereby leading to a larger proportion of valid answer interval within the video and consequently, higher Recall. Conversely, for the (TRC) task, which also derives from the same original annotations and possesses a high proportion of valid answer interval, both Recall and overall performance significantly decrease. This marked performance drop underscores the significant challenge that proactive coherence and understanding contextual information pose for existing models.
8

---

Performance Recall
34-
Model. Categories
Response
VIMs
23
32
Offinne MLLMS
WydeLLMMEycWWo
30-
Cnline MILMs
AP5 (Action Per Secoid)
2tr5 thruhhodd
0.4c
MiniCPM-V
Ragion Value
0.53
Objject
0.51
Performance/Recc
Proctsion
28
Owenz-VL
MiniCPM-V
Owen2-VL
Performanice Speed Tradeoff
Tou Contrat
LINEI.8)
LIVE(0.9)
LLaVi-OneVision
0.
MiniCPMEV
LIVE Base(0.9)
MMDuut
0.0
Cla NeXT-iide
Byewo stage
0.59
Eyewo
0.0
180
15
Types
Performance
Figure 5: Average performance Figure 6: Recall-Precision trade- and Ground Truth interval pro- off for different models and portion across 14 tasks, illustrat- evaluation settings, highlighting ing challenges with coherent and the difficulty in responding only contextual questions.
Figure 7: Action Per Second ver- sus ESTP Score for various mod- els, measured on an A40 GPU, demonstrating synchronization efficiency challenges.
when necessary.
Difficulty in Responding Only When Necessary: Fig. 6 presents the relationship between recall and precision for different models under various evaluation settings. We observe a prevalent negative correlation between recall and precision among most models. For instance, MMDuet achieves exceptionally high recall but at the expense of low precision. This trade-off indicates the struggle of existing models to provide proactive yet precise responses.
Synchronization Efficiency Challenges: Fig. 7 illustrates the inherent performance-speen tradeoff in ESTP tasks by plotting Action Per Second (APS) against Performance Score for various models. Existing methods often lie along a clear tradeoff curve, where higher performance is typically associated with lower APS, highlighting the difficulty in achieving both simultaneously. As seen for offline MILMMs using the Polling strategy, achieving high performance while maintaining sufficient speed for real-time synchronization remains challenging. Even approaches near the input frame rate (e.g., LIVE at ~2 FPS) may demonstrate suboptimal performance. This underscores the significant challenges current models face in achieving both high task performance and effective synchronization with the dynamic video stream.
4.4 Evalutation of VideoL.M-EExeWO
Evaluating Zero-Shot Capability in Online/Offline Tasks. Table 3 presents a performance comparison of our model against the baseline on both online and offline tasks. We selected VideoLLM- online [5] as our baseline, given that it shares the same base model (LLaMA3 [16] and SigLIP [60]) and data source (Ego4D)
own mode. For the online task, utilize OvO-Bench [26]
recognized benchmark. For the offline task, following [12], we evaluate our model on the multiple- choice subset of the QAEGO4D-tess benchmark [3]. The 'Online' setting involves posing questions as soon as the relevant answer segment appears, whereas the 'Offline' setting involves questioning after the entire video has been presented. The experimental results demonstrate the generalization capability of our model across these distinct tasks.
Online Task: OVO-Bench
Offline Task
Model
Real-Time Perception
Backward Tracing
QAEGO4D
OCR
ACR
ATR
STU
FPD
OIR
Avg.
EPM
ASI
HLD
Avg.
Online Offline
V
8.05 23.85
12.07 14.04 45.54 21.20 20.79 22.22 18.80
12,18 17.73 29.80 30.20
Ours (VideoLLM-ByeWW))
24.16 27.52
31.89
32.58 44.55 35.87 32.76 39.06 38.51
6.45
28.00 36.20
33.00
Table Detailed Performance Evaluation on OVO-Bench [26] and QAEGO4D [3] Tasks.

---

Evaluating Architecture Generaliz- ability on Offline Tasks As presented in Tab. 4, our model demonstrated com- prehensive performance improvements on five tasks related to traditional tempo- ral summarization and forecasting prob- lems. The performance gain reached up to +2.8%, which indicates that our pro- posed model architecture can effectively generalize to other offline tasks.
COIN Benchmark
Step
Task
Next
Proc
Proc+
ClipRERT
30.8
65.4
V
[S]
59.8
92.1
44.7
47.9
OMeeolLM-oonline--- 88- [5]
63.1
92.6
49.0
49.7
53.6
LM-DOL [52]
92.7
Ours (LLaMa3 [16, 47])
63.9
92.7
50.9
50.8
54.7
Ours (LLaMa3. (16,47)
66.0
93.3
51.5
51.1
55.5
Table 4: COIN [46] Benchmark Top-1 Accuracy comparison across different methods.
5 Related Work
Streaming Video Understanding. Unlike traditional offline video understanding tasks [14, 66,
1, 50] which only allow for question answering after processing the entire video, Online Video Understanding tasks aim to evaluate the ability to respond to questions that arise at any time, based on past information and current observations from sequential video stream input. Previous work [61, 26, 27] has employed various question types presented during the video stream, such as object or event perception, leveraging different spatiotemporal cues to comprehensively evaluate streaming video understanding capabilities. However, for proactive tasks where questions often require reasoning [65] beyond the current frame, existing methods often exhibit limited question diversity and lack contextual continuity across queries. Furthermore, evaluation in previous work has often overlooked efficiency and timeliness.
To specifically endow MILMS [42] with proactive response capability, LideoLLLM-Oning [5] pro- posed the LIVE training framework, which supervises the model's output at each frame. Subsequent work has focused on improving training efficiency [52], inference speed [33], and response capa- bility [5]]. However, these methods often generate under-responsive over-extended answers and struggle to adapt to continuous, multi-turn conversation.
Egocentric Video Understanding. Compared to traditional third-person video understanding, egocentric video data poses unique challenges, such as the scareity of large-scate scale annotated datasets and the inherent narrow and often unstable viewpoint. However, recent work [17, 18, 9, 44] has contributed massive egocentric video datasets and fundamental annotations, significantly benefiting the community. Prior work has typically focused on classic egocentric video understanding tasks such as activity recognition [25, 38], temporal grounding [55, 31, 41], and hand-object detection [2, 21]. which are often limited by closed vocabulary. Other efforts have focused on offline video understanding tasks, such as QaEgo4D [3], EgoSchema [30], and MM-Ego [57]. While SDQES [13] introduced a streaming detection benchmark for egocentric video, it primarily evaluates similarity- based responsiveness and lacks the scope to assess the open-ended generative and conversational capabilities required from Online MLLMs in proactive scenarios.
6 Conclusion
We definite an novel AI assistant's task of proactive, synchronized question answering from ego- streaming video, targeting the key properties of proactive coherence, just-in-time responsiveness, and synchronized efficiency. Our contributions- the ESTP-Bench with its ESTP-F1 metric for evaluation, and novel technical pipeline incorporating data engine, multi-stage training. and proactive dynamic compression enable our model to effectively tackle these properties. This approach outperforms multiple baselines across diverse online and offline benchmarks.
Acknowledgement: This work is supported in part by the National Natural Science Foundation of China under Grant No.62206174 and No.62576365.
10

---

References
[1] Kirolos Ataallah, Eslam Abdelrahman, Mahmoud Ahmed, Chenhui Gou, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny, Infinihench; A benchmark for large multi-modal models in long-form movies and tv shows, 2025.
[2] Siddhant Bansal, Michael Wray, and Dima Damen. Hoi-ref: Hand-object interaction referral in egocentric vision, 2024.
[3] Leonard Bärmann and Alex Waibel. Where did i leave my keys? - episodic- based question an- swering on egocentric videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages
2024. ISSN: 2160-7516.
[4] Jieneng Chen, Luoxin Ye, Ju Hc, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression, 2024.
[5] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difci Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video, 2024.
[6] Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan- Benchmarking multimodal large language models for human-level planning, 2024.
[7] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Weawen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma. Jiaqi Wang, Xiaoyi Dong. Hang Yan, Hewei Guo, Conghui He. Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang. Bo Zhang. Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, 2024.
[8] Sijie Cheng. Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. Videgothink: Assessing egocentric video understanding capabilities for embodied ai, 2024.
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset, 2018.
[10] DeL Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wo, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang. Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li. Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu. Lean Wang, Lecong Zhang. Lei Xu, Leyi Xia. Liang Zhao. Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang. Minghui Tang. Mingming Li. Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen,
Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen. Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. I. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang. Xiaosha Chen, Xiaotao Nie, Xiaowen Sun. Xiaoxiang Wang. Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li. Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying Hc, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseck-v3 v3 technical report, 2025.
[11] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding, 2019.

---

[12] Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, Hao Jiang, et al. Streaming video question-answering with in-context video kv-cache retrieval. In ICLR. 2025.
[13] Cristobal Eyzaguirre, Fric Tang, Shyamal Buch, Adrien Gaidon, Jiajun Wu, and Juan Carlos Niebles. Streaming detection of queried event start, 2024.
[14] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li. Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis, 2025.
[15] Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos, 2024.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic. Francisco Guzmán, Frank Zhang. Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson. Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, maanol Arrieta Ibarra, Isahel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jacwon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi. Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park. Joseph Rocca. Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala. Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava. Pratik Dubal. Praveen Krishnan. Punit Singh Koura. Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosalc, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan. Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami. Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albicro. Vladan Petrovic, Weiwei Chu. Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharic Delpierre Coudert, Zheng Yan, Zhengxing Chen, 7ee Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo. Anam Yunus. Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong. Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape. Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric- Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers. Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella
12

---

Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi. Zhang, Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph. Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elemt Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptise Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Vecraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen. Lakshya Garg. Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu. Liron Moshikovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus. Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa. Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz. Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun I Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang. Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen. Steve Kehoe, Steve Satterfield. Sudarshan Govindaprasad. Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Vietoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Pocnaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenehen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vetoto, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024.
[17] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger. Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie. Cristina Gonzalez. James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Kartlikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Mercy Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelacz, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bemard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralha, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022.
[18]
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh. Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng. Fu-Jen Chu, Sean Crane. Jing Dong, Maria Escobar. Cristhian Forigua, Abrham Gebreselasie, Jing Huang, Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Kevin J Liang, Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Ramakrishnan, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng. Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song. Audrey Southerland, Huiyu Wang. Yuchen Wang, Takuma Yagi. Mingfei Yan, Xitong Yang. Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel. Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem,
13

---

Judy Hoffman. C V Jawahar, Richard Newcombe, Hyun Soo Park, James M Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng, and Michael Wray. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives.
[19] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial strenngh Natural Language Processing in Python. 2020.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[21] Shuhei Kurita, Naoki Katsura, and Eri Onami. Refego: Referring expression comprehension dataset from first-person perception of ego4d, 2023.
[22] Jie Lei, Linjic Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling, 2021.
[23] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024.
[24] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interllawe Tackling multi-image, video, and 3d in large multimodal models, 2024.
[25] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representa- tions from third-person to first-person videos, 2021.
[26] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang. Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How far is your video-Ilme IIms from real-world online video understanding", 2025.
[27] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. StreamingBench: Assessing the gap for MLLMs to achieve streaming video understanding. In None, 2024.
[28] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao. Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang. Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining, 2022.
[29] Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Reconstructive token compression for extremely long video understanding, 2025.
[30] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-forri form video language understanding. 2023.
[31] Zaira Manigrasso, Mattco Dunnhofer, Antonino Furnari, Moritz Nottebaum, Antonio Finocchiaro, Davide Marana, G. Farinella, and C. Micheloni. Online episodic memory visual query localization with egocentric streaming object memory.
[32] Xavi Puig, Fric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang. Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimir Vondrus, Vincent- Pierre Berges. John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots, 2023.
[33] Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video liss with active real-time interaction via disentangled perception, decision, and reaction, 2025.
[34] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models, 2024.
[35] Qiu, Yi Chen, Yuying Ge, Yixiao Ge. Ying Shan, and Xihui Liu. Egoplan-benche benchmark for multimodal large language model planning real-world scenarios, 2025.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[37] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A Lime-sensitive multimodal large language model for long video understanding, 2024.
14

---

[38] Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi. and Giovanni Maria Farinella. Action scene graphs for long-form understanding of egocentric videos. 2023.
[39] Michael S. Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (hlip-3- video): You only need 32 tokens represent video even in vlms, 2025.
[40] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[41] Yuhan Shen, Huiyu Wang, Xitong Yang, Matt Feiszli, Ehsan Elhamifar, Lorenzo Torresani, and Effrosyni Mavroudi. Learning to segment referred objects from narrated egocentric videos. In 2024 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14510-14520. IEEE.
[42] Cheng Shi, Yizhou Yu, and Sibei Yang. Vision function layer in multimodal Ilms, 2025.
[43] Yan Shu, Zheng Liu, Peitian Zhang. Minghao Qin. Junjie Zhou, Zhengyang Liang. Tiejun Huang, and Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding, 2024.
[44] Yale Song, Gene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[45] Andrew Szot, Alex Clegg, Eric Undersander. Erik Wijmans, Yili Zhao, John Turner. Noah Maestre, Mustafa Mukadam, Devendra Chaplot. Oleksandr Maksymets. Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou Coin: A large-scale dataset for comprehensive instructional video analysis, 2019.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
[49] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: vl: Enhancing vision-language model's perception of the world at any resolution, 2024.
[50] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long vidco understanding benchmark, 2025.
[51] Yueqian Wang. Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format, 2024.
[52] Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mod: Efficient video-language streaming with mixture-of- depths vision computation, 2024.
[53] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.
[54] Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Maren Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, and Ziwei Liu. Egolife: Towards egocentric life assistant, 2025.
[55] Xitong Yang. Fu-Jen Chu, Matt Feiszli, Raghav Goyal, Lorenzo Torresani, and Du Tran. Relational space-time query in long-fome form videos. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6398 6408.660.680 IEEE.
15

---

[56] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Zhou, Cai. Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun. Miniepm-v: gpt-4v level mllm on your phone, 2024.
[57] Hanrong Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, and Yinfei Yang. Mm-ego: Towards building egocentric multimodal Ilms for video 2025.
[58] Xubing Ye, Yukang Gan, Xiaoke Huang. Yixiao Ge, and Yansong Tang. Voco-Ilama: Towards vision compression with large language models, 2025.
[59] Gokul Yenduri, Ramalingam M. Chemmalar Selvi G. Supriya Y. Gautam Srivastava, Praveen Kumar Reddy Maddikunta, Deepti Raj G, Rutvij H Jhaveri, Prabadevi B, Weizheng Wang, Athanasios V. Vasilakos, and Thippa Reddy Gadekallu. Generative pre-trained transformer: A comprehensive review enabling tochnologies, potential applications, emerging challenges, and future directions, 2023.
[60] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023.
[61] Haoji Zhang, Yiqin Wang. Yansong Tang. Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Fltrhavstream: Memory-base real sall understanding for long video streams, 2024.
[62] Jiahao Zhang. Stephen Gould, and Itzik Ben-Shabat. //gi thub. com/ souucmml/vidai 2020.
CVML video annotation tool. https:
[63] Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon, 2024.
[64] Yue Zhao, Ishan Misra, Philipp Krähenbühh, and Rohit Girdhar. Learning vidco representations from large language models, 2022.
[65] Ge Zheng. Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thone ofoth prompting for multimodal reasoning in language models, 2023.
[66] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang. Yongping Xiong, Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi task long video understanding, 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide an accurate and faithful summary of the paper's contributions, with no overstatement or omission of key elements.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
A detailed discussion of the limitations will be included in the supplementary
material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiaiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include thorough descriptions of our experimental setup, implementation, and evaluation protocols.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines thttps://nips.. cc/
public/ seliic/guides// Suuiee//Coeeeueeieee ssiossiinnn ssionPolccy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: ssionPolicy for more details.
/nips. pttpps.cc/pbll... cttppccpppcccccccccccccccccccccc wecii/ggiiii wetiddeee//oeobiiie
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental settings.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We observe substantial performance gains, supported by comprehensive analyses that validate our conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide resources details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tttss//neurrinn ott.iiittttt////rrrr......oooote
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, papersw thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We will release the dataset and benchmark we collected.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not
involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ht httt:::://nuripptt htttts///eerrrps.c///feeeeeeeeee /00pp///////ccceeeeeee///////// for what should or should not be described.
23

---

In the Appendix, we provide additional information regarding,
Ablation Study of Lide Sec. A
ESTP-Bench: Dataset and Evaluation Details in Sec. B
Model Input Example in Sec. C
Training Implementation Detail in Sec. D
ESTP-Bench Evaluation Details in Sec. E
Limitation and Future Work in Sec. F
Data Examples in Sec. G
Qualitative Results in Sec. H
A
Ablation Study of VideoLLM-EyeeO
Single Question
Contextual Question
Method
Performance 1
KV Cache Size
Performance f
KV Cache Size |
LIVE
14.9
9636.0
18.9
31199.5
+ ESTP-IT
22.0
7859.1
25.7
28236.4
Stage-0
7988.2
23.0
17567.6
with increased proactive dynamic compression mechanism
+ +taage-1 ask high frame
34.0
1182.8
38.7
3731.8
Stage-2
33.2
942.0
43.6
3242.8
Table 5: Ablation study results on ESTP bench
Tab. details the results of our ablation study on the ESTP benchmark:
1. (+ESTP-IT) enhanced the LIVE baseline's performance on both Single and Contextual Question tasks, increasing it by +7.1 and +6.8 respectively, thereby demonstrating the effectiveness of ESTP-IT.
2. (Stage-O) addressed the training conflicts stemming from simple binary supervision, enabling performance improvements without requiring any manual threshold tuning, which demonstrates the model's acquisition of a basic ability to trigger responses.
With the increased proactive dynamic compression mechanism, the model's KV cache consump- tion was significantly reduced, requiring on average only about 0.11 % of the baseline.
4. (+Stge--)) significantly boosted Single Question performance 34.0 and Contextual Question performance jumped to 38.7 by incorporating the mechanism for actively requesting high- resolution frames for scrutiny alongside initial compression.
(+Stage-2) further improved contextual coherence and refined compression, enabling the model to achieve gain of +4.9 on Contextual tasks, reaching 43.6. Simultaneously, the more accurate and efficient responses further reduced memory consumption to minimal levels.
B ESTP-Bench: Dataset and Evaluation
In this section, we first introduce the detailed description for each task within ESTP-Bench. Subse- quently, we present the annotation tool and pipeline employed for these tasks. Finally, we provide detailed deseription of the ESTP-F1 metric.
B.1 Task Explanation
Explicit Proactive Tasks require detecting and responding to queries that directly reference visual cues. This category includes eight distinet task types:
1. [OR] Object Recognition: Detect and identify specific objects.

---

Figure 8: Interface of the annotation tool. This tool enables annotators to define valid answer intervals on the video timeline, author questions, and input corresponding answers. For clarity, multiple answers and their associated intervals for a single question are visually linked by shared color.
2. [AP] Attribute Perception: Recognize object attributes.
3. [TRU] Text-Rich Understanding: Interpret and explain text content.
4. [OL] Object Localization: Identify spatial relations between objects.
5. [OSC] Object State Change: Detect object state transitions.
6. [BOL] Ego Object Localization: Localize objects relative the ego.
[BOSC) Ego Object State Change: Track object state changes from the ego's view.
[AR] Action Recognition: Detect and classify specific actions.
Implicit Proactive Tasks require performing inference and scene understanding beyond direct observation. Task types include:
1. [OFR] Object Function Reasoning: Identify functional objects and guide their use.
2. [IFR] Information Function Reasoning: Extract and provide information from source objects.
3. [NAR] Next Action Reasoning: Predict and recommend next user actions.
4. [TU] Task Understanding: Comprehend user goals and offer task guidance.
Contextual Proactive Tasks require maintaining dialogue history and visual coherence over time. Task types include:
[ORC] Object Relative Context: Track object's attributes, position changes, and function across question sequences.
2. [TRC] Task Relative Context: Monitor user task progress, action correctness, and provide sequential guidance.
See Sec. G for data examples.
B.2 Annotation Tool and Pipeline
Annotation Tool. As shown in Fig. 8, annotation tool [62] facilitates the labeling process: annotators can adjust the start and end timestamps of valid answer intervals in the upper region, while authoring questions in the bottom-left area and completing answers in the corresponding region the right. Notably, a single question can be associated with multiple answers and their corresponding intervals, which visually linked through shared color for clarity.
Annotation Construction Pipeline. As a first step, we leverage L.LMs [10] to convert raw annotations into initial question-anseee answer pairs. For Action Recognition (AR) tasks, utilize Ego4D [17]
25

---

basic Narration raw annotations, which comprise descriptions of action-related events and their corresponding timestamps. For Next Action Reasoning (NAR). Task Understanding (TU), and Task Relative Context (TRC) questions, we leverage Ego4D) GoalStep [44] annotations, which provide task goal, multiple task steps, and their corresponding temporal intervals. For these aforementioned tasks, we directly prompt LLMs for question generation. For all other task types, question generation is performed through the data engine.
Subsequently, the data undergoes a human annotation phase. For Action Recognition (AR), Next Action Reasoning (NAR), Task Understanding (TU), and Task Relative Context (TRC) tasks, given their origin from existing dataset [44], annotators primarily focus on three aspects: (1) detecting ques- tion ambiguity; (2) assessing the question-answe
pair's 's matching quality and correcting erroneous wording; and (3) calibrating the answer interval timestamps. For all other task types, annotators are additionally required to generate questions and complete answers, inspired by the initial QA pairs, before executing the aforementioned refinement steps. Simultaneously, double-check mechanism is implemented, where a second annotator independently re-executes the aforementioned three steps for verification.
B.3 Evaluation Metric (ESTP-FI)
.* semil, kx ) denote set of M ground truth items, where is the content, [t ' ! is the temporal interval, and tx [+* (Trhd] is the optimal response timestamp. Let P {pr (ōr, ) represent the set of N predictions, where each prediction Pt consists of the predicted content ot and prediction timestamp f,. Following [13], we also introduce temporal tolerances: Fam (set to second) for allowed anticipation before fstan, and Tha (set to 2 seconds) for allowed latency after txmi.
Let G {9:: = (ok: [*
prediction fo is considered a valid match for a ground truth 9k its timestamp & falls within the interval [+ Tant: gend + nar]. We define Pk {pr E P Por valid match for gu } as the set of all such valid matching predictions for gk-
For each prediction iI that is valid match for Gk. we compute a match quality score S(f,, 9k) E [0, to comprehensively evaluate its quality. This score the average of answer accuracy score and timeliness score:
S(r gk) Scoreanswer (01, ok) Scoresime (hi, gk)
10
The Scoreasser (ō), 0&) E [1, 5] evaluates the semantic correctness of the predicted content oG against the ground truth content ok, typically assessed by advanced Large Language Model [10].
The Score (f,, 9x) E [0, 5] assesses the temporal accuracy of the prediction timestamp i relative to the optimal response timestamp cop The definition of ( varies by task type:
For tasks emphasizing immediate perception upon appearance (e.g., Object Recognition (OR),
Attribute Perception (AP), Text-Rich Understanding (TRU), Object Localization (OL), Object Function Reasoning (OFR), Information Function Reasoning (IFR), and Object Relative Context (ORC)), 1 is set to taan.
For all other task types that may require more observation of event progression, (.th is the midpoint of the interval [istant 18m4), allowing sufficient time for information processing before response.
The timeliness score is then calculated as:
(1.
1 opp
Scoresime (h,, = 5 - 5 x min
1,
scales-
where scale; = max(1, (rend istart) Tani + 7ar) is a normalization factor representing the total effective time window. This factor is used to prevent division by zero for point intervals (where ustart gend) kk
26

---

For each ground truth item Jk. an aggregated score S(gk) E [0, 1] is determined by averaging the quality scores its matching predictions:
Pa E EpEp, g(f if if
S(gk)
(10)
Px
This S(g)) provides a nuanced measure of performance for each ground truth item.
Finally, the ESTP-F1 score is calculated as:
M 2 > 2k=1 S(gn)
ESTP-FI
(11)
N - I(S(g)) 0)
S(gx)
where I[()) the indicator function. This metric offers a comprehensive evaluation of model's just- in-time proactive capability by jointly considering content accuracy and temporal alignment through the (gx) scores. In contrast to classic detection tasks, which may generate redundant proposals for the same region, streaming proactive tasks necessitate a single, well-considered judgment at each timestamp. Therefore, rather than selecting the best among multiple redundant proposals as a True Positive (TP), our evaluation metric calculates the average of all valid matches. This approach comprehensively assesses each of the model's actions, reflecting the unique temporal nature of streaming environments.
B.4 Data Cleaning and Filtering
To ensure the high quality and suitability of the egocentric video data for the ESTP tasks, a compre- hensive data cleaning and filtering pipeline is applied, involving the following key steps:
Initial quality control was performed in accordance with [28] to identify and remove corrupted, incomplete, or visually ambiguous segments, ensuring the foundational integrity of the data.
2. Segments with a duration shorter than 250 seconds were filtered to ensure sufficient temporal context for learning proactive behaviors.
3.
Segments where the narration annotation coverage fell below a threshold of 0.8 were discarded to ensure robust linguistic supervision and data richness.
This multi-stage filtering process results in refined dataset optimized for the challenges of streaming proactive perception, ensuring both visual and linguistic fidelity.
B.5 Data Engine: ESTP-Gen
As shown in Fig. 9, for automatically generates diverse, multi-turn data, propose ESTP-Gen. a data engine that leverages VLMs [56, 49] and LLMs [10] to generate diverse proactive QA data from scage-scale ego video datasets like Ego4D [17]. We apply multi-stage pipeline including Caption Generation, Key Segment Extraction, and Multi-Leved QA Generation. For simplification, in the main paper, consolidate the Caption Generation and Key Segment Extraction steps into the one-to-one stage. For videos spanning several minutes, directly generating QA for all segments proves challenging for entities that are persistently visible and static, defining clear response boundaries for such entities often ambiguous. Instead, we solely generate relevant QA from extracted key segments and complete answers by leveraging all available captions. This approach simultancously ensures clear response boundaries and answer completeness. The specific details of each stage below:
Multi-Perspeece Caption Generation:
Narration: Origin narration [44, 17] provides basic descriptions of human actions and their corresponding timestamps, serving as a foundational reference for our caption generation process.
Event Caption: The original narration is typically structured as simple
phrase, merely including basic category information for the actor, action, and manipulated object. It often lacks detailed information regarding object attributes, spatial-temport location, state, or actions beyond the basic level. To generate more detailed captions specifically for events, we first follow the video clipping approach of Ego VLP [28] obtain event-relevant video clips based
27

---

Law vaten simlirity, Clip
EGO 4B
Caption
Scone Clip
Evenn Clip
human Narration
Terderal
MUM
Prompt facus Scene Event
Sindlahhe Low
Relative Captia
Specific Event
Eve. Cannion
Caption
For
caxts sact the notebook
paslage,
and other lak itees Enlicating
[raws]
Suriest real)
[knarc]: bedgyt palles bucket sits the the otherdse autdd teme of the zeer
61d0 the munter.
VUM sm
Spectic Event
Explict:: Spatiel Understanaing
Question Can you tell me where the garbage are when drop them Answer
up same things there anything that can help me?
Quediion
[02:23] The garbage dropped to the the notebook on the counter.
Key Segment
Answer:
[04:50] You dlaan some thing: into the
yullow bucket near the shalf.
[30:40] The garbage dropped on the lab bench in front of you
LL
+
Contestual GA Object Relonve
Question
Question
Manry
LLM
Many
the yellow buchet
Figure 9: ESTP-Gen Data Engine is multi-stage pipeline for automatically generating diverse, high- quality egocentric proactive QA data, which includes: (a) Muulti-Perspectvv Caption Generation, creating Narration, Event, and Scene Captions to provide comprchensive foreground and background context. (b) Key Segment Extraction, identifying key segments characterized by significant visual changes and specific events. (c) QA Generation Pipeline, proceeding through three stages: One-to- One QA for initial pairs, one-to-Many Expansion, and Many-to-Mann
on the origin narration timestamps. Then, using lexical analysis [19], we extract the actions and interacted objects. Subsequently, use an MLLM [56] provide detailed descriptions of the action and the interacted object.
The Event Captions primarily focus foreground directly involved in human interaction, potentially overlooking the broader background content within ego videos. Therefore, detailed description of the video's background information is necessary complement the Event Captions. First, we perform video scene segmentation. Then, for each video segment, we prompt MILM [56] generate detailed descriptions of the background context. Specifically, we utilize SigLIP [60] to extract features from each video frame and compute the change from the previous frame using cosine similarity. Potential video scene boundaries are identified where the similarity is significantly below the video's average similarity. This segmentation method, based on visual content changes, preserves the comprehensive scene content within ego videos, facilitating effective description by the MLLM.
Scene Caption:
Key Segment Extraction: Avoiding questions about persistently visible visual and static content is a key challenge for data generation of proactive task. We address this by employing two criteria for key segment extraction aimed at identifying moments suitable for generating proactive QA: specific egocentric actions and visual similarity change. The key insight is that moments in ego videos characterized by significant visual changes and scene transitions are necessarily accompanied by specific egocentric events of the actor (e.g.. "turn around," "walk," "step into"). conjunction with visual similarity analysis, we extract visual segments exhibiting viewpoint changes to serve as the basis for QA generation. As illustrated the portion of Fig. 9, extract key segments for QA generation. These segments are identified by significant visual changes (e.g.. "scene 1"" and specific event (e.g., "You look around")
One-to-One QA: Using diverse prompting strategies, we guide L.LMs to generate questions, visual cues, and answers from captions. As illustrated in the portion of Fig. 9, the caption of discarding
28

---

a package prompted the LLM [10] to generate question concerning the "location of dropped garbage". while the yellow bucket prompted the LLM [10] generate question regarding the "function of the item for helping clean".
One-to-Many Expansion: We refine timestamps by aligning QA pairs with captions that fully contain the visual cues needed to answer, improving temporal coverage and QA consistency. As shown in the portion of Fig. 9, utilizing the LLM's Retrieval-Augmentie Generation (RAG) capabilitiis, segments with similar visual cues identified from all remaining captions to complete all related answers.
Many-to-Man Contextual QA: Logically grouped questions from the same video are combined via LLMs to form coherent multi-turn contextual QA sets. As illustrated in the portion of the Fig. 9, the questions "location of dropped garbage" and "function of the item for helping clean" exhibit logical relevance, and are therefore merged into a coherent multi-turn contextual QA set.
C Model Input Example
In this section, utilize input examples to more clearly illustrate the concepts presented in Sec. 3 of the main paper. We first present input examples for LIVE [5] (corresponding to Sec. 3.1 of the main paper); followed by input examples for our multi-stage training strategy (corresponding to Sec. 3.2 of the main paper). Finally, we will demonstrate Compression Token Insertion and related training details, further elucidating key aspects introduced in Sec. 3.3 of the main paper.
LIVE Input. An input example for the LIVE [5] model is provided below. Visual tokens [F] denote the tokens per frame. The number of visual tokens per frame, [[F]], is set to 10, comprising 1 CLS token and x 3 average-pooled spatial tokens for enhanced visual understanding. Tokens marked in purple indicate
output of Orespense- while green tokens correspond to output of acontine- Text highlighted red is subject to language model loss [59, 11, 48], whereas other textual elements receive no supervision. Some chat template strings have been omitted for better visualization.
LIVE Example
[System Prompt]
Observation: [F] [F] User: Can you tell me where the garbage are when I drop them? Observation: [F][F]... [F][[ Assistant:
The garbage is dropped the next to the notebook on the counter. Observation: [F][F]... [F]] Assistant: The garbage is dropped on the lab bench in front of you. Observation: [F]
User: want to clean them up. Observation: [F][F).. [F][F] Assistant: You can throw them from the counter
and the lab bench into the yellow bucket near the shelf. Observation: [F] [F).
Stage-1 1 Input. An input example for the Stage-1 training is provided below. The core distinction from the LIVE [5] example is the transformation of the abrupt supervision signal into a progressive one. Visually, as the purple color deepens, the weight of the supervision signal increases, indicating that the supervision signal becomes stronger as approaches the optimal response timestamp.
Stage-1 Example
[System Prompt]
Observation: [ [F] ... User: Can you tell me where the garbage are when drop them? ?
Observation: F Assistant: The garbage is dropped to the next to the notebook
The garbage is dropped the lab bench in front of you. Observation: [F] [F] User: want to clean them up.
the counter. Observation: [F][F] [F] Assistant:
Observation: [F][F)... [F][F] Assistant: You can throw them from the counter and the lab bench into the yellow bucket near the shelf. Observation: [F).
Stage-2 Input. First, the set of uncertain timestamps Tuncertain (shown in Equ. 5) is automatically identified by perceiving timesteps where the model is likely to be confused or uncertain. Specifically, an initial training phase is conducted for one epoch using the Las high (() loss, with Tuncertain initially set equal to Timervat- illustrative example provided below. Compared to Stage 1, the Oresponse is now replaced by dask high. with the blue tokens indicating receiving this new supervision.
29

---

[System Prompt]
Observation: [F] [F] User: Can you tell me where the garbage are when drop them? Observation: ... [F][HF Assistant:
The garbage is dropped to the next to
Assistant: The garbage is dropped on the lab bench in front of you. Observation: [F] ... User: want to clean
the notebook on the counter. Observation:
them up. Observation: [F][F]... [F[F[[(P Assistant: You can throw them from the counter and the lab bench into the yellow bucket near the shelf. Observation: [F] [F)
Subsequently, by comparing the model's predicted sequence against the ground truth sequence, we identify timestamps of errоneous high predictions (femre). For each such error timestamp ferrers locate the nearest preceding timestamp Evisual_chang saagual_chane_pp__ prey characterized by a significant visual feature change, measured using visual [60] similarity. The set Tenor is then defined as the union of intervals (Evisual_change prev ferror where for each response turn, fenor corresponds to the erroneous prediction with the highest logit if multiple errors present within that turn. An example is provided below, where red tokens indicate timesteps where the ground truth was acomtnue but the model erroneously predicted aakk high-
[System Prompt]
Observation: [F] User: Can you tell me where the garbage are when drop them?
Observation: (FIF).... FIsssssstttt The garbage is dropped to the next to
Assistant:
the notebook on the counter. Observation: [F][F)... [F][F.... [FF Assistant: The
garbage is dropped on the lab bench in front of you. Observation: [F] [F] User:
want clean them up. Observation: [F][F]... [F][F].- [F][F[[FF)ASS Assistant: You can
throw them from the counter and the lab bench into the yellow bucket near the shelf.
Observation: [F] [F]
Tuncertain is then constructed by merging Tenre with the Ground Truth relevant regions Tinerval This allows the model to learn from mistakes and focus training challenging moments, encouraging to utilize scrutiny before making correct decisions in these uncertain scenarios. An illustrative example is provided below.
Stage-2 2 Example
[System Prompt]
Observation: [F] [F] User: Can you tell me where the garbage are [FF garbage is dropped to the next to the notebook on the counter. Observation: [F][F)...[F) [F] [FIF] Assistant: The garbage dropped on the lab bench in front of you. Observation: [F) User: want to clean them up. Observation:
when drop them? Observation:
The
[F][F]... [F]FF... [F[ Assistant: You can throw them from the counter and the
lab bench into the yellow bucket near the shelf. Observation: [F] [F)
Proactive Dynamic Compression Mechanism Example. An example illustrating the insertion of compression tokens is provided below. Compared to the Stage-2 example, (ct) tokens are inserted after segments of input, specifically following multiple low-resolution frames, single high-resolution frames, and text replies.
Compression Example
[System Prompt]
Observation: [F] [F] (ct.)
User: Can you tell me where the garbage are when
drop them? Observation: [F][F]... [F] [HF] ... .F.( [HF](ct) Assistant:
The garbage is dropped to the next to the notebook on the counter. {ct) Observation:
[F][F]... (Ft) (ct) ... [F]()(tt) (ct) Assistant: The garbage is dropped
on the lab bench in front of you. (ct) Observation: (F) (ct) User: want to clean
them up. Observation: [F][F]... [F][].. [HF]
(F((e)) [F][HF]ctt) Assistant:
You can throw them from the counter and the lab bench into the yellow bucket near the
shelf. (ct) Observation: [F] [F]...
30

---

Stage-1
Sluge-2
Stage-3
Data
Dataset
Ego4D:
ESTP-IT
ESTP-IT
Naration Stream +
Single-Tum
Mulri: Turn
GoalStep Stream
Multi Turn
ESTP-IT
Single-Tur
#Samples
113K 21K
60K 20K
20K
Model
Rase I.I.M
ILaMA3
LLaMA3
ILaMA3
Encoder(s)
Sig
SigLIP
SigLIP
Visson Token per Frame
1+3x3
Low Res: 1+3x3
Low Res: 1+3x3
High 1+7x7
llgg 1+7x7
Connector Trainable
MLP
MLP
MLP Connector(full) L.M(LaRA
Connectertull))
Connectortull)) LLM(LoRA
LLM(LORA
1-28, scale=256)
1-128, scale=256)
1=128, scale=256)
Initialization
Connector
N/A
Stage-1 Connector Stage-1 LoRA
Stage-2 Connector
L.M
[LaMA3
Stnge2 LoRA
Training
Batch Size per Device
1
1
Gradient Accumulation
8
8
8
Learning Rate
2e-4
le-4
Se-5
Warm Ratio
0.05
0.05
0.05
LR Scheduler
Cosine
Cosine Adammw
Cosine
Optimizer
Adamw
Adamw
Epochs
Precision
br16 fp16
bf16 fp16
hf16 fp16
Table 6: Multi-Stage Training Plan in ESTP
During training, inspired by [43], the LLM is trained to process response turns sequentially. The initial sequence provided to the LLM is as follows:
[System Prompt] Observation: (F).
Past visual tokens are compressed into a (ct). The System Prompt, containing essential system- instructions [53], remains uncompressed. The subsequent sequence processed by the LLM as follows, where gray portions indicate tokens stored in the KV Cache:
[System Prompt] (ct)
User: Can you tell me where the garbage are when I drop them? Observation: [F]
[HF](ct) (ct) (ct) Assistant: The garbage is dropped
to the next to the notebook on the counter. (ct)
Following this, solely the user's query content and the (ct) are maintained in the KV cache.
D Training Implementation Detail
In this section, we provide detailed implementation configurations of our training methodology.
ESTP-Bench. Our training methodology employs a three-stage strategy to progressively endow the VyyeeO model with advanced proactive capabilities. Tab. 6 summarizes key details of each stage's specific configuration and learning objectives, while Tab. 5 presents the corresponding ablation results.
COIN-Benchmark. Tab. 7 presents the training plan on the COIN [46] dataset. For a fair comparison with VideoLLM-Online [5], we utilized the same base model (LlaMa3 [16] and SigLIP [60]) and adopted identical training hyperparameters.
E ESTP-Bench Evaluation Details
In this section, we first present the parameter seales and hyperparameters for various models, Subse- quently, illustrate the corresponding prompts used in the evaluation.
31

---

Data
Dataset
COIN
Model
Base LLM
LLaMA3
Vision Encoder(s)
SigLIP
Vision Token per Frame
Low Res: 143x3
High Res: 1+7x7
Connector
ML.P
Trainable
Connector(full)
LLM(LoRA
1=228, scale=256)
Initialization
Connector
N/A
LLM
LLaMA3
Training
Batch Size per Device
Gradient Accumulation
8
Learning Rate
1.5e-4
Warm-up Ratio
0.05
LR Scheduler
Cosine
Optimizer
Adamw
Epochs
5
Precision
6 fp16
Table 7: Training Plan in COIN [46]
Model
Params
Frame
Query Frequency
Offline MLLMs Response-in-Lan Last
LLaVA-One Vision
7B
32
Ask per Question
Qwen2-VL VL
8B
0.2-1 fps
Ask per Question
MiniCPM-V
8B
64
Ask per Question
LLaVA-NeXT-Video
7B
32
Ask per Question
Intern VL--22
8B
32
Ask per Question
VLMs for Streaming Detection
CLIP
Base
fps
Hz Hz
LaViLa
Base
fps
EgoVLP
Base
fps
Hz
Offline MLLMs Polling Strategy
LLaVA-One CeeV-----VVVV-- VieVin
7B
32
0.175 Hz
Qwen2-VL
8B
0.2-1 fps
0.175 Hz
MiniCPM-V
8B
64
0.175 Hz
LeaVA-NeXT-Vider
7B
32
0.175 Hz
InterVV--V
8B
32
0.175 Hz
Online MILLMs
(Ire(thre)))ld=00.)
8B
fps
2 Hz
(hre
8B
fps
2 Hz
MMDuet
8B
2 fps
2 Hz
AideoLLM-EyeWOOOm
8B
2 fps
2 Hz
Table 8: Parameter Scales and Hyperparameters for Various Models
E.1 Various MLLMs Hyperparameter
For Offline MILMs, considering Synchronized Efficiency, we employed models around the 7B/8B scale. For the Response-in -n-ossseeii---tttt strategy, a single query was issued after processing the complete video segment. Regarding the Polling strategy, the querying frequency was set to 0.175 queries per second. This frequency was chosen to match the model's response efficiency (Action Per Second), as illustrated in Fig. 7. Given that our videos are often significantly longer than the typical input frame capacity of these MLLMs,, we followed the open-source code from [27], utilizing the corresponding sampling frequencies or input frame counts for different models.
32

---

For VLMs, we adopted model parameters and input sampling frequencies consistent with [13]. Given their inherent lack text generation capabilities and consequently, their relatively low computational cost, the query frequency was set to match the input sampling frequency.
For Online MILMM,, utilized model parameters, input frequencies, and query frequencies as reported in [5,
E.2
Evalutation Prompt
Below, present the prompt used for the Offlin MILMs Response-in-Last strategy. Inspired by [37], we guided the model within the system prompt to output all answers corresponding to given question, along with their respective frame indices. These frame indices then converted into corresponding timestamps via defined sampling relationship. Contextual information, encompassing both timestamps and content, is incorporated into the prompt following the approach of [27].
Below, we present the prompt used for the Offline MLLMs Polling Strategy, following the approach of [27]. The sole distinction lies in that proactive tasks in [27] solely require the model to judge if can answer a question at the current moment, outputting only 'yes' or 'no'. In contrast, our approach additionally requires the model to generate textual reply. Therefore, at timesteps where the model indicates readiness (by outputting 'yes"), we execute an additional query, prompting the model to generate the answer.
Prompt Used for Response-in-L.ast
You are advanced video assistant. Given video and a question, carefully analyze each frame of the video, identify all relevant moments that help answer the question, and provide the corresponding frame numbers along wi th the answer. The format should be: '[frame idx] answer'.
For example, [6] The object is cup.
[60] The object is a cup. [100] The object
yellow cup.
Here are the contextual information related to the video. Please answer the questions based on the contextual information:
At timestamp (}. the following question occurred: O
At timestamp (}, the following answer occurred: [}
Here is the question. Answer and don't confuse it with the previous conversation Question:
The answer is:
Prompt Used for Polling Strategy
You are an advanced video question-answering AI assistant. You have been provided with video and question related to the video. Your task is to carefully analyze the video and provide the answer to the question. You need to carefully confirm whether the video content meet the conditions of the question, and then output the correct content.
Here are the contextual information related to the video. Please answer the questions based on the contextual information:
At timestamp (]. the following question occurred: [ At timestamp (}. the following answer occurred: (}
Here is the question. Answer and don't confuse it with the previous conversation
Question: the right time to answer the question "(""? You need answer yes or no. Please answer the question: "(""
The answer is:
33

---

Limitation and Future Work
To build a truly intelligent agent akin to Jarvis, essential to move beyond passive perception and incorporate active interaction with the environment. In this work, we focus on constrained setting where the agent operates solely on pre-recorded videos. The objective is to train model that can perceive the visual world and respond promptly within fixed observation window. While this setting enables us to study perceptual grounding and response capabilities controlled manner, it inevitably limits the agent's s ability to engage in real-time decision making, action planning, or closed-loop interaction.
This limitation highlights an important future direction: extending perceptual agents into embodied settings where they can interact with the environment through physical actions. For instance, integrat- ing vision- vision-language models into real-world robots-such as home assistants, warchouse manipulators, or autonomous drones would require the ability reason about action consequences, update beliefs based on new observations, and adapt behavior on the fly. Existing platforms like Habitat [32, 40, 45] for embodied AI or Meta's HomeAssist offer promising testbeds for such developments.
Therefore, future work should explore how to bridge the gap between static visual perception and dynamic agent-environment interaction. This includes developing methods for real-time perception- action loops, task-aware decision making, and cross-modal alignment under continual feedback. Strengthening interaction with the environment is crucial step toward building general-pin purppss- autonomous agents that go beyond observation to perform useful actions in the real world.
34

---

G
Data Examples
Task Type: Object Recognition
Q Qustioo::
What is the object on the desk near the tablet?
Siience
Silence
Silence
Silence
L304.005
.00G
50s
The object on the desk near the tablet is a white cup.
Silence
Siience
Slience
Siience
Silence
Silence
1429 505
1433 D75
430 505
1478 DOS
1467 505
1600 30s
1470 255
1473 00s
A white cup.
Sience
Sience
Silence
Sieence
Silence
Silence
Silence
Task Type: A ttribution Perception
Q uestion:
What coloris the frisbee held by the person in the frame?
blue
Siience
0,50s
5505
11.005
30.505
00.00s
23.50s
5050s
53, $3,505
Siencce
Siience
Slience
Slence
Slience
Siience
Siience
Stience
50s
blue
1:9
182 CON
176 108-50x
290
214.0Ck
225 005.00x
Figure 10: Data examples for Object Recognition, Attribution Perception tasks.
35

---

Task Type: Text-Rich Understanding
Q Question: Can you remind me of the specific brand name written on the signboard above the store entrance?
Silence
Sieence
Siience
Sifence
Siience
Siience
Silence
Silence
00s
K3 755
505
000
000
105 c0t
"tamimi markets express"
Siience
Sleence
Slience
Siience
Siience
Silence
Siience
00s
203. 50s
137 505
"tamimi markets express"
"Welcome Tamimi Express
243
Task Type: Object Location
Q uestion: W here's the blue bike's position in the room? The blue bike positioned elevated on
repair stand in the room stience
Silence
Siience
Siience
Silence
Silence
Siience
The blue bike is positioned elevated repair stand in the room.
653 D05
565.565
653.755
676.00
099 505
729 S0s
230 205
207
Figure 11: Data examples for Text-Rich Understanding, Object Location tasks.
36

---

Task Type: Object State Change
Question: Can you remind me when the state of the fridge changes?
The fridge has been opened.
Siience
Siience
Siience
Siience
Slience
Siience
Siience
A
DOs
203 50c
312 5Ds
24 D3s
$35.00
135 505
388 S25
The fridge has been closed.
Siience
Silence
Silence
Silence
The fridge has been opened.
The fridge has been closed.
372
173,50s
175.00s
7.75 DDS
2:75 DDC
375 S0b
TTC D0S
777 DCS
Task Type: Ego-Object L Localization
Q uestion: How far the copper- colored pot is from me?
The copper-colored pot is on the sink's edge, within arm's reach.
Silence
D
10.504
11,756
20.504
25.0
$1,00s
10,00
2D 50%
Silence
Siience
Siience
Silence
Siience
Siience
Silence
50s
copper-colored colored is located to your right on the countertop, within arm's reach
The copper-
148.00s
IGB 505
170.50s
285 50s
995553
205.500
220 505
223 005
Figure 12: Data examples for Object State Change, Ego-Object Localization tasks.
37

---

Task Type: Ego-Object State Change
Question:
When does the tree change its position relative to me?
Silence
Slannc
Silence
Silence
Siience
Siience
Silance
Silence
moves closer to you and finally disappear on your left side.
It moves closer to you.
595 000
601 005
608 505
605 005
512 005
615.0 00.00
623. 505
682
Task Type: Action Recognition
Q Qustioon: I want to know when I pick up items from the shelf in the market. Could you let me know?
Sllence
Siience
Slience
Siience
Siience
Silence
Silence
Siience
0015
2b5
Bcl
FOS DUS
You pick up the box from the shelf.
You pick up the packet from the shelf.
Silence
Silence
Siience
Silence
You pick up the milk from the shelf.
Siience
Silence
175.2 -0s
103 250
187. 50s
280. 00s
280 D0s
300.20 505
215 005
218.25
Silence
Sieence
Siience
Silence
silence
Siience
Silence
Silence
17..005
Figure 13: Data examples for Ego-Object State Change, Action Recognition tasks.
38

---

Task Type: Object Function
Q Quetioo:
W hat could help if want to load lot of cargo to a far position?
00c
You could use the tractor to help you with the task.
Siience
Slanne
Siience
Silence
Silence
Silance
Silence
A50
150 00s
20%
D0s
505
525
315.305
You could use the tractor to help you with the task.
Silence
Silence
Siience
Siience
137 10c
17% 50s
581 00m
181 n0x
161 n0x
185 50x
100
160 5.2
Task Type: Information Function
Q Qustioo: What time is it now?
Silence
Silence
Silence
Slience
Silence
Siience
Silence
270.00s
50
50s
A ccording to the time displayed the phone, it's 16:56.
335 D0N
324 CCs
50
Figure 14: Data examples for Object Function, Information Function tasks.
39

---

Task Type: Task Understanding
Please explain how to achieve my goal of seasoning vegetables, starting from the very beginning and detailing each step.
Question:
To begin, pour lime juice over the vegetables, This will
add tangy flavor and help the seasoning adhere better, Siaece
Siience
Slience
Siience
915 305
51. 53s
DEA DDs
RO7 10%
SOR STs
29 D7%
051 50
RTM D00
Next, add your chosen seasoning to the vegetables. This could include salt, pepper, herbs, or any other spices you prefer.
Silence
Silence
Siience
Silence
Silence
Silence
Finally, mix the seasoning thoroughly into the vegetables ensure an even distribution of flavors.
1000.005
1003.502
1002, DOS
1003 D02
1000.505
2004 DDS
2004 350
1005 50G
Siinnce
Slience
Siience
Siience
Siience
Siience
Silence
Siience
JOLS
LGUS
Sits
2010
Task Type: Next A ction R Reasoning
uestiinn: What should do next to continue dicing ingredients after completing the earlier tasks: washing utensils and slicing meat? Please provide clear instructions
520 50s
C32.50
CO5.00 D0S
$33.505
$34.00s
$54.50s
738 50s
542 505
First, wash cucumber thoroughyy under running water to ensure it clean and ready for preparation.
Next slice the cucumber into even pieces. This will make easier to dice them into smaller, uniform cubes.
515.00.
Sc5.50
$64.00 00
S45.5CL
500
554.50+
SEC SC.
SELOOS
Figure 15: Data examples for Task Understanding, Next Action Reasoning tasks.
40

---

Task Type: Object Relative C ontext
Question: [270s] Could you remind me when the dog is pooing?
Sience
Siience
270 D0e
276 D0s
272 50t
202 D0s
260 nOx
504
7D5 DCs
300 50x
uestion:
[332s] want to clean it up. Give me some advice.
The dog is pooing.
Siience
Silence
Silence
You can use the green plastic bag in your right hand to collect them.
337 50s
342 50s
345 005
S45 035
353 S05
257 00.
360, 50.
You can drop it into the dark bin on the left of you..
Siience
Slience
Slience
Siience
Siience
Siience
50c
372 D05
00x
50x
408
You can drop into the dark bin on the left of you.
Silence
Silence
Siience
Silance
Silence
STD.
433 CDs
431 S0s
SDs
453.500
4144.00
410 D0s
Figure 16: Data examples for Object Relative Context task.
41

---

Task Type: Object Relative C Cntext
Question: [98.1s] Can you analyze my current approach?
Currently, you' re moving the mushrooms around the pan, which is great for ensuring even cooking.
Siience
Siience
$1.00s
COS
noticed you't re adjusting the heat this is good practice maintain control over the cooking process.
siience
silence
silence
silence
108
113 505
J.5. 005
118
122,005
L23,005
L23 505
You've just added paprika spice, which will enhance the flavor profile.
Now that you' added chili flakes, the dish is gaining nice kick of heat
Q uestion: [150s] W hat are the resources I'll need for the next task?
For the next task, you'll need spatula or spoon to continue moving the mushrooms, heat source maintain the cooking temperature, and possibly timer to keep track of the cooking duration.
350 00x
355 DON
LES SON
Q Qestion: [162.4s] W hat are the resources I'll need for the next task?
Your current goal is to cook the mushrooms thoroughly while ensuring the spices (oregano, paprika, and chili flakes) are evenly distributed.
Silence
Silence
Siience
Silence
Silence
You're re checking the recipe from the pack this is good step ensure you't 're on track.
Siience
Silence
Silence
Silence
121.305
075
075.005
375
377.00
378.305
L80 COS
Figure 17: Data examples for Task Relative Context task.
42

---

H
Qualitative Results
Q Question: Where is the fire extinguisher located relative to the microwave?
Silence
11RU
1153
1152.00s
1154.00
5ts
[1187.5s]The fire extinguisheris mounted on the wall to the left of the microwave.
silence
VideoLLM MiniCPM-V
Ours
Silecce
Sieece
Slience
Siience
Siience
Silance
Silence
1185
1129.203 1150.205
J127.C08
1157-302
21291305
bits
L130 bos
Question: Can you outline the subsequent steps for making rice sushi roll after completing the previous steps: setting nori sheet on the rolling mat, spreading rice on th nori sheet, and transferring rice from rice cooker into the bowl with a spoon? Provide detailed guidance.
[783.0s] First, season the rice in the bowl with a mixture of rice vinegar, sugar, and salt to enhance its flavor.
[787. 00s4814s] Put Add the rice on the nori sheet.
MMDuet
Cers
Sienne
Silance
703.000 00s
703 00c
7S0 DDS
795 50c
20C 00c
033. 726
$14.000 104.00s
224.20s
[815.0s] The next step is spread the rice evenly on the nori sheet. This ensures a consistent layer for rolling.
[823.0s-888 0s-80ss33 Place rice on the nori sheet.
Ours
MI DueD
S15. 50c
817 000
$15.50 50c
RO0 DOG
520 275
523
226 00
SOC
[840.0s]Once the rice is spread, you begin shaping the noni sheet into roll. Ours
MiniEPM-
DCG
Figure 18: Qualitative Comparison with Baseline.
43

---

Question: W hhat can help case of fire?
[1903.55] A fire extinguisher mounted on the wall near the kitchen entrance, slightly to the right of the central workspace. Quus
[1903.5s] The puts the the turns the
[1922.56] the fire hose.
MMDuet
MMBuet
Sience
Siience
1890 O0s
1096 75t
1901 D05
3903 50s
2004 DOs
1905.00s
1915 50g
1527 00s
[1975.5s] put the
extinguisher the kitchen.
MMDuee
100%
1877
[2045s] The object fire extinguisher.
[2050.5s] fire extinguisher is mounted the wall near the kitchen entrance
MiniCPM-V
Ours
22044.5,,005 www.ss,,25550000 2,,220555,,,55656 2056660 put fire extinguisher the kitchen.
2057.0s,2055. 2.007.5s,2058 20,,2058.s,,055 .0s) put the bag into the refrigerator.
[2063.5s) A fire extinguisher is mounted on the wall near the kitchen entrance
M MDuet
MMDuet
Qurs
Sience
Silence
Silence
Siience
Silence
Figure 19: Qualitative Comparison with Baseline.
44


Reviews:
[{"review_id": "LfQvvRArXp", "reviewer": "Reviewer_Cx8y", "summary": "This work introduces a novel task for an AI assistant that can proactively answer a wide range of evolving questions from an egocentric, streaming video input. The authors identify three key properties for this task: Proactive Coherence, Just-in-Time Responsiveness, and Synchronized Efficiency. To address these properties, the paper makes several contributions. First, it introduces the Ego Streaming Proactive Benchmark (ESTP-Bench) and a new evaluation metric, ESTP-F1, designed to assess these capabilities. Second, it proposes a comprehensive technical pipeline to enable models to perform this task. This pipeline includes a data engine for generating diverse, multi-turn questions and answers, a multi-stage training strategy to progressively build the model's capabilities, and a proactive dynamic compression technique to manage the growing volume of video data efficiently. The proposed model, VideoLLM-EyeWO, is shown to effectively address the challenges of the task and achieves SOTA performance on the COIN benchmark.", "questions": "1. The proactive dynamic compression mechanism is a key component of your proposed pipeline. Could you provide more details on how the model learns to determine when to compress and which compression level to apply? Is this a learned policy, and if so, how is it trained?\n2. The paper mentions that the model can request high-resolution frames when it is uncertain. Could you elaborate on how the model's uncertainty is quantified and used to trigger this request?", "limitations": "Please see weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Novel and Important Problem Formulation: The paper tackles a timely and significant challenge in AI: moving from passive observation to proactive assistance in real-world, dynamic environments. The formulation of the Ego-Streaming Proactive (ESTP) task, with its emphasis on proactive, just-in-time, and synchronized responses, is a valuable contribution to the field.\n2. Comprehensive Benchmark and Evaluation: The development of ESTP-Bench and the ESTP-F1 metric provides a much-needed framework for evaluating models on this complex task. The benchmark's inclusion of diverse question types (explicit, implicit, and contextual) and its focus on response timing and precision are particularly noteworthy.\n3. Strong Empirical Results: The paper presents compelling experimental results, demonstrating the effectiveness of the proposed model, VideoLLM-EyeWO. The model's significant performance improvements over existing baselines on the ESTP-Bench and its state-of-the-art results on the COIN benchmark validate the proposed approach.\n\nWeakness:\n\n1. Limited Discussion of Generalization: While the paper demonstrates the model's performance on the Ego4D dataset, it would be beneficial to see a more in-depth discussion of how the proposed approach might generalize to other datasets and real-world scenarios. More evaluation results on online and offline video understanding and reasoning benchmarks are expected.\n2. The main novelty of the framework seems to add an additional special token requesting a high-resolution frame. The ablation studies lack an analysis of this aspect.\n3. Ablation Studies: While the paper presents a comprehensive evaluation, more detailed ablation studies could provide further insights into the specific contributions of each component of the proposed pipeline. For example, it would be interesting to see the impact of each stage of the training strategy or the effect of different compression levels.\n4. Complexity of the Pipeline: The proposed technical pipeline is quite complex, with multiple components (data engine, multi-stage training, dynamic compression). This complexity could make it challenging for other researchers to reproduce and build upon the work.\n5. The basic setting of the model VideoLLM-EyeWO is unclear. What is the input format for the video frames? Are they processed frame-by-frame or several frames together? How is the information from the historical trajectory stored and managed within the KV cache? What are the specific resolutions for the high and low-resolution frames?", "was_revised": true, "final_justification": "My concerns have been solved well including the limited discussion of model generalization, the ask_high’s ablation study, basic settings of the method, etc. However, the complexity of the proposed method is still a problem, which may affect the reviewer's evaluation of the overall availability of the work.", "rebuttal": "Thank you for your valuable suggestions for strengthening the evaluation and clarifying the technical details. We have provided detailed answers to each of your points below, supported by corresponding experimental tables.\n\n> **W1: Limited discussion of model generalization.**\n> \n\nThanks for the suggestion! We've expanded our evaluation to include more benchmarks and additional baselines.\n\n1. **More benchmarks:**\n\nWe select VideoLLM-online as a baseline due to its shared base model (LLaMA3 and SigLIP) and data source (Ego4D) with our VideoLLM-EyeWO. Our model consistently outperforms the baseline in both online and offline tasks, demonstrating strong generalization across diverse scenarios.\n\n- **OVO-Bench** for online video understanding.\n    \n    \n    | Table 1 | **OvOBench Real-Time Visual Perception** | **OvOBench Backward Tracing** |\n    | --- | --- | --- |\n    | VideoLLM-online-8B | 20.79 | 17.73 |\n    | Ours (VideoLLM-EyeWO-8B) | 32.76 | 28.00 |\n- **QaEgo4D** for offline video understanding.\n    \n    \n    | Table 2 | **QaEgo4D test-online** | **QaEgo4D test-offline** |\n    | --- | --- | --- |\n    | VideoLLM-online-8B | 29.8 | 30.20 |\n    | Ours (VideoLLM-EyeWO-8B) | 36.2 | 33.00 |\n    \n    **QaEgo4D test-online** evaluates the model's query performance in an online scenario, where inquiries are made *after* the answer interval appears. In contrast, **QaEgo4D test-offline** assesses performance when queries are made *after* the entire video has been loaded.\n    \n- **COIN dataset** for offline temporal prediction.\n    \n    \n    | Table 3 | **Step** | **Task** | **Next** | **Proc** | **Proc+** |\n    | --- | --- | --- | --- | --- | --- |\n    | VideoLLM-online-8B | 63.1 | 92.6 | 49.0 | 49.7 | 53.6 |\n    | VideoLLM-MOD | 63.4 | 92.7 | 49.8 | 49.8 | 53.3 |\n    | Ours | 65.9 | 92.7 | 50.9 | 50.8 | 54.7 |\n\n2. **More baselines:** We adopt the most recent **LiveCC** [1] as our baseline. **Table 3** in our response to Reviewer jDVL consistently demonstrates the improvements achieved by our method, showcasing our model's strong generalization capability. \n\n[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale\n\n> **W2: The ask_high’s ablation study and other key novelty.**\n> \n- Indeed, **ask_high** is one of our core novelties, making it highly suitable for streaming and proactive scenarios. It processes key frames at high resolution for frame-by-frame input, providing crucial information for response decision-making and answer generation. Table 4 (lines 4 and 5) further confirms its critical role in these processes.\n- Beyond ask_high, our framework introduces several other significant core contributions:\n    1. **Interval Responsiveness:** A new training method and loss function to handle the \"silence to response\" shift without manual thresholds (in Sec 3.2 Stage-1). Without this, a trade-off between over- and under-responsiveness emerges, requiring manual tuning and leading to suboptimal performance. Unlike the abrupt supervisory signals in LIVE, our method smooths the signal over an interval, fostering a more stable and effective training process.\n    2. **Proactive Dynamic Compression :** A novel compression strategy tailored for streaming, multi-turn scenarios (in Sec 3.3). Traditional compression fails in these contexts because single-turn schemes don't suit interleaved visual/textual streams, and pre-compressing prevents the LLM from perceiving every frame, hindering decision-making.\n    3. **Data Engine:** A new multi-stage pipeline for generating high-quality, proactive training data. Unlike prior video QA, proactive task requires just-in-time and contextual information. Our framework automatically derives captions with time interval for a video , then generating QA pairs for critical segments, finally completing answers and merging questions via captioning.\n    4. **Holistic Multi-Stage Strategy :** The curriculum learning approach itself, which outperforms single-stage training.\n\n> **W3: Ablation studies.**\n> \n- Our ablation studies, detailed in the table below and Appendix B, confirm the significant impact of **ask_high** (Table lines 4-5). These studies also highlight the crucial roles of **ESTP-IT**, **multi-stage training**, and the **proactive dynamic compression mechanism**.\n- As suggested, the Table 4 also includes **KV Cache Size statistics for dynamic compression** (lines 3-4), demonstrating a significant reduction without performance compromise.\n\n**Table 4: Ablation Study on ESTP Bench.**\n\n| **Index** | **Method** | **Performance (Single Question) ↑** | **KV Cache Size (Single Question) ↓** | **Performance (Contextual Question) ↑** | **KV Cache Size (Contextual Question) ↓** |\n| --- | --- | --- | --- | --- | --- |\n| 1 | LIVE | 14.9 | 9636.0 | 18.9 | 31199.5 |\n| 2 | + ESTP-IT | 22.0 | 7859.1 | 25.7 | 28236.4 |\n| 3 | Stage-0 | 24.9 | 7988.2 | 23.0 | 17567.6 |\n|  | *with increased proactive dynamic  compression mechanism* |  |  |  |  |\n| 4 | + Stage-1 ask low frame | 27.3 | 623.1 | 25.1 | 1592.3 |\n| 5 | + Stage-1 ask high frame | **34.0** | 1182.8 | 38.7 | 3731.8 |\n| 6 | + Stage-2 | 33.2 | 942.0 | **43.6** | 3242.8 |\n\n> **W4: Complexity of the pipeline.**\n> \n\nOur pipeline comprehensively explores the ESTP task from three distinct yet interconnected perspectives: **data, training, and architectural design**. While each angle offers a unique focus, they share a common, intuitive design motivation: to efficiently and accurately answer questions at precise moments within a streaming scenario.\n\nWe intend to open-source the entire project, including ESTP-Bench, VideoLLM-EyeWO (encompassing training data and model weights), to facilitate reproducibility and further research within the academic community.\n\n> **W5: Basic settings.**\n> \n1. **W5.1 Input Format** : The model processes input **frame by frame at 2 FPS**. After each frame, the model outputs a decision: continue to the next frame or generate a response. (See Appendix D for a LIVE Input example.)\n2. **W5.2 historical trajectory storage:** Specifically, after the model decides to respond and generate an answer, Proactive Dynamic Compression is executed, storing only compressed, compact tokens in the KV cache. We also keep the System Prompt at the very beginning of the KV cache. (Appendix D, \"Proactive Dynamic Compression Mechanism Example,\" offers more detail.)\n3. **W5.3 Specific Resolution**: Following SigLIP, our input frames use a **384x384 resolution**. Low-resolution frames are pooled into 10 tokens (1+3x3), consistent with VideoLLM-online, while high-resolution frames are represented by 50 tokens (1+7x7). (Appendix E provides more detail.)\n\n> **Q1: Training of proactive dynamic compression.**\n> \n\n**Q1.1: How the model learns to determine when to compress and which compression level to apply?**\n\nGiven the nature of streaming proactive tasks—(1) where each response turn naturally segments context and (2) information from \"request-triggering\" positions is more crucial than from \"silent\" ones—we employ the following strategy:\n\n1. **Compression is performed after a response is completed.** This approach prevents interruption of ongoing text generation or the response judgment during frame input.\n2. **Equal amounts of compression tokens are appended to both \"request-triggering\" regions (e.g., high-resolution frames, answers) and \"silent\" regions.** This inherently results in a **dynamic compression ratio**: lower for request-triggering regions and higher for silent ones, as the original token count in request-triggering regions is significantly smaller.\n\nThis mechanism, combined with the model's proactive behavior in streaming tasks, imbues the model with the ability to perform proactive, dynamic compression through training on the original task.\n\n**Q1.2:  How is it trained?**\n\nFor a multi-turn conversation, we forward **turn by turn**, inputting only the **system prompt, compression tokens from historical turns, and all tokens from the current turn**. This approach enforces compression during training. We provide an example in Appendix D Proactive Dynamic Compression Mechanism Example.\n\n> **Q2: Quantify uncertain interval.**\n> \n\nWe define the **model's erroneously triggered response regions** as uncertain intervals. Specifically, we first train for one epoch using the annotated answer intervals as the initial uncertain intervals. Subsequently, we perform an inference pass over the data, identifying positions where high-resolution requests occur outside the annotated intervals and designating these as uncertain intervals. This process, along with input examples, is detailed in Appendix D Stage-2 Input."}, {"review_id": "0m1FbKrFa4", "reviewer": "Reviewer_UR4Q", "summary": "The paper introduces ESTP-Bench, a benchmark for ego-centric streaming QA that emphasises (i) proactive coherence, (ii) just-in-time responsiveness and (iii) synchronised efficiency. It further proposes VideoLLM-EyeWO, a three-stage LoRA-based training pipeline with a data-generation engine, an action space that can request high-resolution frames, and a proactive compression token. On ESTP-Bench and offline tasks (COIN), EyeWO reportedly outperforms prior VLMs and VideoLLM variants.", "questions": "Please see weakness", "limitations": "Please see weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: (i) first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric—a novel framework designed for their rigorous assessment. (ii) propose a comprehensive technical pipeline to enable models to tackle this challenging task.\n\nWeaknesses:\ni. Many components appear incremental: (a) the streaming ability is directly from LIVE.  (b) uniform compression resembles earlier token-compression work (Video-XL, VoCo-LLaMA). The data engine largely follows existing LVLM-based QA generation pipelines. Overall methodological novelty is limited.\nii. EyeWO is trained on ESTP-IT derived from Ego4D, whereas key baselines (VideoLLM-online, MMDuet) are evaluated zero-shot using released weights, as the authors explicitly state.  A fair comparison would (a) fine-tune those baselines on ESTP-IT or (b) report EyeWO in zero-shot.\niii. Experiments are only conducted on Ego4D-derived ESTP-Bench and COIN; no results are reported on other relevant ego-centric, streaming benchmarks (e.g., LIVE) or other offline datasets, making generalisation claims unsubstantiated.", "was_revised": true, "final_justification": "Thank author for the detailed response. The additional rebuttal especially experiments from the other authors appears sufficient to me; therefore, I will raise my final rating.", "rebuttal": "Thank you for your feedback and for recognizing the contributions of our new benchmark. To address your concerns, we have provided detailed responses below that further clarify the methodological novelty and present additional experimental results to support our claims regarding strong performance and generalizability. \n\n> **W1: Novelty**\n\nBoth our **complete framework**—from data engineering to training and architecture (i.e., compression)—and **its components are novel, rather than mere incremental extensions** of prior work. The components mentioned by the reviewer cannot be readily adapted to ESTP, and straightforward combinations fail to address its challenges.\n\nWe outline the core innovations of our framework, demonstrating how each part **meets the unique demands of ESTP** and why the mentioned components **fundamentally fall short**.\n\n| **Core Contribution**                                        | **Prior Approaches**                                         | **Inherent Challenge of ESTP**                               | **Limitation of Prior Approaches**                           | **Our Principle**                                            | **Our Proposed Solution and Result**                         |\n| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| **1. Multi-Stage Training (Stage 1): Interval Responsiveness** | **LIVE:** Employs frame-level supervision and timestamp-based annotations. | while adjacent frames are nearly identical, the ground-truth label mandates a **sharp, binary shift from silence response.** | The training conflict results in a trade-off between over-responsiveness and under-responsiveness, **requiring manual tuning of response thresholds for suboptimal performance**. | To train the model to perceive subtle inter-frame changes by gradually transitioning the supervisory signal within an event-relevant interval, enabling a response only when sufficient information is aggregated. | **(1) Region-Based Modulation Loss:** Smooths the supervisory signal over an interval. **(2) Interval-annotated Data.** Achieves optimal performance without manual threshold tuning. *(See Ablation Study, Lines 2 & 3)* |\n| **2. Multi-Stage Training (Stage 2):** Ask High frame for just-in-time responsiveness and accurate answering. | **(1) VideoLLM-online:** Uses low-resolution frames exclusively.  **(2) Offline MLLMs (e.g., MM-Ego, VideoEspresso):** Select keyframes for accurate answering, but not in real-time. | (1) Low-resolution frames lack the spatial detail needed for decisive response timing and accurate answer generation.  (2) Inserting a high-resolution frame only *after* a response decision has been made does not aid the decision-making process itself. | Models struggle to achieve timely responsiveness and accurate answer generation. | The model learns to proactively request high-resolution details at key moments of interest. This enhanced information serves a dual purpose: first, to make a definitive judgment on whether a response is necessary, and second, to furnish the rich context required for an accurate and comprehensive answer. | The introduction of an `**ask_high**` **token**, trained in Stage 1, which enables the model to actively request high-resolution frames to improve both response timing and answer accuracy. *(See Ablation Study, Lines 4 & 5)* |\n| **3. Proactive and Dynamic Compression**                     | **(1) LLM-based compression (e.g., Video-XL):** Compresses single-turn visual inputs.  **(2) Input-stream compression (e.g., Flash-Vstream):** Merges frames before they reach the LLM. | (1) Single-turn compression schemes are ill-suited for multi-turn nature of streaming scenarios, which involve continuously **interleaved streams of visual and textual data.** (2) Pre-compressing the input stream prevents the LLM from perceiving every frame, hindering its decision-making. | (1) Compressing with a fixed-size window can interrupt ongoing reasoning or answer generation.  (2) Frame loss impairs the model's ability to learn when to respond. | (1) Each response turn forms a natural contextual boundary for compression.  (2) Information at moments that trigger a model request (e.g., for a high-res frame, answer) is more critical than at moments of silence. | **(1) Post-Response Compression:** Compression is only initiated after a response-generation cycle is complete, preventing interruption.  **(2) Two Level Compression Ratio:** A lower compression ratio is applied to more critical data segments (e.g., high-resolution frames, generated answers). |\n| **4. Data Engine**                                           | Existing data engine. (e.g **LLaVA-Next-Video** and **VideoEspresso**) | (1) Requires identifying all relevant moments for a potential question.  (2) Questions should target the very first appearance of an object or the start of an event.  (3) Data must support multi-turn, proactive dialogue. | (1) Incomplete recall of relevant moments can introduce training conflicts.  (2) Generated questions often pertain to persistent objects rather than novel events, failing to train true proactivity.  (3) Lack of multi-turn conversational flow. | (1)  identify key temporal moments by leveraging the event narrations from the Ego4D in conjunction with detected shifts in visual feature similarity. This allows us to precisely localize the scene changes, event occurrences, and initial object appearances. (2) Use an LLM to iteratively expand and consolidate these moments into coherent QA pairs. | Multi-Stage Data Engine and ESTP-IT. *(See Ablation Study, Lines 1 & 2)* |\n| **5. Holistic Multi-Stage Training Strategy**                | **Single-Stage Training**                                    | Different tasks exhibit a wide spectrum of complexity.       | Co-training disparate tasks leads to suboptimal performance, as the learning process for simpler skills can be compromised by more complex ones. | The training follows an \"easy-to-hard\" structure: it begins by establishing foundational responsiveness, then proceeds to requesting high-resolution information for precise answers, and concludes with fine-tuning for multi-turn dialogue. | The ablation studies in Tab Ablation demonstrate the effectiveness of each stage. |\n\n> **W2: Effect of ESTP-IT**\n\nIn Appendix B, we conducted ablation experiments on ESTP-IT. As shown in Appendix Table 1, lines 1 and 2, ESTP-IT effectively improves the performance of VideoLLM-online. However, constrained by the training methodology, it was unable to achieve optimal performance. This further validates the effectiveness of our multi-stage training strategy.\n\nIn the table below, we present the requested experimental results.\n\n| Table 1   | **Performance (Single Question) ↑** | **KV Cache Size (Single Question) ↓** | **Performance (Contextual Question) ↑** | **KV Cache Size (Contextual Question) ↓** |\n| --------- | ----------------------------------- | ------------------------------------- | --------------------------------------- | ----------------------------------------- |\n| LIVE      | 14.9                                | 9636.0                                | 18.9                                    | 31199.5                                   |\n| + ESTP-IT | 22.0                                | 7859.1                                | 25.7                                    | 28236.4                                   |\n| Ours      | **33.2**                            | **942.0**                             | **43.6**                                | **3242.8**                                |\n\n> **W3: **Limited discussion of model generalization****\n\nThanks for the suggestion! We've expanded our evaluation to include more benchmarks and additional baselines.\n\n1. **More benchmarks:**\n\nWe select VideoLLM-online as a baseline due to its shared base model (LLaMA3 and SigLIP) and data source (Ego4D) with our VideoLLM-EyeWO. Our model consistently outperforms the baseline in both online and offline tasks, demonstrating strong generalization across diverse scenarios.\n\n- **OVO-Bench** for online video understanding.\n\n  | Table 2                  | **OvOBench Real-Time Visual Perception** | **OvOBench Backward Tracing** |\n  | ------------------------ | ---------------------------------------- | ----------------------------- |\n  | VideoLLM-online-8B       | 20.79                                    | 17.73                         |\n  | Ours (VideoLLM-EyeWO-8B) | 32.76                                    | 28.00                         |\n\n- **QaEgo4D** for offline video understanding.\n\n  | Table 3                  | **QaEgo4D test-online** | **QaEgo4D test-offline** |\n  | ------------------------ | ----------------------- | ------------------------ |\n  | VideoLLM-online-8B       | 29.8                    | 30.20                    |\n  | Ours (VideoLLM-EyeWO-8B) | 36.2                    | 33.00                    |\n\n  **QaEgo4D test-online** evaluates the model's query performance in an online scenario, where inquiries are made *after* the answer interval appears. In contrast, **QaEgo4D test-offline** assesses performance when queries are made *after* the entire video has been loaded.\n\n2. **More baselines:** We adopt the most recent **LiveCC** [1] as our baseline. **Table 3** in our response to Reviewer jDVL consistently demonstrates the improvements achieved by our method, showcasing our model's strong generalization capability.\n\n[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale"}, {"review_id": "gi3zRPSNTJ", "reviewer": "Reviewer_Lpzn", "summary": "This paper presents a compelling advancement in proactive video-language models, with a well-designed benchmark and innovative techniques. However, broader applicability, reproducibility, and statistical rigor could be strengthened. The work sets a strong foundation for future research in real-time, interactive AI systems.", "questions": "1. In Figure 7, why the model trained only on multi-turn QA (Stage 3) achieves higher APS compared to Stage 2?\n\n2. In L212-218, the description of the compression is confusing. I could not understand (1) what are the two levels; (2) how many tokens are compressed; (3) what low/high compression rates did you choose.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a well-defined, challenging task (ESTP) that bridges the gap between passive observation and proactive AI assistance in streaming video scenarios.\n2. The proposed benchmark rigorously evaluates the three key properties (proactivity, timeliness, synchronization) with diverse question types and contextual dependencies.\n\n\nWeaknesses:\n1. The \"scrutinize loss\" in Eq. (7) is not clearly referenced.\n\n2. The authors should also report the performance of their proposed model on other online/streaming benchmarks, such as StreamingBench [1] and OVO-Bench [2], to prove its generalization ability.\n\n3. In L79-L81, the authors claimed data engine as a key component in solving the ESTP task, however, I could not find more details about how this engine works except from L82-87.\n\n4. The formatting of the references is completely incorrect; the authors should carefully review them before submission.\n\n[1] Lin et al. StreamingBench: Assessing the gap for MLLMs to achieve streaming video understanding.\n\n[2] Li et al. OVO-bench: How far is your video-LLMs from real-world online video understanding?", "was_revised": true, "final_justification": "Most of my concerns have been addressed by additional experiments. I consider slightly raise my score.", "rebuttal": "Thank you for your valuable feedback and for acknowledging our contributions in defining the ESTP task. Following your suggestions, we have provided detailed clarifications on the \"scrutinize loss\" and data engine below, along with new experimental tables for OVO-Bench.\n\n> **W1: Typo of \"scrutinize loss\".**\n\nWe apologize for the oversight; the term \"**scrutinize loss**\" in Equation (7) indeed refers to the \"**ask_high loss**\" introduced in Equation (5). We will rectify this inconsistency in subsequent revisions.\n\n> **W2: Performance on other online benchmarks.**\n\nThank you for your valuable suggestion. We have conducted experiments on OvO-Bench to demonstrate our generalization capability. Concurrently, we selected VideoLLM-online as our baseline, given that it shares the same base model (LLaMA3 and SigLIP) and data source (Ego4D) as our own model.\n\n| Table 1            | Frames | OCR       | ACR       | ATR       | STU       | FPD       | OJR       | Avg.      | EPM       | ASI       | HLD       | Avg.      |\n| ------------------ | ------ | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| VideoLLM-online-8B | 2fps   | 8.05      | 23.85     | 12.07     | 14.04     | **45.54** | 21.20     | 20.79     | 22.22     | 18.80     | **12.18** | 17.73     |\n| Ours (VideoLLM-EyeWO-8B)  | 2fps   | **24.16** | **27.52** | **31.89** | **32.58** | 44.55     | **35.87** | **32.76** | **39.06** | **38.51** | 6.45      | **28.00** |\n\nCompared to the baseline, our model demonstrates significant improvements. We also incorporate the latest **LiveCC [1]** as a baseline. It's one of the most outstanding online video LLMs built on Qwen2-VL, and experimental results effectively showcase the efficacy of our method.\n\n| Table 2                                     | # Frames | OCR           | ACR        | ATR            | STU       | FPD       | OJR       | Avg.      | EPM          | ASI         | HLD       | Avg.      | REC         | SSR        | CRR            | Avg.      | Overall Avg. |\n| ------------------------------------------- | -------- | ------------- | ---------- | -------------- | --------- | --------- | --------- | --------- | ------------ | ----------- | --------- | --------- | ----------- | ---------- | -------------- | --------- | ------------ |\n|                                             |          | **Real-Time** | **Visual** | **Perception** |           |           |           |           | **Backward** | **Tracing** |           |           | **Forward** | **Active** | **Responding** |           |              |\n| ***Open-source Multimodal Models-Offline*** |          |               |            |                |           |           |           |           |              |             |           |           |             |            |                |           |              |\n| Qwen2-VL-72B                                | 64       | 65.77         | **60.55**  | 69.83          | 51.69     | 69.31     | 54.35     | 61.92     | 52.53        | **60.81**   | **57.53** | **56.95** | **38.83**   | 64.07      | 45.00          | 49.30     | **56.27**    |\n| LLaVA-Video-7B                              | 64       | **69.13**     | 58.72      | 68.83          | 49.44     | **74.26** | 59.78     | 63.52     | **56.23**    | 57.43       | 7.53      | 40.4      | 34.10       | **69.95**  | 60.42          | **54.82** | 52.91        |\n| LLaVA-OneVision-7B                          | 64       | 66.44         | 57.80      | **73.28**      | **53.37** | 71.29     | **61.96** | **64.02** | 54.21        | 55.41       | 21.51     | 43.71     | 25.64       | 67.09      | 58.75          | 50.50     | 52.74        |\n| Qwen2-VL-7B                                 | 64       | 60.40         | 50.46      | 56.03          | 47.19     | 66.34     | 55.43     | 55.98     | 47.81        | 35.48       | 56.08     | 46.46     | 31.66       | 65.82      | 48.75          | 48.74     | 50.39        |\n| InternVL-V2-8B                              | 64       | 67.11         | **60.55**  | 63.79          | 46.07     | 68.32     | 56.52     | 60.39     | 48.15        | 57.43       | 24.73     | 43.44     | 26.5        | 59.14      | 54.14          | 46.60     | 50.15        |\n| LongVU-7B                                   | 1fps     | 53.69         | 53.21      | 62.93          | 47.75     | 68.32     | 59.78     | 57.61     | 40.74        | 59.46       | 4.84      | 35.01     | 12.18       | 69.48      | **60.83**      | 47.50     | 46.71        |\n| ***Open-source Multimodal Models-Online***  |          |               |            |                |           |           |           |           |              |             |           |           |             |            |                |           |              |\n| Flash-VStream-7B                            | 1fps     | 24.16         | 29.36      | 28.45          | 33.71     | 25.74     | 28.80     | 28.37     | 39.06        | 37.16       | 5.91      | 27.38     | 8.02        | 67.25      | 60.00          | 45.09     | 33.61        |\n| VideoLLM-online-8B                          | 2fps     | 8.05          | 23.85      | 12.07          | 14.04     | 45.54     | 21.20     | 20.79     | 22.22        | 18.80       | 12.18     | 17.73     | -           | -          | -              | -         | -            |\n| Dispider                                    | 1fps     | 57.72         | **49.54**  | 62.07          | 44.94     | 61.39     | 51.63     | 54.55     | 48.48        | 55.41       | 4.3       | 36.06     | 18.05       | 37.36      | 48.75          | 34.72     | 41.78        |\n| LiveCC*                                     | 2fps     | 71.14         | 43.11      | 64.65          | 42.69     | 67.32     | 59.24     | 58.00     | 58.24        | 52.70       | **92.47** | 67.80     | 34.52       | 65.50      | 54.16          | 51.40     | 58.81        |\n| Ours (LiveCC-EyeWO)                                | 2fps     | **78.52**     | 44.95      | **71.55**      | **45.50** | **69.30** | **63.04** | **62.14** | **59.60**    | **60.13**   | 89.78     | **69.83** | 32.80       | 64.22      | **64.58**      | **53.87** | **62.00**    |\n\nLiveCC*: We re-evaluate LiveCC's performance on our own devices.\n\n[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale\n\n> **W3: Data engine detail.**\n\nDue to main text page limitations, relevant details are provided in Appendix C.4 and C.5, with a framework diagram presented in Appendix Fig. 2.\n\n> **W4: Reference format error.**\n\nThank you for pointing out this oversight. We sincerely apologize for not thoroughly checking this issue, which stems from our reference export process. We will correct all citation formats throughout the paper.\n\n> **Q1: Stage 3 and stage 2 APS comparison.**\n\nThe KV Cache Size decreased after training with multi-turn QA (Stage 3) (from 3731.8 to 3242.8 as shown in Appendix Tab. 1). This reduction indicates that the model decreased the frequency of triggering responses. This, in turn, suggests that after multi-turn, more refined data training, the phenomenon of over-responding was reduced, leading to an improvement in APS.\n\n> **Q2: Detail of proactive dynamic compression.**\n\n**Q2.1 What are the two levels?**\n\nGiven the inherent characteristics of streaming proactive tasks—specifically, that information at locations where the model initiates a request is more critical than information at silent locations—we assign a lower compression ratio to \"request-triggering\" locations (e.g., high-resolution frames, answers).\n\n**Q2.2 How many tokens are compressed?**\n\nAs shown in Appendix Table 1, an average of 90% of tokens are compressed.\n\n**Q2.3 What low/high compression rates did you choose?**\n\nWe do not choose specific, fixed numerical values for the low and high compression rates. Instead, the compression ratio is dynamic and relative, emerging naturally from our compression mechanism, which is designed for the unpredictability of streaming scenarios.\n\nHere is the principle:\n\n- Our method applies a **fixed number** of compression tokens to all regions, regardless of whether they are \"request-triggering\" or \"silent.\"\n- **A relatively low compression rate** is inherently applied to \"request-triggering\" regions (such as high-resolution frames or answers). Because these critical regions contain significantly fewer original tokens, adding the fixed amount of compression tokens has a smaller relative impact, thus preserving their information with higher fidelity.\n- **A relatively high compression rate** is applied to \"silent\" regions. These regions are often long, containing many consecutive low-resolution frames. The same fixed number of compression tokens represents a more aggressive compression for these larger segments.\n\nThis approach is crucial for streaming environments where the length of visual or textual inputs cannot be determined beforehand, as it allows the compression to adapt dynamically without relying on static, predefined ratios."}, {"review_id": "8I7WWHTeKr", "reviewer": "Reviewer_jDVL", "summary": "This paper defines the task of proactive question answering for AI assistants, using ego-streaming video as the input. The key properties emphasized are proactive coherence, just-in-time responsiveness, and synchronized efficiency. The authors construct a benchmark for this task, complete with an evaluation metric. They also propose a technical pipeline that addresses the challenge and demonstrates state-of-the-art (SOTA) results on both the proposed benchmark and a traditional video QA benchmark.", "questions": "1. The process for generating QA pairs and the corresponding response time annotations is not clearly explained in the main paper. Could you provide more details on how these annotations are generated?\n2. In the ESTP-F1 metric, how are the components $S_{time}$ and $S_{answer}$ aggregated to obtain $S(g_k)$? A clearer explanation of this fusion process would be helpful.\n3. Is there a more generalizable model solution that could address the various challenges mentioned in the task, beyond the current specific approach?", "limitations": "Not discussed in the main paper. Maybe limitations about the scalability and efficiency of training on more large-scale data can be discussed. Additionally, the scalability of the data generation process itself could be a concern.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper introduces a novel proactive egocentric streaming video QA task aimed at helping developing AI assistants. The task design addresses key real-world application characteristics such as Proactive Coherence, Just-in-Time Responsiveness, and Synchronized Efficiency, making it highly valuable for research.\n2. The authors propose a comprehensive benchmark and evaluation metric for the task, alongside a tailored solution to address the specific challenges of proactive video question answering. The results achieved are promising, and the work is thorough.\n\nWeaknesses\n1. The description of the proposed three-stage generation pipeline is insufficiently detailed. For instance, the use of RAG techniques for the one-to-many expansion is not clearly explained. Additionally, the integration of multi-turn QA and how to define the response time is unclear. Any manual annotations are needed? If needed, how to combine the automatically generated data and annotations is also not fully clarified. This lack of detail diminishes the perceived innovation in the dataset construction process and its potential for scalability.\n2. The ESTP-F1 metric and the VideoLLM-EyeWO solution presented in the paper appear to have limited novelty. VideoLLM-EyeWO follows the framework of VideoLLM-online, with additional loss functions introduced at different stages to impose constraints on the model. However, these constraints are not exhaustive, which raises concerns about the scalability and generalizability of the approach.", "was_revised": true, "final_justification": "After the discussion with the authors during the rebuttal, my major concerns have been solved. Regarding the relatively limited novelty in model and metric design, I keep my score.", "rebuttal": "Thank you for your valuable feedback and for recognizing our contributions in establishing the ESTP task. To address your concerns, we provide detailed responses below that clarify the workings of our data generation pipeline and articulate the specific detail of my ESTP-F1 metric and VideoLLM-EyeWO model, distinguishing them from prior art.\n\n> **W1 and Q1: Data engine detail**\n> \n\n**W1.1 One-to-Many**\n\nIn the **One-to-Many stage**, the core objective is to identify all segments within the captions that can answer the current question-answer (QA) pair. We explored two technical approaches:\n\n1. **Inputting all captions into the Large Language Model (LLM)**: This method involves feeding the entirety of the captions into the LLM, tasking it with identifying all relevant segments and subsequently generating an answer. While this approach yields optimal results, it incurs substantial token consumption.\n2. **Keyword-based Retrieval and Supplementary Generation**: During the one-to-one QA generation phase, we instruct the model to generate keywords pertaining to visual cues. These keywords are then used for retrieval, selecting relevant captions from the Multi-Perspective Caption. These selected captions are then input into the LLM, which is tasked with discerning their ability to answer the question and generating supplementary answers as needed.\n\n**W1.2 Integration of Multi-turn QA**\n\nMulti-turn QA is constructed by combining multiple single-turn QA pairs that pertain to the same video segment. Specifically, we leverage an LLM to identify logically related QA pairs and subsequently sequence them. Following this established order, the subsequent question is appended after a randomly selected number of preceding answers (one or more). This process is entirely rule-based and does not necessitate any manual annotation. Finally, an LLM is utilized to refine the generated text, ensuring it conforms to a fluent, streaming conversational style.\n\n**W1.3 Definition the Response Time**\n\nDuring training, response times are randomly sampled from the latter 50% of the answer interval. The generation of the answer interval itself is derived from the time intervals of relevant captions. The methodology for obtaining these caption intervals is detailed in **Appendix C.5: Multi-Perspective Caption Generation**. Briefly, for Event Captions, we follow the approach of EgoVLP for calculating interval start and end points. For Scene Captions, we leverage inter-frame similarity to determine potential interval boundaries.\n\nFor evaluation, the optimal Response Time definition can be found in Appendix C.3.\n\n**W1.4 Manual Annotation**\n\nFor ESTP-Bench, each data is manually annotated, with specific annotation details presented in Appendix C.2. For Data Engine and ESTP-IT, the data is entirely automatically generated without any manual annotation; more detailed generation processes can be found in Appendix C.4 and C.5.\n\nThank you for your suggestion. We will subsequently add more details regarding data generation in the Appendix, such as how to prompt the LLM for RAG and the concatenation of multi-turn QA. Concurrently, we plan to open-source the data generation process.\n\n> **W2: ESTP-F1 metric and the VideoLLM-EyeWO**\n> \n\n**W2.1 ESTP-F1**\n\nRegarding the design of ESTP-F1, our aim was for it to be both simple yet comprehensive. We are the first to introduce F1 as a unified metric in streaming proactive tasks to simultaneously measure both response quality and timeliness. We note that reviewer Cx8y recognized the novelty of ESTP-F1, a point which was also appreciated by reviewer UR4Q.\n\n**W2.1 VideoLLM-EyeWO**\n\nVideoLLM-online, with its simple training and architecture, is our streaming proactive video LLM baseline. However, it falls short on **just-in-time responsiveness, accurate answers, and contextual understanding**—all essential for ESTP tasks. Our multi-stage training strategy, proactive dynamic compression, and data engine introduce these critical capabilities. These novel components are unique to our approach and are not found in prior proactive video LLM methods.\n\nWe also agree that **scalability and generalizability** are critical.  To validate these aspects, we conduct additional experiments:\n\n**(1) Extended Benchmarking:** We test our model on other **online and offline video benchmarks**. This demonstrates its performance across a wider range of video understanding tasks.\n\n| Table 1 | **OvO-Bench Real-Time Visual Perception** | **OvO-Bench Backward Tracing** |\n| --- | --- | --- |\n| VideoLLM-online-8B | 20.79 | 17.73 |\n| Ours(VideoLLM-EyeWO-8B) | 32.76 | 28.00 |\n\n**Table 2: Performance comparison on the QaEgo4D test set.**\n\n| Table 2 | **QaEgo4D test-online** | **QaEgo4D test-offlline** |\n| --- | --- | --- |\n| VideoLLM-online-8B | 29.8 | 30.20 |\n| Ours(VideoLLM-EyeWO-8B) | 36.2 | 33.00 |\n\n**QaEgo4D test-online** evaluates the model's query performance in an online scenario, where inquiries are made *after* the answer interval appears. In contrast, **QaEgo4D test-offline** assesses performance when queries are made *after* the entire video has been loaded.\n\n**(2) Comparison with State-of-the-Art:**  We also incorporate the latest **LiveCC [1]** as a baseline. It's one of the most outstanding online video LLMs built on Qwen2-VL, and experimental results effectively showcase the efficacy of our method.\n\n| **Table 3** | **# Frames** | **OCR** | **ACR** | **ATR** | **STU** | **FPD** | **OJR** | **Avg.** | **EPM** | **ASI** | **HLD** | **Avg.** | **REC** | **SSR** | **CRR** | **Avg.** | **Overall Avg.** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  |  | **Real-Time** | **Visual** | **Perception** |  |  |  |  | **Backward** | **Tracing** |  |  | **Forward** | **Active** | **Responding** |  |  |\n| ***Open-source Multimodal Models-Offline*** |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n| Qwen2-VL-72B | 64 | 65.77 | **60.55** | 69.83 | 51.69 | 69.31 | 54.35 | 61.92 | 52.53 | **60.81** | **57.53** | **56.95** | **38.83** | 64.07 | 45.00 | 49.30 | **56.27** |\n| LLaVA-Video-7B | 64 | **69.13** | 58.72 | 68.83 | 49.44 | **74.26** | 59.78 | 63.52 | **56.23** | 57.43 | 7.53 | 40.4 | 34.10 | **69.95** | 60.42 | **54.82** | 52.91 |\n| LLaVA-OneVision-7B | 64 | 66.44 | 57.80 | **73.28** | **53.37** | 71.29 | **61.96** | **64.02** | 54.21 | 55.41 | 21.51 | 43.71 | 25.64 | 67.09 | 58.75 | 50.50 | 52.74 |\n| Qwen2-VL-7B | 64 | 60.40 | 50.46 | 56.03 | 47.19 | 66.34 | 55.43 | 55.98 | 47.81 | 35.48 | 56.08 | 46.46 | 31.66 | 65.82 | 48.75 | 48.74 | 50.39 |\n| InternVL-V2-8B | 64 | 67.11 | **60.55** | 63.79 | 46.07 | 68.32 | 56.52 | 60.39 | 48.15 | 57.43 | 24.73 | 43.44 | 26.5 | 59.14 | 54.14 | 46.60 | 50.15 |\n| LongVU-7B | 1fps | 53.69 | 53.21 | 62.93 | 47.75 | 68.32 | 59.78 | 57.61 | 40.74 | 59.46 | 4.84 | 35.01 | 12.18 | 69.48 | **60.83** | 47.50 | 46.71 |\n| ***Open-source Multimodal Models-Online*** |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n| Flash-VStream-7B | 1fps | 24.16 | 29.36 | 28.45 | 33.71 | 25.74 | 28.80 | 28.37 | 39.06 | 37.16 | 5.91 | 27.38 | 8.02 | 67.25 | 60.00 | 45.09 | 33.61 |\n| VideoLLM-online-8B | 2fps | 8.05 | 23.85 | 12.07 | 14.04 | 45.54 | 21.20 | 20.79 | 22.22 | 18.80 | 12.18 | 17.73 | - | - | - | - | - |\n| Dispider | 1fps | 57.72 | **49.54** | 62.07 | 44.94 | 61.39 | 51.63 | 54.55 | 48.48 | 55.41 | 4.3 | 36.06 | 18.05 | 37.36 | 48.75 | 34.72 | 41.78 |\n| LiveCC* | 2fps | 71.14 | 43.11 | 64.65 | 42.69 | 67.32 | 59.24 | 58.00 | 58.24 | 52.70 | **92.47** | 67.80 | 34.52 | 65.50 | 54.16 | 51.40 | 58.81 |\n| Ours (LiveCC-EyeWO) | 2fps | **78.52** | 44.95 | **71.55** | **45.50** | **69.30** | **63.04** | **62.14** | **59.60** | **60.13** | 89.78 | **69.83** | 32.80 | 64.22 | **64.58** | **53.87** | **62.00** |\n\nLiveCC*: We re-evaluate LiveCC's performance on our own devices.\n\n[1] LiveCC : Learning Video LLM with Streaming Speech Transcription at Scale\n\n> **Q2: ESTP-F1 Detail**\n> \n\nSpecifically, we utilize an LLM to determine the score between the generated text and the ground truth, while also employing the temporal distance of timestamps to quantify response timeliness. Further details are comprehensively explained in Appendix C.3.\n\n> **Q3: VideoLLM-EyeWO shows strong potential to extend beyond its current tasks**\n> \n1. VideoLLM-EyeWO is not a model tailored solely for the tasks proposed in our paper; instead, VideoLLM-EyeWO follows a general paradigm that supports adaptation to a wide range of tasks (see our response to W2.1).\n2. The proactive task proposed in our paper is not only tailored for specific AI assistant scenarios, but also serves as a unified paradigm for conventional tasks. For example, it reduces to answering a single question given an image in typical image-based QA, and to performing short-range reasoning over a video clip in standard video QA. Therefore, a model like VideoLLM-EyeWO—designed to handle more generalized task formulations—holds strong potential for broader applications beyond those discussed in our paper.\n\n> **L1: VideoLLM-EyeWO shows strong Scaling Laws**\n> \n1. **The model scales well with size, adapting effectively to various tasks.** For the newly proposed proactive task, we successfully trained a model of size 8B, demonstrating its effectiveness in handling the task and achieving promising results.\n2. **The data scales well with size, with more data leading to significant performance improvements.** This is evident from our experiments, where increasing the dataset size consistently improved the model’s accuracy and generalization ability, particularly in complex tasks.\n3. Since the data generation is fully automated, it allows for efficient scaling, ensuring a continuous supply of diverse and high-quality data without manual intervention."}, {"review_id": "tBxn8nuZ6H", "reviewer": "Reviewer_VD5g", "summary": "The paper proposes a new task where, given ego-streaming video input, an assistant proactively answers questions while maintaining synchronized perception and reasoning. To enable this capability, the paper proposes an MLLM with calibrated benchmarks (ESTP-Bench) and metric (ESTP-F1). The model is trained in three stages and includes a dynamic compression technique to enable multi-turn QA while being efficient. The final model outperforms prior Online MLLMs as well as modified Offline MLLMs.", "questions": "My biggest concerns lie in the clarity. Please refer to the weakness section for more details.\n\nOne more question: any open-source plan for code and dataset?", "limitations": "no. \n\nI would suggest the authors think of this carefully and add it in the revision. Possible limitations would include: how fast does the model run? Is it ready for real-time purpose? If so, on what kind of hardware device?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Originality in terms of (1) task, (2) benchmark, and (3) loss design.\n\n(1) The paper proposes a new task for MLLMs to proactively answer diverse given egocentric streaming videos.\n\n(2) A new benchmark (ESTP-Bench) with a new metric that emphasizes (a) Proactive Coherence, (b) Just-in-Time Responsiveness, and (c) Synchronized Efficiency.\n\n(3) A modified loss that addresses the training conflict due to the high similarity of adjacent frames in streaming inputs when using vanilla binary cross-entropy loss. \n\n2. Significance of results.\n\n(1) Clear improvement over prior Online MLLMs as well as modified Offline MLLMs.\n\n(2) Generalizable on Offline Tasks (i.e., COIN benchmark). \n\nWeaknesses\n\n1. Notations are not always clear; sometimes inconsistent.\n\n(1) L168, $\\mathcal{T}\\_\\text{timestamp} $ = $ \\mathcal{T}\\_\\text{GT} $? $ \\mathcal{T}\\_\\text{GT} $ has never been used ever since.\n\n(2)  Given the description, Eq (4) is simply a modulated version of Eq (3) with weighted supervision. $ \\mathcal{T}\\_\\text{interval} $ should be the same as $\\mathcal{T}\\_\\text{timestamp} $?\n\n(3) Eq (5): $\\mathcal{T}\\_\\text{uncertain} $ needs to be defined in the main text for completeness. \n\n(4) Eq (6): It is $ O_t^h$  rather than $H_t$ that has to be highlighted.\n\n(5) Eq (7): $\\mathcal{L}\\_\\text{scrutinize}$ is never defined.\n\n2. The two-level Compression makes sense. However, the implementation is not clear (L219 - L230). According to the paper, you are using a learnable special token \\<ct\\> to compress the input low-res multi-frame (or high-res single-frame). My question is, how do you supervised train this? How does the training enforce the compression? Some technical details and intuitions need to be added here.\n\n3. \"Triangle of Impossibility\" in Figure 2 caption. This term refers to the trilemma where you cannot simultaneously achieve three goals. This contradicts the claim of the paper that the proposed method does well on all axes. I also believe that they are not essentially exclusive, and it's just a matter of dataset and model, which the paper's contribution lies in. \n\n4. To get the number on COIN, what training data did you use? According to L149, the training dataset is Ego4D. If so, it is somewhat surprising to see that the model trained on egocentric videos does well on exocentric videos too.\n\n5. Supplementary L130, since $\\tau\\_\\text{ant} = 1$, $\\tau\\_\\text{lat}=2$, $\\text{scale}_k$ should always be greater than 3 even without $\\max(1, \\cdot)$?", "was_revised": true, "final_justification": "My concerns are addressed. I hope the details/corrections will be incorporated in the future draft/revision.", "rebuttal": "Thanks for your exceptionally thorough review and your detailed, positive summary of our work's originality and significance. We appreciate the precise suggestions for improvement, which we address in detail below.\n\n> **W1: Inconsistent notations**\n> \n1. **W1.1:** Yes, in the LIVE training dataset, only answer timestamps are provided, not answer intervals. Therefore, $\\mathcal{T}\\_{timestamp}=\\mathcal{T}\\_{GT}$.\n2. **W1.2:** No, we generate the ESTP-IT dataset, which includes answer interval annotations. To alleviate the conflicts that arise from timestamp-level supervision between adjacent frames, we incorporate interval-level temporal supervision signals. Concurrently, we employ a modified loss within these intervals to enable the model to discern subtle variations among frames within the given interval.\n3. **W1.3:** Thank you for your suggestion. We provide a detailed definition of $\\mathcal{T}\\_{uncertain}$ in Appendix D (Stage-2 Input), and we'll revise the main text to include this information.\n4. **W1.4:** Yes, thank you for pointing that out; we'll make the necessary revisions.\n5. **W1.5:** Thank you for your correction. We'll amend this issue, as \"scrutinize loss\" in Eq. (7) refers to \"ask_high loss\" in Eq. (5).\n\n> **W2: Detail of Proactive Dynamic Compression**\n> \n\nFor a multi-turn conversation, we forward **turn by turn**, inputting only the **system prompt, compression tokens from historical turns, and all tokens from the current turn**. This approach enforces compression during training. We provide an example in Appendix D Proactive Dynamic Compression Mechanism Example.\n\n> **W3: Caption for Figure 2**\n> \n\nThank you for your suggestion. We will revise the caption for Figure 2, replacing \"**Triangle of Impossibility**\" with a term like \"Multi-objective Trade-off Triangle\" to enhance clarity. We also appreciate your recognition of our work's contribution.\n\n> **W4: COIN training set**\n> \n\nIn our COIN experiments, we aim to demonstrate the **generalizability of our model architecture** (VideoLLM-EyeWO). Therefore, we follow the training setup of the baseline (VideoLLM-online), **also utilizing the COIN training set**. Thank you for your correction; we will subsequently supplement the main text with more detailed settings for COIN training.\n\n> **W5: Correction on $scale\\_k$ Calculation**\n> \n\nYes, thank you for your correction. Values without the maximum also exceed 3.\n\n> **Q1: Open-source plan**\n> \n\nWe **plan to open-source the entire project**, including ESTP-Bench and VideoLLM-EyeWO (including training data and model weights). This effort aims to help researchers reproduce and build upon our work.\n\n> **L1: How fast does the model run? Is it ready for real-time purpose? If so, on what kind of hardware device?**\n> \n\nOur model, as depicted in Fig. 7, operates at an average rate exceeding **2 APS (Answers Per Second)** with a **2 FPS (Frames Per Second)** input. This experiment runs on an **8-A40 GPU cluster** configured with **96 Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz cores**. On our 8-H20 cluster, the running speed increases by 1.7 to 2 times. We appreciate your suggestion; further accelerating model execution and identifying suitable hardware setups remain meaningful challenges."}]

Author rebuttal:

Meta-review: This paper introduces a proactive ego-centric Video-LLM framework for streaming video understanding, enabling real-time processing through a novel mechanism that anticipates future frames while maintaining temporal coherence. 

Reviewers agree that the work addresses an important and timely problem of efficient streaming video understanding, offering strong technical contributions, solid experiments, and meaningful improvements over existing baselines. 

Strengths include the originality of the proactive streaming design, clear motivation, and comprehensive evaluation across diverse tasks. Weaknesses mainly concern limited analysis of scalability and ablation depth, as well as the need for clearer articulation of broader applicability beyond ego-centric scenarios, which should be included in a revision. 

The rebuttal period saw constructive responses that addressed concerns to a reasonable extent.  Finally, all reviewers voted for accept, though they suggested clarifying broader impact and extending empirical validation.  Given the novelty, rigor, and potential impact on the growing field of streaming multimodal agents, I recommend acceptance (poster).
Decision: Accept (poster)