LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
Authors: Zhenyu Yang
Kairui Zhang
Yuhang Hu
Bing Wang
Shengsheng Qian
Bin Wen
Fan Yang
Tingting Gao
Weiming Dong
Changsheng Xu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Online Video Understanding
Streaming Video Understanding
Primary area: applications

Abstract: Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous frame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative coherence. To address these limitations, we introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53× faster inference. We also construct an OmniStar dataset, a comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evaluation tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-art performance, achieving an average 19.5\% improvement in semantic correctness with 18.1\% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0\% across all five OmniStar tasks. Our model and dataset can be accessed at https://github.com/yzy-bupt/LiveStar.

LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
Zhenyu Yang"?, Kairui Zhang', Yuhang Hu', Bing Wang', Shengsheng Qian"?; 12;
Bin Wen*, Fan Yang', Tingting Gao*, Weiming Dong" Changsheng Xu
Institute Automation, Chinese Academy of Sciences, UUiiversity of Chinese Academy of Sciences, "ShnnghaiTech University, "Kuaishou Technology. 'Peng Cheng Laboratory yangzhenyu202201a. ac.cn
Abstract
Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous rrame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative co- herence. To address these limitations, introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) cusponse-silence decoding framework that determines optimal proactive response timing via single forward pass verification; (3) memory-aware accel- eration via peak-end memory compression for online inference 10+ minute videos, combined with streaming key-value cache to achieve 1.53x faster inference. We also construct OmniStar dataset, comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evalua- tion tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-arr performance, achieving an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0% across all five OmniStar tasks. Our model and dataset can be accessed bittps///gihhne cww cotm/yyy-buppt/Lieettar
Introduction
Recent advancements in Large Vision-Language Models (LVLMss) 1+5 have substantially propelled
the development of Video Large Language Models (Video-LLMs) 66-0 These cutting-edge models
have been instrumental in pushing the boundaries of multimodal AI systems through innovations in spatial-temporal reasoning [TH33) memory-enhanced processing (4H16, and long-term video understanding [17 19]. particularly enhancing capabilities in offline scenarios.
Building on these advances, researchers are increasingly exploring live streaming assistants for online video understanding 25) which remain always-on, reason through time, and provide real-time responses. However, existing online Video-II (25-27 struggle to process frame-by- frame inputs and determine response timing, as shown in Fig a). To effectively process continuous video streams and generate timely responses, such models must possess temporal decision-making capabiliiies--rspoonn at contextually appropriate moments rather than producing repetitive outputs
"Corresponding author: bacnnshong.qja@@nlppiicc
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(6) Streaming Narration
Frame-by-Frame Input
Video Stream
Proseit:
tine.
Narrattion
MsNarrationn
NiVorraiion
(b) Existing Methods
(c) LiveStar (Ours)
(d) Training Perspective
Narration
Pre-trained
Narration
Training
<DS>>
VideoLLM
Training
LiveStr
Narmation
Narration
(e)
Atthe gray kitton
Inference
ruening green graver Shortyy oftes another
<EOS>
Namatio
froms appreaches
Decade
Check
firs onc. Then, soone shifts they are now
Carren Frome
Bingle Poneard
(DMtol)
Figure 1: Illustration of online video understanding. (a) Taking the RNG task as an example, online vidco understanding requires Vidco-LLMs to handle continuous streams and output at appropriate times; (b) Existing methods overly rely learning the EOS token, leading to poor inference performance; (c)-(c) LiveStar establishes an effective response-silence training and inference framework by SCAM and SVcD without compromising basic video understanding capabilities.
default "I don't know" responses for each frame. To address this, VideoLLM-onlin (0) designs a streaming EOS SEnd-Of-Sequence prediction mechanism (Fig. [b)). It ingests video streams, determines response timing based on user queries, and generates EOS tokens to mark silence intervals. Subsequent advancements like VideoLLM-MoD [21] and LION-FS (2) have further refined this EOS-based framework, achieving improved computational efficiency and response accuracy.
However, current online Vidco-LLMs exhibit excessive reliance on the EOS token, which leads to four critical issues: (1) Response-Silence Silence Imbalance: Frames requiring EOS tokens as output vastly outnumber those generating normal responses [23]. For example, in 1-minute video at 3 FPS with response intervals, the response-tos cesponse-to-silence ratio becomes 1:35. (2) Consecutive Frame Inconsis- tency: As shown in Fig. [b), adjacent visually similar frames may produce conflicting outputs -one one generating full narration while the other outputs only the EOS token. This inconsistency disrupts model convergence during fine-tuning. (3) Pre-training Misalignment: While pretraining aligns image-text pairs, silent states enforce mapping from input frames to EOS tokens, contradicting the pretraining objective of meaningful visual-language correspondence. (4) Vocabulary Confusion: Since the EOS token embedded in the vocabulary as a regular token, its frequent appearance in responses pollutes semantic coherence, introducing ambiguity and conflicting with normal outputs. These issues hinder model training effectiveness and degrade base video understanding capabilities, as the forced responee-silence mechanism compromises the lisuul-language aligmment. Motivated by this critical gap, we aim address Challenge How to establish an effective response-silees silence training and inference framework without compromising basic video understanding capabilities?
Moreover, online Video-LLMs face challenges due limited diversity in training data and evaluation scope, failing to fully encompass the online applications encountered in real-world scenarios. For primarily focus first-person video analysis through training on Ego4D (8). Although StreamMind 23 extends its reach to sports scenarios with SoccerNet (29] its scope remains severely constrained. Recent efforts like SVBench (30] OVO-Bench [31], and StreamBench (32) have pioneered online evaluation by assessing models' ability to process videos synchronously during playback. However, these benchmarks focus exclusively on video question answering and lack coverage of diverse real-world tasks. In practice, beyond first-person devices, online scenarios span live streaming platforms, surveillance systems, and cinematic tools, requiring capabilities like real-time interactive editing [35 34), streaming narration, and temporal grounding continuous streams. Current evaluation benchmarks present three critical limitations: (1) Nar- row scenario coverage limited to specific domains, (2) Single-task evaluation focused on video question answering, and (3) Offline assessment of temporal reasoning and response timing. These shortcomings motivate us tackle Challenge 2: How to construct comprehensive dataset and
instance, prominent models 20 2

---

benchmark encompassing diverse real-world scenarios and tasks for holistic evaluation of online video understanding?
To deal with the above challenges, we introduce LiveStar, an innovative live streaming assistant that enables online video understanding across diverse scenarios, delivering context-aware responses semantically relevant moments through adaptive streaming decoding. For Challenge 1, we establish streaming response-silence paradigm featuring two key innovations: (1) streaming video-language alignment strategy using Streaming Causal Attention Masks (SCAM) during training, and (2) a Streaming Verification Decoding (SVeD) mechanism for inference. During training. we construct interleaved ftame-caption sequences where revealed video frames progressively align with their corresponding captions through causal masked attention constraints (Fig. [d),, maintaining consis- tency with standard multimodal pre- pre-training paradigms. For streaming inference. SVeD employs single-pass verification to determine optimal response timing: it suppresses redundant outputs by comparing current content with previous responses, maintaining narrative continuity through strategic silence. To handle extended video contexts, we implement a peak-end memory compression strategy that prioritizes salient keyframes and final captions, enabling efficient processing of long videos by focusing on semantically rich content while accelerating inference through streaming key-value cache. For Challenge 2, we introduce OmniStar, a dataset for training and benchmarking that encompasses diverse real-world scenarios and evaluation tasks in online video understanding. The dataset includes carefully annotated real-time narration, streaming QA, and streaming video grounding tasks 20,137 videos, evaluated through an online setting that better simulates real-world conditions compared to traditional offline settings. Extensive experiments conducted on three benchmarks demonstrate that LiveStar achieves state-of-the-ard results for online video understanding.
In summary, our contributions can be summarized as follows:
We present LiveStar, innovative live streaming assistant for online video understanding, capable of performing real-time comprehension on continuous video streams across diverse real-world scenarios and online tasks, while responding at contextually appropriate moments.
We design streaming video-language alignment training strategy using Streaming Causal Attention Masks (SCAM). By constructing interleaved frame-caption sequences, we train LiveStar generate temporally consistent captions incrementally for variable-length streaming video inputs, maintaining narrative coherence across evolving frame prefixes.
We propose Streaming Verification Decoding (SVeD), a response-silence decoding framework that determines optimal response timing via a single forward pass verification. Furthermore, SVeD mimics human-like memory consolidation through a peak-end rule-based strategy, distill- ing long video contexts by prioritizing salient frames accelerate processing.
We introduce OmniStar, a comprehensive dataset for training and benchmarking that encom- passes 15 diverse real-world scenarios and 5 evaluation tasks in online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the- art perfor- mance, achieving an average 19.5% improvement in semantic correctness with 18. 1 % reduced timing difference compared to existing online Video-LLMM across all five OmniStar tasks.
2
Related Work
Video Large Language Models Recent advancements in Large Language Models (LLMs) 35 have paved the way for a significant research focus on Video-I.LMs 6-10], particularly for enhancing video understanding tasks such as video captioning 42, video question answering 45 and temporal grounding [46448] in offline settings. Currently, in addition to the popular closed-source
LVLMs such as GPT-4V 49), GPT-4o (37] and Gemini 1.5 Pro [50], an increasing number of open-source Video-LLMs, including LLaVA- -Ne-T- (11). Video-LLaVA (52), and VILA 533 have also demonstrated impressive capabilities these tasks. Despite these advances, achieving seamless human-com computer interaction demands capabilities beyond static video analysis. Specifically, must process real-time online videos while exhibiting temporal decision ceaisiio-mainn---eee responding at
contextually appropriate moments and synthesizing histories to maintain coherence 20 25). Current
Video-I.I.Ms .Mss however, remain limited in handling the dynamic complexities of continuous streaming video processing and often struggle to fully capture the nuances of real-world contexts.
3

---

Video-LLMs for Online Video Understanding Recently, there has been growing interest in
online video understanding 25
20 driven by the availability of diverse streaming video sources.
including online video platforms
54 live streaming services [55] and wearable camera footage (28)
These resources enable Video-LLLMs to learn from real-world contexts, enhancing their ability to interact with streaming content. For instance, VideoStreaming [25] is designed to understand video content in real-time through streaming encoding and memory selection. However, existing models [25 struggle to determine response timing and process frame-by-frame inputs. To address this, VideoLLM-onling [20] introduces a streaming EOS (Cnd-Or-Sequence) prediction mechanism, which continuously ingests video streams and generates EOS tokens to indicate periods requiring silence. Subsequent models [2] 22 refine this framework, improving computational efficiency and response aceuracy. Despite this, excessive reliance on EOS tokens disrupts training and performance. This imbalanced dependency 23 causes most frames to be suppressed into silent outputs, while visually similar adjacent frames produce conflicting predictions (e.g., alternating between captions and silence). Furthermore, it misaligns with pretraining objectives that prioritize visual-language alignment [56. as silent states incentivize trivial mappings rather than meaningful understanding, To bridge this gap, we introduce a live streaming assistant designed to perform real- time understanding of continuous video streams, delivering responses at contextually appropriate moments while maintaining the core capabilities of its pre-trained video-language alignment.
Online Video Understanding Benchmarks The exponential growth of video data has elevated video understanding to a critical field. To rigorously evaluate model capabilities, researchers have established range of standardized benchmarks 44 58-68) for offline Video-LLMs. While these benchmarks enable comparative evaluations and drive progress in Video-LLMs, they primarily rely on complete videos individual clips for offline analysis. Recent efforts like SVBench (30), OVO- Bench [31, and StreamBench [32 have pioneered online evaluation by assessing models' ability to process videos synchronously during playback. However, these benchmarks focus exclusively video question answering and lack coverage of diverse real-world scenarios. Current Video-LLM evaluations remain constrained, with most models primarily evaluated on the first-person Ego4D dataset (28). Therefore, we introduce a comprehensive dataset for training and benchmarking that encompasses diverse real-world scenarios and evaluation tasks in online video understanding.
3 Method: LiveStar
3.1 Training Strategy
We design streaming training strategy for aligning streaming video and language using Streaming Causal Attention Masks (SCAM). By constructing interleaved frame-caption sequences, we train LiveStar to incrementally generate temporally consistent captions for streaming video inputs of varying lengths, maintaining narrative coherence across evolving prefixes as the frames progress.
Streaming LanguaLa Alignment
Current Video-LLMs primarily rely on static vision- 69 pre-trained on image-text pairs, exhibiting limited adaptation to These models typically optimize the objective by image/video-
language foundation models 2
streaming video-text relationships 10 text pair alignment:
max P([Trt]] [Imgs//VVdd))
(1)
which fails to address two critical challenges in online video understanding: (1) incremental process- ing of streaming frames, and (2) dynamic alignment between evolving visual contexts and linguistic outputs. To address them, we propose multi-turn, frame-by-frame instruction tuning that bridges streaming video inputs with language generation through the reformulated objective:
max P[[Txt*] | [Ctr
[Frm*t]). Vt, Ck.
(2)
where [Frm denotes the video frame at timestamp 4, [Ctx:t)) represents the accumulated multi- modal context, and Cs = {t indicates a semantic clip (frames from I' to tn within the same scene/event/action) sharing semantic text [Trt*]. This addresses the pretraining misalign- ment in existing EOS-based models [30 that enforce end-of-sequence token prediction (i.e., max PEEOS| [Ctr [Frm+]]) across non-response frames. Unlike these models, LiveStar achieves streaming consistency through incremental context integration over variable-length video streams. Our method maintains narrative coherence by dynamically updating frame prefixes while preserving evolving visual contexts, establishing new paradigm for streaming video understanding.

---

Streaming Vorification Dacodlnn
Fraee J- Franed Captaan
Frane Fravell Capteed
Prane Prave2r Captbe-d
Inference Framework
Oillder mettral Decuding Forward Inpet
Deey
Dee
Drwditg atep
VLM
Daxel Layer
Framo-by-Frame Input
2PG
Dec
Cheipane wdlh
Coption
Caption 2
VLM
Decoder Layer
Mlic
Jught
Streaming Verification Decoding
Docoder
Decoder
Poop Drg
Figure 2: Overview of the streaming verification decoding (SVeD) inference framework: A dynamic response-silenes decoding framework designed to determine optimal response timing for online video understanding.
Interleaved Trame-Caption Sequences To enable multi-turn, camne-b--frame instruction tuning, we construct interleaved frame-caption sequences using chat-inspired format, which allows LiveS- tar to incrementally ingest visual inputs while preserving temporal awareness for accurate scene transition detection. Each conversational turn includes: (1) a frame [Frm**] (at timestep 4.) and (2) a corresponding caption [Cap*] (for the k-th semantic clip). Consecutive frames from the same semantic clip share captions with identical semanties. To mitigate overfitting from repeated caption exposure, we randomly sample a caption [Capğ] from pool of M paraphrased captions.
To train LiveStar au- toregressively on interleaved frame-caption sequences, we address three challenges: (1) Preventing leakage: Already- generated captions within the current semantic clip must be masked when generating the current caption to prevent trivial copying, as they are identical in content. (2) Token- Token-specific context: During autoregression, the model must retain visibil- ity into previously predicted tokens for the current caption to maintain coherence. (3) Scene transition signaling: The last caption of each semantic clip must persist across subsequent frames to explicitly demarcate semantic boundaries. To ad- dress these, we propose Streaming Causal Attention Masks (SCAM) (see Fig. for the mask matrix), replacing standard causal attention masks. Formally, when generating the cap- tion for frame Frm in the clip Ck,, the model constrained to attend only to all video frames from the preceding clips (C1, Cz..... C } and the last captions of those clips. The final optimization objective is as follows:
Streaming Causal Attention Masks
Ertg
Cup
Cap
Img
Img
Cup
decoding
Figure 3: Mask matrix of SCAM.
max P([Cap}) [Cta (Mask (, Vt, E Ct.
Here, the mask matrix Mask is designed to block attention to all non-terminal caption tokens from semantic clips (C1, .2...., C prior to timestep t,, as illustrated in Fig. Id).
3.2 Inference Framework
While LiveStar can generate real-time captions each timestep, determining when update the response while maintaining content coherence and avoiding redundant outputs remains critical challenge. To address this, we propose a dynamic response-silence decoding (SVeD) framework (see Fig. and employ memory-aware techniques for acceleration.
Streaming Verification Decoding We present SVeD, a dynamic response-silence decoding frame- work designed to determine optimal response timing for online video understanding. It intro-
5

---

duces a decoding gate that selectively triggers caption generation based on streaming verifica- tion mechanism. At each triggered decoding step i,, we compute the perplexity of the gener- ated caption [Dec] as: "PL (
\/T/P[[Dec] [ ([rm=t [Frm*]]) where N represents
the token count of [Dee], and P(.) is the autoregressive probability derived from token logits. For each incoming frame [Frm*]] SVeD performs a single forward pass to verify the latest caption's ss validity by recomputing PPL (PLee( If PPL* ([Dec) > PPL ([Dee)) (where a is a tunable scaling factor), the gate activates decoding at i,, generating an updated caption. Otherwise, move [Dec] to the end of [Ctx] without decoding, preserving temporal coherence while minimizing latency. Under the same model architecture, SVeD achieves faster inference than decoding EOS token to indicate silent response. This lightweight verification step en- sures adaptive response timing, balancing accuracy and efficiency. For details, refer to Alg.
Peak-End Memory Compression Modern streaming videos often span hours with high frame rates, posing computational challenges for long-term understanding. Inspired by the hu- man memory retention prioritizes salient mo- ments (keyframes) and recent experiences (sum- maries)-Wy we propose memory compression framework tailored for 10+ minute video anal- ysis at 3 fps. Our method leverages two criti- cal signals: (1) Keyframe detection: We have computed the perplexity PPL'((Dec)) for each frame, where lower values indicate higher se- mantic importance. (2) Temporal summariza- tion: The final frame's caption of each seman- tic clip encapsulates event semantics. To opti- mize memory, we probabilistically prune frames older than window W, with deletion likelihood proportional to both the relative PPL. within its semantic clip and elapsed time. Experiments demonstrate that it achieves optimal semantic accuracy (SemCor) and minimal timing differ- ence (TimDiff) compared to Uniform Dropout and FIFO Forgetting, as shown in Tab. 4
Algorithm 1: Streaming Verification Decoding
Input: Video frame stream
Output: Dynamically generated caption [Dec]
Peak-End cognitive rule [70
Initialize [Dec,, [Ctxi] < f
Initialize reference timestamp f, f
for each incoming frame [Frm">] do
Append [Frm's] to [Cta]]
if [ec] * 0 then
Compute verification perplexity:
PPL. ([Dec) V P([Dec]][Ctstt
if PPI,5 ((Dee)
(PPt,""[[Del)) then
Activate decoding: Generate new tokens [Declaew using [Ctx:tt]]
Update [Dec] + Deec new
Append [Dec] to (Ctrr
t: t+tt t
else
Swap the last two elements in [Ctz]]
// Move [Dec] to the end
else
Perform initial decoding to generate [Dee] Append [Dee] to [Ctx]
4 4 Ly
Streaming Key- Value Cache We implement an effective streaming key-value (KV) cache mecha- nism for multi-turn conversational interactions. We introduce dual-level caching architecture that maintains both intra-dialogue KV caches for frame-level processing and inter-dialogue streaming caches for long-context preservation across conversations. This eliminates redundant recomputation of historical context representations (keys and values) during new video frame processing while ensuring low-latency inference. It specifically addresses two critical challenges: (1) Cache sequence integrity under swap operations in our SVeD module, and (2) Dynamic length adaptation through Peak-End Memory Compression that strategically prunes redundant tokens while preserving temporal coherence. Experiments (see Tab. 4 demonstrate that it achieves 1.53x faster inference than methods without KV caching while incurring negligible performance loss on video streams.
4 Dataset: OmniStar
Datasets Statistics. We introduce OmniStar, a comprehensive online video understanding dataset featuring 15 diverse real-world scenarios (46 specific categories) and online evaluation tasks. The dataset comprises 20.137 expert-annotated video streams with temporally dense annotations, rigorously split into 19,137 training and 1,000 evaluation instances (200 per task) without overlap. It establishes five synergistic evaluation tasks through unified streaming protocol: (1) Real-time narration generation [RNG], (2) Online temporal grounding [OTG], (3) Frame-level dense QA [FDQ), (4) Contextual online QA [COQ], and (5) Multi-turn interactive QA [MIQ]. More detailed online tasks are described detail in I The 15 scenarios include: Travel & Events, Sports, Pets & Animals, Music, Autos & Vehicles, Film & Animation, Nonprofits & Activism, Science &

---

Technology, Education, Howto & Style, News & Politics, Entertainment, Comedy. People & Blogs, and Gaming, each subdivided into 2-4 fine- aueegraaned categories using YouTube's 's native annotation
system.
Temporally Dense Annotation. In contrast to prior online video understanding benchmarks, which task and lack coverage of real-world scenarios [30]- 32.. OmniStar adopts a semi-automated, temporally dense annotation pipeline to obtain denser labeled data. Every video is annotated with multiple caption segments, where each segment is precisely aligned with its corresponding start and end timestamps. Instead of providing disjointed descriptions, OmniStar's s annotations are constructed as form coherent storylines (e.g.. "First, the woman gathers the ingredients... - "Then, she mixes them in large bowl... - "Finally, she places the dish in the oven..." "), thereby facilitating the RNG task by effectively modeling long-range narrative flow.
are typically limited to single
caption sequences that collectively
Building on this foundation, OmniStar introduces two forms of streaming QA supervision: (1) temporally distributed multi-turn QA chains (Task: MIQ. COQ), where earlier and later questions and answers are semantically linked, and (2) frame-level streaming QA (Task: FDQ), where a standing query corresponding to time-varying answers (e.g.. Q: "What is the person showing?"
"A pen."
+ "A book." + "A smartphone. "). Both supervision paradigms are designed to move beyond static, isolated QA settings by encouraging models maintain temporal awareness. integrate past context, and update answers according to the video streams. Case examples are listed in Tabs. 7 g and 9
5 Experiments
5.1 Experimental Setup
Datasets. We validate LiveStar's effectiveness across offline and online settings, including Ego4D 22 (egocentric narration stream), SVBench (30 (streaming QA) and OmniStar (5 real-world online tasks). We build the Ego4D Narration Stream benchmark using dense timestamped narrations, following VideoLLM-onling
[20]. We leverage SVBench's evaluation set for dialogue and streaming QA. For real-world online evaluation, we benchmark Vidco-LLMs on OmniStar through five online tasks, with real-time narration generation assessed in both offline and online settings, We design a two-phase progressive training paradigm for LiveStar, detailed in App- A.2
Implementation Details. We conducted full fine-tuning of LiveStar 8X NVIDIA A800 GPUs. Building upon Intern Video2.5 [10]. the model consists of a vision encoder (Intern ViT (I), an MLP projector, and large language model (Internl.M2.5-7B (3]). For the vision encoder, we employ ViT to extract video frame embeddings at a rate of 1-4 FPS, with each frame represented by 16 tokens. To optimize efficiency in online video understanding, we fine-tuned the model using a static resolution strategy, enabling it to process multi-minute vidco content within an 8K-token context window. The extracted frame embeddings are then fed into an MLP projector to generate frame tokens, following the approach used in LLa [74). These frame tokens are interleaved with language tokens and input into the large language model, InternLM22.5 Additional details about model architecture and model training can be found in App. A.1
Parameter Settings. During training, we conducted full fine-tuning of LiveStar using total of 83K data, which includes the OmniStar training set. We trained the models for epoch with a learning rate of 4 10-5 using the Adam optimizer (3, 0.9, B2 0.999, weight decay = 0.05), a per-device batch size of 1, and gradient accumulation over 4 steps to achieve an effective batch size of 32. We adopted cosine learning rate scheduling with a warmup ratio of 0.03. Input frames were uniformly resized to 448 x 448, with a patch downsampling ratio of 0.5. The vision encoder was frozen during training, while the MLP projector and language model components were fully updated. Each training sequence contains up to 8192 tokens, consisting of interleaved frame and language tokens following the Intern VL.2.5 conversational template. We optimized the model using the standard autoregressive cross-entropy loss computed over the language tokens, where loss was computed only on assistant response tokens, and inter-frame language segments were excluded via our SCAM strategy. For inference, the tunable scaling factor in SVeD was set to 1 1.03 by default, the prune window W in peak-end memory compression was set to 40 frames, and the size of the paraphrased caption pool of streaming video-language alignment was set to M=1 by default for better temporal alignment.
7

---

Table 1: Evaluation results of various models OmniStar- RNNG--NGG in offline and online evaluation. PPL is a commonly used metric for narration, but due to the different vocabulary sizes of LLMs, is only for reference. "." indicates the inability perform the required test.
Method
Offine Evaluation (Fixed decoding)
Online Evaluation (Response-silence decoding)
PPL. TokAcct SemCort SumFluent
TimDitt, TimRedun! TimCover' SemCort Sum Fluen"
Human
6.73
7.17
108
1.24
0.84
6.09
6.81
Offline Video- LLM
GPT-4V
4.97
5.37
GPT-4o
5.03
LLaVA- Video
12.42
0.53
3,40
2.88
Intern Video2.5
6.91
0.56
4.32
3.61
Intern VL2.5
9.81
0.51
3.40
2.94
MiniCPM-V
4.34
4.13
Qwcn2.5-VL 13.80
0.59
4,42
4.24
Online Assistants
LM-onine 9.73
0.49
3.01
0.69
2.67
2.15
0.80
1.68
0.59
VideoLLMMMMan 9.93
0.48
2.89
0.65
2.54
2.49
1.66
0.55
MMDuet 5.69
0.60
4.29
3.40
2.32
0.62
0.51
1.93
2.69
LiveStar (Ours) 5.14
0.62
4.62
4.55
1,91
0.95
0.71
3.19
4.25
5.2 Online Experiments
To simulate real-world scenarios of online video understanding, we evaluated Video-LLMs during the inference phase using the OmniStar test set. Unlike the Ego4D Narration Stream benchmark, which is offline due to the absence of scoring for inference results and fixed decoding points (as is the case with SVBench), our OmniStar allows Video-LLMs to determine the response-silence timing across five tasks in online setting. This evaluation not only assesses the accuracy of the output timing but also examines whether the output quality aligns with the ground truth.
Evaluation Metrics. We use the following metrics to evaluate the model online video assistant:
DifDiff (Timing Difference) quantifies the absolute time difference between model responses and ground-truth annotations (missing outputs are penalized with full scene duration; multiple outputs are penalized by cumulative time), evaluating real-time response precision. Lower values indicate better synchronization with visual events.
TimRedun (Timing Redundancy) measures redundancy through the average number of unneces- sary responses per scene, where a perfect single response could fully capture the scene content.
TimCover (Timing Coverage) computes the average coverage of responses across all scenes, where scene is scored 1 if it contains at least one valid response and O otherwise.
SomCor (Semantic Correctness) scores semantic alignment between responses and ground truth using GPT-4o across three axes: (1) Semantic Accuracy, (2) Language Quality, and (3) Information Completeness. The final SemCor averages all dimensions (0-10 each). See App. for details.
SumFluen (Summarize Fluency) evaluates holistic fluency of concatenated responses against aggregated ground truth using GPT-4o, with scores for (1) Writing Logicality, (2) Language Fluency, (3) Writing Conciseness, (4) Semantic Consistency, and (5) Narrative Completeness.
Results. The right side of Tab.
Table 2: Evaluation results of online Video-IIMs on OmniStar: "L" in OTG means generation needed for scoring; "_" in COQ and MIQ indicates real-time QA. Note that Videol M.-M-oline not achieve the reported 10 FPS.
presents the results of various mod- els OmniStar-RNG in online evaluation (under each model's 's own response-silence strategy), indicat- ing that LiveStar achieves lower re- sponse latency and higher seman- tic precision. Notably, VideoLLM- online and VideoLLM-MoD output on nearly every frame, leading to the highest TimCover, but this aggres- sive strategy also results in poorer performance on other metrics. contrast, MMDuet produces much sparser outputs, resulting the
Online Evaluation (SemCor//TimDitt) FPS (5min)
Method
RNG
OTG
FDQ
COQ
MIQ
Human
6.09/1.00 .08 -/1.81 9.12/1.01 7,96/-
7.83/-
VideoLLM-online
1.68/2.67
-1969 2,35/2.15 4.01/
3.83/
3.37
VideoLLM-MoD
1.66/2.54
-1983 2.11/2.23 3.99/
3.75/-
3.41
MMDuet
1.63/2.32
./4.42 4.78/2.65
5.71/
5.62/-
0.91
LiveStar (Ours)
3.19/191 -/3.57 6.44/180 5.85/- 5.78/-
3.82
lowest TimRedun, but at the cost of lower scores other metrics. These deficiencies clearly

---

demonstrated in Tab. 2 where LiveStar consistently outperforms all other online Video-LLMs across tasks of the OmniStar benchmark while maintaining the fastest inference speed-with 19.5% increase SemCor score, an 18. 18.1% 1% reduction TimDiff value, and 12.0% increase in FPS compared to the second-best model.
5.3 Offline Experiments
Unlike online evaluation where models autonomously determine response timing, our offline evalua- tion on Ego4D, OmniStar-RNG, and SVBench pre-specifies decoding timing. For fairnsss, we strictly adopt the evaluation protocol and metrics from previous works 20.22) despite their non-generative constraints (i.c., ground-truie verification only). On OmniStar, we extend beyond standard perplexity and token accuracy verification for ground truth implement full model generation with online scoring a critical advancement enabling comprehensive online capability assessment.
Evaluation Metries. For rigorous bench- marking, we adopt established offline met- rics: PPL. TokAcc (Token Accuracy, previ- ously referred to as LM-Correctness). TimeD- iff, and Fluency from prior works (20)22) Additionally, we implement online genera- tion evaluation with offline prescribed timing, measured through SemCor and SumFluen metries, both scored based on GPT-4o.
Table 3:
Evaluation results of online Video-LLMs Ego4D Narration Stream benchmark offline settings. "L" indicates the model lacks EOS tokens for Fluency.
Method
PPL.
TimeDiff Fluency! TokAcct
VideoLLMMoon
2.43
2.04
45.1%
48.1%
MM-DLLLM-MoD
2.41
2.04
45.2%
48.9%
LION-FS
2.09
2.15
46.1 %
52.4%
MMDuct
4.51
1.97
39.3%
LiveStar (Ours)
1.97
1.76
61.1% 1
Results. The left side of Tab. presents the results of various models on Maniittar-NNG in offline evaluation (given decoding timing conditions), showing that LiveStar surpasses all other online assistants and open-source offline Video-LLMs/LVLMM across all metrics, though there remains performance gap compared to human performance and GPT-4V/4o. Tab. 3 demonstrates that LiveStar comprehensively outperforms all other online assistants on the Ego4D Narration benchmark, particularly achieving an 8.7% higher TokAce than the second-best LION-FS. Experimental results and analyses for dialogue and streaming evaluations on SVBench can be found in App. C.1
5.4 Ablation Study
Impact of Response-Silence Threshold. LiveStar employs dynamic response-silenee decoding framework governed by an adaptive threshold mechanism. The decoding thresh- is tunable
2:25
Mitr Walues
Turable Threshold
2.00
1.35
TeBn
*PL (Decc), where scaling factor applied to the timestep-specific perplexity. To rigorously evaluate the parameter sensitivity of a, we conducted an empirical analysis over the interval [1.0, 1.1] using the OmniStar- RNNG benchmark. Our online evalua- tion (see Fig. 4 demonstrates a's critical role in balancing timing difference, timing redundancy, and timing cover- age. Controlled tests revealed optimal performance within the narrow range 1.02-1.04, with 1.03 selected as the final configuration.
old is defined
1:00
0.2
1.24 1.00
1.20
1:3
Tanable Thoestotd
Figure 4: Ablation study on the impact of response-silence threshold.
Strategy for Inference Acceleration. Tab. presents ablation studies memory dropout and KV cache strategies for inference acceleration. For memory dropout mechanisms: (1) Uniform dropout causes a 4.70% SemCor degradation due critical recent frames being discarded; (2) FIFO forgetting impairs temporal reasoning by discard- ing historical event captions, increasing TimDiff by 9.42% with 3.76% SemCor reduction; (3) Our Peak-End Memory Compression preserves seman- tic clip summaries and keyframes through precomputed PPL dropout of non-critical frames, achieving optimal SemCor and minimal TimDiff. Regarding KV cache: While removing both inter/inrra-dialog caching or intra-diann intra-dialog caching shows negligible performance loss, adopting both strategies boosts
Table 4: Ablation study on strategies for inference acceleration on OmniStar-RNG.
Dropout
KV Cache
SemCort TimDiff FPS1
Uniform
3.04
2.01
3.77
Both
FIFO
3.07
2.09
3.91
Neither
3.19
1.95
2.50
Peak-End
w/o Inter-Dialog
3.17
1.87
2.92
Both
3.19
1.91
3.82

---

FPS by 1.53x/1.3 respectively under 5-minute-videe videos inference by eliminating redundant recom- putation of historical key-value representations during new frame processing, enabling low-latency inference without quality degradation.
Size of Paraphrased Caption Pool. To mitigate overfitting in- duced by repeated caption exposure during training. we employ
Table 5: Ablation study on the size of the caption pool.
stochastic sampling approach that selects captions [Capf]
from paraphrased caption pool of size M. Experiments re-
Pool Size
SemCort TimDiff).
veal critical trade-off between semantic richness and tempo-
M=1
3.19
1.91
ral consistency: Increasing M enhances semantic correctness (SemCor), with M
M=2
3,23
1.98
achieving +1.57% improvement over M = 1, but simultaneously degrades temporal alignment (TimDiff) by +3.14%. While larger pools better capture semantic diversity, the incurred temporal inconsistency becomes prohibitive for real-tima applications. Given the heightened sensitivity of temporal alignment in online tasks, we adopt M as the default configuration. This framework allows task-aware adaptation, enabling practitioners to optimize the pool size according to their specific robustness-latency requirements.
M=3
3.24
1.97
6
Case Study
We conduct qualitative comparisons across multiple tasks, including RNG, MIQ COQ, and FDQ, be- tween our LiveStar model and existing representative online video understanding models, VideoLLM- online and MMDuet [24). with details shown in Figs. [ and 7 These cases demonstrate that VideoLLM-online and MMDuet often suffer from limited contextual understanding, hallucina- tions, and insufficient fine- geee-rraied recognition, while LiveStar consistently provides more accurate, grounded, and timely responses. These comparisons collectively highlight the advantages of LiveStar and reveal the limitations of current LVLMs in handling ine-grained and context-sensitive video question answering.
7 Conclusion
This paper introduces LiveStar, an innovative live streaming assistant that pioneers always-on proac- tive responsiveness through adaptive streaming decoding. We establish streaming response-silence paradigm featuring two technical contributions: (1) streaming video-language alignment framework utilizing Streaming Causal Attention Masks (SCAM) during model training, and (2) Streaming Verification Decoding VeD) mechanism for real- time inference optimization. Through extensive experiments across three benchmark datasets, LiveStar demonstrates state-of-the-ar performance in online video understanding while achieving practical deployment capabilities of processing 10+ minute video streams at FPS. This paper significantly advances the field by introducing novel response-silent Sipence paradigm and establishing a comprehensive benchmark, thereby stimulating future research in developing advanced models for complex online video understanding tasks.
Limitations. Despite the superior performance of our proposed LiveStar on several benchmarks for online video understanding, there remain some limitations: First, to improve inference efficiency, we compress each video frame into 16 visual tokens. While this compact representation significantly reduces computational cost, it inevitably compromises the model's ' ability to capture fine- tine-grained visual details. As a result, the model may miss critical cues in scenarios involving subtle motion changes or complex scene dynamics. Second, the current version of LiveStar only supports vision-text modalities and does not incorporate audio information. This limitation constrains the model's capacity to fully leverage multimodal cues in video understanding tasks. In future work, we aim to extend the model include audio modalities for more comprehensive multimodal reasoning.
Acknowledgment
We would like to present our appreciation to the anonymous reviewers and ACs for their constructive suggestions. This work is supported by the National Key Research and Development Program of China (No.2023YFC331000) the Beijing Natural Science Foundation (JQ23018, I.252032), and the National Natural Science Foundation of China (No.6203601,, 62276257).
10

---

References
[1] Z. Chen. W. Wang. Y. Cao. Y. Liu, Z. Gao. E. Cui. Zhu, S. Ye, H. Tian, Z. Liu al., "Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling," arXiv preprint or X11:22412,05771, 2024.
[2] P. Wang. Bai, S. Tan, Wang. Fan, J. Bai, Chen, X. Liu, Wang. W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, "Qwen2-v1: Enhancing vision-language model's perception of the world at any resolution," arXiv preprint arXiv:2409.
2024.
[3] Y. Yao, T. Yu, A. Zhang, C. Wang. Cui, H. Zhu, T. Cai, H. Li. W. Zhao, Z. He et al., "Minicpm-v: A gpt-4v level milm on your phone," arXiv preprint arxiv:2408 01800, 2024.
[4] P. Zhang, X. Dong, Y. Zang, Y. Cao, R. Qian, L. Chen, Q. Guo, H. Duan, B. Wang, L. Ouyang et al., "nteemim-xcomppser-2.55 A versatile large vision language model supporting long-contextual input and output," arXiv preprint arXiv:2407.03320 2024.
[5] T. GLM, A. Zeng. B. Xu, B. Wang, C. Zhang. D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao et al., "Chatglm: A family of large language models from glm- 130b to glm-4 all tools," arXiv preprint Xxxv:2406.12793, 2024.
[6] K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, D. Zhu, J. Ding, and M. Elhosciny, "Minigpt4-vides Advancing multimodal Iims for video understanding with interleaved visual-textual tokens," arXiv preprint arXiv:2404.00 00441333,,, 2024.
[7] M. Maaz, H. Rasheed, S. Khan, and Khan, "Video-chatgpt: Towards detailed vidco understanding via large vision and language models," arXiv preprint arXiv:2336.05422. 2023.
[8] K. Li, He, Y. Wang, Y. Li, Wang. P. Luo, Y. Wang. L. Wang, and Qiao, "Videochat: Chat-centric video understanding," arXiv preprint arXiv:2305.063: 2023.
[9] A. Yang, A. Nagrani, P. H. Sco, A. Micch, J. Pont-Tuset, I. Laptev, Sivic, and C. Schmid, "Vid2seq: Large-scale pretraining of a visual language model for dense vidco captioning." in Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10714-10726.
[10] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang et al., "Intemvideo: Gen- eral video foundation models via generative and discriminative learning," arXiv preprint 2rXiv:2212.03191, 2022.
[11] Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo. D. Zhao et al., "Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms," arXiv preprint arXiv:2406. 240.:4006.77766 2024.
[12] Z. Liu, Y. Dong. Z. Liu, W. H., Lu, and Y. Rao, "Oryx mllm: On-demand spatial-tempontt understanding at arbitrary resolution," " arXiv preprint arXiv:2409. 12961, 2024.
[13] S. Ren, L. Yao. S. Li, X. Sun, and L. Hou, "Timechat: A time-sensitive multimodal large language model for long video understanding," in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14313-14325,
[14] H. Zhang, Wang, Y. Tang, Y. Liu, J. Feng. J. Dai, and Jin, vstreem: Memory-based real- tile understanding for long video streams," " arXiv preprint 20t::2200000088. 2024.
[15] E. Song, W. Chai, T. Yc, J.-N. Hwang, X. Li, and G. Wang, "Movicchat+: Question- aware sparse memory for long video question answering," arXiv preprint 4xXiv:2444.17177 2024.
[16] B. He. H. Li, K. Jang. M. Jia, X. Cao, A. Shah, Shrivastava, and S.-N. Lim, "Ma-lmm: Memory- augmented large multimodal model for long-term video understanding." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13.504-13 514.
[17] X. Wang, D. Song. S. Chen, C. Zhang, and B. Wang, "Longllava: Scaling multi modal ilms to 1000 images efficiently via hybrid architecture," 2024. [Online]. Available: httpss//arxiv.orr/abb
028889
[18] F. Xue, Y. Chen, D. Li. Q. Hu, L. Zhu, X. Li. Y. Fang, H. Tang. S. Yang. Z. Liu, Y. He, H. Yin, P. Molchanov, J. Kautz, L. Fan, Y. Zhu, Y. Lu, and S. Han, "Longvila: Scaling long-context visual language models for long videos." null, 2024.
[19] P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang. Y. Zhang, Z. Wang H. Tan, C. Li, and Z. Liu, "Long context transfer from language to vision." arXiv preprint arXiv:2406. 16852, 2024.
11

---

[20] J. Chen, Z. Lv. S. Wu, K. Q. Lin, C. Song. D. Gao. J.-W. Liu, Z. Gao, D. Mao, and M. Z. Shou, "Videollm- online: Online video large language model for streaming video." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024, pp. 18407-18411 418.
[21] S. Wu, J. Chen, K. Q. Lin. Q. Wang, Y. Gao, Q. Xu, T. Xu, Y. Hu, E. Chen, and M. Z. Shou, "Videollm- mod: Efficient video-language streaming with mixture-of-deptee -eetth vision computation," Advances in Neural Information Procexsing Systems, vol. 37. pp. 109 922-109 947, 2024.
[22] W. Li. B. Hu, R. Shao, L. Shen, and L. Nie. "Lion-fs: Fast & slow video-language thinker as online video assistant," " arXiv preprint arXiv:2503. 033663 2025.
[23] X. Ding, H. Wu, Y. Yang, S. Jiang, D. Bai, Z. Chen, and T. Cao, "Streammind: Unlocking full frame rate streaming video dialogue through event- event-gated cognition," arXiv preprint urXiv:2503.062200 2025.
[24] Y. Wang, X. Meng. Wang, Liang, Wei, H. Zhang, and D. Zhao, "Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format," arXiv preprint arXiv:2411. 177911 2024.
[25] R. Qian, X. Dong, Zhang, Zang. S. Ding, D. Lin, and Wang, "Streaming long video understanding with large language models," " Advances in Neural Information Processing Systems, vol. 37. pp. 119 336- 119360, 2024.
[26] X. Zhou, A. Arnab, S. Buch, Yan, A. Myers, X. Xiong, A. Nagrani, and C. Schmid, "Streaming dense video captioning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 243-18222.
[27] H. Xiong. Z. Yang. J. Yu. Y. Zhuge, L. Zhang, Zhu, and H. Lu, "Streaming video understanding and multi-round interaction with memory
memory-enhanced enhanced knowledge," arXiv preprint arXiv:2501. 13468, 2025.
[28] K. Grauman, A. Westbury, E. Byme, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu,
X. Liu et al., "Ego4d: Around the world in 3,000 hours of egocentric video," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 18 18999-199 19012.
[29] S. Giancola, M. Amine, T. Dghaily, and B. Ghanem, "Soccernet: A scalable dataset for action spotting in saccer videos," " in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018,1111 pp. 1711-1721.
[30] Z. Yang, Y. Hu. Z. Du, D. Xue, S. Qian, J. Wu, F. Yang, W. Dong, and C. Xu, "Sybench: A benchmark with temporal multi-turn dialogues for streaming video understanding," " arXiv preprint arXiv:2502. 10810. 2025.
[31] Y. Li, Niu. Z. Miao, C. Ge. Y. Zhou, Q. He, X. Dong, H. Duan, S. Ding, R. Qian et al., "Ovo-bench: How far is your video-Ilms from real-world online video understanding?" arXiv preprint arXiv:2501.05510. 2025.
[32] H. Xiong. Z. Yang. J. Yu. Y. Zhuge. L. Zhang, J. Zhu, and H. Lu, "Streaming video understanding and multi-round interaction with memory-enhaneee knowledge," 2025. [Online]. Available: https:/arriv.org//b//55..L..
[33] Z. Yang, D. Xue, S. Qian, W. Dong, and C. Xu, "Ldre: Llm-based baseb divergent reasoning and ensemble for zero zero-shot composed image retrieval." in Proceedings of the 47th International ACM SIGIR conference on research and development in information retrieval, 2024, pp. 80-90.
[34] Z. Yang, S. Qian, D. Xue, J. Wu, F. Yang, W. Dong, and C. Xu. "Semantic editing increment benefits zero- shot composed image retrieval," in Proceedings of the 32nd ACM International Conference on Multimedia, 2024. pp. 1254.
[35] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosalc et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv prep 2023.
[36] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu. R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Mil- lican al., "Gemini: a family of highly capable multimodal models." arXiv preprint arXiv:2112.11805, 2023.
[37] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya. F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et ol., "Gpt-4 technical report," arXiv preprint arXiv:2.033.088. 087744 2023.
[38] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," Advances in neural information processing systems, vol. 35, pp. 27 730 27 744, 2022.
12

---

[39] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever al., "Improving language understanding by generative pre-training," Unknown, 2018.
[40] L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang. Z. Chen, H. Duan. Z. Tang. L. Yuan et al., "Sharegptévideo: Improving video understanding and generation with better captions," " Advances in Neural Information Processing Systems, vol. 37, pp. 19472-19 495. 2024.
[41] L. Xu, Y. Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng. "Pllava: Parameter-fice free llava extension from images to videos for video dense captioning." arXiv preprint arXiv:2404. 16994, 2024.
[42] M. M. Islam, N. Ho. X. Yang. T. Nagarajan, L. Torresani, and G. Bertasius, "Video recap: Recursive captioning of hour-long vidcos," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 198-18 208.
[43] D. Ko. J. S. Lee, W. Kang, B. Roh, and Kim. "Large language models are temporal and causal reasoners for video question answering," arXiv preprint arXiv:2310. 15747, 2023.
[44] K. Li, Y. Wang, Y. He, Li, Y. Wang, Y. Liu, Z. Wang, Xu, G. Chen. Luo al., "Mvbench: comprehensive multi-modal video understanding benchmark," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195-22 206.
[45] M. Maaz, H. Rasheed, S. Khan, and F. Khan, "Vidcogpt+: Integrating image and video encoders for enhanced video understanding," " arXiv preprint arXiv:2406. 09418, 2024.
[46] Y. Guo, Liu, M. Li, D. Cheng, X. Tang, D. Sui, Q. Liu, X. Chen, and K. Zhao, "Vig-llm: Integrating timestamp know ledge into video Ilms for enhanced video temporal grounding," " in Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, no. 3, 2025, pp. 3302-3310.
[47] Y. Xu, Y. Sun, Z. Xie, B. Zhai, and S. Du, "Vig-gpt: Tuning-free zero-shot video temporal grounding with gpt," Applied Sciences, vol. 14, no. 5, p. 1894, 2024.
[48] Y. Wang, X. Meng. J. Liang, Y. Wang, Q. Liu, and D. Zhao, "Hawkeye: Training video-text ilms for grounding text in videos." arXiv preprint arXiv:2403. 10228, 2024.
[49] Z. Yang. L. Li, K. Lin, J. Wang, C.-C. Lin. Z. Liu, and L. Wang. "The dawn of Imms: Preliminary explorations with gpt-4v (ision)," arXiv preprint 2r39v:2309.117 17421, vol. 9, no. 1. 1, 2023.
[50] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b, Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser et al., "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context," arXiv preprint wxxv:2403.005500 2024.
[51] Y. Zhang, B. Li h. Liu, Y. Lee, L. Gui, D. Fu. J. Feng, Z. Liu, and C. Li. "Llava-nnxt: A strong zero-shot video understanding model," April 2024. [Online). Available: -ellvvvveex-vieooo
https://lava-vl..gtnt www.hhhhboooboolg/222200000
[52] B. Lin, B. Zhu, Y. Ye, M. Ning. Jin, and L. Yuan, "Video-Ilava Learning united visual representation by alignment before projection," " arXiv preprint arXiv:231 I. 10122, 2023.
[53] J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, "Vila: On pre-training for visual language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 689-26699,
[54] R. Zellers, Lu, X. Lu, Yu, Y. Zhao, M. Salchi, A. Kusupati, Hessel, Farhadi, and Y. Choi, "Merlot reserve: Neural script knowledge through vision and language and sound," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 16 375-16387
[55] Gao, Lian, Z. Zhou, Y. Fu, and Wang. "Livechat: A large-scale personalized dialogue dataset automatically constructed from live streaming." arXiv preprint arX1v:2306.08-001 2023.
[56] B. Zhu, B. Lin, M. Ning. Y. Yan, Cui, H. Wang, Pang, W. Jiang. Zhang. Z. Li et al., "Languagebind: Extending langulan pretraining to n-modality by language-based semantic alignment," arXiv preprint arXiv:2310.00
2023.
[57] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, "Flava: A foundational language and vision alignment model." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 15 638 15 650.
[58] K. Mangalam, R. Akshulakov, J. Malik et al., "Egoschema: A diagnostic benchmark for very formform video language understanding." Advances in Neural Information Processing Systems, vol. 36, 2024.
13

---

[59] Z. Yu, D. J. Yu, T. Yu. Z. Zhao. Y. Zhuang. and D. Tao, "Activitynet-qa: A dataset for understanding complex web videos via question answering." " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019. pp. 9127-9134.
[60] L. Li, Y.-. Chen, Y. Cheng. Z. Gan, L. Yu, and J. Liu, "Hero: Hierarchical encoder for video+ language reni-eepresnnntiionn pre-training." arXiv preprint arXiv:2005.00 000000000.000 2020.
[61] V. Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y. Yang, C. Doersch et al., "Perception test: A diagnostic benchmark for multimodal video models," Advances in Neural Information Processing Systems, vol. 36, 2024.
[62] A. Zadeh, M. Chan, P. Liang. E. Tong, and L.-P. Moreney, "Social-iq: A question answering benchmark for artificial social intelligence," in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8807-8817.
[63] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang. X. He, and Y. Zhuang. "Video question answering via gradually refined attention over appearance and motion," in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 1645-1653.
[64] J. Lei, L. Yo, M. Bansal, and T. L. Berg, "Tvqa: Localized, compositional video question answering," arXiv preprint 10XXv:88090016990 2018.
[65] J. Xiao, X. Shang, A. Yao, and T.-S. Chua. "Next-qa: Next phase of question guestion-answering to explaining temporal actions," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2021, pp. 9777-9786.
[66] E. Song, W. Chai, G. Wang. Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye. Y. Zhang al., "Moviechat: From dense token to sparse memory for long video understanding." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18121118 232.
[67] Wang, Z. He, W. Hong. Y. Cheng, X. Zhang. J. Qi, S. Huang, B. Xu, Y. Dong, M. Ding et al., "Lvbench: An extreme long video understanding benchmark," arXiv preprint arXiv:2406.09000 08035, 2024.
[68] Y. Jang, Y. Song. Y. Yu, Kim, and Kim, "Tgif-qa: Toward spatio-temporal reasoning in visual question answering," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2758-2766.
[69] H. Liu, C. Li, Q. Wu, and Lee, "Visual instruction tuning." " Advances in neural information processing systems, vol. 36, pp. 34 898994 916, 2023.
[70] D. Kahneman, "Evaluation by moments: Past and future," " Choices, values, and frames, pp. 993-708, 2000.
[71] D. Kahneman, B. L. Fredrickson, C. Schreiber, and D. A. Redelmeier, "When more pain is preferred to less: Adding a better end," Psychological science, vol. 4, no. 6. pp. 401-405, 1993.
[72] B. L. Fredrickson and D. Kahneman, "Duration neglect retrospective evaluations of affective episodes." " Journal of personality and social psychology, vol. 65, no. 1, p. 45, 1993.
[73] Z. Cai. M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu el al., "Internlm2 technical report," arXiv preprini arXiv:2403.17297 2024.
[74] H. Liu, C. Li, Y. Li, and Y. J. Lee, "Improved baselines with visual instruction tuning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296-2696000 306.
[75] Han, L. Yang, X. Chang, and H. Wang, "Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos." arXiv preprint arXiv:2312.10330 2023.
[76] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., "Gpt-40 system card," arXiv preprint arXiv:2410.21276, 2024.
[77] Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, Feng, Z. Liu, and C. Li. "Llava-next: A strong zero-shot video understanding model," April 2024. [Online]. Available: http::/lava-vll wttphb.io/blog//u222. 04- 1114-00-llvaa llava- next- video/
[78] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song, K. Dang, P. Wang. S. Wang. J. Tang et al., "Qwen2. 5-vl technical report," arXiv preprint 2502 13923, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state contributions and summarize them at the end of Sec. and the claims match the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed instructions for replicating the results are provided in Sec. 5.1 and App. A Additionally, we provide detailed code, model weights, datasets, and the com- plete training/inference process in the anonymous GitHub repository httts://anonymoon: 4open wwwinnce///iivettrrreeeeee
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Detailed instructions for replicating the results are provided in Sec. 5.1 and App. A Additionally, we provide detailed code, model weights, datasets, and the com- plete training/mnferenee process the anonymous GitHub repository 4open 4open. www.annnee////iiettrrraaaa. 5272
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guides http://uuuddee/eeeuuubbieeeeee
(oodeSubmissionooll1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cilips.cc/publle http://ullccgguuddeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Sec. 5.1 and App. A for experimental settings/details, and Sec. 4 for data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to resource constraints, we did not include error bars in our analysis, following the common practice in prior studies within this field maintain consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report details on computational requirements to run the experiments in App. A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres the NeurlPS Code of Ethies, Anonymity is preserved during the current stage.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential impacts are discussed in App.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such models or datasets involved, so there are no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Yes, we have properly credited the creators original owners of the assets used in the paper, and we have done so in appropriate ways under the CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
All assets introduced in this paper will be well documented upon their release.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We hired 20 professional annotators from the outsourcing department of our company. All participants received appropriate and fair compensation according to the standards of their respective regions. The annotation guidelines can be found in App
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: There were no potential risks incurred by the study participants. The hiring and annotation work were approved by our company's 's legal department.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: addition to correcting grammatical errors and editing the format, this paper also used LLMs score the model output part of the evaluation. Detailed evaluation criteria can be found in App. E. and the prompt is provided in App.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy /https://neurib: http://upppspppooeeeeeeeeeeeeeee /coc//ooeeeeccce//225//////////e for what should or should not be described.
21

---

Appendix
Implementation Details
22
Model Architecture
22
A.2 Training Datasets.
22
B Baselines
23
Additional Experiments
24
Dialogue and Streaming Evaluations on SVBench.
24
Online Tasks
24
D.1
Real-time Narration Generation...
25
D.2
Online Temporal Grounding ................................
25
D.3
Frame-level Dense Question-Answerin
25
D.4
Contextual Online Question-Answering
25
D.5
Multi-turn Interactive Question-Answering
26
Online Evaluation Metries
29
Semantic Correctness
29
E.2
Summarize Fluency
30
Visualization of LiveStar
34
Impact Statements
34
H Prompt
35
A Implementation Details
A.1 Model Architecture
Inspired by Intern Video2.5-Chat--88 [TO), the model consists of vision encoder (Intern (. an MLP projector, and large language model (InternL.M2.5-75 (73) For the vision encoder, we employ Intern ViT, a model pre-trained on hybrid dataset combining image captioning and OCR- specific data, to extract video frame embeddings at rate of 1-4 FPS, with each frame represented by 16 tokens. To optimize efficiency in online video understanding, we fine-tuned the model using a static resolution strategy, enabling it to process multi multi-minute video content within an 8K-token context window. The extracted frame embeddings are then fed into an MLP projector to generate frame tokens, following the approach used in LLaVA-1.5 (74). These frame tokens are interleaved with language tokens and input into the large language model, InternLM2.5.
A.2 Training Datasets
Our proposed LiveStar builds upon the pre-trained Intern Video2.5 foundation model 10, which demonstrates robust capabilities in offline video comprehension through spatial-temporal reasoning and long-term temporal modeling, achieved through large-scale pretraining on 3.7M video-text pairs. To bridge the gap between static video understanding and dynamic streaming perception, we design two-phase progressive training paradigm, implementing streaming video-language alignment through Streaming Causal Attention Masks (SCAM) described in Sec. 3.] enable effective adaptation to online vidco understanding scenarios.
Phase I: Temporal Alignment Pretraining, We conduct dense video-language alignment using 63K carefully curated video segments from carefully selected high-quality subsets of three complementary datasets: ActivityNet Captions (59] (9K selected from 20K raw samples), Shot2Story 775 (33K selected from 43K raw samples), Ego4D Narration Stream (28) (20K selected from 113K raw samples), and MVBench (4) (1K selected from 4K raw samples). This stage focuses on establishing frame-level semantic correspondences and enhancing the model's 's capability to process streaming visual input through SCAM.
22

---

Phase II: Multi-Task Online Adaptation. We employ the OmniStar benchmark s 20K training samples across five distinct online video understanding tasks: (1) Real-time narration generation, (2) Online temporal grounding. (3) Frame-level dense QA, (4) Contextual online QA, and (5) Multi-turn interactive QA. Through task-specific adapters, we achieve simultaneous multi-objective alignment that preserves general video understanding capabilities while specializing for streaming scenarios.
This hierarchical training strategy demonstrates two key advantages: 1) Effective knowledge transfer from static to streaming domains through progressive alignment, and 2) Efficient utilization of limited annotated data (83K total samples) via our SCAM-enhanced streaming video-language alignment framework.
B
Baselines
In this paper, we present a comparative analysis of online and offline LLLeoL,LMsLLLLMMs, evaluating their capabilities in two critical aspects of video understanding: (1) determining optimal response timing and (2) processing continuous video streams. Our study focuses on benchmarking their performance in dynamic, real-world scenarios. We categorize baseline models into groups based on their temporal processing paradigm. Online models are evaluated for their ability to incrementally analyze streaming video frames with minimal timing difference, with the key feature being their ability to autonomously decide when to output. In contrast, offline models process complete video sequences as static inputs or handle streaming video but cannot autonomously decide when to output. This dichotomy allows us to systematically compare temporal reasoning strategies and online video understanding capabilities across different architectures.
Online Video-LLMs:
VideoLLM-onie 200 a grreaming-orientter architecture employs continuous key-value cache to enable low-latency incremental processing of video streams. It designs streaming EOS (End-OffSequence) prediction mechanism to determine response timing based on user queries and generates EOS tokens to mark silence intervals.
VideoLLM-MoD (reproduced) 21: a modified variant of VideoLLM-online utilizing mixture-of-dest dephhs token selection to balance computational efficiency and temporal reason- ing accuracy in streaming video analysis. Since the code is not open-sourceds we reproduced it based on VideoL..M-onling
MMDuet 24 a multimodal dialogue- shalogue-oriented oriented model featuring dual-stream encoders for synchronized audio-visus processing is specifically designed for interactive video applica- tions. Unlike VideoLLM-onlle and VideoLLM-MoDD this model uses classification head to decide whether to proceed with decoding and output.
LION-FS (paper-reported results) [22]: : dual-path video-language framework that resolves the efficacy-efficiene trade-off in online video assistance through dynamic routing- based real-time response determination and multi-granular keyframe augmentation with spatial- interaction modeling. Since it is not open-sourced, we used the reported results from the original paper for comparison on the Ego4D Narration Stream benchmark.
These models use their respective response-silence decoding strategies to effectively process continu- ous video streams and generate timely responses. They possess temporal decision- capabilities, responding contextually appropriate moments, processing frame-by-frame inputs, and determining response timing.
Offline Video-LLLM/LLVLM..
GPT-4V & GPT-4o (76): both models process real-time video streams alongside synchro- nized audio, text (e.g., subtitles), and visual frames, enabling holistic comprehension of dynamic scenes. Their autoregressive architecture supports sequential processing of video frames, capturing temporal dependencies critical for tasks like action recognition, event prediction, or summarizing long-form content.
I.aVA--eeT-Videe Video [77]: an open-source source chatbot advances video understanding through zero-shot AnyRes representation and linear scaling for long-context processing.
23

---

Intern Video2.5 I0 & Intern [] the first open-soure MLLMs to achieve over 70% on the MMMU benchmark, matching the performance of leading closed-source commercial models like GPT-4o.
MiniCPM-V 2.6 3] the latest and most capable model in the MiniCPM-V series.
Qwen2.5-VL [78] the flagship multimodal model in the Qwen series, advances visual- language intelligence through native dynamic-resolution ViT with Window Attention, enabling efficient high-fidelity processing of variable-sized images and hour-long videos with second-level temporal localization.
Implementation Notes: Due to the lack of open-source code for VideoLLM-MoD, we reproduced it based on the VidooLLM framework. For LION-FS, which is also not open-soureed, we utilized the results reported in the original paper for comparison the Ego4D Narration Stream benchmark. StreamMind was excluded from our comparisons due to the absence of open-source datasets and model weights.
Additional Experiments
C.1 Dialogue and Streaming Evaluations on SVBench
We benchmarked open-source Lidoo-LLMsLLVLMMs and closed-souree LVLMs aligned with OmniStar on SVBench. The dialogue and streaming evaluation results on SVBench, outlined in Table provide a comprehensive comparison among various Video-LLMs. We have published the evaluation results on the official leaderboard website Notably, closed-source models such GPT-4o and GPT-4V attain significantly higher scores across all metrics, with GPT-4o achieving an Overall Score (OS) of 62.57 in dialogue evaluation and 59.97 in streaming evaluation. LiveStar's zero-shot performance SVBench in dialogue and streaming evaluations outperforms LLaVA-NeXT-Video InternVL2.5, and Intern Intern Video2.5, but underperforms MiniCPM-V 2.6 and Qwen2.5-VL. However, after fine-tuning on the SVBench training set, LiveStar achieves an average improvement of 15.37% across metries, surpassing all source Mideo-LLMMsLLLMMMs and closing the gap with GPT-4V. This demonstrates LiveStar's robust online video understanding capabilities.
Table 6: Evaluation results of various models on SVBench dialogue and streaming evaluation. f denotes fine-tuning the SVBench training set.
Dialogue Evaluation
Streaming Evaluation
Model
AVG
SA
CC
LC
TU
IC
OS
SA
CC
LC
TU
IC
OS
Closed-source LVLMs
GPT-VV
62.61
69.09
05.36
53.73
60.30
56.37
61.41
65.80
59 18
57. 16
57. 93
59.12
GPT-4o
58.26
64.76
70.75
67.68
55,82
62.57
$7.99
63.52
67.72
60.18
59.25
59.97
61.27
Open-source Video-LLMssLVL.Ms
LLaVA-NeXI-Video
37.71
44.59
52.05
41.80
36.58
41.40
34.29
39.68
47.65
35.33
36.68
36.12
38.76
InlemVVL2.5
43.73
50.70
56.61
55.03
43.46
48.73
40.44
48.34
52.84
46.93
45.10
45.04
46.89
Intern Vide222
46.83
53.48
58.22
58.91
47.02
51.73
41.76
49.72
53.25
48.44 44
47. 10
46.58
49.16
MiniCPM-V22
51.70
59.50
65.33
61.72
50.09
56.63
46.44
02.73
58.35
53.48
48.32
49.67
53.15
Qwea2.5VVL
52.54
59.85
65.52
61.61
51.23
57.57
48.21
56.12
60.31
56.33
52.46 46
52.81
53.21
Online Assistants
LiveStar
53.75
59.36
57.29
45.64
51.37
43.56
55.71
50.79
LiveStar"
54.06
61.08
66.43
66.06
52.67
58.95
52.19
59.00
62.85
58.35
54.95
55.87
57.41
D Online Tasks
Our dataset is organized into 5 distinct online evaluation tasks designed to comprehensively assess Video-LLMs' capabilities in streaming video understanding. Each task targets a specific aspect of temporal reasoning and contextual awareness under causal constraints. Detailed case examples for each task are provided in Tabs. and

---

D.1 Real-time Narration Generation
This task requires Video-LLMs to generate accurate and temporally coherent descriptions for stream- ing video in real time, where descriptive segments selectively triggered at transitions between semantic clips to avoid redundancy. Rather than producing isolated captions, the output is expected to form coherent and continuous narrative that evolves with the visual content. Each segment should be logically connected to the previous ones through transitional phrases or discourse markers first, then, meanwhile, as result, finally) ensure a smooth and natural flow. Coreference resolution is also required: entities introduced earlier (e.g., "a man") ") may be subsequently referred to using appropriate pronouns (e.g. "he"), demonstrating contextual awareness. Although generated in multiple parts, the narration should ultimately come together as single, fluid, and unified story- one that maintains semantic clarity, narrative cohesion, and temporal alignment with the unfolding video.
D.2 Online Temporal Grounding
This task requires Video-L.IMe LLMs to detect and localize temporal segments in a streaming video that semantically correspond given textual query, enabling real-time moment retrieval without access to future frames. For each query, the task involves identifying the start point where the visual content begins to align with the deseribed event and the end point where the scene transitions to a different semantic clip. Importantly, multiple distinct segments in a video may independently satisfy the same query-for example, the phrase "Video of a car in travel." " might correspond to several con-contiguous intervals throughout a long video, each relevant interval should be detected independently and promptly as it unfolds. It assesses a model's 's capacity for fine- are-grraddd, online temporal localization and its ability to detect multiple instances of the same semantic clip over time.
Since current Video-LL.Ms lack native online temporal grounding capability, we propose a surrogate evaluation strategy. We first divide the dataset based on the length of the relevant window (e.g.. 10s, 20s, 30s), forming candidate temporal intervals of varying durations. For each video and query pair, we compute the model's per-frame perplexity (PPL) with respect to the query, treating lower perplexity as an indicator of stronger semantic alignment. Among the candidate intervals of the specified length, we select the one with the lowest average perplexity as the predicted segment. Finally, we compare the predicted segment with the ground-truth relevant window to compute a temporal deviation metric (TimDiff) that quantifies alignment accuracy.
D.3 Frame-level Dense Question-Answering
This task requires Video-LLMMs generate multiple temporally grounded answers single persis- tent question throughout a video, producing dense yet non-redundant responses that are triggered only by meaningful semantic changes in the visual content. For instance, given the question "How many people are there in the video now?", the model should initially respond with "One person" " and remain silent until second individual appears, at which point the response should be updated to "Two people". The task emphasizes the production of sparse but informative answer sequences that reflect event-driven and temporally consistent reasoning. It ensures that an online video model's outputs align closely with significant shifts in the video's 's semantic clips, prioritizing relevance and conciseness over frequency.
D.4 Contextual Online Question-Answering
This task requires Video-LLLMs to perform contextual question answering on a continuously streaming video by addressing queries issued at arbitrary time points, based on the visual content observed up to that moment. These questions are organized into QA Mkains-temporaly, distributed, multi-turn interactions grounded in the same video segment, which may exhibit implicit dependencies between successive questions and answers. Each question must be answered using only the video frames that have been observed at the time of the query, without access future information. As the video progresses, new questions may arise, which may exhibit contextual dependencies on both historical video segments and dialogues. For example, given the QA chain Q1: "Who is the competitor in this event?", ?", a follow-up question Q2: "Whal did he continue to do?" relies on coreference resolution, where "he" " refers to the previously identified "competitor". The task emphasizes incremental understanding, memory retention, and context-aware reasoning under causal constraints. It evaluates
25

---

the ability to utilize historical information and engage in coherent multi-turn dialogue within real-time video stream.
D.5 Multi-turn Interactive Question-Answering
This task requires Video-LLMs to perform multi-turn question answering on a continuously stream- ing video by responding multiple point-based on the visual content observed to that moment. Similar to the Contextual Online Question-Answering Task D.4 all queries are organized into QA chains, with each chain repre- senting temporally distributed sequence of questions and answers pertaining to the same semantic segment of the video. this interactive setting, multiple QA chains may evolve parallel, and the questions within or across chains may exhibit implicit contextual dependencies, requiring the model reason jointly over the video history and multiple dialogue threads. The task emphasizes the ability to handle concurrent multi-turn interactions in a streaming context, assessing whether the model can track and resolve complex inter-question relationships while maintaining temporal grounding and causal consistency.
issued simultaneously the same time
26

---

Table 7: Case examples of OmniStar (RNG, OTG, COQ, MIQ).
Online Task
Case Examples
<RNG task promp t> Please describe the content of the video real time and use cohesive words to keep the narrative coherent.
00:00: In room, there is woman with short purple hair wearing black top, and there characters ahove her head.
Real-time Narration
Generation
00:01 - 00:03: Silent.
00:04: Then the cartoon characters above her head keep changing, and finally stay cartoon
character of little red man.
00:05 00:10: Silent.
00:11: Afterwarss the woman plays the cartoon character. She wears black top hat decorated
with row of white teeth and red suit against red background.
00:12 00:20: Silent.
00:21: Finally, the woman holds a white balloon with black mouth and eyes drawn on it, She
keeps staring the balloon. und the picture zooms in her eyes.
<0TG task promp t> A man reporting from outside of a destroyed house.
00:00 01:13: Slient. (Not related)
01:14: Start.
01:15 01:23: Slient. (Related)
Online Temporal
01:24: End.
Grounding
01:25 ~ 01:29: Slient. (Not related)
01:30: Start.
01:31 ~ 01:39: Slient. (Related)
01:40: End.
01:41 02:00: Slient. (Not related)
<C00 task prompt>
00:00 - 00:16: Silent.
00:17: What were the ladies talking about at the beginning of the video clip?
A: They were discussing entering the house.
Contextual Online
00:18 00:22: Silent.
QA
00:23: Q: What are they discussing?
A: They are discussing how to express gratitude for each other's help.
00:24 00:33: Silent.
00:34: Q: Why are they talking about the topic of gratitude?
They talking about the topic of gratitude because of them helped the other,
prompting this discussion.
<HIQ task prompt>
00:00 - 00:06: Silent.
00:07: Q1: What appears at the beginning of the video?
A1: A row of bright windows appears.
Q2: What is above the window?
A2: It's white roof.
Q3: What is in front of it?
A3: There are several white tables.
00:08 00:14: Silent.
00:15: Q1: Where is this row of windows located?
Multi-turn Interactive QA
A1: Located within indoor space.
Q2: What is inside this indoor space?
A2: There is lady and some equipment.
Q3: What does equipment include?
A3: Includes yoga mats and fitness balls.
00:16 00:27: Silent.
00:28: Q1: What is there at the beginning of the video?
A1: There is room for fitness and exercise.
Q2: What are the equipments the indoor space?
A2: There are yoga mats, fitness balls, dumbbells, and other exercise equipment.
Q3: What other furniture and decorations
the room?
A3: In the room, there are also mirrors, potted plants, and a basket among other
decorations.
27

---

Table 8: Case examples of OmniStar (FDQ-1).
Online Task
Case Examples
<FDQ task prompt> What the person showing to the camera?
00:00: The person shows nothing to the camera.
00:01 00:05: Silent.
00:04: The person is showing white cup the camera.
00:07 - 00:10: Silent.
00:11: The person is showing grey cup the camera.
00:12 - 00:16: Silent.
00:17: The person is showing red book the camera.
00:18 00:23: Silent.
<FDQ task prompt> How many times the person launch the toy car on slanted
plane?
00:00: The person did not launch the toy car on the slanted plane.
00:01 00:04: Silent.
00:05: The person launched the toy car once on the slanted plane.
00:06 00:12: Silent.
00:13: The person launched the toy car twice on the slanted plane.
00:14 - 00:19: Silent.
00:20: The person launched toy car three times on slanted plane.
00:21 ~ 00:30: Silent.
<FDQ task prompt> What repeated actions did the person do?
00:00: The person did not do repeated actions.
00:01 - 00:16: Silent.
Frame-level Dense
00:17: The person clapped hands.
QA
00:18 (0::1: Silent.
00:22: The person tapped the table with a ruler.
00:23 00:27: Silent.
00:28: The person folded paper.
00:29 - 00:34: Silent.
00:35: The person did do repeated actions.
00:36 - 000:5:: Silent.
<FDQ task prompt> What letters is the person writing on the paper?
00:00: Not writing.
00:01 - 00:07: Silent.
00:08: The person is writing the letter 'B'.
00:09 00::4: Silent.
00:15: The person is writing the letter 'E',
00:16 00:18: Silent.
00:19: The person is writing the letter "D',
00:20 - 00:23: Silent.
00:24: Not writing.
00:25 - 00:41: Silent.
<FDQ task prompt> What is the order of the letters on the table?
00:00: The order of the letters the table is 'LOVE'.
00:01 00:18: Silent.
00:19: The letters on the table is being shuffled.
00:20 00:42: Silent.
00:43: The order of the letters on the table is 'VOLE'.
00:44 - 01:01: Silent.
28

---

Table 9: Case examples of OmniStar (FDQ-2).
Online Task
Case Examples
<FDQ task prompt> How many times did the objects the video collide?
00:00: No collision occurred.
00:01 00:04: Silent.
00:05: One collision occurred.
00:06 00:10: Silent.
00:11: Two collisions occurred.
00:12 - 00:19: Silent.
<FDQ task prompt> How many objects are there in the video?
00:00: There are four objects the video.
00:01 - 00:02: Silent.
00:03: There five objects in the video.
00:03 - 00:07: Silent.
00:08: There are six objects in the video.
00:09 00:19: Silent.
<FDQ task prompt> How many moving objects are in video?
00:00: No objects moving in the video.
00:01 (0::3: Silent.
00:04: Two objects are moving in the video.
00:05 - (0):8:: Silent.
00:09: Three objects are moving in video.
Frame-level Dense
00:10 - 00:19: Silent.
QA
<FDQ task prompt> t> What is the scene in the video like?
00:00: The scene in the video shows several people the interrogation room the police station.
00:01 ~ 00:15: Silent.
00:16: The scene in the video changes five women outside in garden.
00:17 - 00:30:
Silent.
<FDQ task prompt> How many people are wearing glasses in the video now?
00:00): There is person wearing glasses in the video.
00:01 00:08: Silent.
00:09: There are no people wearing glasses in the video.
00:10 00:15: Silent.
00:16: There are two people wearing glasses the video.
00:17 - (0:21: Silent.
<FDQ task prompt> What actions is the person performing in the video during the cooking
preparation?
00:00 00:03: Silent.
00:04: The person takes out cucumber wrapped in plastic.
00:05 00:08:
Silent.
00:09: The person takes out knife.
00:10 00:15:
Silent.
00:16: The person takes out cutting board.
00:17 - 00:22: Silent.
00:23: The person takes plate.
E
Online Evaluation Metrics
We utilize GPT-4o to evaluate the generated answers in terms of semantic correctness and summary fluency.
E.1 Semantic Correctness
The semantic correctness evaluation used to assess the ability of Video-LLMMs understand video segments after segmenting the complete video, focusing whether the model can accurately capture and comprehend all the information the video segments. When evaluating the semantic correctness of the answers, we input both the model-generated answers for the corresponding video segments
29

---

and the ground truth into GPT-4o, and ask GPT-4o to evaluate the model -eneerted answers from metrics of Semantic Accuracy, Language Quality and Information Completeness, then output scores for these three metrics.
Semantic Accuracy. It evaluates the alignment of core meaning between generated answers and ground truth, evaluating semantic compatibility without errors or hallucinations. The evaluation metric focuses on assessing the overall correctness of the generated answers. The more accurate and closer the generated answer is the corresponding parts of the ground truth, the higher the score, and vice versa. For this metric, we design five scoring criteria: a score of 10 indicates perfect matching with ground truth and no deviation; score of 7-9 indicates minor semantic differences but core information intact; a score of 4-6 indicates significant errors omissions in key details; score of 1-3 indicates mostly incorrect or irrelevant; a score of 0 indicates completely wrong or unrelated.
Language Quality. It evaluates grammatical correctness, lexical precision, and readability, ensuring clarity and fluency in expression. If the generated answer has less grammatical issues or spelling errors, the score will be higher; otherwise, the score will be lower. For this metric, we design five scoring criteria: a score of 10 indicates flawless grammar, precise wording and no redundancy; a score of 7-9 indicates minor grammatical slips but do not affect understanding; a score of 4-6 indicates obvious grammar issues or unclear phrasing; a score of 1-3 indicates chaotic language and multiple errors hinder comprehension; a score of 0 indicates unintelligible or symbol-filled.
Information Completeness. evaluates the breadth and depth of information coverage relative to ground truth, including key events and dynamic actions without omissions. This evaluation metrie focuses on assessing whether the generated answer is sufficiently comprehensive compared to the ground truth. The more ground truth information the generated answer covers, the higher the score, and vice versa. For this metric, we design five scoring criteria: score of 10 indicates including all key information from ground truth; score of 7-9 indicates main points covered but missing minor details; a score of 4-6 indicates partial key information missing but retaining the core content; a score of 1-3 indicates only fragments of information retained; a score indicates no valid information provided.
E.2 Summarize Fluency
The summary fluency evaluation is used to assess whether the model, after segmenting the complete video, can notice the information contained in previous video segments and thus incorporate this information into its answers. When evaluating the summary fluency of the answers, concatenate the model- -enerraeed responses for all video segments and the corresponding ground truth for different segments, then input them into GPT-4o, and request GPT-4o evaluate the from metrics of Writing Logicality, Language Fluency, Writing Conciseness, Semantic Consistency and Narrative Completeness, then output scores for these five metries.
answers
Writing Logicality. It evaluates the clarity and strength of causal, temporal, or inferential relationships between sentences, ensuring coherent progression of ideas. The clearer and more coherent the cause-and-effeee logic in the generated answer, the higher the score it receives. Conversely, if the logic is confused, the score will be lower. For this metric, we design five scoring criteria: score of 10 indicates flawless logic with no gaps or contradictions; a score of 7-9 indicates minor logical jumps but generally coherent; a score of 4-6 indicates weak connections requiring reader inference; score of 1-3 indicates frequent illogical breaks or contradictions; a score of 0 indicates no logical structure.
Language Fluency. It evaluates precision in wording and avoidance of redundant content. If the generated answer contains sentences with abrupt transitions, the score will be low. If the transitions are smooth, the score will be high. For this metric, we design five scoring criteria: a score of 10 indicates effortless flow with no awkward phrasing; a score of 7-9 indicates smooth overall with rare hiccups; a score of 4-6 indicates choppy transitions and some re-reading needed; score of 1-3 indicates relies heavily on forced connectors; score of 0 indicates unreadable due to disjointedness.
Writing Conciseness. It evaluates precision in wording and avoidance of redundant content. If the generated answer contains redundant or repetitive expressions, the score will be low.
30

---

If the writing is concise and succinet, the score will be high. For this metric, we design five scoring criteria: a score of 10 indicates precise wording and redundancy; score of 7-9 indicates minor repetitions but efficient overall; a score of 4-6 indicates verbose sections needing trimming; score of 1-3 indicates excessive repetition filler; a score of indicates entirely redundant or irrelevant.
Semantic Consistency. It evaluates uniformity in terminology, pronouns, and tense usage across the text. If the generated answer contains inconsistencies terminology pronoun usage, the score will be low. If the pronouns, tenses, and terminology are consistent throughout the answer, the score will be high. For this metric, we design five scoring criteria: score of 10 indicates perfect consistency in all elements; score of 7-9 indicates slight inconsistencies with no impact: score of 4-6 indicates multiple inconsistencies requiring correction; a score of 1-3 indicates confusing shifts in terms or perspective; score of indicates incoherent due inconsistency.
Narrative Completeness. It evaluates coverage of all key information without omissions. This evaluation metric focuses on assessing whether there is any missing information in the overall narrative of the generated answer. The more information missing, the lower the score; conversely, the less information is missing, the higher the score. For this metric, we design five scoring criteria: score of 10 indicates including all critical details; score of 7-9 indicates covering main points but having minor gaps possible; score of 4-6 indicates missing some key elements; score of 1-3 indicates only fragmented or irrelevant information; score of 0 indicates no relevant information.
31

---

POOON
O
poooo
DOOOO
OoooO
Timestamp:
In bathroom, a pair of hands are placed under a black fancet, with orange liquid on them and light purple Nuffy scrunchic on the wrist. On washstand on the side of the picture, there red oraamntt white duckling and sign the potted plant with greenery.
Ground Truth:
that says "Keep going today". the washstand right side of the picture, there small
Li--Olieeee You hold bottle.
MMDuet: Slient. (nothing output)
LiveStar: A hand wearing pink wristband is placed on white table. The wristband has the words "Sunny Day" written it in black. There are some items on the table, including a small white duck-sbaped object and pot of green plants.
Timestamp: 00:02
Ground Truth: Then hand sweeps upward across black faucet, the shot changes. White foam flows down from the upright hand the picture, flowing down the light white plush wristband on the wrist.
your MMDuet: Slient, (oothing output)
VideoL.LMM MaL.LMM-Ollies
LiveStur: Then, the hand wearing the pink wristband is placed under running faucet. The water from the faucet washes over the white liquid on the palm, gradually washing it away.
Timestamp: 00:04
Ground Truth: Immediately afterward, the camera pans to a woman wearing red shirt and a yellow headband, squatting the washing her
LideoLLM-Online: You nub your lands.
MMDuet: Apply the cream the face.
LiveStar: Then seene switches to a bathroom. woman wearing a pink T-shirt standing in front the sink, washing bor face. The from the fauect flows down drain.
Timestamp: 00:06
Ground Truth: Then the camem switches again. left hand renches out the other side of the screen, and the other right hand arranging the cuff of the left band. The clothes are beige, and the cuffs have pink and white checkered puttern.
VideoL. MideoLLLM-nnlinn M-Onnine:
your hands.
MMDueet: Slient. (oothing oulgul)
LiveStar: The then switches the woman's hands. She wearing and white wristbands, and hand
adjusting the cuff of the other.
Timestamp: 00:08
Ground Truth: Finally, the camera tums, and four different colors of plush wristhands are shown in the picture. The four plush wristhands neatly arranged hathroom
VideoL You rub your hands.
MMDuet: Slient. (nothing output)
LiveStar: The scene then switches to gray wall. where the women is arranging four wristbands of different colors.
This case shows that VidooLLM-Online and MMDuet, perform poodly on the RNG task. VideoL MM-Online's answers are brief, limited to first-person perspective ("Yow ") and often redundant, while MMDust misses key points and provides simple, responses. Compared these, LiveStar exhibits timely and precise understanding video and sometimes finer details beyond the annotations (e.g, timestamp 00:04).
Comment:
Figure 5: Comparison of M-dooin, MMDuet, and LiveStar on the RNG task.
32

---

Timestamp: 60:000
00:25
Question I: What animal appears on the sureen a: 00:00001 of the video!
Ground truth: An sipacs appeses, being led by men.
VideoLLLM Online: looks like lion's head appearing on the screen.
MMDuet: Ilama appears on the screen.
alpoco appears
Question Why they bring alpwcas?
Ground truth: They bmught alpacas help students
ansiety.
VideoL MMOlninn: I'm
possible they're being used form therapy entertainment. Alpacas often aninaal-assisos assisted thenapy programs in help people reduce
They brought alpacas help people anxiety.
LiveStar: They brought alpucas
ancicty.
Timestamp: 00:26 00:28
Question I: Why do universities use 'herapy algucss the beginning of the viden?
Ground truth: becouse these univenities hope to help studeuts reduce stress and auxiety by using 'terapy alpacas'.
VideoL.LM-O M-Online: look around.
MMDueet help
deal
anviery.
LiveStur: Universities use 'hhrapyy alpacas' help students fight stress anxiety.
Question
these thempy alpwess studkuts
auxisty?
Ground truth: Therapy alpacas' help alleviate and anxiety students through their gentle and friendly students rehaxed when interscting with
any It's possible Out you may have misinterpreted or misnemembered the information. Can you provide more context or clarify what you therapy alpocas? ?
V-deoLLMMOOollnc
universities using 'hhrapy alpacasi heginaing
MMDuet: The students the alpacss and smile.
LiveStar: The therapy alpacas' have calming effect on the students.
Timestamp:
Question
healing alpacal
beginning anxicty.
truthic It's
thempy help
Videol LM-Olinn associated relaxution and stress relief. common practice use alpacio thempy animads help people unwind and fool calm.
question: The 'healing alpaca' refers the alpaca's and which often
Duett The healing alpaca' is therpyy animal.
LiveStar: The hhalling alpace' means the alpaca has colming effect on the students.
Question 2: snimals appear in video?
VideoLLM- LM--line appears
you're referring tell you
group people.
Could you please clarify or provide more context??
MMDuet: A cabbit appenes in video.
LiveStar: rabbit appears video.
Question
gray.
VideoLLLM La-LLLL---llee: Could you please clarify provide more context?
what animals you're referiing tell you that the video group people. grey with LiveStur: gray.
This case shows that Vidcol LM-Online performs poorly on the MIQ task (inclading CDQ), with limited contextual understanding and frequent hollsciaaioos, For its timestompp 00:42 00:54 plausible. but likely generated relatively better but still suffes from halluciaaions and limited contest zwarmess. LiveStar, by compasison, demonstrates videa understadinng.
Comment:
clearly olign
video
price knowlodge MMDuet performs
stronger
Figure 6: Comparison of VideoLL MMDuet, and LiveStar on the MIQ task (also
representative of the COQ task).
33

---

Question: What this showing camern??
Timestamp: 00:00
Ground truth: This person does not show objecct.
M--OnlLe-- Slient. (nothing ourpat)
MMDuett: Slient. (nothing output)
LiveStar: The person is not showing anything to the camern.
Timestamp: 00:07
Ground truth: This person showing a phone.
Videol.LM-Qnline You are in a
MMDuct: person holding mobile phone frout of
LiveStar: The person showing black mobile phone to the eamera.
Timestamp: 00:13
Ground truth: The man showing packet of tissues.
VideoLLM-Onnline You in
MMDuet: person holding black box with white object
LiveStar: The person showing box tissues to camera.
Timestamp: 00:21
Ground truth: This person is showing a black book.
VideoLLLMOnlliee You in gym. person holding book hands. LiveStar: The person is showing book to the camera.
Timestamp:
Ground truth: This person is showing a black
Video1.1 M-Onnline: You gym.
MMDuet: Slient, (nothing output)
LiveStar: The penson is showing black pen to the camera.
This shows that VideoLLM-Onlin performs poorly the FDQ mask, repentedly generating response without understanding the question. MMDuet performs relatively better but still struggles with fine-grained recognition and detection (e.g, at Timestamps 00:13 and 00,30) comparison, LiveStar delivers the best performance.
Comment:
Figure 7: Comparison of VideoLL MMDuet, and LiveStar on the FDQ task.
F
Visualization of LiveStar
develop a web-based demo for LiveStar using Gradio, which enables users to upload arbitrary videos and perform real-time streaming chat. The interface also allows flexible adjustment of inference parameters (e.g., the tunable scaling factor or) to accommodate various application scenarios. The demo interface is illustrated in Figure 8
G Impact Statements
Our proposed LiveStar, pioneering live streaming assistant with temporal decision-making capabili- ties, presents significant socictal implications. On the positive side, LiveStar introduces paradigm
34

---

LiveStar: Pioneering Live Streaming Assistant
System Prompt Quastin (Ootionat
You aie an ropart in ma me eireniiing video dener prion. rull proutde sidno mome
sequent siyy you need to comprettnnt each hore contant in dime while dy dymamiically jenenating concise descrittonss transtiionlll piassea to muintaaan burtual coherennd avoid repuzting a descfibedd contant.
Model Ourpun
poking whitt plostic piaic with piace pleate imaggt groy yolloo nourty.
papes
O X
Frame Nothing
Framed::
connect
botiomel hume..
Frame-t Nuthiing
Frane Nutling
Frame Lt: bebseeuuenty, a used hout the out plostic buttl.
Tunable Sea Fartore
Frome Ln seathin
Foome- ia fixalty, the scnee suteche haind haiding white plantt e with afthaeeres
plaate botta attuched. A green mark appears below.
Clear Output
Start Processing
Figure 8: A web demo of LiveStar. Green points indicate generated outputs, while white denotes output.
shift in online understanding by dynamically determining when and how to generate outputs. This enables efficient processing of real-time online data with reduced latency and computational overhead. This innovation holds promise for applications such real-time online platforms, where it could en- hance viewer engagement through instant content summarization, automated moderation, or adaptive subtitling. Additionally, its ability to prioritize critical events in surveillance systems could improve public safety by enabling faster emergency responses in scenarios like traffic monitoring or crowd management. The technology also has potential in assistive tools for individuals with disabilities, such as real-time visual interpretation or context-aware navigation aids.
However, the technology's adaptive decision logic also introduces risks. When integrated into drones or public cameras, it raises concerns about mass privacy infringement, particularly if deployed without transparent governance. In conclusion, while LiveStar advances real-time online video understanding. its societal adoption demands robust governance frameworks. We advocate for strict regulatory compliance in privacy-sensitive domains and interdisciplinary collaboration to mitigate misuse risks while maximizing its benefits for
applications.
H Prompt
35

---

Prompt to Evaluate Semantic Correctness Using GPT-40 Score
Task Description:
You are an expert in evaluating captions generated for various scenes in videos. I will provide you with the ground truth for each scene caption and the caption generated by the model. You need to score the given caption generated by the model based on following three dimensions and output the respective scores for each dimension:
Semantic Accuracy: Measures the match in core meaning between the generated caption and Ground Truth.
Scoring criteria: 10: Perfect matching with Ground Truth, no deviation. 7-9: Minor semantic differences but core information intact. 4-6: Significant errors omissions in key details. 1-3: Mostly incorrect or irrelevant. O: Completely wrong or unrelated.
2. Language Quality: Evaluates grammatical correctness, word choice, and readability. Scoring criteria: 10: Flawless grammar, precise wording and no redundancy. - 7-9: Minor grammatical slips, but do affect understanding. 4-6: Obvious grammar issues or unclear phrasing. 1-3: Chaotic language, and multiple errors hinder comprehension. - O: Unintelligible symbol-filled.
3. Information Completeness: Assesses breadth and depth of information coverage relative to Ground Truth.
Scoring criteria: 10: Including all key information from Ground Truth. 7-9: Main points covered, but missing minor details. 4-6: Partial key information missing, but retaining the core content. 1-3: Only fragments of information retained. 0: No valid information provided.
Requirements:
will give you the ground truth with "Ground Truth:" and the model-generated caption with "Model Response:".
When outputting the score for the Semantic Accuracy dimension, it must start with "Semantic Accuracy:". When outputting the score for the Language Quality dimension, must start with "Language Quality:". When outputting the score for the Information Completeness dimension, it must start with "Information Completeness." Apart from the scores for the three dimensions, no other characters are allowed in the output. Ground Truth: (ground truth) Model Response: [
response)
36

---

Prompt to Evaluate Semantic Correctness Using GPT-4o Score
Task Description:
You are expert in evaluating video content caption generation. I will provide you with the ground truth of video caption and the caption generated by the model. You need to score the given caption generated by the model based following five dimensions and output the respective scores for each dimension:
1. Writing Logicality: Evaluates causal, temporal, inferential relationships between sentences.
Scoring criteria: 10: Flawless logic with no gaps contradictions. 7-9: Minor logical jumps but generally coherent. 4-6: Weak connections requiring reader inference. 1-3: Frequent illogical breaks contradictions. O: No logical structure.
2. Language Fluency: Assesses natural transitions between sentences.
Scoring criteria: 10: Effortless flow with no awkward phrasing. 7-9: Smooth overall with rare hiccups. 4-6: Choppy transitions, some re-reading needed. 1-3: Relies heavily on forced connectors. Unreadable due to disjointedness.
3. Writing Conciseness: Measures brevity and avoidance of redundancy. Scoring criteria: 10: No redundancy, precise wording. 7-9: Minor repetitions but efficient overall. 4-6: Verbose sections needing trimming. 1-3: Excessive repetition or filler. O: Entirely redundant or irrelevant.
4. Semantic Consistency: Checks uniformity in terminology, pronouns, and tense.
Scoring criteria: 10: Perfect consistency in all elements. 7-9: Slight inconsistencies with no impact. 4-6: Multiple inconsistencies requiring correction. 1-3: Confusing shifts in terms perspective. O: Incoherent due to inconsistency.
5. Narrative Completeness: Evaluates whether the caption covers all key information. Scoring criteria: 10: Including all critical details. 7-9: Covering main points, but having minor gaps possible. 4-6: Missing some key elements. 1-3: Fragmented irrelevant info only. 0: No relevant information. Requirements:
will give you the ground truth with "Ground Truth:" and the model-generated caption with "Model Response:".
2. When outputting the score for the Writing Logicality dimension, it must start with "Writing Logicality:".. When outputting the score for the Language Fluency dimension, it must start with "Language Fluency:"" When outputting the score for the Writing Conciseness dimension, must start with "Writing Conciseness.". When outputting the score for the Semantic Consistency dimension, it must start with "Semantic Consistency:". When outputting the score for the Narrative Completeness dimension, it must start with Narrative Completeness:". Apart from the scores for the five dimensions, no other characters allowed in the output. Ground Truth: (ground_truth)
Model Response:
37

---

Prompt to Generate Second-level Caption
You are an expert in generating descriptions of streaming videos. Now, given all the frames of I-second video, analyze the visual content and generate complete subtitles. Input: real-time frames of video clip or video stream.
Steps: Analyze the video content: identify the main events, characters, scenes, and actions in the video. 2. Extract key information: extract key information from the video that can represent the video content. 3. Generate description: generate concise and rich text description based on the extracted information. 4. Proofreading and optimization: check the accuracy and fluency of the description to ensure that it conforms the video content.
Notes: 1. Make sure the description is directly related to the video content. 2. The description prohibited from using modifiers such adjectives and adverbs. Avoid verbosity and keep it concise. 4. Pay attention to the logic and coherence of the description. 5. It is forbidden to mention content that not mentioned in the video.
Output example: "A basketball player (wearing red jersey) jumps vertically in the middle of the court under the bright court lights, with arms reaching towards the net. The ball spins clockwise in the air, while two defensive players (in blue uniforms) prepare to grab the rebound."
Prompt to Generate Real-Time Narration
Task: Generate fluid, cinematically-cohereee real-time narration that creates continuous visual storytelling. Integrate persistent historical elements with instantaneous observations using temporal reasoning and causal chaining.
Inputs: Historical Context (H): [History)
2. Current Second Data (C): SCurrent_catiions
Current Video Frames
Processing Requirements:
Dynamic Temporal Modeling: Maintain object/interaction state vectors across sliding windows Calculate kinematic signatures (jerk vectors, path integrals) Anticipate action se- quences using momentum calculus Differentiate transient vs sustained states ("progressively veering" vs "abrupt swerve")
2. Scene State Engine: Preserve spatial memory lattice for persistent entities Refresh object kinematics (pose derivatives, interaction potential) Implement occlusion resilience through Markovian tracking Maintain projective geometry constraints for spatial coherence 3. Narrative Flow Synthesis: Forge event domino sequences with causal integrity Em- ploy aspectual verb framing tattaannngoommnening//llllinnino
Apply filmic continuity principles (eyeline matches, temporal ellipses)- Optimize attention allocation through depth- ordered significance
[Subject][action] [prepostionncenniionneell [[envirommett] while [par- allel process], with [situational awareness] (phase indicator)"
Output Format:
Example: Input H: [=[00:00-00:05] Red sedan accelerates past stationary truck, establishing eastward movement") Input C: ["[00:05-00:00) Vehicle decelerating near crosswalk. [00:06- 00:07] Pedestrian initiating crossing sequence"]
Output: "[00:05-00:07] The crimson sedan progressively moderates speed approaching crosswalk, its path converging with pedestrian now committing to traverse, while the historically logged truck persists in the rear-left zone"
38

---

Prompt to Generate Real-Time
Time Narration
Task:
As a real-time narration generator, create vivid and coherent video commentary by combining:
Historical context from previous narration: [Insert past narration)
Current video clip
Per-second captions: (List timestamped captions)
Guidelines:
Temporal continuity: Maintain strict time synchronization between captions (seconds 00:00-
X0:XX) and narration
Context weaving: Blend historical context with new developments while avoiding repetition
Fluency: Use natural transitions between sentences and temporal markers (c.g.. "Meanwhile," " "Suddenly")
Focus: Highlight key actions/objects from captions while maintaining narrative flow
Output Requirements:
1-2 sentences per major event/transition
Active voice and descriptive verbs
Neutral observatory tone without speculation
Example Output Format: "[00:05-00:07] The crimson sedan progressively moderates speed approaching a crosswalk, its path converging with pedestrian now committing to traverse, while the historically logged truck persists in the rear-left zone"
39


Reviews:
[{"review_id": "9l12qqhtvt", "reviewer": "Reviewer_j4Ki", "summary": "In this paper, the authors focus on the video content narration problem and propose LiveStar model. Besides, they also propose OmniStar, a comprehensive dataset for training and benchmarking.", "questions": "Please refer to Strengths And Weaknesses", "limitations": "Please refer to Strengths And Weaknesses", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n1. The paper is well written and easy to follow\n2. The proposed model works well on the proposed benchmark\n\n### Weakness\n1. My main concern is the goal of the paper. The paper focuses on the narrative of video content, but it is only one function of online video understanding, so it is not proper to use 'Live Streaming Assistant' to name a model that can only tell me what happened in the video. The exploration of video narration is a novel idea at the time of VideoLLM-online (2024.1), but I do not think it is meaningful now. There are also some explorations about more capable streaming/online video LLMs for interactive (ViTA[1], MiniCPM-o[2]) and general response decision (Dispider[3]).\n\n2. Following 1, the paper mainly reports the results on the Omni-Star. It is intuitive that a model trained with similar data will perform better, so it is not convincing without additional benchmark results. I'm wondering about the online/streaming video benchmark performance, for example, the OVOBench (mentioned in the introduction but not reported), the OVBench[4], etc.\n\n3. The introduction part is a bit chaotic. There are 4 challenges proposed, but the solution part only considers the first two challenges. Are the latter two solved by the PPL based decision in SVeD? A reorganized introduction would be helpful for understanding.\n\n4. Following 3, I do not think EOS is a big problem that leads to model confusion. It is a simple indicator for salient that any unused token in the LLM vocabulary could serve this role.\n\n \n[1] VITA: Towards Open-Source Interactive Omni Multimodal LLM\n\n[2] MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone\n\n[3] Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction\n\n[4] Online Video Understanding: OVBench and VideoChat-Online", "was_revised": true, "final_justification": "Based on the benchmark results. I tend to keep my score as Borderline reject", "rebuttal": "Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n> [W1] The paper focuses on the narrative of video content, but it is only one function... it is not proper to use 'Live Streaming Assistant' to name a model...\n\nWe appreciate the feedback on the paper’s scope. Our work extends beyond *solely* video narration, addressing 15 diverse real-world scenarios and **five distinct evaluation tasks**: (1) real-time narration, (2) online temporal grounding, (3) storyline QA (multi-temporal answers), (4) streaming QA, and (5) multi-turn interactive QA. Detailed definitions and cases are provided in App. F and Figs. 7–9. LiveStar achieves an average **19.5%** improvement in semantic correctness with **18.1%** reduced timing difference versus existing online Video-LLMs while improving throughput by **12.0% FPS** across 5 tasks. This demonstrates its capability as a comprehensive assistant for real-world online video understanding (including narration, grounding, and QA), and is therefore called 'Live Streaming Assistant'. We acknowledge that the illustrative focus on narration in Figs. 1–2 may have caused confusion; this will be clarified in the revision.\n\n> [W2]  the paper mainly reports the results on the Omni-Star... the OVBench[4], etc.\n\nWe agree broader benchmarking strengthens validation. However, beyond OmniStar (our primary benchmark due to its alignment with real-world streaming tasks), we reported state-of-the-art results on **Ego4D Narration** (Table 3) and **SVBench** (Table 6). Per the reviewer’s suggestion, we conducted additional experiments on OVBench: LiveStar significantly outperforms VideoLLM-online, MovieChat, and Flash-VStream (reported in the original paper), though it trails VideoChat-Online marginally. We attribute this to two factors:\n\n- LiveStar optimizes for *text-generation tasks* (e.g., narration, QA), while OVBench emphasizes multiple-choice QA.\n- OVBench evaluates *passive-response models* (no output-timing decisions), whereas LiveStar and VideoLLM-online are *active-response* systems.\n  Task objectives thus differ, and we will discuss this nuance in the revision.\n\n| Model            | FP        |           |           | THV       |           |           | PM        |           |           | SP        |           | STP       |           | TP        |           |           | Avg       |\n| ---------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|                  | AA        | GSP       | MP        | AP        | SV        | OP        | AR        | PR        | TR        | AL        | OP        | AT        | OT        | AS        | SL        | OES       |           |\n| VideoLLM-online  | 0.00      | 1.80      | 20.90     | 5.20      | 5.90      | 32.60     | 0.00      | 2.30      | 26.70     | 0.60      | 26.60     | 0.90      | 19.90     | 0.90      | 1.70      | 8.30      | 9.60      |\n| MovieChat        | 23.10     | 27.50     | 23.60     | 58.40     | 43.90     | 40.30     | 25.60     | 31.10     | 23.90     | 26.90     | 39.60     | 24.40     | 28.90     | 29.30     | 25.50     | 21.90     | 30.90     |\n| Flash-VStream    | 26.90     | 37.60     | 23.90     | 60.10     | 41.90     | 40.00     | 23.40     | 35.30     | 26.10     | 24.70     | 28.80     | 27.00     | 21.40     | 29.80     | 25.60     | 26.80     | 31.20     |\n| LiveStar         | 37.50     | 57.06     | **25.00** | 48.21     | 50.69     | **69.15** | 34.92     | **65.58** | 52.36     | 34.85     | **41.64** | 69.49     | 46.58     | 65.07     | **29.66** | 22.71     | 46.90     |\n| VideoChat-Online | **64.10** | **59.70** | 16.60     | **63.10** | **58.30** | 62.80     | **42.20** | 54.40     | **70.60** | **54.10** | 24.80     | **88.70** | **48.50** | **73.00** | 25.90     | **71.70** | **54.90** |\n\n> [W3]  There are 4 challenges proposed, but the solution part only considers the first two challenges...\n\nThe Introduction presents **two core challenges** (not four), with *four sub-issues* arising from EOS-based decision methods under the first challenge. Since there is no EOS token training, our PPL-based SVeD framework resolves **all four issues** (including the latter two noted by the reviewer). We will restructure the Introduction to summarize and distinguish challenges and issues, improving coherence.\n\n> [W4]  I do not think EOS is a big problem that leads to model confusion. It is a simple indicator for salient that any unused token in the LLM vocabulary could serve this role.\n\nWe agree that any unused vocabulary token could technically serve as an EOS signal. However, in VideoLLM-online’s implementation, the EOS role was assigned to `','` (comma), a high-frequency token in text that risks conflating linguistic and timing functions. We will remove the \"*Vocabulary Confusion*\" claim in revision, as the core concern is implementation-driven rather than theoretical.\n\nFinally, we emphasize our deep respect for VideoLLM-online and EOS-based works: they pioneered this space and inspired our research. Our goal is not to replace EOS-based methods, but to offer a complementary approach; both paradigms may prove synergistic in future work. \n\n---\n\nThank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work."}, {"review_id": "Ri0WHObFtp", "reviewer": "Reviewer_wXqt", "summary": "This paper introduces LiveStar, a new live streaming assistant designed to enhance real-time online video understanding. LiveStar proposed three innovations: a training strategy for incremental video-language alignment, a response-silence decoding framework for optimal response timing, and memory-aware acceleration for efficient online inference. The authors also present the OmniStar dataset, which includes 15 diverse real-world scenarios and five evaluation tasks for online video understanding.", "questions": "1. **Novelty Clarification**: You mentioned that LiveStar addresses limitations in existing online Video-LLMs. Could you provide more specific details on how LiveStar's approach to adaptive streaming decoding and response-silence framework is distinct from previous models like VideoLLM-online, VideoLLM-MoD, and LION-FS?\n\n2. Evaluation Metrics: You introduced new metrics like SemCor and TimDiff. Could you elaborate on how these metrics are calculated and why they are more effective than traditional metrics in evaluating online video understanding?", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. **Comprehensive Dataset**: The OmniStar dataset covers diverse real-world scenarios and tasks, providing a robust benchmark for online video understanding.\n2. **Performance Improvement**: LiveStar demonstrates significant improvements in semantic correctness and inference speed compared to existing models.\n\nWeaknesses:\n1. **Limited Benchmarking**: The paper does not evaluate LiveStar on widely recognized video benchmarks like MVBench, VideoMME, and LVBench, which could provide a more comprehensive assessment of its capabilities.\n2. **Incremental Innovation**: The work may lack groundbreaking innovation, as similar approaches have been explored in existing models like VideoLLM-online.\n3. **Evaluation Scope**: The evaluation focuses primarily on the OmniStar dataset, potentially limiting the generalizability of the results across other datasets and tasks.", "was_revised": true, "final_justification": "I believe there is no fundamental difference between offline video understanding and online video understanding. If a model can perform well in online video understanding, it should also excel in offline video understanding. Similarly, only by achieving good performance in offline video understanding can we subsequently achieve good performance in online video understanding.", "rebuttal": "Many thanks to your valuable comments and questions, which help us a lot to improve our work. \n\n> [W1 & W3] Limited Benchmarking: The paper does not evaluate LiveStar on video benchmarks like MVBench, VideoMME, and LVBench... primarily on the OmniStar dataset...\n\nWe appreciate the reviewer's suggestion regarding broader benchmarking. We would like to clarify that our evaluation already encompasses **three benchmarks** covering diverse online video understanding scenarios and tasks: OmniStar (Tables 1 & 2), Ego4D Narration (Table 3), and SVBench (Table 6). We are grateful that the reviewer recognizes the comprehensiveness of OmniStar, stating it \"covers diverse real-world scenarios and tasks, providing a robust benchmark,\" which further underscores the breadth of our primary evaluation.\n\nCrucially, the benchmarks highlighted by the reviewer (MVBench, VideoMME, LVBench) are primarily designed for **offline video understanding**, where models analyze complete videos. In contrast, our work focuses explicitly on **online/streaming video understanding**, a distinct paradigm requiring capabilities like real-time processing, active response triggering, and continuous comprehension of streaming inputs. These capabilities are central to our contribution but are not the primary targets of the mentioned offline benchmarks.\n\nFurthermore, it is standard practice in online Video-LLMs that models like VideoLLM-online, VideoLLM-MoD, and LION-FS are only evaluated on benchmarks specifically designed for the online setting, as reflected in their papers.\n\nNevertheless, to further validate the generalizability of LiveStar and address the reviewer's concern, we have conducted additional evaluations on MVBench, LongVideoBench, and VideoMME. \n\n| Benchmark    | MVBench   |           |       | LongVideoBench (64 frames) |          | VideoMME (64 frames) |          |\n| ------------ | --------- | --------- | ----- | -------------------------- | -------- | -------------------- | -------- |\n| **Settings** | 16 frames | 64 frames | 1fps  | w subs                     | w/o subs | w subs               | w/o subs |\n| **Results**  | 69.05     | 69.77     | 66.95 | 57.0                       | 56.3     | 64.4                 | 60.8     |\n\nThe results demonstrate that LiveStar maintains strong performance and achieves competitive results even on these traditional offline video understanding benchmarks. To further enrich the evaluation, we have also added results on the OVBench benchmark, as detailed in our response to Reviewer j4Ki.\n\n> [W2 & Q1] Novelty Clarification: The work may lack groundbreaking innovation, as similar approaches have been explored in existing models like VideoLLM-online.\n\nThe novelty of LiveStar lies in its completely shift away from the paradigm of models based on EOS tokens. We introduce a new response-silence framework built on two innovations: Streaming Causal Attention Masks (SCAM) for training and Streaming Verification Decoding (SVeD) for inference.\n\n1. Training Paradigm:\n\n   - *EOS based Models*: These models are trained to solve a classification-like task for every frame: either generate a response or predict a special EOS token. For the vast majority of frames (silent intervals), the training objective is to map a visually rich frame to a single, static EOS token. It forces the model to learn a trivial, non-descriptive mapping for most of its training data.\n\n   - *LiveStar (with SCAM)*: We completely eliminate the EOS token from our training process. Instead, we use SCAM to train the model on interleaved frame-caption sequences. As described in Section 3.1 and illustrated in Figure 1(d), our model is trained to always generate meaningful, temporally consistent responses. It addresses the issues of pre-training misalignment and vocabulary confusion, encouraging the model to maintain its comprehension capabilities.\n\n2. Inference Mechanism:\n   - *EOS based Models*: During inference, these models must perform a full forward pass and a decoding step for every single frame to decide whether to output a response or the EOS token. Although generating an EOS token may seem sample, in high-frame-rate videos where most frames are silent, this still incurs the cost of generating a token per frame, leading to inefficiencies in overall inference performance.\n\n   - *LiveStar (with SVeD)*: Our SVeD mechanism, detailed in Section 3.2 and Algorithm 1, introduces a more efficient and intelligent decision-making process. For silent frames, we do not decode anything. We perform a single, lightweight forward pass to verify if the previously generated response is still semantically valid by checking its perplexity. This verification step is faster than decoding an EOS token (as empirically shown in Figures 7 & 8). A full decoding process is triggered only when a semantic shift is detected, which leads to faster overall inference.\n\n> [Q3] Evaluation Metrics: You introduced new metrics like SemCor and TimDiff. Could you elaborate on how these metrics are calculated...\n\n**Calculation Methods**\nFor each video, we have:\n\n- A set of ground-truth (GT) semantic clips  $G = \\lbrace g_1, g_2, \\dots, g_N \\rbrace$, where each clip $g_i$ is defined by: a start time $t_{\\text{start},i}$, an end time $t_{\\text{end},i}$ and a GT caption.\n\n- A set of model-generated responses $R = \\lbrace r_1, r_2, \\dots, r_M \\rbrace$, where each response $r_j$ has an associated response time $t_{\\text{resp},j}$.\n\nFor each $ g_i \\in G $, define the set of matching responses as:\n$$\nM_i = \\lbrace r_j \\in R \\mid t_{\\text{resp},j} \\in [t_{\\text{start},i},\\ t_{\\text{end},i}] \\rbrace\n$$\n\n\n- If $ |M_i| = 0 $, no response is generated for $ g_i $; in this case, we assign an **empty string** $ r_j = $ ' '  as the placeholder response for evaluation.\n- If $ |M_i| \\geq 1 $, we select the **first matching response** $ r_j \\in M_i $ (i.e., with minimal $ t_{\\text{resp},j} $) as the representative output.\n\nAs detailed in Appendix G, Semor and SumFluen is calculated using GPT-4o as an evaluator. The process is as follows:\n\n1. **SemCor (Semantic Correctness):** This metric evaluates the quality of an individual response generated for **a specific semantic clip**. We prompt GPT-4o to score the pair $(r_j, g_i)$ on three dimensions (each on a 0–10 scale): *Semantic Accuracy*, *Language Quality*, and *Information Completeness*.\n\n\n2. **SumFluen (Summarize Fluency):** This metric evaluates the holistic narrative quality of the model’s complete output sequence, treating all responses as a unified story. Let the **full model narrative** be the concatenation of all responses in temporal order:\n   $$\n   \\mathcal{N}\\_{\\text{model}} = \\text{Concat}\\_{j=1}^M (r_j)\n   $$\n   Similarly, construct the **ground-truth narrative** by concatenating all GT captions in temporal order:\n   $$\n   \\mathcal{N}\\_{\\text{GT}} = \\text{Concat}\\_{i=1}^N (g_i)\n   $$\n   We evaluate $ \\mathcal{N}\\_{\\text{model}} $ against $ \\mathcal{N}\\_{\\text{GT}} $ by prompting GPT-4o to assess them on five storytelling-relevant dimensions(each on a 0–10 scale): *Writing Logicality*, *Language Fluency*, *Writing Conciseness*, *Semantic Consistency*, and *Narrative Completeness*.\n\n\nThe metrics TimRedun, TimCover and TimDiff are calculated as:\n\n3. **TimRedun (Timing Redundancy)**\n   Measures the deviation from the ideal of exactly one response per semantic clip:\n\n$$\n\\text{TimRedun} = \\frac{1}{N} \\sum_{i=1}^N \\left||M_i| - 1\\right|\n$$\n\n4. **TimCover (Timing Coverage)**\n   Indicates the proportion of semantic clips that were covered by at least one response:\n\n$$\n\\text{TimCover} = \\frac{1}{N} \\sum_{i=1}^N \\mathbb{1}[|M_i| > 0]\n$$\n\n5.  **TimDiff (Timing Difference)**\n    Captures the cumulative response latency for all matched outputs within a semantic clip, while penalizing missed clips by their full duration:\n\n$$\n\\text{TimDiff} = \\frac{1}{N} \\sum_{i=1}^N \\left( \\mathbb{I}\\big[|M_i| = 0\\big] \\cdot \\big(t_{\\text{end},i} - t_{\\text{start},i}\\big) + \\mathbb{I}\\big[|M_i| > 0\\big] \\cdot \\sum_{r_j \\in M_i} \\big(t_{\\text{resp},j} - t_{\\text{start},i}\\big) \\right)\n$$\n\n**Advantages:**\nThe key distinction of our evaluation framework lies in its holistic assessment of both temporal alignment and semantic quality, specifically designed for **proactive response models**. This stands in stark contrast to traditional paradigms that evaluate models in a passive or offline manner.\n\n- Integrated Temporal and Semantic Evaluation: Traditional offline metrics (e.g., BLEU, CIDEr) or even some online QA benchmarks(e.g., OVBench) focus solely on the semantic correctness of an answer, assuming the response timing is either given or irrelevant. Many streaming video benchmarks also follow a passive paradigm, where the model is prompted at a pre-defined timestamp. While this setup tests understanding at a specific moment, it overlooks a crucial capability for real-time assistants: **the ability to autonomously decide when to speak.**\n\n- Our framework addresses this gap by introducing integrated temporal and semantic evaluation. Metrics like TimDiff and TimRedun quantify the model's temporal decision-making: *Did the model respond at the right time?* *Did it respond too much?* Meanwhile, SemCor and SumFluen evaluate the quality of what was said at that moment. By combining these, our metrics move beyond simple content accuracy to provide a multi-faceted evaluation of a model's comprehensive ability to perceive, reason, and act in a continuous, dynamic environment. This makes them uniquely suited for benchmarking the next generation of proactive online video understanding models.\n\n---\n\nThank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work."}, {"review_id": "hMaGgv0D3L", "reviewer": "Reviewer_aC31", "summary": "The paper introduces LiveStar, an online video understanding framework that addresses key limitations of existing Video-LLMs, such as poor real-time response timing and narrative inconsistency. LiveStar employs adaptive streaming decoding with: (1) a training strategy using Streaming Causal Attention Masks for incremental video-language alignment, (2) a Streaming Verification Decoding framework to determine optimal response timing, and (3) memory-aware acceleration via peak-end compression and KV caching. The authors also present OmniStar, a comprehensive dataset covering real-world scenarios. Experiments show LiveStar achieves state-of-the-art performance compared to other methods. Key contributions include a novel response-silence paradigm, a diverse benchmark, and practical deployment capabilities for long-video streams.", "questions": "1. The SVeD decoding threshold parameter α is shown to be sensitive. Is there a way to dynamically learn or adapt this threshold rather than fixing it empirically?\n\n2. The perplexity is used to determine whether a response should be generated, then a complete response must be produced at each forward pass. Wouldn't this significantly slow down the inference speed, since generating a full response involves producing multiple tokens, whereas generating a single [EOS] token allows for much faster termination?", "limitations": "Yes, in appendix", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The technical contributions are well-motivated and clearly articulated. LiveStar effectively integrates innovations in training and inference to enable efficient, coherent, and temporally appropriate responses in streaming video settings.\n\n2. Using perplexity instead of having the model explicitly generate an EOS token to decide whether to update the response is a brilliant idea. It effectively transforms the proactive response problem, thereby mitigating the disruption to the original capabilities of the MLLM.\n\n## Weakness\n1. The authors have only demonstrated the effectiveness of their approach on their proposed benchmark and SVBench. It is recommended to provide evaluation metrics on additional online video benchmarks (e.g., OVBench) and offline video benchmarks (e.g., videomme) to better illustrate the generalizability of the method.\n\n2. Compressing frames into 16 tokens may miss subtle visual cues (e.g., small object interactions, fine-grained actions), as acknowledged in the limitations.\n\n3.  A direct comparison with recent models like StreamMind, LION-FS and VideoChat-Online would better position LiveStar.", "was_revised": true, "final_justification": "In my opinion, this paper does not have any obvious flaws that would lead to a direct rejection. After carefully reading the author's response, I believe that although LiveStar still has some flaws, the relatively weak evaluation in the initial version has become quite solid after the reviewers' Rebuttal, and the active response design in this paper provides a new idea for such models. Althougth the performance improvement it achieves and the novelty of its approach are not particularly high, it also exceeds the acceptance bar.", "rebuttal": "Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n> [W1] The authors have only demonstrated the effectiveness of their approach on their proposed benchmark and SVBench... It is recommended to provide ... additional online video benchmarks (e.g., OVBench) and offline video benchmarks (e.g., videomme) ...\n\nWe appreciate the reviewer's suggestion for broader benchmarking. Our initial evaluation already spanned **three distinct benchmarks** designed to cover a variety of online video understanding scenarios: our proposed OmniStar benchmark (Tables 1 & 2), Ego4D Narration (Table 3), and SVBench (Table 6). These benchmarks collectively assess the core capabilities of proactive, real-time response generation that are central to our work.\n\nNevertheless, we fully agree that validation on additional standard benchmarks is important. Following your suggestion, we have conducted new experiments on both online and offline benchmarks to further demonstrate the generalizability of LiveStar.\n\n**Performance on Additional Online Benchmark (OVBench):**\nwe conducted additional experiments on OVBench: LiveStar significantly outperforms VideoLLM-online, MovieChat, and Flash-VStream (reported in the original paper), though it trails VideoChat-Online marginally. We attribute this to two factors:\n\n- LiveStar optimizes for *text-generation tasks* (e.g., narration, QA), while OVBench emphasizes multiple-choice QA.\n- OVBench evaluates *passive-response models* (no output-timing decisions), whereas LiveStar and VideoLLM-online are *active-response* systems.\n  Task objectives thus differ, and we will discuss this nuance in the revision.\n\n| Model   | FP  |  |  | THV |  |  | PM  |  |  | SP  |  | STP |  | TP  |  |  | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|   | AA  | GSP | MP  | AP  | SV  | OP  | AR  | PR  | TR  | AL  | OP  | AT  | OT  | AS  | SL  | OES |  |\n| VideoLLM-online  | 0.00   | 1.80   | 20.90  | 5.20   | 5.90   | 32.60  | 0.00   | 2.30   | 26.70  | 0.60   | 26.60  | 0.90   | 19.90  | 0.90   | 1.70   | 8.30   | 9.60   |\n| MovieChat  | 23.10  | 27.50  | 23.60  | 58.40  | 43.90  | 40.30  | 25.60  | 31.10  | 23.90  | 26.90  | 39.60  | 24.40  | 28.90  | 29.30  | 25.50  | 21.90  | 30.90  |\n| Flash-VStream | 26.90  | 37.60  | 23.90  | 60.10  | 41.90  | 40.00  | 23.40  | 35.30  | 26.10  | 24.70  | 28.80  | 27.00  | 21.40  | 29.80  | 25.60  | 26.80  | 31.20  |\n| LiveStar   | 37.50  | 57.06  | **25.00** | 48.21  | 50.69  | **69.15** | 34.92  | **65.58** | 52.36  | 34.85  | **41.64** | 69.49  | 46.58  | 65.07  | **29.66** | 22.71  | 46.90  |\n| VideoChat-Online | **64.10** | **59.70** | 16.60  | **63.10** | **58.30** | 62.80  | **42.20** | 54.40  | **70.60** | **54.10** | 24.80  | **88.70** | **48.50** | **73.00** | 25.90  | **71.70** | **54.90** |\n\n**Performance on Additional Offline Benchmark (VideoMME):**\n\n| Benchmark  | VideoMME (64 frames) |   |\n|---|---|---|\n| **Settings**  | w subs   | w/o subs|\n| **Results**   | 64.4  | 60.8 |\n\n\nThese results on both OVBench and VideoMME confirm that LiveStar not only excels at its primary task of online, proactive video understanding but also maintains robust foundational capabilities. For a more detailed discussion of our evaluations on other benchmarks, such as MVBench and LongVideoBench, we invite you to see our response to Reviewer wXqt.\n\n> [Q2] Compressing frames into 16 tokens may miss subtle visual cues (e.g., small object interactions, fine-grained actions), as acknowledged in the limitations.\n\nYou are absolutely correct, and we acknowledge this trade-off in our limitations section. Compressing each frame into a fixed-size representation of 16 tokens is a deliberate design choice aimed at balancing computational feasibility and semantic comprehension, which is crucial for real-time processing of long video streams.\n\nTo enable real-time, long-context video understanding, LiveStar represents video frames with 16 tokens, which can efficiently process more than 10 minutes of video within an 8K context window. This compact representation supports high-level semantic tasks like narration and scene comprehension, while future work will explore adaptive tokenization to allocate more detail to semantically rich frames without sacrificing overall efficiency.\n\n> [Q3] A direct comparison with recent models like StreamMind, LION-FS and VideoChat-Online would better position LiveStar.\n\nWe appreciate this valuable suggestion. We recognize the importance of benchmarking against the latest models to accurately position LiveStar within the field.\n\n**Comparison with StreamMind and LION-FS:**\nUnfortunately, as of now, both StreamMind and LION-FS had not publicly released their model weights or inference code. This prevented us from conducting a direct, head-to-head evaluation on our proposed OmniStar benchmark or other benchmarks like OVBench.\n\nHowever, to provide a point of reference, we have included the performance metrics reported in their original papers on the Ego4D Narration Stream benchmark, a task that all three models (LiveStar, StreamMind, and LION-FS) have been evaluated on. As shown in the table below:\n\n| Model | PPL↓ | TimeDiff↓ | Fluency↑ | TokAcc↑ |\n|----|----|----|----|----|\n| LION-FS  | 2.09  | 2.15   | 46.1% | 52.4%   |\n| StreamMind  | 2.02  | 1.89   | 60.2% | **77.3%** |\n| **LiveStar**   | **1.97**  | **1.76**   | - | 61.1% |\n\nThis comparison, while indirect, shows that LiveStar achieves the lowest PPL and TimeDiff, while maintaining strong competitiveness in Token Accuracy, highlighting the effectiveness of our approach.\n\n**New Evaluation on VideoChat-Online:**\nWe were able to conduct a new evaluation with VideoChat-Online, which is open-sourced. We tested it on our OmniStar-RNG (Real-time Narration Generation) task. It is important to note that VideoChat-Online is not designed with a proactive response mechanism (i.e., it cannot autonomously decide when to generate a response). Therefore, we could only evaluate it in an offline setting, where we provide it with all video frames up to a pre-specified response timestamp and prompt it for a description.\n\nThe results are as follows:\n\n| Model  | PPL↓  | TokAcc↑ | SemCor↑ | SumFluen↑ |\n|---|---|---|---|---|\n| VideoChat-Online   | 5.62  | 0.59 | 3.37 | 1.65   |\n| **LiveStar** | **5.14**  | **0.62** | **4.62** | **4.55**   |\n\nThe results show that even in a controlled offline setting, LiveStar significantly outperforms VideoChat-Online across all metrics. This demonstrates LiveStar's superior foundational ability for video-language alignment and coherent text generation.\n\n> [Q1] The SVeD decoding threshold parameter α is shown to be sensitive. Is there a way to dynamically learn or adapt this threshold rather than fixing it empirically?\n\nWe agree that a dynamic or learnable threshold could offer significant advantages in terms of adaptability and robustness. However, we acknowledge that designing such a dynamic mechanism is a non-trivial research challenge that requires careful consideration of the trade-offs between complexity and performance. \n\nSpecifically, α needs to be tailored to different task demands—for instance, being more sensitive for online live streaming and more stable for sports summarization. Additionally, since models of different scales exhibit distinct perplexity distributions, α could be normalized based on baseline perplexity. \n\nTo enable more adaptive response strategies for streaming video understanding, we will explore making the threshold parameter α dynamically adjustable in the future. But for the current work, we opted for a fixed, empirically-tuned α to establish a clear and reproducible baseline for our core contributions (SCAM and SVeD).\n\n> [Q2] The perplexity is used to determine whether a response should be generated, then a complete response must be produced at each forward pass. Wouldn't this significantly slow down the inference speed ...\n\nThank you for raising this critical point about inference efficiency. There appears to be a slight misunderstanding of how our Streaming Verification Decoding (SVeD) mechanism operates, which we are happy to clarify. The key is that we do not generate a complete new response at every forward pass.\n\n1. \"Generate\" Case: When SVeD's verification step determines that a new response is needed (i.e., the perplexity exceeds a threshold), we then perform a full decoding process to generate a new, complete caption. This happens only at moments of **significant changes in the semantic clip**.\n\n2. \"Silence\" Case: For the majority of incoming frames where the scene is stable, SVeD's verification check passes. In this \"silence\" case, the process is lightweight: We perform a single forward pass to **compute the logits** for the previously generated caption based on the updated context (which now includes the new frame). We use these logits to calculate the perplexity. Crucially, **no decoding or token generation occurs**, which is the main time-consuming step. We simply verify that the existing caption is still valid.\n\nIn summary, generating EOS still requires **a full forward pass plus at least one decoding step**. Our SVeD in the silent case requires **only the forward pass**. We also provide direct empirical evidence for this in our paper. Appendix E.2 and Figures 7 & 8 explicitly visualize this efficiency. The green \"SVeD\" curve (representing the \"silence\" case) has a lower inference time compared to the \"One Step DD\" curve (which simulates the generation of a single EOS token).\n\n---\nThank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work."}, {"review_id": "gw35AUE5pV", "reviewer": "Reviewer_dLwS", "summary": "This paper introduces LiveStar, a live streaming assistant designed for real-world online video understanding. It addresses the limitations of existing online Video Large Language Models (Video-LLMs) that struggle with simultaneously processing continuous frame-by-frame inputs and determining optimal response timing, which often compromises real-time responsiveness and narrative coherence. LiveStar proposes SCAM to preserve consistency across dynamically evolving frame sequences. And the paper introduces OmniStar, a comprehensive dataset for training and benchmarking online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-art performance compared to existing online Video-LLMs.", "questions": "Please see the weaknesses. If the authors adequately address these concerns in their rebuttal, I will reconsider my score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper makes substantial contributions by establishing a comprehensive framework for an online video assistant, encompassing both methodological innovations and dataset construction. And the paper conducts extensive experimental validations that rigorously demonstrate the effectiveness of LiveStar.\n\n2. Unlike previous online Video-LLMs rely on learning the EOS token, this paper proposes SCAM to solve the challenge of response-silence imbalance and consecutive frame inconsistency, which is interesting.\n\n3. LiveStar consistently outperforms other online Video-LLMs, showing significant improvements in semantic correctness and reduced timing differences.\n\n**Weaknesses:**\n\n1. In the ablation of Impact of Response-Silence Threshold, the divergent trends in the three metrics with varying decoding thresholds need a more rigorous theoretical analysis.\n\n2. The calculation methods for the evaluation metrics have not been provided. I want to know the specific computational procedures for these metrics.\n\n3. In your paper, you suggest that the EOS token may lead to pre-training misalignment and vocabulary confusion issues, which appears to be an intuitive conjecture. However, modern large language models (LLMs) possess substantial comprehension capabilities, meaning the EOS token might not necessarily cause these problems or may only marginally impair their conversational performance. Could this issue be empirically investigated through experimentation?\n\n4. In Table 1 (online split), the other models were not trained on the dataset’s training set. Could this introduce potential fairness issues in the comparative evaluation?", "was_revised": true, "final_justification": "After reviewing the performance of Livestar on additional benchmarks and considering the feedback from other reviewers, I maintain my positive score. Further analysis of the results on additional benchmarks requires additional refinement in future versions of this work.", "rebuttal": "Many thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n> [W2]The calculation methods for the evaluation metrics have not been provided. I want to know the specific computational procedures for these metrics.\n\nFirst, We would like to address Weaknesses 2 by introducing the evaluation metric we proposed, as this will help clarify the context and facilitate a better understanding of Weaknesses 1.\n\nFor each video, we have:\n\n- A set of ground-truth (GT) semantic clips  $G = \\lbrace g_1, g_2, \\dots, g_N \\rbrace$, where each clip $g_i$ is defined by: a start time $t_{\\text{start},i}$, an end time $t_{\\text{end},i}$ and a GT caption.\n\n- A set of model-generated responses $R = \\lbrace r_1, r_2, \\dots, r_M \\rbrace$, where each response $r_j$ has an associated response time $t_{\\text{resp},j}$.\n\nFor each $ g_i \\in G $, define the set of matching responses as:\n$$\nM_i = \\lbrace r_j \\in R \\mid t_{\\text{resp},j} \\in [t_{\\text{start},i},\\ t_{\\text{end},i}] \\rbrace\n$$\n\n- If $ |M_i| = 0 $, no response is generated for $ g_i $; in this case, we assign an **empty string** $ r_j = $ ' '  as the placeholder response for evaluation.\n- If $ |M_i| \\geq 1 $, we select the **first matching response** $ r_j \\in M_i $ (i.e., with minimal $ t_{\\text{resp},j} $) as the representative output.\n\nAs detailed in Appendix G, SemCor and SumFluen is calculated using GPT-4o as an evaluator. The process is as follows:\n\n1. **SemCor (Semantic Correctness):** This metric evaluates the quality of an individual response generated for **a specific semantic clip**. We prompt GPT-4o to score the pair $(r_j, g_i)$ on three dimensions (each on a 0–10 scale): *Semantic Accuracy*, *Language Quality*, and *Information Completeness*.\n\n\n2. **SumFluen (Summarize Fluency):** This metric evaluates the holistic narrative quality of the model’s complete output sequence, treating all responses as a unified story. Let the **full model narrative** be the concatenation of all responses in temporal order:\n    $$\n    \\mathcal{N}\\_{\\text{model}} = \\text{Concat}\\_{j=1}^M (r_j)\n    $$\n    Similarly, construct the **ground-truth narrative** by concatenating all GT captions in temporal order:\n    $$\n    \\mathcal{N}\\_{\\text{GT}} = \\text{Concat}\\_{i=1}^N (g_i)\n    $$\n    We evaluate $ \\mathcal{N}\\_{\\text{model}} $ against $ \\mathcal{N}\\_{\\text{GT}} $ by prompting GPT-4o to assess them on five storytelling-relevant dimensions(each on a 0–10 scale): *Writing Logicality*, *Language Fluency*, *Writing Conciseness*, *Semantic Consistency*, and *Narrative Completeness*.\n\nThe metrics TimRedun, TimCover and TimDiff are calculated as:\n\n3. **TimRedun (Timing Redundancy)**\n   Measures the deviation from the ideal of exactly one response per semantic clip:\n\n$$\n\\text{TimRedun} = \\frac{1}{N} \\sum_{i=1}^N \\left||M_i| - 1\\right|\n$$\n\n4. **TimCover (Timing Coverage)**\n   Indicates the proportion of semantic clips that were covered by at least one response:\n\n$$\n\\text{TimCover} = \\frac{1}{N} \\sum_{i=1}^N \\mathbb{1}[|M_i| > 0]\n$$\n\n5.  **TimDiff (Timing Difference)**\n    Captures the cumulative response latency for all matched outputs within a semantic clip, while penalizing missed clips by their full duration:\n\n$$\n\\text{TimDiff} = \\frac{1}{N} \\sum_{i=1}^N \\left( \\mathbb{I}\\big[|M_i| = 0\\big] \\cdot \\big(t_{\\text{end},i} - t_{\\text{start},i}\\big) + \\mathbb{I}\\big[|M_i| > 0\\big] \\cdot \\sum_{r_j \\in M_i} \\big(t_{\\text{resp},j} - t_{\\text{start},i}\\big) \\right)\n$$\n\n> [W1]In the ablation of Impact of Response-Silence Threshold, the divergent trends in the three metrics with varying decoding thresholds need a more rigorous theoretical analysis.\n\nFrom a theoretical standpoint, an ideal model should **trigger a response only once at the first frame of each semantic clip** and remain silent otherwise. However, in our proposed LiveStar model, the core of our SVeD mechanism lies in its decoding condition: \n\n$$\n\\text{PPL}\\_{\\text{current}} > \\alpha \\cdot \\text{PPL}\\_{\\text{ref}}\n$$\n\nHere, the tunable scaling factor α directly influences the model's sensitivity to semantic changes. A higher α value raises the perplexity threshold required to trigger a new response, which means the model will wait for a more significant or prolonged semantic shift before acting. \n\nSpecifically, when α is small, the model tends to decode more frequently, potentially generating an output at every frame in the extreme case. This leads to an increased number of unnecessary responses within each semantic clip, with responses occurring earlier in time. As a result, the temporal evaluation metrics reflect a **smaller TimDiff** , a **larger TimRedun**, and a **higher TimCover** . In contrast, as α increases, the model becomes more conservative in decoding, resulting in fewer outputs and delayed response timings. In the extreme case, no output may be generated at all. Under such conditions, both **TimRedun and TimCover decrease**, while **TimDiff increases** significantly—particularly when a semantic clip receives no response, leading to a substantial penalty.\n\nThe experimental results, computed according to the formulas defined in **Answer 1**, align perfectly with the above theoretical analysis, thereby validating the role of α in regulating the model's response behavior.\n\n> [W3]In your paper, you suggest that the EOS token may lead to pre-training misalignment and vocabulary confusion issues ... Could this issue be empirically investigated through experimentation?\n\nWe agree that modern LLMs are remarkably robust and can adapt to various fine-tuning objectives. Our intention was not to suggest that the EOS-based paradigm is fundamentally flawed, but rather to explore an alternative that might better align with the core strengths of vision-language models in a streaming context.\n\nFor pre-training misalignment, our core idea is to better align the fine-tuning objective with the model's pre-training. In online video, a majority of frames are silent. The EOS paradigm trains the model to map these rich visual inputs to a single, non-descriptive EOS token. While effective, this differs from the pre-training goal of generating descriptive text. Our SCAM training strategy, by contrast, consistently focuses the model on the descriptive task by only computing loss on meaningful frame-caption pairs. We believe this contributes to LiveStar's strong performance on third-party benchmarks like Ego4D (Table 3), where it surpasses other online assistants.\n\nFor vocabulary confusion, we agree that this point requires nuance. Our concern was more practical than theoretical, noting that VideoLLM-online’s implementation used common token `','` (comma) for the EOS role. To avoid overgeneralization, we will remove this claim in our revision and clarify that the issue is implementation-dependent.\n\nFinally, we wish to reiterate our deep respect for the pioneering EOS-based works that inspired this research. Our goal is to propose a complementary approach that contributes to the diversity of methods in this exciting field. We believe that exploring different paradigms is valuable, and we sincerely thank you for helping us improve our work.\n\n> [W4]In Table 1 (online split), the other models were not trained on the dataset’s training set. Could this introduce potential fairness issues in the comparative evaluation?\n\nWe strictly separated the training and testing datasets to ensure a fair evaluation protocol. The model was first trained using our proposed SCAM to learn the alignment between visual and linguistic representations, without any exposure to test data during training.\n\nTo provide a more direct and fair comparison of the models' intrinsic capabilities, we conducted evaluations on a pre-existing, third-party benchmark where all models are on a level playing field. As shown in Table 3 (Ego4D Narration Stream benchmark), LiveStar significantly outperforms all other online assistants.\n\nFurthermore, we evaluated this version of LiveStar (without fine-tuning on OmniStar) on the OmniStar-RNG task. The results are as follows: \n| **Method** | **Offline Evaluation (Fixed decoding)**  |   |   |   | **Online Evaluation (Response-silence decoding)**  |     |     |     |     |\n|---|---|---|---|---|---|---|---|---|---|\n|  | **PPL↓** | **TokAcc↑** | **SemCor↑** | **SumFluen↑** | **TimDiff↓** | **TimRedun↓** | **TimCover↑** | **SemCor↑** | **SumFluen↑** |\n| **Online Assistants**   |\n| VideoLLM-online | 9.73 | 0.49 | 3.01 | 0.69 | 2.67  | 2.15  | 0.80  | 1.68 | 0.59  |\n| VideoLLM-MoD | 9.93 | 0.48 | 2.89 | 0.65 | 2.54  | 2.49  | **0.90**   | 1.66 | 0.55  |\n| MMDuet | 5.69 | 0.60 | 4.29 | 3.40 | 2.32  | **0.62**   | 0.51  | 1.93 | 2.69  |\n| **LiveStar (Ego4D verison)** | 5.48 | 0.60 | 4.35 | 4.20 | 1.93 | 1.12  | 0.67  | 2.85 | 3.91 |\n| **LiveStar (Ours)** | **5.14** | **0.62**  | **4.62** | **4.55** | **1.91** | 0.95 | 0.71 | **3.19** | **4.25** |\n\nIf you are interested in LiveStar's performance on additional online (e.g., OVBench) or offline (e.g., MVBench, VideoMME, and LVBench) benchmarks, please refer to our response to Reviewer wXqt and j4Ki, where we provide further evaluation results and analysis across a broader set of tasks.\n\n---\nThank you once again for your thoughtful and constructive reviews. We have carefully considered all your comments and will incorporate these changes into the revised version. We hope that our responses and planned revisions have addressed your concerns, and we would be very appreciative if you might reconsider your score of our work."}]

Author rebuttal:

Meta-review: This paper proposes LiveStar, a framework for proactive online video understanding that departs from EOS-based designs by combining Streaming Causal Attention Masks (SCAM) during training and a perplexity‑based Streaming Verification Decoding (SVeD) at inference, and introduces OmniStar, a large multi-task online-video benchmark; the authors report substantial gains in semantic correctness, temporal accuracy, and inference speed over prior online Video‑LLMs. Reviewers generally agree the paper is well written and that the core technical ideas—particularly the verification-based response decision—are well motivated and practically effective; the OmniStar dataset and extensive experiments are also appreciated. The primary weaknesses noted are the limited external benchmarking (missing comparisons on widely used offline and online video benchmarks and some recent baselines), concerns about whether the contribution is incremental relative to recent EOS‑based advances, sensitivity and theoretical understanding of the SVeD threshold α, potential loss of fine-grained visual information from aggressive token compression, and missing details on metric computations and some fairness issues in training comparisons. Balancing these factors, I recommend acceptance as a poster: the methodological novelty (response‑silence paradigm and verification decoding), strong empirical improvements, and the new benchmark make this a valuable contribution warranting dissemination and community follow-up, while the evaluation gaps and open analysis questions argue against a spotlight/oral slot. I encourage the authors to (1) expand evaluations on standard online/offline benchmarks and recent baselines, (2) provide metric definitions and deeper analysis of the threshold behavior, and (3) clarify training/evaluation fairness in the final version.
Decision: Accept (poster)