StreamingVLM: Real-Time Understanding for Infinite Video Streams
Authors: Ruyi Xu
Guangxuan Xiao
Yukang Chen
Liuning He
Kelly Peng
Yao Lu
Song Han
Venue: ICLR 2026 Poster (2026)
Keywords: Machine learning
Vision Language Model
ML System
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present StreamingVLM, a model designed for real-time, stable understanding of infinite visual input.

Abstract: Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage.
Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation.
In this paper, we introduce **StreamingVLM**, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. 
During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. 
This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts.
For evaluation, we build **Inf-Streams-Eval**, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text.
On Inf-Streams-Eval, **StreamingVLM** achieves a **66.18%** win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100.
Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96.
Code will be released upon publication.

Published as a conference paper at ICLR 2026
STREAMINGVLM: REAL-TIME UNDERSTANDING FOR INFINITE VIDEO STREAMS
Ruyi Xul
Guangxuan Xiao 1+
Yukang Chen Liuning He'
Kelly Peng
Yao Lu
Song Han
MIT
NVIDIA FFirst Intelligence
https://github. hwtt://tttmmmtmmmmmaaaeeea Labhhaann-aaaaaaee www.-/aaasssreaniiiiiiii
ABSTRACT
Vision-language models (VLMs) could power real-time assistants and au- tonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor per- formance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redun- dant recomputation.
model de- signed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During in- ference, maintain compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and long window recent text to- kens. This streaming ability is instilled via simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effec- tively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-E-ww videos averaging over two hours that requires dense, per-second alignment be- tween frames and text. On inf-Streams-Ein win rate against GPT-4O mini and maintains stable, real-time performance up to 8 FPS on single NVIDIA H100. Notably, our SFT strategy also enhances gen- eral VQA abilities without any VQA-specific fine-tuning, improving performance Long VidcoBench by +4.30 and OVOBench Realtime by +5.96.
this paper, we introduce StreamingVLMM
new benchmark with
StreamingVL. LLM achieves a 66.18% %
INTRODUCTION
VLMs could power autonomous driving, embodied agents, and real-time assistants, but they face critical challenges: understanding near-infinite video, responding in real time stably. To accept infi- nite input, common ideas are Sliding Window Attention with or without overlapping. As shown in Figure L (a) Full Attention suffers from heavy memory and latency; (b) Sliding Window (w/o Over- lapping) resets context frequently and breaks coherence; (c) Sliding Window Attention (w/ Overlap- ping) keeps recent tokens but recomputes attention many times, which hurts efficiency.
Aligning training with inference adds further challenges. Real streaming requires taking infinite visual input in real time and replying with very low delay, but training cannot use extremely long videos. Current approaches to KV cache eviction often lack alignment with the training phase. How to train short videos and still enable the model reason over very long streams remains underexplored. This leads to our core question: How can we train VLMs to understand video chunks in real time and reason stably over infinite video, moving toward human- like intelligence?
In this paper, we propose Streaming VLM, a unified framework that aligns training with streaming inference and dataset curation pipeline. The key ideas are: (1) Train the VLM with full attention on short, overlapped video chunks. (2) At inference, use an attention sink and a sliding window with to handle infinite video, aligned with training. (3) Reuse past KV states and use contiguous position IDs to keep inference stable.
"Equal contribution

---

Published as a conference paper at ICLR 2026
(b) Sliding Window (wlo Overlapping)
(c) Sliding Window (w/ Overlapping)
(d) StreamingVL.M (Sliding Window Reuse KV)
(u) Full Attention
Quickly execed training length. efficiency and ODM on long video.
Need lange chunks for caherence. Cannot serve realtime at peak context length.
Cach sliding window requires recomputing attention.
Keep lateney stably low.
Reuse states as compact KV with contiguous RoPE.
Latency prevents real-time inference.
O(T)
Win Rate:
O(TW) Win Rate: 23.54 %
O(TW")
Rate; 66, 54%
O(TW)
Rute: 66.18
T: token raum
W: window size E Ateation Sink =nt Vision Token EI Text Token
Generted Token
Figure 1: Illustration of StreamingVLM vs. existing VLMs. Let T be video length and W the sliding-window size. (a) Full Attention: O(T2) cost; unbounded memory: degrades beyond training length. (b) Sliding Window (no overlap): bounded memory but short chunks break coherence; long chunks raise latency. (c) Sliding Window (overlap): recomputation per window yields high latency. (d) Streaming VLM (Sliding Window + Reuse KV); reuses states of attention sinks, short vision window and long text window, preserving history at low latency. "Win rate" the pairwise win share vs. GPT-4o mini (judge: GPT-5).
Using this framework, we build Inf S-Sttrrass-rriin. sports commentary SFT dataset of over 4000 hours and Inf-Streams-Eval a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. Then, we fine-tune 2.wen-2.--VL--7B--nnii for real-time commentary, yielding Streaming VLMM that can understand infinite video and response in real time. We evaluate Streaming VLM captioning and VQA tasks, including LiveCC-Sports- 3K CC and Inf-Streams-Evee for captioning, and Long VideoBench (and related VQA benchmarks) for video understanding (Chen et al. 2025a Wang al. 2025a).
On captioning tasks, Streaming VLM, with its infinite video understanding, outperforms existing 7--rcc---Instructt As shown Figure 2 Streaming performs well on practi- cal tasks: it can provide continuous commentary for more than two hours on sports games. On VQA tasks, even without any VQA fine-tuning, Streaming VL.M still improves on Long VideoBench by +4.30. In terms of efficiency, Streaming VLM maintains a low and stable latency, making it highly suitable for real-world streaming understanding tasks.
models such
Qwen2255-1--7 --Vl------structt (w/e SFT) X Cannet Generate Colerently
StreamingV LLM (Sliding Window Reuse KV); / (50 mnstak)
muksk field, warming up 03.00 ot. Players from both teams are on the field, warmingg up
Players warming up before kickoff.
Fans will have tonight, let's take look the kickoff. accasse: On the right-hand side, got Portugal Red.
00:0r0:: Players from teams
00:00000 ne. And then at the other end, it's Spuin setting for kickotT:
Fockaa Redia
LiveCC-7B TB-ttstrect (Pull Attentioal: X Enved Vraining Length
Streaming VLM (Sldding Window Reuse KV): / motakl Ronaldo against David De htapi-si penalty.
shot shot shot shot ... (611 snludy
LaveCC-7B- (Sliding Window): X Zampstere Monory 0131:3:: Will Ronaldo be able to score the first penalty? (158 mshnks
Streaming L (Sliding Window Reuse KV): // 010151: - Portugal got three points with Ronaldo's is three goals!
Figure 2: Issues with existing VLMs. (1) Without SFT, models cannot generate cross-round
tent coherently. (2) With full attention, the context exceeds the training length after processing 2-5 minutes of video and latency becomes prohibitive. (3) With sliding window, models cannot re- tain enough context to benefit from efficiency. In contrast, Streaming VLM addresses these issues, enabling coherent commentary, real-time generation, and long-term history.
METHOD
In this section, we introduce our method for the model and the data. This part has three components: (1) inference scheme for vision-langues processing that supports low-latency updates on infinite video used by StreamingVLMN (2) a training strategy that equips Streaming with streaming inference capability; and (3) the data curation pipelines that provides for training and a new benchmark, Inf-Streams.
real-time data
2

---

Published as conference paper at ICLR 2026
Round
3
7
0 1 2... RoPF. Index
</> Attention Sink
Round
4
$
7
8
Evicted Token
Vision Token
Previous Text
Round 3
3
7
Current Text
Round
2
3
4
7
Figure 3: Inference scheme of Streaming We keep 512 attention-sing tokens to stabilize attention, a long text window of 512 recent tokens to preserve long-term memory, and a short vision window covering 16 seconds to track ongoing actions. We use Contiguous RoPE: indices are shifted to stay within fixed range, keeping positions in-distribution and within the training length.
2.1 INFERENCE SCHEME OF STREAMINGVLM
This section describes the Streaming VLM inference structure shown in Figure 3 These design choices reduce the computation in Figure TTc) while maintaining comparable performance.
Streaming-aware KV Cache The key idea is to maintain a compact and stable KV cache by reusing previous states during streaming inference. As new video frames arrive, we reuse the states of (i) a set of sink text tokens including the system and previous text of length Tainki (ii) long window of the most recent text tokens of length Twindow: and (iii) short window of the most recent vision tokens of length Vwindow- In Figure the cache lengths are Tank 1, T window 3, and Wwnnoww 4.
With this structure, older vision tokens are evicted first; early text is evicted only when the budget is exceeded. Instead of recomputing previous tokens, this asymmetric retention keep the lowest com- putation while maintaining sufficient context for coherent generation time, yielding comparable performance with Sliding Window with Overlapping (Figure c)).
Contiguous RoPE To prevent positional drift after eviction, we apply contiguous rotary positional embeddings (RoPE). When earlier tokens removed, the RoPE indices of subsequent and incom- ing tokens are shifted so that their positions remain numerically contiguous with the last retained token. Once the video length surpasses the total window size, the effective RoPE indices stop grow- ing and remain within a bounded range. This keeps positional values in in-distiibution and stabilizes long-horizon streaming inference.
When applied to the Qwen-VI VL family, which uses 3D positional embeddings for visual tokens, we use contiguous 3D RoPE. The RoPE index is still left-shifted to stay contiguous; for vision tokens, we build 3D indices (time, height, width) and assemble them by the 3D rule, matching the interleaved vision-text layout.
2.2 TRAINING STRATEGY
To endow the model with the ability to follow the streaming inference pattern in Figure while keeping training simple, we adopt an overlapped-chunk, full-attention strategy (see Figure The left panel of Figure illustrates the attention at inference time. In this Figure the cache lengths are the same to Figure 3 with 1, T T3intow-3, and Vwindoww
During training (middle panel of Fi rather than replicating the exact sliding- schedule used inference, we split long video stream into consecutive chunks (C1, C2....) length W frames, with temporal overlap O frames between C, and C,+1 (0 O < W). Each chunk is treated as a training instance in which vision and text tokens (V/T) are sampled and interleaved at s intervals. We apply full attention within chunk, i.e., every token may attend all tokens inside the same chunk.
As highlighted in the right panel of Figure this overlapped full-attention supervision closely ap- proximates the effective attention pattern at inference attention sink, a longer window of recent text, and shorter window of recent vision retained in the compact KV cache. Aligning training
3

---

Published as a conference paper at ICLR 2026
supervision with the test-time context teaches the model the intended recency bias and yields stable streaming behavior without training on prohibitively long, quadratic-cost contexts.
Importanll, mirroring the inference-time schedule. we interleave vision and text tokens within each training chunk rather than adopting the com- mon VLM paradigm that places all vision tokens before text. We compute loss only on positions aligned to the per-second narration; when a sec- ond has no narration, we insert a placeholder token in that slot while keeping the interleaved V/T layout. This supervision teaches the model to synchronize generation with the stream-learning when to speak and when to remain silent-and consequently endows Streaming with reliable streaming narration behavior at inference.
2
3
SFT Attention Map Figure 4: Training Strategy. We train with overlapped full attention that mimics test-time attention. (1), (2), (3) and (4) four training samples, both keeping the attention sinks and overlap later in time.
Inference Attention Map
2.3
DATA CURATION PIPELINE
Suckerbit
Caption
Steven Curry pulls up!
Min Words Filtering
SFT & Eval Data
Caption
Caption
Curry
Stephen
Filtering
GPT
HQ Annealing Data
Figure 5: Data Curation Pipeline. We collect games from five sports basketbl, soccer, Ameri- can football, ice hockey, and baseball. We use GPT to edit or reject low-quality segments, yielding 2,449 full games. We then build two datasets through separate pipelines: an SFT dataset using overlapped chunking, and a high-quality annealing dataset focused on real-time actions.
2.3.1 VIDEO COLLECTION AND ASR
As shown in Figure 5 we collected game videos from five sports: basketball, soccer, ice hockey, baseball, and American football, including 712 basketball games, 544 soccer games, 402 ice hockey games, 399 baseball games, and 392 American football games. The commentary language is En- glish. To ensure video quality and read speed, we constrained the video resolution 360P-720P with frame rate of 24 FPS. First, we used the WhisperX model to extract real-time speech (ASR) from these games, obtaining an initial corpus of videos with total duration over 6,000 hours and their corresponding real-time commentary.
2.3.2 DATA CLEANING
In complete commentary videos, there are often many useless segments, such as advertisements and host monologues. These segments have weak connections between visual content and ASR semantics, making it impossible for the model to infer content from the footage. In addition, the ASR model sometimes fails correctly recognize details such as player names and team names.
Therefore, we set rules and used GPT to clean these data. We first split a game into 120-second segments and concatenate the commentary within each segment, then split into sentences. Using the segment and the video title (including game time and both teams) as context, we ask the GPT- 5 model make decision according to the rules, with options "keep." " "delete," and "edit" each sentence in one chunk. "Keep" means the content is game commentary and is correct. "Edit" means commentary but needs to modify some details, such as incorrect names, and the corrected complete sentence is returned. "Delete" means non-compliant content that should not appear in the training data.
For kept sentences, the timestamps are consistent with the ASR results; for edited sentences, evenly distribute the original sentence duration over each word of the edited sentence (since sen- tence typically lasts about 3-5 seconds, the error is within tolerable range). In the original ASR data, 46.22% were kept, 37.89% were edited, and 15.79% were deleted, ultimately forming the raw video-commentary pairs of our data.
4

---

Published as a conference paper at ICLR 2026
2.3.3 SFT AND EVALUATION DATA SEGMENTATION
For the train and validation sets, we build the data as follows. Under the training setup Section 2.2, we split videos with W = 24 and O 12 s. To ensure enough commentary labels per sample, we require at least 2* W words as min words filtering. All commentary before the segment is treated previous text. During training, we take the first Tink tokens and the last T window tokens from this previous text to match the inference setup.
For evaluation, create new benchmark, Inf Inf-Streams-Eval It contains 20 full games with an average length of 2.12 hours. We split each game into 100 segments, selecting those with at least 200 words. Commentaries of these segments model (we use gpt 5) votes between two model outputs with access to ground-truth references. The model with more votes (higher win rate) is judged to provide better commentary.
considered as ground truth. For scoring, larger
Inf-Streams-Fii Eeall has two settings: chunk and infinite, denoted by" and x respectively in following tables. In Figure ! the chunk mode panel (b), and the infinite mode is panel (d). For models that cannot do infinite inference, we cut the vidco into chunks; the model receives the previous text and the current chunk to produce caption. For models that support infinite inference, the model runs on the full stream; we keep its past outputs as previous text and continue captioning until the video ends.
2.3.4 HIGHI-QUALITY ANNEALING DATA
The above dataset can sft the model's ability for real-time video understanding. However, it contains a lot of content such as team information and season history; for the human experience of the com- mentary task, we prefer the model to provide real-time commentary on on-field events. Therefore, we created high-quality annealing data.
We first slice all data without overlap, requiring each clip to be 16-64 seconds long with internal silence no longer than 3 seconds; each clip must also contain at least 2 D (duration in seconds) words. Across all games, we obtained 52,530 new samples. Then, we define the standard of "real- time commentary." For each sample, time commentary" exceeds 80% to decide whether to keep it. In the end, only 14,786 samples were retained. Subsequent experiments in Table that after applying this portion of data for sft, the model's capability and commentary quality further improved.
GPT-5 to determine whether the proportion of "real-
EXPERIMENTS
In this section, we first describe the implementation details, then evaluate on video captioning and VQA against strong baselines. We next test the efficiency of StreamingVLM. Finally, we run abla- tions to better understand its behavior.
3.1 EXPERIMENTAL SETUP
Training We fine-tune Streaming VLM from Qwwen2.5-VL-Instruee) (Bai et al. 2025). Step teaches the model the infinite streaming inference pattern. We train on our SFT set (525K streaming samples) and LiveCC's Live-Whisperal WhisperX-526K (526K streaming samples) (Chen et al. 2025a). Step 2 uses our high-quality annealing data (14K streaming samples, each 16-64 s with detailed actions) to boost real-time action commentary and improve human experience. After these two stages, we obtain Streaming V..M. The total compute is about 128 H100-days.
Baselines We select strong baselines to compare with Streaming VLM. For the captioning task, we use GPT-4o mini to show commentary strength, and Livecc-7B-Instruue which trained on 5.5M YouTube video clips (30 240 s) and 178K Video Aideo-Question-Answee samples, working well on short videos commentary (OpenAl 2024 Chen et al. 2025a). We also include ReKV, a strong training- free streaming-inferenc method (Di et al. 2025). Due to design limits, GPT-4o mini is evaluated on Inf-Streams- Eval in the chunk setting, not the infinite mode used by StreamingVLMM LiveCC- Qwen2.5-Vi
7B-Instruct is tested in both chunked and infinite settings. For the VQA task, IB-Instt which is the base model before SET for Streaming LLM to show that our SET pipeline improves the base ability (Bai et al. 2025).
5

---

Published as a conference paper at ICLR 2026
Table 1:
Captioning accuracy (win rate vs. baselines). Base-
Win Rate vs. GPT-4G-minl
LiveCC
StreemingVLM
lines with/without chunking fall short; StreamingVL.M surpasses strong models such as GPT-4o and produces compelling commen- (Spperssisssssssss for Inf-Streams- length 100s. On Livecc-Sports-3K CC, LiveCC has only one mode and cannot be compared against itself, so we show "-"
80
86.5
66.8
67.4
66.2
60
oo infinite; chunk
40
20
8.4
13.2
15.7
4.0
500
Win Rate A vs. B
Inf-Streams- Eval
Livece SpectsSports-333 3K oc
Chunk Size
Model
GPT-46' Livece" Livece
LLaVA GPT-4o Gemini Livere
Model A
Figure 6: For existing VLMs, balancing cross-chunk coher- ence with training- -lngth lim-
20.44
95.97
24.50
16.25
28.38
34.11
Liveoe-BB-instruct
15.73
Sivecc-7B-Instreen
1.82
41.50
40.06
39.73
Streaming VLM x
66.18
87.81
99.12
47.33
45.59
44.21
56.19
its is challenging.
Benchmark We evaluate real-time captioning and video understanding across broad set of tasks. For captioning, we use our Inf-Streams-Fr Evall (average length 2.12 hours), which tests long- long-horizon commentary and the LiveSports3K-CC benchmark (49 sports, 416 clips, each 2 10 s) (Chen et al.) 2025a). For video understanding, we evaluate Streaming VLM on four public suites. VideoMME: multi-task set (QA, caption, grounding) covering short and long videos for general comprehension (Fu al.] 2025). MVBench: fine-graid fine-grained skills on short clips (actions, objects, counting, temporal order) (Li et al.) 2024b). LongVideoBench. long-video QA that requires long-tern memory and cross-segment reasoning (Wang et al., 2025a). OVOBench: video QA that tests real-time under- standing and streaming perception (Li 2025
3.2
ACCURACY RESULTS
3.2. CAPTIONING
We first compare our inference strategy with ReKV on the captioning task. We observe paradox for training-free ReKV: models without task-specific fine-tuning perform poorly, yet models that are spe- cially fine-tuned (e.g.. StreamingVLMM) rely fixed context format that ReKV's eviction policy output. In contrast,
Training-inference consistency surpasses ReKV. Non-fine-tuned models lack capability of real-time captioning, while with fine-tuning models ReKV's eviction pol- icy disrupts context, frequently resulting in no output. (Superscripts for Inf Inf-Streams- Eval: infinite; = chunk length 100s.)
Table 2:
disrupts, often yielding Streaming VL.M 's training-inference consistent de- sign resolves this issue.
Win Rate
Inf-Streams-Eie
Model
GPT-4o' Livece' Liveco
Then, we evaluate StreamingVLMM, Qwen-2.5-VL. 7B-Instruct, and LiveCC-7BB-Insrruet on LiveCC- 3K-Sports-CC and Inf-Streeam--Eval As shown in
Model
Qwen (+ ReKV)
0.00
19.56
63.57
Streaming (+ ReKV)
0.00
0.00
0.00
Table D on Inf-Streams- Eval, Q-- Instruct cannot keep continuous commentary and thus performs poorly. LiveCC-77-IInsrrucc works better with chunked inference. Figure 6 further shows that short chunks break coherence; these de- signs do not support infinite inference, and with long chunks they soon exceed the training length and degrade.
SureaningVLM (+ Ours) "
66.18
87.81
99.12
contrast, Streaming VLM runs infinite mode; its long-term memory and streaming video percep- tion give it clear edge, surpassing GPT-4o mini in commentary quality. Figure figure shown) illustrates real case where Streaming VLM maintains coherent output, real-time latency, and long- term memory, addressing the core challenge of real-time perception for infinite video streams. On Streaming VLM also performs better than baselines, showing stable stream- ing captioning videos of various length.
3.2.2 VQA
We evaluate Streaming VLM and its base model, Qwen-2.5-VLL 2.5--LL-7B--nstruu on four VQA tasks. As shown in Table 3 even without any VQA SFT, StreamingVI.M outperforms the base all tasks,

---

Published as a conference paper at ICLR 2026
Table VQA results comparing StreamingVLM with its base model. Without any VQA fine-tuning, Streaming VLM delivers consistent accuracy gains across all tasks, with the strongest improvements on long-horizon and real-time settings.
MVBench Video MME (w/ sub.) Long VideeBeenn OVOBench (Realtime)
1wen-2.5-VI-7B--nnmme StreamingVLM. 69.16
67.34
65.10
54.70
56.00
65.10
59.00
61.96
Generation latency token Video length
Table 4: Ablation of RoPE on captioning (win rate). Native RoPE drops on infi- nite streams; 100 chunking partly recov- ers but hurts long-term memory: contigu- ous RoPE keeps indices bounded and sus-
[a] Full Attention
0.6
0OM
Sliding Window Overiappingl
latency
b.5
(c) Sliding Window /w/ Overlapping)
StreamingVLM (Sliding
0.4
0.3
Not Realtime Realtime
tains infinite performance. (Superscripts for -e-Streeams-Eval.. x infinite: chunk length 100s.)
0.2
0.1
200 400
600
800
1000
Win Rate A vs. B
Inf-Streams-Eval
Processed video length
Model B
GPT-4o" Livece" Livece 00
Figure 7: Per-token latency vs. video length. Full at- tention hits OOM; sliding window w/o Overlapping spikes above real time; sliding window w/ Overlap- ping remains inefficient; Streaming latency stays low and stable. The dashed line marks the real-time threshold (10 tokens/s = S 0.1 $ per token).
Model A
Native Native
63.23
74.00
98.07
25.09
59.42
60.32
Contiguous 66.18
87.81
99.12
showing that our SFT improves general visual ability. OVOBench Realtime tests understanding of the immediate, streaming scene. On this streaming perception task, Streaming VLM improves by 5.96%. This highlights the strength of nrf-Streams-Train and our training strategy, which enhances the model's core abilities.
3.3 EFFICIENCY TESTS
As shown in Figure commentary: VLMs with full attention, sliding window attention (w/o overlapping), sliding window attention (w/ overlapping), and the inference strategy of Streaming VLM, respectively correspond to panels (a), (b), (c), and (d) in the Figure] 1
report per-token latency for the three methods in Figure [ infinite
Real-time replies require latency below fixed threshold the dashed line. Full attention exceed the limit and OOM. Sliding window (w/o overlapping) needs large chunks for coherence, so it shows periodic latency pattern: the start of each chunk the model rebuilds context and the commentary is not coherent with the past; later in the chunk, latency rises sharply and fails to meet real-time needs. Sliding window (w/ overlapping) remains inefficient for computation redundancy. Streaming VLM keeps fixed context length and reuses KV, maintains lower and stable latency, and supports real-time commentary FPS on a single NVIDIA H100.
3.4
ABLATION STUDY
3.4.
CONTIGUOUS ROPE
We study the effect contiguous RoPE indices. Since we train with full attention, training only uses the native RoPE. At inference, compare contiguous RoPE with the native version. As shown in Table 4 native RoPE degrades sharply on infinite streams because its index grows fast and exceeds the training range. Splitting the video into 100 chunks can partly recover accuracy, but harms long-term conherence. With contiguous RoPE, the position index stays bounded, so the model supports infinite inference without loss.

---

Published as a conference paper at ICLR 2026
Table 5: Ablation of sliding window and sink size with accuracy on captioning tasks (win rate). Left: effect of sink and T wimdow, trained with Window = 1.s. Right: effect of V Winkow trained with Tann 512 and T window 512. (Superscripts for Inf Inf-Streams-Bve Eval: do infinite; chunk length 100s.) )
Infer args
SFT args
Inf-Streams--ww (Basketball)
V Viindow
Inf Inf-Streams-Fit
Tink
Twindow
Tink
T Twindww
GPT-4o"
Livece
Livece
Win Rate vs.
GPT-4o"
Livecc"
Livece
512
0
512
512
69.68
89 89.42
99.19
Os
52.90
77.49
97.56
512
512
512
66.76
86.03
98.69
1
63.46
83.24
98.18
256
256
512
512
70.17
91.79
99.62
4
66.08
83.86
98.73
1024
1024
512
512
71.43
91.69
99.84
8
65.66
85.09
99.14
60.41
72.08
98.55
32
65.49
85,58
99.06
16
66.18
87.81
99.38
512
512
512
512
73.64
92.33
99.38
Table 6: Ablation of SFT strategy and dataset on captioning and VQA. Overlapped SFT strategy improves over the Live-Whispe Wive-Whipper-55266 base, and adding the high-quality annealing data brings further improvements, especially for infinite streaming task S---treemm--vall (Superscripts for Inf- Streams-EEv oo infinite: chunk length 100s.)
Win Rate A vs.
Inf Streams- Eval
Livece Livece-Spetts-B Spocts Poott--KKcc
MVBench Video MME VideeBecec OVOBeach
Model
Model
GPT-40 Livece'
LJaVA GPTT-4 Livece Score
sul
Realtime
D..V-225.VLL-7B- Innsstuut Instruct
0.01
20.44
95.97
24.50
16.25
28.38
34,11
67.34
65.10
$4.70
56.00
Live- WhisperX-526K s
42.77
+ Inf Inf-Strcamy- Train Train
63.46
83.82
98.95
46.45
45.48
44.27
$3.07
68.66
64.90
59.00
60.55
High-Quulity Annealing Data
66.18
87.11
99.12
47.33
45.59
44.39
56.19
69.16
65.10
59,00
61.96
3.4.2 SLIDING WINDOW AND SINK
We firstly verify the value of evicting text during training. Then we search for the best inference settings of Tink Twindows V Window
First, the left table in Table 5 ablates the lengths of the attention sink and text window. Here Taink and Twindow are the lengths of previous attention sink and text window kept during both training and inference. We take a basketball-only subset of the SFT data and train two models: one with text eviction using Tink==112 and T 5512, and one without eviction. On the Inf-Streamms-iva (basketball subset), we evaluate each model under its matching policy (evict vs. no-evict). The left table in table [5] shows that, for infinite inference, evicting previous text tokens is important and improves performance.
Next, we study different choices of Vwindow The right table in Table 5 shows that 16s visual window a good choice: it is long enough to cover recent actions, yet short enough to stay efficient. In contrast, keeping Os of vision context leads to clear drop, confirming that retaining recent vision tokens for continuous actions is essential.
3.4.3 TRAINING STRATEGY AND DATASET
We study the effect of our SFT data and high-quality annealing data. The SFT set teaches the model the infinite streaming inference pattern, while the high-quality annealing data further improves com- mentary quality.
SFT Strategy shown in Table with our overlapped training strategy, our SFT subset helps the model adapt to the interleaved vision-text pattern and understand very long videos. Compared with model trained only on Live- Live-WhisperX-S5 WhisperX-526K, training on the overlapped SFT data strengthens perception of infinite video, yielding clear gains +31.29 (win rate against GPT-4o-mini) Inf- Streams-EEv Eval and +3.68 (win rate against LL VVV--ide--P77 (iddeo-7BB-Oe (2B-Qee22) on Livecc-Sports-33 cc.
As shown in Table overlapped-chie SFT strategy (i.e., using standard non-overlapping chunks). We hypothesize two things: (1) This new model performs significantly worse, proving the data alone is insufficient. (2)
train
baseline model using full SFT dataset but without
8

---

Published as a conference paper at ICLR 2026
Table 7: Ablation of SFT strategy on captioning and VQA. Overlapped SFT strategy performs significantly better than the ablation model trained without our overlapped-chunk SFT strategy. (Su- perseripts for inf-Streams-Eve 0o infinite; chunk length 100s.)
Win Rate A vs.
Inf Streams-Eve Eevll
Livece Livece-Spurts- 3K cc
MVBencc Video MME LongVideoBench OVOBench
Model B
GPT-4o!
LLaVA GPT-4o Gemini Livece Score
w/o sub.
Realtime
Model A
Non-overlapping Strategy
62.51
46.24
45.08
43.21
56.19
68, 69.16
65.50 65.10
58.90 $9.00
59.20
Overlapped Stratogy
47.33
45.59
44.39
56.19
61.96
This model will fail or perform poorly when run with our streaming KV cache, proving our SFT strategy is essential for the training-inference inference alignment.
High-quality Annealing Data Our high- annealing data focus real-time content and fur- ther boosts model ability. As shown in Table 6 we compare training with and without the high- quality annealing data. We can observe significant gains on both captioning and VQA benchmarks.
4 RELATED WORK
Vision Vision-Language Models Early multimodal models start from images and then extend to videos by adding temporal modules or token schedulers. Recent open models improve video understanding and transfer across tasks. Examples include VLaVA-One Vision for unified transfer across images, multi-image inputs, and videos (Li et al. 2024a), Video-LLaMA 2 for pratial-temporal and audio cues (Cheng et al. 2024), Intern 2/deo2/2.5 for scaling video encoders and long context (Wang al.] 2024 20. Long LongVIL.. VILA for long video training system (Chen et al.) 20256). and Qwen2.5-VI. for strong grounding, document parsing, and long-video skills (Bai et al. 2025). Most systems process finite clips and often place all vision tokens before text, which can hurt alignment in streaming and limit real-time interaction practice. In contrast, we interleave vision and text steps match real-time commentary and interaction, and we observe gains on both commentary and VQA.
Long-Context and Streaming Inference in Text LLMs memory and delay, the text community has proposed several lines of work: (1) Attention sink + slid- ing window: StreamingLLLM keeps a small set of early "sink" tokens plus a recent window, which stabilizes very long decoding (Xiao et al. 2024 (2) RoPE extension and continuity: YaRN, Lon- gRoPE, and LongLoRA for efficient fine-tuning tuning improve position embedding extrapolation (Peng al.) 2023 Ding al. 2024 Chen al.) 2024bb; our contiguous RoPE follows this idea but tar- gets cross-modal, step-wiss updates. (3) KV cache compression/eviction: HzO, SnapK and ReK V reduce KV size by selecting heavy hitters or gating heads (Zhang et al. 2023 Li al. 2024c Di et al., 2025). However, these methods are mostly tested on text, and alignment between streaming training and inference remains underexplored. We bring the "sink sliding window contiguous position" recipe to cross-modal streaming and introduce a training strategy for streaming inference.
To handle near-infinite inputs under fixed
Streaming and Online Video LLMs Several concurrent works target streaming video directly. MidooLLM (LIVE) converts offfine data into streaming dialogue for long context and low latency (Chen et al.
2024a). VideoStreaming uses a fixed video token budget to handle long videos
(Qia et al.] 2024). LiveCC aligns large-scale ASR with video frames to push real-time sports commentary (Chen et al. (202a).. In practice, on videos longer than 5 minutes (at least 200 frames), these methods show clear performance drops, and their latency is still far from infinite real-time interaction. Compared with these, we (i) train with overlapped short chunks and full attention to match the sink sliding window test pattern, and (ii) keep contiguous RoPE across modalities to enable real-tima time understanding over infinite videos.
VLMs Benchmarks and Evaluation VideoMME covers 900 videos (254 hours) with multimodal
inputs and tests both short and long time ranges (Fu et al. 2025). LiveSports-3K-CC compares real- time commentary quality and often uses the "LLM-as as- judg"" win rate metric (Wang et al. 2025a). LVBench targets ultra-long videos and long-term memory (Wang al. 2025a). However, Current benchmarks often focus on retrieval or summary over long videos and do not require frame-level un- derstanding, very low FPS sample may pass. Our Inf Inf-Streams-Fww is built for near-infinite

---

Published as a conference paper at ICLR 2026
commentary (over 2 hours). It requires second-level alignment between frames and responses and tests high-FPS, long-video understanding closer to real-world needs for VLM assistants, robots, and autonomous driving.
5 CONCLUSION
In this paper, we introduce StreamingVLM, a unified time streaming perception existing VLMs. We first present an efficient strategy for training streaming VLMs and data curation pipeline that together boost performance on both streaming tasks and VQA. We then show on real-world cases that our inference design enables real-time video understanding, delivering stable commentary for over 3 hours at up 8 FPS single NVIDIA H100. Finally, we release Inf-Streams, a new SFT dataset and benchmark that tests second-level real-time understanding videos averaging over 2 hours. Taken together, this work paves the way for practical deployment in real settings.
framework that brings real-
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang. Humen Zhong. Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https::/arxii.coo
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video, 2024a. URL
http:/orrrabb////......// 2406 11816
Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livece: Learning video Ilm with streaming speech transcription scale, 2025a. URL htttps://arxxii.rrrrgg/.
2504.1603
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- glora: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations, 2024b.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu Xiuyu Li, Yunhao Fang, Haotian Tang. Shang Yang, Zhijian Liu, Yihui He. Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling context visual language models for long videos. In International Conference on Learning Representations, 2025b.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms, 2024. URL https: org/ abs/
14006004770
Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video kv- cache retrieval, 2025. URL
00540
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending I1m context window beyond 2 million tokens, 2024. URL htt.:/ppss//aiii.trrraaa////////
13753
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen. Caifeng Shan, Ran He. and Xing Sun. Video-mme: The first-ever comprehensivee evaluation benchmark of multi-modal Ims in video analysis, 2025. URL
htt.p:://arxxv.orrr/a////// 2107
10

---

Published as a conference paper at ICLR 2026
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a. URLhttps: /attas////riiiie aarii.iiii.....ii org/ abs/ 2408 03326
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video under- standing benchmark, 2024b. btRps:/t::///atttt http:://axxvvv..o
2311. 17005
Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How is your video-Ilms from real-world online video under- standing?, 2025. URLattts:
org/ abs abs/
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation, 2024c. URL htttpp://aradi..rr//ab///02222
OpenAI. Gpt-4 technical report, 2024. URL UtL/tttt:///arri/.rr/////////
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding. Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models, 2024. URL. https:/
arxiv.org/ abs 2405. 16009
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng. Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong. Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2025a. URL. TRI.https:: //tp///avvvv///////////..e..wwv
http://awwwvw.aaaa////// wwt./oo////////eooooooooooo /206 08035
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang. Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvide02: Scaling foundation models for multimodal video understanding, 2024. URL wwtphhtt////at pettps:/aarxvv.rgg abs/2403. 5377
Yi Wang. Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang. Changlian Ma. Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2.5:: Empowering video milms with long and rich context modeling, 2025b. URL /eletps://rxivve httrg//bb/25511 12386
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks, 2024. URL/tttps: arwii.iiiiv.///..///aaaa/a/aa..a org/ abs 2309.0000 11999...44 17453
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference of large language models, 2023. URL hctps:
7/ 7/arxiv. org abs/ / /306 14048

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LLM USAGE STATEMENT
We acknowledge the use of Large Language Models (specifically Claude and GPT-5) in the prepa- ration of this manuscript. The LLMs were used exclusively as writing assistants to:
Polish and refine the language for clarity and conciseness
Improve grammar and sentence structure
Suggest alternative phrasings for technical descriptions
Help organize and structure sections for better flow
All research ideas, experimental design, theoretical derivations, and scientific contributions en- tirely our own. The LLMs did not contribute to research ideation, hypothesis formulation, or any core scientific aspects of this work. We used LLMs in manner similar grammar-checking tools, but with more sophisticated language capabilities. All content, including any LLM-assisted text, has been carefully reviewed and verified by the authors. We take full responsibility for all contents of this paper, including their accuracy and originality.
A.2 STABILITY OVER TIME
We split each video into five segments at 20% intervals and evaluate on the 2-hour test set. As shown in Figure 8 Streaming VLMM does not degrade across later segments and reaches performance close to Sliding- Window w/ Overlap. This indicates that Streaming VLM maintains quality as videos grow and effectively supports unbounded inference.
Full Attention
90
Sliding Window Overiappingi Sliding Window (w/ Overlapping) StreamingVLM (Sliding Window+Reuse KV)
Win Rate against GPT
67.5
San.
65.7ee::3
66.4 67.5
66.796.8
24.1
24.2
23.3
22.5
15.2
0.0
0.0
0%6-20%
20%-40%%
40%-60%
60%-80%
80%-100%
Video Segment Range
Figure 8: Stability over time. Each test video split into five segments at 20% intervals. Stream- ingVLM (Sliding Window + Reuse KV) maintains nearly constant win rate across segments and matches the performance of Sliding Window w/ Overlap, while Full Attention and Sliding Window w/o Overlap degrade remain far lower.
Table 8: Sensitivity analysis over T sink Inf- S---ttrr---vall
SFT Tank
Eval Tain
GPTT-4"
Livecc
Livece
64
64
72.04
90.32
99.46
128
128
73.65
92.94
99.47
256
256
73.73
93.28
99.38
1024
1024
74.82
93.51
99.53
512
512
73.64
92.33
99.38
12

---

Published as a conference paper at ICLR 2026
A.3 SENSITIVITY ANALYSIS OF SINK TOKEN WINDOW SIZE
To address how finite token length limits impact model performance across varying scenarios, we conducted sensitivity analysis the attention-sink window size (Tink).. As discussed in the Limi- tations section, these window sizes serve as key hyperparameters.
Table presents the ablation results for varying Tink sizes (64, 128. 256, 512, and 1024) during both Supervised Fine-Tuning (SFT) and Evaluation. The results demonstrate that the sink token size noticeably impacts the final performance. Generally, larger Tink capacities yield better win rates against GPT-4o and Livecc metrics, as the model retains more initial contextual tokens.
However, performance gains plateau larger window sizes, indicating trade-off between context retention and computational efficiency. This confirms that Tink should be carefully tuned based on the specific scenario's 's context length requirements.
A.4 DEMO
We provide demo video in the supplementary materials showing StreamingVLM 's commentary after 100 minutes of continuous inference. The video is randomly selected and edited to remove long pauses and mid-length ads. As the base model is modest in size, occasional hallucinations may occur. Please see the supplementary materials for details.
13


Reviews:
[{"review_id": "5QimyLuvDa", "reviewer": "Reviewer_bkLD", "summary": "This work presents StreamingVLM, a framework capable of understanding continuous visual input in real time. Specifically, the key ideas include: (1) an efficient inference scheme that maintains sink text tokens, a long window of the most recent text tokens, and a short window of the most recent vision tokens; (2) a dedicated training strategy that splits the input into overlapping chunks and trains with full attention to approximate the aforementioned efficient inference scheme; and (3) a dataset providing long-horizon data for fine-tuning and evaluation. Experiments conducted on publicly available datasets, as well as the newly created dataset, demonstrate improved real-time captioning and video understanding capabilities compared to both in-house and open-source models.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid presentation: I especially appreciate the contribution of the newly created SFT dataset, and I found the demo video to be a convincing demonstration of the practical value of this work.\n\n2. Clear performance improvement over baselines: The improvements in captioning and video understanding are significant, supported by both qualitative and quantitative comparisons in the manuscript.\n\n3. Comprehensive coverage of prior work: The paper provides a thorough literature review and is overall well written.", "weaknesses": "1. Clarification of differences in streaming-aware KV cache: The distinction between the proposed approach and StreamingLLM is not sufficiently clear. Based on my understanding, the main difference lies in using different window sizes and eviction strategies for text and visual tokens. It would be helpful to explicitly explain this difference and discuss whether the proposed StreamingVLM training strategy could be applied to StreamingLLM.\n2. Generalizability of hyperparameters: The window sizes for text and visual tokens are clearly important factors (as shown in Table 5). However, I am concerned that these hyperparameters may be highly task-dependent. For example, a 16-second visual token window might work well for basketball videos but may not generalize to other scenarios. Additional discussion on how to tune or generalize these hyperparameters would strengthen the work.\n3. Smaller gains in VQA compared to captioning: While the method improves performance on both VQA and captioning tasks, the gains in VQA are relatively smaller. Providing an explanation for this observation would help readers better understand the strengths and limitations of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fnJQ8BDPM4", "reviewer": "Reviewer_Ep3A", "summary": "This paper proposes to tackle that challenge of enabling vision-language model to process long video streams in real-time. To achieve that, the authors propose to to align the training and streaming inference using two main design ideas:\n* It uses a compact streaming-aware KV cache mechanism, which only keeps a small of attention-sink tokens, a text window and a short visual token window. \n* It introduces a contiguous RoPE to prevent positional drift by reindexing the tokens to stay within a bounded range. \n\nTo evaluate the algorithm, it introduced two new datasets for streaming the video understanding. \n* It introduce Inf-streams-train, a 4000-hour dataset sports commentary dataset curated using ASR and GPT. \n* A benchmark dataset Inf-Streams-Eval with good per-second alignment of frames and text to test the infinite streaming. \nThe authors provide extensive evaluations using model comparisons and show the proposed design is effective to handle long videos, and can improve step by step when incorporating the proposed mechanism.", "questions": "I enumerate a few questions in the weakness part, which are mostly about how the paper improve the clarity. Hope the authors can provide me a few evidences to address them in the easiest setting, or pointing me to the right source if I missed any. \n\nThe dataset used for training and benchmark is very important part in this part to make model great and evaluation solid. I wonder whether they are available to the community in some ways. Together with the trained model, I wonder whether are open source plans.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall the paper is well motivated, and demonstrates convincing results that can advances real-time long-horizon video understanding. \n* The proposed KV-cache mechanism composed with attention-sink tokens, long text window and short vision window is effective from the evaluation, and the contiguous ROPE can further yield stable output with improved performance. \n* It further proposed a dataset that can train the model with higher quality dataset, and an evaluation mechanism to benchmark the progress. The process to create the dataset is legit and considers the important factors that limits existing datasets. If they author can share them out, it will benefit the community a lot.", "weaknesses": "Overall the paper aces well in a number of engineering factors to make the current system solid. However, there are many design choices that embed strong heuristics in them and unclear what they will terminate at. \n* The model is trained on overlapped short video chunks and never experiences true streaming behavior with recurrent KV use. It is not quite clear whether it is \"training-inference alignment\" claimed by the author. In Table 2, it shows alignment is important (where ReKV completely fails), while it is a very differnet mechanism and not optimized for multimodal use. It could be done better if the author can show a comparison to the same model developed (but without stream-aware training), and shows the difference. \n* There are also finite token length limit for attention-sink, visual token and text token windows. It is not quite clear to me how they impact the final results if the scenario varies. \n* From the results, we can clearly see the model can already achieve really good performance compared to the baselines (GPT-4, LiveCC) even without using T_sink or T_window. I wonder how much does the base model and training on the created dataset contribute. Ideally, we want to factor out them out in performance evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VSA8Mzm7KJ", "reviewer": "Reviewer_tsyX", "summary": "This paper introduces StreamingVLM, a vision-language model framework designed for real-time, long-horizon video understanding. \n\nThis framework aligns training and inference by training on short chunks with original attention, and during inference maintaining a compact KV cache with attention sink, short visual window, and long text window.\n\nThis paper also introduces a new dataset and Inf-Streams-Eval benchmark which consisting of long sports videos with ASR and commentary annotations.", "questions": "1. Could the authors provide more details on the data annotation and filtering process, such as examples of removed or edited segments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.Clear motivation. The paper focuses on a real and underexplored problem—achieving real-time video understanding under limited latency and memory constraints.\n\n2. Simple and clear method. The proposed attention sink + sliding window + contiguous RoPE mechanism is simple, elegant, and demonstrates good empirical performance.\n\n3. Valuable dataset. The introduced dataset makes a meaningful contribution to the community of real-time long-video understanding.\n\n4. Clear writing and easy to follow.", "weaknesses": "1. Experiments focus mainly on sports videos. It remains unclear how well the model generalizes to other domains such as egocentric or instructional videos.\n\n2.  Although the Inf-Streams-Eval benchmark is valuable, it relies on GPT-based judgment for scoring, which may introduce bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMke7rPt1C", "reviewer": "Reviewer_qjFR", "summary": "This work presents StreamingVLM, a vision–language model trained for near-infinite video understanding. To enable low-latency inference under a fixed GPU-memory budget, the method maintains a compact KV cache that reuses (i) persistent attention-sink states, (ii) a short window of recent vision tokens, and (iii) a long window of recent text tokens. To teach the model this input pattern without training on prohibitively long contexts, the authors adopt an overlapped-chunk, full-attention SFT strategy that mimics the inference-time attention layout. They also curate a sports-focused training set and introduce a benchmark, Inf-Streams-Eval, targeting long-video understanding(Captioning) with real-time narration. On this benchmark, StreamingVLM reports gains over strong long-video baselines and GPT-4o, and it shows modest improvements on general VQA benchmarks as well.", "questions": "1. From Table 5, we can see that when the values of $T_{sink}=512$ and $T_{window}=512$ in the inference stage are the same as those in the training stage, the overall performance is the best. This suggests the model is tightly tuned to a single window geometry, and performance degrades when the inference budget diverges, indicating limited robustness. What happens if you vary these values during training? Have you tried other values in SFT or using a curriculum scheme(start larger, anneal to target), then evaluating under both matched and mismatched inference windows? \n\n2. Why does StreamingVLM leverage 512 sink tokens? How sensitive is performance to reducing number of sinks​? and is there a saturation point where adding more sinks yields little or no gain?\n\n3. Table 3 does not compare StreamingVLM with Livecc-7B-Instruct, what does Livecc-7B-Instruct perform on those general VQA tasks? \n\n4. How much of the reported improvement (e.g. Table 6) is due to the streaming method (KV-reuse + contiguous RoPE + in-domain SFT) versus the in-domain SFT data?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The KV cache design including persistent attention sinks, a long text window, a short vision window, plus contiguous RoPE shifting is conceptually sound and easy to implement. It delivers lower latency and reduced GPU memory usage while outperforming baselines on the proposed benchmark.\n\n\n2. The overlapped-chunk, full-attention SFT strategy is an intuitive way to teach the model the streaming cache pattern without training on prohibitively long contexts.\n\n\n3. The curated long-video captioning corpus and the Inf-Streams-Eval benchmark fill a gap for real-time, long-horizon evaluation in VLMs and should be valuable to the community.\n\n\n4. The paper is well organized and clearly written, making the method and experiments easy to follow.", "weaknesses": "1. The choice to retain 512 sink tokens, a 512-token text window, and a 16-second vision window appears empirical and manually tuned. Moreover, the ablation supporting this setting is run primarily on a basketball-only subset, which raises concerns about domain generality. A single fixed policy may either waste KV budget in slow scenes or evict critical evidence too early in fast ones. Therefore, more adaptive and fine-grained design would be expected. \n\n2. Current KV cache eviction follows a naive FIFO rule, and there is no scoring or compression to retain semantically salient frames. This may harm long-horizon reasoning, especially for sparse-action videos. Therefore, this weakness can undermine the generalization of the proposed method.\n\n3. The proposed benchmark has leakage risk. Sports broadcasts are heavily duplicated and often reuse the same commentary audio, so near-duplicates can slip across training dataset and benchmark. The paper does not document near-duplicate filtering, making memorization of phrasing/style a real possibility that could inflate results. \n\n4. This work only shows results on SFT Qwen2.5-VL-Instruct-7B, making it hard to claim the method is architecture agnostic or base model agnostic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)