Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions
Authors: Kecheng Zhang
Zongxin Yang
Mingfei Han
Haihong Hao
Yunzhi Zhuge
Changlin Li
junhan zhao
Zhihui Li
Xiaojun Chang
Venue: ICLR 2026 Poster (2026)
Keywords: Online video understanding; Video Question Answering; Vision-Language Models; Decision
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Progressive, Causal Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions Methods

Abstract: Visual agents operating in the wild must respond to queries precisely when sufficient evidence first appears in a video stream, a critical capability that is overlooked by conventional video LLMs evaluated in offline settings. The shift to an online, streaming paradigm introduces significant challenges: a lack of decision transparency, the difficulty of aligning response timing with visual evidence, and the need to maintain a global, causally consistent understanding under tight computational budgets. To address these issues, we propose a novel framework that decouples reasoning control from memory integration. We introduce Thinking-QwenVL, an instantiation of this framework with two core components. First, the Active Thinking Decision Maker (ATDM) is a transparent reasoning controller that externalizes its decision process using observable progress ($\boldsymbol{\rho}$) and confidence ($\boldsymbol{c}$) metrics. This allows it to precisely time its response $t_r$ to match the first-sufficient-evidence timestamp $t^\star$ while streaming its reasoning to the user. Second, the Hierarchical Progressive Semantic Integration (HPSI) module acts as an efficient memory system. It employs a set of learnable, multi-level aggregation tokens that are propagated across clips to build a rich, global cognitive state without exceeding token budgets. Extensive experiments demonstrate the effectiveness of ATDM and HPSI, e.g., Thinking-QwenVL improves the accuracy of the previous state-of-the-art from 67.63\% to 71.60\% on the StreamingBench benchmark.

Published as a conference paper at ICLR 2026
PROGRESSIVE ONLINE VIDEO UNDERSTANDING WITH EVIDENCE-ALIGHT TIMING AND TRANSPARENT DE- CISIONS
Kecheng Zhang' Zongxin Yang² Mingfei Han Haihong Hao' Yunzhi Zhuge' Changlin Li
Junhan Zhao Zhihui Li Xiaojun Chang!
University Science and Technology of China Harvard University DDeparmeent CV, MBZUAI Dalian University of Technology Stanford University Chicago University
ABSTRACT
Visual agents operating in the wild must respond queries precisely when suf- ficient evidence first appears in video stream, a critical capability that is over- looked by conventional video LLMs evaluated in offline settings. The shift an online, streaming paradigm introduces significant challenges: lack of decision transparency, the difficulty of aligning response timing with visual evidence, and the need to maintain a global, causally consistent understanding under tight com- putational budgets. To address these issues, we propose a novel framework that decouples reasoning control from memory integration. We introduce Thinking- VwenVL, an instantiation of this framework with two core components. First, the Active Thinking Decision Maker (ATDM) is transparent reasoning controller that externalizes its decision process using observable progress (p) and confidence (e) metries. This allows it to precisely time its response i, to match the first- sufficient-evidennee timestamp t* while streaming its reasoning to the user. Sec- ond, the Hierarchical Progressive Semantic Integration (HPSI) module acts an efficient memory system. It employs set of learnable, multi-leve aggregation to- kens that are propagated across clips to build a rich, global cognitive state without exceeding token budgets. Extensive experiments demonstrate the effectiveness of ATDM and HPSI, e.g., Thinking- -wegn improves the accuracy of the previous tate-of-theearr from 67.63% to 71.60% on the StreamingBench benchmark.
INTRODUCTION
Visual wvidence-aligned response timing is central visual agents operating in the wild: an assistant should answer only once the video first contains sufficient evidence, and it should show when and why (Cai et al., 2025; Subramanian et al., 2024). Consider a domestic robot asked, "is the kettle boiling?" It should wait for visible steam or a rolling boil and report immediately at the first frame these signals appear to avoid danger (Li et al., 2019). driver-assistanee agent queried, "is safe to turn right?" must defer until the crosswalk and signal are jointly favorable.
Despite rapid progress, representative iideo-understanding LLMs such VideoLLaMA3 (Zhang et al., 2025), InternVI.3 (Zhu et al., 2025), and Qwen2-VI. (Wang et al., 2024a) are commonly evaluated in idealized offline regimes. The entire video is preloaded; frames clips may be retrieved and re-encoded multiple times; and global reasoning precedes response generation. This practice diverges from interactive, real-world operation which users ask at time ty. but the earliest sufficient evidence may not appear until t*. A system should respond at é, only when f, s t*; otherwise, avoidable compute and queuing delays degrade responsiveness and user experience. These issues motivate the online video understanding setting, which constrains the model to act only on currently accessible visual evidence while enabling perceivable and controllable interaction.
In online use, three aspects become decisive. First, decision transparency and real-time feedback. Collapsing timing into a black-box gate ("answer" vs. "defer") leaves no visibility into timestamps,
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
intermediate conclusions, progress, undermining controllability and trust during streaming inter- action. Second, evidence-alignel response timing. With t.. tr, and t" as defined above, the goal is to minimize 8 |t, t*| under streaming uncertainty and latency constraints without sacrificing correctness; recent benchmarks (e.g., OVOBench (Niu et al., 2025), RTVBench (Xun et al., 2025)) stratify tasks by the relation between b and t², yet many systems fix 4, bg or use centered win- dows. Third, global, causal updates under tight budgets. Let V {UI,.... et denote the observed stream and f2 compact cognition state summarizing entities, events, and relations supported by Vr- As new clips arrive, the model should revise hypotheses and propagate temporal/spatial constraints globally not merely apply myopic, clip-local updates that break the storyline or causal consistency.
We address these needs with two complementary ideas that separate reasoning control from mem- ory/inlegratinn. i) Evidence-aligned, transparent timing (reasoning controller). We replace a single opaque gate with multi-stage, observable decision process that surfaces evidence-aligned timestamps, stage-wise progress p. concise rationales, and an estimated response time ty; the con- troller self self-triggers cross-clip reflection when confidence is low, so users can see why now or why wait. ii) Progressive and global causal state (memory & integration) with evolving visual ev- idence. We maintain and refine a compact, relation-aware ht under token/latency budgets so that cross-clip evidence updates the global understanding the stream unfolds. The online framework proceeds stepwise: ingest the next clip and update /c+1; the controller consults (Ax+11, q). advances p and c, and decides to answer (emitting 4p) or wait/reflect; timestamps and interim conclusions are streamed to users for auditable, real-time interaction.
Building on these ideas, we present Thinking-QwenVe which instantiates the framework with two modules. Active Thinking Decision Maker (ATDM) implements the controller: it factorizes timing into sub-goals with observable progress and confidence predicts an evidence-aligne i, via the quantitative indicators (p. c), and self-uriggers cross-clip reflection when needed. In doing so, it streams timestamps, interim conclusions, and rationale snippets to the user in real time, macisi becoming transparent, observable, and quantifiable with real-time progress and response feedback. Hierarchical Progressive Semantic Integration (HPSI) implements memory and integration inside the Aision-languae decoder: at multiple decoder depths (e.g., lower/middll/upper thirds), it inserts small set of learnable multi-level aggregation tokens p that attend to frame/clip tokens via structured sparse attention. The p tokens are carried forward across clips as part of hy that is refined new clips arrive, enabling causal, relation- prrtrion-reeeviingg updates the global visual view without inflating the token budget.
We evaluate on benchmarks designed for online video understanding, including Streaming- Bench (Lin al., 2024), OVOBench (Niu et al., 2025). OVBench (Huang et al., 2024), and RTVBench (Xun et al., 2025), where Thinking-Qwen's ATDM of 71.6%, 46.9%, 35.6%, and 35.9%, respectively. Thinking- Thinking-QwenVi VL also maintains competitive long-video performance- up to 67.7 % on VideoMME (Fu et al., 2024) and 68.3% on MLVU (Zhou et al., 2024) primariyy due to HPSI that enables segment-wise attention perception and cross-clip causal relations preservation. In summary, our contributions are:
attains strong results due to HPSI and
We formalize evidence-aligned timing in the online regime via tr,tt,))) and deviation 8, elevate decision transparency to a first-class objective for streaming interaction, and propose a two-part framework Thinking Seinking-QwenVVi for online video understanding.
Combining ATDM and HPSI, we instantiate the framework with a controller that exposes and and aligns f,- to first-suffccient evidence with self-triggered reflection, and a hierarchi- cal integration module with learnable multi-depth, multi-level aggregation tokens that guides segment-wiss attention enhancement and preserves cross-clip relations, enabling globally con- sistent updates of ht under tight budgets.
RELATED WORK
Offline Long Video Understanding. Research on long-form video understanding investigates how to process vast numbers of visual tokens within limited context windows and constrained compute. Recent efforts have extended capability from short clips to videos exceeding ten minutes (Shen et al., 2024; Xue et al., 2024; Wang et al., 2024c; Zohar et al., 2024). Representative lines include adapt- ing image-centric LMMs to long videos (e.g., Long VA building on LLaVA (Zhang et al., 2024b; Liu et al., 2023)), retrieval over graph/tree indices to shorten effective context (VideoRAG (Luo et al., 2024), Ommi-AdaVideoRAA VideoRAG (Xue et al., 2025)), and improved temporal selection and train-
ing curricula (Videol. I LaMA3 with differential frame pruning and vision-centric multi-stage train-
2

---

Published as a conference paper at ICLR 2026
Time
512s
1263: L.
1276x (+) at.
1380s
Question at 1263s:
F(Q. Min
The and the wuman am holding nocklae their hands, what do they do Lae necklace nect?
Perception
" Bopprti
1 Get Guidances CI -2 Split to (S,
Offline
Ingger Self motivated Refine when Clip
Chp
No decision
CH
Sub-que eu--uucttinn Status
timestung before the and the vićeo response?
asprepricte
1op
Dispidur
Reaction
+
Questiz
Amsanr
What..
Binary
Q1
Response
Response
Confitance
-
8
Visual clue 1276s:
Aooile
forroahd rnferatioo
Decision
D
Get anew-- o
setrotad progmss
nswer for
Give Caption under
Time
A: The man put on the woman's neck.
by shes feecbwck
(S,) if Relevant
[a) Current VLLTe
(b) Ours
Figure 1: Comparing paradigms vs. Ours. Given query Q, offline VLLMS answer only after the full video is available (t = 7), while streaming models answer at the query moment (t = fgg) neither ensures siddence-alignedd timing with the earliest evidence time t*. Our method decomposes Q into sub-goals and maintains a progress estimate p. emitting real-time, stage-wise feedback at every step and selecting a response time I, Rs 1*, thereby reducing latency without sacrificing correctness and avoiding iitemaatio-vvccumn waiting.
ing (Zhang al., 2025)). InternVL3 (Zhu al., 2025) further explores variable visual position encoding and text-time scaling to better align temporal and textual streams. While recent advances improve offline reasoning over long videos, most methods assume full-video access and prioritize token reduction. So, offline pipelines sidestep ineeraction-critccn needs: evidence-aligne response timing. These gaps motivate our online formulation, which preserves progressive understanding and couples inference with timely feedback.
Online Video Understanding, To better define and evaluate online video understanding. recent benchmarks such as OVOBench, StreamingBench, and RTVBench (Niu et al., 2025; Lin et al., 2024: Xun et al., 2025) have initiated systematic investigations in open-source settings. Existing methods largely split into two families. In fixed-response streaming (simply t, = ty). Stream- Bridge (Wang et al., 2025), StreamChat (Xiong et al., 2025), VideoStreaming (Qian et al., 2024), Flash- VStream (Zhang et al., 2024a), and VideoLLM-Onlie streaming readout, alignment, and memory, but do not make decision or align f, to t", In timestamp- deciding methods, Dispider (Qian et al., 2025) compresses incoming clips and applies a binary head for answerability, yet the decision is opaque, repeatedly invoked without a principled stopping rule, and prone to prolonged al., 2025) ties answerability scene transitions, but scene change does not guarantee sufficient ev- idence, rendering, it brittle and threshold-sensitive. By contrast, our formulation provides evidence- aligned timing and transparent decision progress, directly addressing these limitations. We employ the same kinlle-pass, single-turn streaming regime rather than the multi-round vidco processing described in StreamBridge (Wang et al., 2025) to align with traditional streaming methods.
(Chen et al., 2024a) mainly optimize
an-aaweeeerbee states that appear stalled to users; Timechat-Onin Onnlnne (Yao
3 THINKING- -WENVL
Overview. We pursue visual siidence-aligned progressive, causal understanding of a video stream. Let V, = (uI...., 0t} be the visible clips and he a compact cognition state. With each new clip 1441. HPSI updates the state via fi+1 = ((,,, 1)+1), using a small set of multi-depth aggrega- tion tokens with structured sparse attention to aggregate locally, integrate hierarchically, and prop- response-timing decision (t, = min { [i]F(he, Q) = A}) into a sequence of sub-goals S and maintains time-indexed tuples (t), (t), (t)) (u(tt)subb-ee sub-answer a, confidence c, and progress p-to quantify reasoning and ex- pose rationales. F denotes decision function, A denotes answer for user question Q. ATDM returns the final response time f, when each sub-goal s E S is solved.
agate causally. On top of h4, ATDM decomposes the evidence-aligned
3.1 HIERARCHICAL PROGRESSIVE SEMANTIC INTEGRATION (HPSI)
To address the grag-progressivee, causal understanding of the ever-expanding visible set V,-we introduce Hierarchical Progressive Semantic Integration (HPSI). HPSI equips the model with a compact, relation- preserving cognition state that is advanced as new clips arrive. Concretely, insert a small number of learnable aggregation tokens p at multiple depths and enforce structured sparsity that evidence is aggregated locally, integrated hierarchically, and propagated causally.
3

---

Published as a conference paper at ICLR 2026
Input tokens
(a)
Tnggerr aggregation takens insert operation
Text
Video
First-level Second-leve
Adapter Pooling number Thhrndlleve
Talegration williins
Toksns of first frame in video
One video clip
(b)
(c) A Altoolion Mask
Mosk
[ frames]
18/7t 0l
As 9g
LLM Deooder Layer
Jevel
[0, 14/3
LLM Deooder Layer
P franes
ICM Disoddr Layer
[14/3,2433]
frames
LLM Decoder Layer
ILM Decoder Layee
[2L/3,
LCM Dasoddr Layer
Figure 2: (a) Visual information aggregation flow diagram. (b) The dynamic integration oper- ation in LLM with a single clip as an example. The aggregation tokens are initialized in layer 1, layer 1 and layer 2I//3 according the aggregation tokens of the previous level that can sup- port dynamic resolution style, and these tokens are passed forward layer by layer within the LLM to aggregate the visual information of the clip by the causal ability of the LLM and the coefficient attention mask constructed in (c). (c) Attention mask and its changes.
Dynamic- Resoluttno Progressive Integration Overview. We segment the video into n clips and, for each clip,, append a dynamic number of aggregation tokens after its visual tokens. These tokens summarize the semantic content of each clip while leveraging the causal reasoning capabilities of LLMss. Let the original input be I concat( u, w), with text tokens and visual tokens (.elip......, Velip.,). ). We introduce three aggregation levels j E {1, 2, 3), progressively inserted at transformer depths é, E {0, 2//33 with target token ratios ry E (3x, 2x, 1x). }. For clip,, level-j produces n25 (?) tokens (i) Pelip," After inserting the last (level-3) aggregation tokens, the input
sequence becomes
(2) I = concat(w. Vclip, Peip, Pelip, : Pelip,,"... Velip, Pelip, Pelip, Pelip., w).
(1)
Aggregation Tokens Initialization. Each aggregation token is initialized via adaptive average pool- ing over its clip's 's visual tokens; let j 1, 2, denote the aggregation level, Peip, Velp, and N.c denote the final level's token count (adjustable match different video resolutions):
(0)
P Pelp: AdapterPool (Palp.) Noe))Noe)
(
(2)
where Velip; Rd. represents the n, visual tokens of the i-th clip, and AdapterPool : Ra. xd - X outputs = (4 - No tokens of dimension d.
To guide the model to integrate visual information into these tokens, we construct sparse, struc- tured attention masks (see Fig. 2) that enforce hierarchical visibility: each level-j aggregation token attends only to the preceding level's tokens, ensuring directional semantic consolidation. Text to- kens attend causally only the last-level aggregation tokens each layer. Additionally, retain visibility for the first-frame tokens of each clip to preserve crucial anchor cues.
Progressive Integration. Unlike single-layer average pooling (e.g., Long VA (Zhang et al., 2024b)), HPSI exploits decoder depth L by assigning different aggregation strengths across three layer groups: 1) layers [0, IL/3] integrate raw visual tokens; 2) layers [LL/3, 2L/3] integrate the pre- vious level's 's tokens; and 3) layers [2L/3, L.] refine high-level semantics. With token ratios 3:2:1 1, information gradually condensed into fewer, more meaningful tokens. Let L.- (0, L/3, 2L/3).
(k) 7 concat (w. (Velip,. (Pelp,) m(f)) n
)== w).
m(f) = 1
38 L
l Le C,, (3)
where n, L, and m(() denote the number of clips, the layer index that triggers insertion, and the highest visible aggregation level per clip layer l. The output h; at layer E {1,..., L} is
h TransformerBlockk (I(() @ Tiec, h (1 TTe,,) )
(4)
where Itec, is 1 when / E L3 and 0 otherwise.
Finally, the progressive integration objective in the semantic space of LLM can be defined as:
min T Imoogration
ΣΣ
(||
(00) Pain:
Pool (Delip)) Pool(vellp))
||,
Palnn
()0))
Pelin,
(s-1)(0)
l1) (5)
1=0
4

---

Published as a conference paper at ICLR 2026
1080s
1101s F*
Visual Encoder
Part-3: Give Caption under Cl
boy weating T-aliri -y after
His leasing?
Guidances Cig
Part-1: Get
to (S,
Part-2: Split
HPSI
Sub question Status
ATDM
for (S,
Part-4; 4: Fill Answer
at
LLM
captioning under
Type
relevant
Empower Update
Sqi
atppn
Part-S: Active Thinking
Answer: shoots an arrow.
Get Aqd)
New
solved
Estimated
Until 100%
Response Now
Figure 3: Pipeline of -wenkiLngQQeeLL..... Given streamed clips and a query Q, ATDM generates cresstion-uided caption instructions, decomposes Q into sub-questions, and iteratively extracts ev- idence from each clip (with progressive visual integration using HPSI), updating sub-answers with progress p [0, 1] and confidence [o, 1]. This process runs in parallel across clips and permits to trigger active reflection according to c. The model emits an answer at t, = t; once p(t.) 1.
which encourages faithful integration toward clip evidence and smooth refinement across levels.
Layer-wise Task Decomposition for Hierarchical Aggregation. Conceptually, HPSI treats the depth of a transformer as a division of labor for aggregation, rather than a single pooling step. We explicitly assign different aggregation roles different layer segments: shallow layers focus preserving fine-grained local evidence, middle layers consolidate mid-range temporal and structural patterns, and deep layers perform strong semantic condensation summary tokens. In this view, the three levels are not ad-hoc tricks but a progressive aggregation pipeline that incrementally transforms dense visual streams into a small, semantically rich state.
a compact set of high-level
3.2 ACTIVE THINKING DECISION MAKER (ATDM)
Overview. ATDM converts online answering into compact, observable chain-of-thought that car- ries explicit telemetry. Given streamed video (segmented into clips) and query g. ATDM (i) derives question-guide caption requirements and produces a per-elip summary, (ii) decomposes q into K concrete sub-questions (iii) extracts and updates sub sub-answers with per-step progress p E [o, and confidence с E [0, 1]x, and (iv) declares readiness and answers when all required sub-answers are confidently resolved-thereby aligning I, with the first-sufficient evidence (*. A modular wrapper schedules the per-elip evidence extraction and sub-answer updates in parallel across consecutive clips clip, clip, : reducing idle time and preserving responsiveness.
Active, Self-triggered Thinking, Beyond the fixed CoT flow, ATDM monitors and over time; when confidence remains low or the stream exhibits major semantic shifts, it self-triggers reflection that revisits prior summaries, constructs cross-clip causal links, and revises hypotheses and metrics (p.c) c). This mechanism prevents myopic updates, improves evidence alignment, and yields more accurate, timely responses under evolving visual evidence.
Combining the above ideas, the Five-Part Chain-of-Thought Active Thinking Decision Maker Process (ATDM) is as follows. Only Parts 3 and 4 require reasoning to be processed iteratively across video clips rather than forcing five parts
executed sequentially for each clip.
Part-I: Question-Guided Captioning instructions. Unlike general video captioning models that produce either overly generic descriptions (e.g., "a person is cooking in cluttered kitchen")
or irrelevant ones due to unaligned attention, we first ask the model to analyze the question and generate its own captioning guidelines, termed caption instructions Cly. These instructions focus the captioning process on questioning-relevan elements.
Task: Analyze the user's question and define exact observation requirements for video cap-
tioning to help answer it. Output: [< Calpaiion Requi rements List |>]
Part-2: Question Decomposition. Inspired by the progression of human-liks reasoning, where answering complex questions involves progressively addressing multiple semantic dimensions, we decompose the original question into set of sub-questions {Sy). These sub-questions structure the reasoning process and allow us quantify decision progress.
Task: Break the user's question down into set of precise, concrete uub-questions. Each sub- question can focus on an observable aspect of the video (c.g., object, person, action, spatial
5

---

Published as a conference paper at ICLR 2026
Question What text is visible on the right side of the street right now?
Excavator
B. CRANE.
@ WEST NEW Y YORK.
D. L. Loader
guedtonn accuntedy. caphinn
rigitemnnt
3ar Tae spmEs 8:23:04 02.00. cainy New City,
contral its jocation
aavigeiig through
idcntffictioo the
CRANE TWEST YORK Jazd Loader
sid the condtrction hamkees
tanding indicde
lased on their shapa posiitore
A reatnn WEST NEW YORK viabln, provitngg outtot the locdios. The proseeee rullact the deveeerse sky, ethancing uhbn simsspthen. Consmrction equipenett like an excerator and lester am partally stable behnn the bameer he scene captane the braeting city life smodet maryy waster consitions
contact incioniing the locaton being rnfernel TW NEW YORK'
dollowtep quedtoons are demood:
right
tustis visible the skde stroe??
Part Ouguut usdaael cuesioos
theme other styect visiie
the street
(Tas itiba Easl spdsted state
custtoos listed follows)
type: tast
ppp lect
type otject
2 hhrm any lent si right thestmme??
1: Whd het siwibble right side stred" value
a tasre ass object visibl sqatt of strett
coetidesce
D.
confidhecs::
Estirated proprss 3
Oppe:
ype object
Estimattd progres :200
them best vedble iigh dide srree!
bect seetieer the nggtt iide the street? voloe MM WSS NPW YORK"
q them ary other sdgect vtabio en the right atte of the srret?
Vallae WVPST NEW YORK
Value cacoraaat crast, Joader
condidoke: 0.35
confidenns
Part 5/ 0:::0:30:: "Cuual chaio. All cip No relovaat evidetee So cucor ats bunes, keyp dlue states uxdurgee
Figure 4: Visualization of qualitative example showcasing how our ATDM framework achieves successful decision and video reasoning.
relation, etc.). These sub-questions represent the key things that must be visually or aurally ver- ified in the video to answer the main question. Output: [<] Required Subquestions }>]
Part-3: Video Clip Captioning, Based on the caption instructions CIyy from part-1, 1,, the model generates a summary (C,) of the clip content. This streaming captioning continues until the model determines that sufficient information is available to answer the question.
Task: Watch the current video clip and generate descriptive caption, you must focus your
caption on the following key observation points: <I Caption Requirements List Output: < /eetalled capt ion that fulfills the requirements
Part-4: Sub-answer Extraction and Filling. Using {S₂] and the current clip caption {C₂}. the model attempts to answer each sub-question, forming a set of partial answers {SA,). At each time step, the most recent (5Ag) is fed back into the model, enabling it to track historical answer states across frames. This is crucial for effective task decomposition, corroborated in (Jang al., 2025).
Read [Question], [<| Required Subquestion: |>] and the caption of the cur- rent vidco clip [<| Capt ion |>] For each subquestion, determine whether the caption pro- vides enough information answer it: If yes: provide appropriate answer and confidence score e E 1]. If or uncertain: set value is '?' and e = 0.0.
Task:
Output: < Updated subquestion ion state (value, c) and progress pl>
Low Confidence or Major Shifts. Rigid step-by-step rea- soning can lead to tunnel vision, causing the model to miss globally coherent information and the relationships between continuously changing information. To mitigate this, we monitor confidence scores for each {SAa). When scores exhibit sharp drops or remain low across time, the model trig- gers active thinking: it reviews prior {C₂), detects temporal shifts, constructs causal chains across clips, and re-evaluates sub-answers accordingly.
Part-5: Active Thinking Trigger:
Task: Given [question] Past reasoning state: [past cot state] and the [new clip caption]. 1) Cross-clip causal reasoning. Build an explicit, ordered chain that shows how evidence from each new clip supports, contradicts, refines the current hypothesis. 2) Consis- tency check. Detect attributes that are contradicted, supported with higher certainty, still low- confidence (5 0,50), or missing. 3) Update the attribute list and return. Output: < </Updat ed subquestion state and progress!
The explicit progress and confidence signals (p, c) transform ATDM from a sequence of memoryless binary decisions into a genuinely bistory-aware control pro- cess. Each judgment about whether the current visual evidence is "sufficient" is not an isolated yes/no query. Treating it as a mere collection of independent binary decisions breaks the informa- tion chain. In contrast, ATDM does not repeatedly decide "answer or wait" based only the current visual chunk; it observes its own past decisions and scores and can refine or revise them as addi- tional evidence arrives. The continuous pair (p. c) therefore carries substantially higher information content in context than a single 0/1 gate, as it not only encodes a bare "stop/continue" signal but also compresses the entire history of intermediate judgments into compact quantitative state.
History-aware Decision Process.

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy (100%) comparison on StreamingBench focusing on Real-Time Visual Under- standing tasks. f indicates the reproduced results. The meaning of each subtask is in Appendix A.5.
Model
Size
Framos
Pub
Subcasks
OP
CR
CS
ATP
BU
TR
PR
SU
ACP
CT
All
Human
89.47
92.00
93.00
91.47
95.65
92.52
88.00
88.75
89.74
91.30
91.46
Propristary MILLMs
Gemini 1.5 pro
fps
79002
80.47
83.54
79.67
90.00
84.34
71.18
64.25
71.95
48.70
GPT-4o
77.11
80.47
33.91
70.19
$3.80
69.12
49.22
Clande 3.5 Sonnet
75.33
90.47
$4.09
75. 35.39
79:53
61.11
61.79
69.
43.09
72.44
Open-source Offinne Long
Video LLMs
Vade-IJaMA2
7R
32.
ARXIV34
55.86
55.47
57.41
55.17
52. 80
43.61
39.81
42.68
45.61
35.23
49.52 00.52
VILA-1.5
8B
14
ARXIV25
53.68
89.22
70.98
56.86
53.42
53.89
54.63
48.78
50.14
17.82
$2.32
Video-CCMM
14B
96
ARXIV24
56.40
57.81
65.30
62.75
64.60
51.40
42.59
47.97
49.55
31.61
53.96
LongWA
ARXIV24
70.03
63.28
61.20
62.73
39.50
61.11
53.66
54.67
34.72
59.96
ARXIV24
60.94
67.70
62.93
54.96
63.72
Kangaroo
ARXIV34
71.12
70.66
67.08
56.48
62.04
38.86
LLaVA-NoXT-Ville
32B
BLO022
78.20
70.31
73,80
76.80
63.35
69.78
57.41
56. 56.10
64.31
86
96
Min.CPM---2266
8R
ARXIV25
71.93
71.00
77.02
75.82
64.60
65.73
70.17
56.10
62.32
53.37
67.44
LLaVA-OneVisinn Vnsennsoon
7B
32
CVER25
80.38
74.22
76.03
90.72
72.67
71.65
67.59
65.45
65.72
45.08
71.12
Qwen2.5VVL
7B
ARXIV24
78.32
80.47
78.86
80.45
76.73
78.50
79.63
63.41
66.
53.19
73.68
Offline Long VLLMs
62:75
00.73
53.28
Open-source Onlinne Video LL
Hlah- VSueam
JCCV25
25,89
43:57
24.91
27.33
13.08
18.52
25.20
23.87
49:70
V.deooIMM-oliie M-oeliiee
2/ps
CVPR34
39.07
40.06
34.49
31.05
45.96
32.40
31.48
42.49
27.88
Disgider
7R
1fps
CVPR25
74.92
75.53
74.10
73.08
74.44
59.92
76.14
62.91
62.
45.80
67.63
Thinking-QwenVL. QwennVL
7B
1pps
70.27
66.67
80.00
77.97
79.31
66
78.26
68.18
72.31
$2.38
71.60 10011
Flash VStream!
7B
1fps
JCCV25
24.52
21.53
21.45
19.00
26.42
26.56
22.22
22.36
21.45
24.35
22.53
Flach VStrem +ATDM
7B
1pps
ICCV25
28.53
27.34
24.68
26.15
31.01
27.00
25.00
24.90
27.64
26.80
26.55 #4001
Model
ACR
FPD
Real.
Back.
Forw.
Overall
Human Agents
92.6
91.1
93.2
92.3
92.9
92.8
Gemini 1.5 Pro
67.0
68.3
70.8
62.3
$7.2
65.3
LLaVA-NeXT Videe-78
59.6
72.3
41.7
53.1
Visii--n
58.7
71.3
62.8
45.0
50.9
52.9
Qwen2-NL-7B
53.2
66.3
60.7
48.6
48.9
52.7
40%
LongVUL-7
49.5
68.3
57.4
39.5
48.5
48.5
+5.3%
Open Online Video-III. I.I.s
Mlahh-SSteaam-77 Dispide--7B
32.1
29.7
29.9
25.4
44.2
33.2
23.9
45.5
20.8
17.7
49.5
61.4
54.5
36.1
34.7
41.8
Ours (L 93,75%)
54.9
67.5
55.8
47.4
28.6
46.9
TimeChat-Onling (100%)
45.8
09.3
41.7
46.7
Ours (100%)
57.2
75.0
64.7
44.3
37.6
52.5
Figure 5: Accuracy improvements over Table 2: Accuracy on OVOBench. Real.: Real-Time Vi- our baseline model on sub-tasks of the sual Perception, Back.: Backward Tracing, Forw.: For- RTVBench under the same experimen- ward Active Responding. The specific requirements of tal conditions as the RTVBench paper. Forw. task resulted in V not being able to The overall accuracy of our model in- response in demanded format.
creased from 32.75% to 35.87%.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
Training Details. Our model is built upon the VL- architecture and all experiments are conducted using 4x A100-80G GPUs The learning rate is set to 10-6, and the model is configured with a maximum input resolution of 448 x 448. More hyperparameter details can be found in Appendix A.3. In our setting, the number of clip frames 32, the number of the three aggregation tokens is 3, 2. and 1 frames's 's tokens of the videos. So, the number of the final aggregation tokens can be changed with the video resolution setting.
Benchmarks. Our evaluation spans complementary online and offline long-video QA suites that jointly stress real-time perception, temporal alignment, and long-horizon reasoning. Streaming- Bench (Lin et al., 2024) targets low-latency. timestamped queries under streaming constraints. OVOBench (Niu et al., 2025) enforces answer-when-readyy timing-models defer responses until sufficient future evidence (real-time perception, forward tracking, active responding). RTVBench and OV OVBench (Xun et al., 2025; Huang et al., 2024) probe continuous perception and online spatio- anchoring.
temporal reasoning via multi-time timestamp. hierarchical questions and Past/Current/Futte For offline long- form understanding, VideOMME, MLVU, Long VideoBench, and LV LVBench (Fu et al., 2024; Zhou et al., 2024; Wu et al., 2024; Wang et al., 2024b) cover short clips hour-long
7

---

Published as a conference paper at ICLR 2026
Table 3: Accuracy on offline long-video benchmarks: MLVU, LongVideoBench, VideoMME (w/o subtitles), and LVBench. Videos are divided into 16-frame clips in HPSI (493.75% signi- fies 75% reduction in video frames) and up to 256 frames are sampled per video. "100%" for TimeChat-Online (based on Qwen2.5-VL) denotes no dropping-only model parameters; we repro- duce this setting in the last row. "(or%""ours) indicates no insertion-only attention redistribution.
Vide
Model
Frumes
MLVU
LongVideoBench
LVBench
Overall
Long
Video Length
3~ 120 min
8 sec~600
1~60 min
30~60
30~120 min
Open-source Offline Video LLMs
LLaMA-VID-7B
[BCCV24]
1fps
33.2
23.9
MovieChat--B
[CV9R24)
2048
25.8
38.2
33.4
22.5
LLaVA-NcXT-Viddo--
[BL0G24)
32
43.5
46.6
32.2
VideoChat2-7B
[CVP924]
16
47.9
39.3
39.5
33.2
32.5
VA-7V
[ARXIV24]
128
56.3
$2.6
46.2
35.7
Kangaroo-7B
[ARXIV34]
64
61.0
54.2
56.0
46.6
39,4
Video-XI-7B
[CV9R25)
128
64.9
55.5
49.2
Qwen2.5-V1L-78
[ARXIVI5]
16ps
61.5
63.2
50.4
43.1
VISTA-7B
[CV9R25]
62.1
53.1
55.5
49.2
39.0
Open-source Online Video LLMs
Dispider-7H 7B
(CVI)25)
1fps
61.7
$7.2
VideoChat-Online-SB
(CVPR25)
2fps
52.8
44,9
Shinking-Qwente
lfps 75%)
56.3
49.1
TimeChat Chat--nniine-77-
(ACMDS)
Tips (1009)
62.6
55.4
62.4
48.4
A - Q-wn2.5-VL.
-4.5
-6.1
-0.8
-1.6
Thinking-Qwen's
1pps (100%)
62.0
67.7
56.4
43.6
A -VW
+1.4
+0.5
+4.5
+6.0
+0.5
videos, emphasizing granular recall and cross-scale reasoning. We follow official scoring protocols (per-suite QA accuracy and aggregates); full task/metric definitions are in Appendix $A.5.
Comparative Models. 1) Proprictary Assistants. For completeness, the strong closed-source models as upper-bound references are included: GPT-4o (OpenAI, 2024), Gemini 1.5 Pro (Team et al., 2023), and Claude 3.5 Sonnet (Anthropic, 2024). 2) Offline Long-Video Video MILMs. We com- pare to the SOTA long-context video understanding models: Video-LLaMAA (Cheng et al., 2024), VideoChat2 (Li et al., 2024b), Video-CCAM (Fei et al., 2024), VILA-1.5 .5 (Lin et al., 2023), LLaMA- VID (Li et al., 2025), Long VA (Zhang et al., 2024b), Kangaroo (Liu et al., 2024b), MiniCPM-V. 2.6 (Yao et al., 2024) and Vidco-XL (Shu et al., 2024), along with commonly reported baselines ( LLaVA-OneVision (Li et al., 2024a), LLaVA-NeXT-Videe (Liu et al., 2024a), InternVL-VE (Chen al., 2024c), Qwen2.5-VL. (Wang et al., 2024a)). 3) Online Video LLMs. Online methods include VideoLLMMonniin (Chen et al., 2024a), Suts--Streamm (Zhang et al., 2024a), Dispider (Qian et al., 2025), and TimeChat(-Onling (Ren et al., 2024).
4.2 MAIN RESULTS
StreaminggBnch. In Table compare our model with recent state-of-the-an systems, including Dispider. Our model achieves an accuracy of 71.60 %, setting a new benchmark for this task. Com- pared to previous online and streaming models, have improved the state-of-the-arr performance by 3.97% on the online task, increasing the accuracy from Dispider's 67.63% % to our 71.60% % Fur- thermore, we also evaluated the elfectiveness of our ATDM approach on models without decision- making capabilities, such as Flash-V and Qwen2.5-VL. The results indicate that, in the case of WlshhVVStramm the model's accuracy increased from 22.53% to 26.58%, representing an im- provement of 4.01 %. This demonstrates the general applicability of our decision- maasiinnmmking method for online video understanding.
OVOBench, RTVBench, and OVBench. In Table 2, we compare our proposed method, Thinking- VLe with existing models on OVOBench. Compared -atsemm which lacks decision- making capabilities (33.2%). and Dispider, which incorporates binary opaque decision-making (41.8%), our model achieves an accuracy of 46.9%, marking an improvement of 4.9% over Dispi- der. Compared to our baseline, the overall accuracy of our model increased on RTVBench from 32.75% to 35.87%. We achieved 35.6% accuracy on OVBench. The performance on sub-tasks is in Fig. 5 and Table 6& 7. The meaning of each symbol in Fig. 5 is: TP- Temporal Perception, VP Visual Perception, SP Scene Perception, PU Phenomenological Understanding, GU Global Understanding, IA- Intent Analysis, FP- Faithfulness Prediction, SR Similarity Reasoning.
8

---

Published as a conference paper at ICLR 2026
Table 4: Impact of level aggregation on VideoMME w/o subs and OVOBench. We ablate by directly removing the corresponding level tokens. + denotes that the first-stage compressed-tokee count set as the final token budget (1x) equivalent to applying adaptive pooling visual tokens before the LLM, as in prior long-video models. FF: First Frame. LV: Level. burden of tokens.
FF LN-1
LN-2
LX-3
OVOBench
VideaMME
AVG
Overall
Real.
Back.
Forw.
Overall
Short
Medium
Long
V
1x
46.9
55.8
47.4
28.6
56.3
66.0
53.9
49.1
51.6
V
X
2x
46.0
53.2
48.5
29.1
56.01
65.6
53.6
49.0
51.0
X
X
3x
49.6
56.7
53.9
31.4
54.7
61.9
54.7
47.6
52.2
42.6
49.1
42.2
30.2
49.7
55.9
48.6
44.7
46.2
X
X
1x
43.4 3335
45.4
52.7
29.9
48.9
1..4
52.4
49.0
45.1
46.2
1330
tmpp v.
Compossisa
Figure Impact of ATDM components. All rep- resents the complete model performance when use ATDM. Each column beyond this represents the ab- lation of the corresponding part of ATDM.
Figure 6: The impact of ATDM on OVO- Bench and StreamingBench subtasks.
VideoMME and MLVU. Although our model is optimized for online scenarios, it still demonstrates competitive performance on long-video benchmarks. Our model achieves 56.3% % on VideoMME, 49.1% on VideoMMMELLngg. and 61.2% on MLVU, outperforming several models specifically de- signed for offline long-video understanding. When the experimental setup is configured to use only the modified attention weight distributions (100%), the accuracy reaches 68.3% on MLVU, 62.0% on Long VideoBench, 67.7% on VideoMME, and 43.6% on LVBench, surpassing existing state- of-the-art offline videv models. Notably, VideoMMFF-Long (30 60 min), it outperforms the leading Qwen2.5-VLL VL.-BB by 6% in accuracy. This strongly demonstrates the effectiveness of our HPSI module for video understanding, as this progressive causal approach that incrementally enhances the model's I's cognitive state proves effective for tasks requiring long-term dependencies.
4.3 ABLATION STUDY
Overview. We conduct a comprehensive ablation study in two dimensions: 1) the impact of hierar- chical integration across different layers, and 2) the contribution of each part in ATDM.
HPSI and Three-Level Aggregation Tokens. Table 4 ablates the per-level insertions of HPSI. A salient finding is that removing levels 2-3 and forcing level (the first LLM layer) to downsample directly to the same token budget as our level-3 setting-ia. i.e., single-shot AdapterPooling baseline accuracy by 3.5% on OVOBench and 7.4% on VideoMME-LLng
applied before the This confirms that one-stage pooling discards fine-grained cues and disrupts long-range, cross-clip dependencies; HPSI cannot be replaced by simple pooling. On offline long-video benchmarks (Ta- ble 3), Thinking Qwen VL further surpasses the baseline by 4.5% on VideOMME, and under the same backbone and comparable data
outperforms TimeChat-Online by 5.9% on MLVU, 6.6% Long VideoBench, and 7.6% % on VideoMMEE-Lng.. Together, these results show that HPSI's multi-depth aggregation tokens and structured sparse attention preserve semantics under tight budgets and enable stronger causal reasoning over extended evidence than single-step pooling.
ATDM and its Components. We evaluate the decision-making capability of ATDM across three models on OVOBench and StreamingBench, as shown in Fig. 6. Models without decision- making capabilities show significant performance improvements with ATDM. For example, on the OVOBench-EPM sub-task, all three models achieve more than 5% accuracy boost. In Table 1, Flash- VStrem'ss performance StreamingBench increases from 22.53% to 26.58%, 4.05 % gain. These results demonstrate that streaming and offline video understanding models, when op- erating under paradigms like f, fy or i, = T, suffer from performance limitations. However, when equipped with decision decision-making capabilities aligned with visual evidence, model accuracy significantly improves. We further isolate the contribution of each component (Pi-P;) on Thinking- QwenVL and Flash- VStream in Fig. 7. Each part is either removed or replaced with alternative operations. More detailed experimental setups are provided in Appendix A.2.

---

Published as a conference paper at ICLR 2026
Robust
Setting
100%
80%
70%
Talmery IMATH
Acc. (%)
71.60
68.75
67.81
Applicability
Framework
T-VSirremm
Ours
Base Model LLaVA-7B
QwenVL-3B QwenVL--7
Acc. (%)
26.58
62.62
71.60
Table 5: Robustness and Applicability. Top: Stress-testing streaming robustness
under abnormal conditions by uniformly
tbroughput Onkensoo
dropping frames after 1 FPS extraction Figure 8: Efficiency on NVIDIA A100 GPUs. Im- (retaining 100%, 80%, 70%). Bottom: pact of the aggregation rate HPSI on FPS and token
The ATDM controller is applied multi- throughput. At 93.75% aggregation rate, our method ple backbones (Flash-VStrean VStream YSSreamm-LLaV-77B matches Flash-VStream VStream's FPS (8.49 vs. 8.45) with Our -weng.--wwnn.2.-.ee..
show- 78x higher avg. token throughput (1261 vs. 16), and slight latency increase (13.2ms vs. 9.5ms).
ing its framework-agnostic utility.
Robustness and Applicability. We evaluate stability under abnormal streaming conditions (miss- ing frames and abrupt scene transitions) by starting from the FPS protocol and uniformly retaining only 80% and 70% of frames. As shown Table 5, the degradation is mild: even at 70% retention, StreamingBench accuracy remains 67.81 %. Combined with the ablations in Table 4, this indi- cates that HPSI and ATDM preserve performance under aggressive frame loss. Beyond robustness, the two modules transfer across backbones and frameworks. Instantiating ATDM the real-time Flash-VStream pipeline (built on LLaVA) consistently activates time-stamped decision making and yields 4.0% accuracy gain on StreamingBench (Table 1). Applying HPSI and ATDM smaller Qwen2.5-VL. backbone, Thinking Qwen VL-3B reaches 62.62 % in Table 5, only about 5% below the 7B Dispider model and far above the 8B VideoLLM-On
model (35. 33..9)). Method- ologically, HPSI requires only a deep I.M,, which we partition into three segments with increasing aggregation strength, without backbone changes (Fig. 2). ATDM relies only on basic instruction- following and visual comprehension, arousing the decision-making ability of generic VLMs easily.
Efficiency and Feasibility. On StreamingBench (NVIDIA A100 GPUs), our 93.75% aggregation setting matches Flash-VStream in frame throughput (8.49 vs. 8.45 FPS) while yielding higher accu- racy (71.60% vs. 22.53%). The explicit decision pipeline adds a modest ~ 3.7s end-to-end latency (13.2s vs. 9.5s). Sweeping aggregation (75% (75%~97.5% 97.5%% monotonically improves total average FPS (5.28-9.11 12) and token throughput (tokens/ / (786~1351) scaled x0.01 in Fig. 8), providing a simple speed-qualit knob. A 3B variant 93.75% still attains average 7.07 FPS / 1050 throughput.
Qualitative Effect of HPSI on ATDM. On the painting clip in Fig. 13, HPSI supplies ATDM with a temporally consolidated memory, yielding captions that explicitly encode state changes over time (e.g.. "the brush moves from right to left" and "the hand adjusts its angle"), rather than a single, static snapshot. In contrast, the baseline-lacking hierarchical integration-produceee short, largely scene-static descriptions with weak cross-frame cohesion. This qualitative gap indicates that HPSI's multi-level aggregation preserves and stabilizes evolving visual evidence across frames, which ATDM then leverages to issue timestamped, evidence-aligned decisions; the same synergy remains observable even when frames are missing or hard cuts introduce abrupt scene transitions. We also provide an intuitive comparison of our model and Flash- VStream'ss 's output examples in Fig. 11& 12.
CONCLUSION
We introduced Thinking-Qwent which integrates Hierarchical Progressive Semantic Integration (HPST) with an Active Thinking Decision Maker (ATDM). HPSI maintains compact, relation- preserving cognition state that is progressively updated as evidence accrues under structured sparsity, while ATDM complements this with a decision process that decomposes tasks into observable sub- goals, enriched by progress metries, confidence estimates, and readiness head aligned to first- sufficient evidence. Empirical evaluation shows that Thinking- Queen VL achieves strong results on online benchmarks and remains competitive on offline long-video tasks, with ablations confirming that HPSI's multi-depth aggregation and ATDM's decision process are key to both accuracy and timely responses.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially supported by New Generation Artificial Inteligence-Natiioaal Science and Technology Major Projectiom(22552201333000 and by The National Natural Science Foundation of China (NSFC) under no. 62573399 and U25A20530.
REFERENCES
Anthropic. Claude 3.5 sonnet, 2024. URL https: httpss//www.anhhropi.co
claude-33 --rode-3-5--o --nnnet.
Min Cai, Qixuan Jin, Jia Zhou, and Xinggang Luo. How transparency shapes the quality of human- robot interaction: An examination of trust, perception, and workload. International Journal of Social Robotics, 17:1335-1362, 05 2025. doi: 10. 11600/22369--0--00222
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin. Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognition, pp. 18407-18418, 2024a.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint arXiv:2406.04325, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-tempont modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.0744 4047077776, 2024.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408. 012440884422. 14023, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal 1lms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: A comprehensive benchmark and memory-augmented method. arXiv preprint arXiv:2501.0053 2024.
Youwon Jang, Woo Suk Choi, Minjoon Jung, Minsu Lee, and Byoung- Tak Zhang. Confidence- guided refinement reasoning for zero-shot question answering, 2025. URL https://arxiv.
org/abs/ 2tt.aa/////0002.0.....
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint wwXiv:2408.03332,, 2024a.
Huao Li, Stephanie Milani, Vigneshram Krishnamoorthy, Michael Lewis, and Katia Sycara. Per- ceptions of domestic robots' normative behavior across cultures. In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 345-3511 2019.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024b.
11

---

Published as a conference paper at ICLR 2026
Yanwei Li, Chengyao Wang, and Jiaya Jia. Alama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2025.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for milms achieve streaming video understanding. arXiv preprini arXiv:2411.03622 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:// 1lava- vl. ww..ithub. ww/booog//0/222 2024 01 30 seeva-next..
Jiajun Liu, Yibing Wang, Hanghang Ma. Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408. 15542, 2024b.
Yongdong Luo, Xiawu Zheng, Xiao Yang. Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval- aermeval- long video com- prehension. arXiv preprint arXiv:2411.13093, 2024.
Junho Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, al. Ovo-bench: How far your video-Ilms from real-world online video understanding? In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18902-18913, 2025.
OpenAl. Hello GPT-4o, 2024. URL https: oww.opeen...omoooo weeenaii.com//ndx//eeelelee http:////hhooooptooooooo--
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models, Advances Neural Information Processing Systems, 37:1 191111933611999 119660, 2024.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang- Dispider: Enabling video iims with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint 2rXiv:2501.032/8. 2025.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14323, 2024.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, al. Longvu: Spatiotemporal adap- tive compression for long video-language understanding. arXiv preprint arXiv:24 10. /7444, 2024.
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409. 240..4454.4 2024.
Karthik Subramanian, Liya Thomas, Melis Sahin, and Ferat Sahin. Supporting human-robot teraction in manufacturing with augmented reality and effective human-computer interaction: review and framework. Machines, 12(10):706, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. Streambridge: Turning your offline video large language model into a proactive streaming assistant. arXiv preprint arXiv:2505.0544 2025.
12

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languen model's ' perception of the world at any resolution. arXiv preprint mmXiv:2499.12191 20244,
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark.
arXiv preprint arXiv:2406.08000 08035, 2024b.
Ziyang Wang, Shoubin Yu, Elias Stengel-Esking Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024c.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. arXiv preprint arXiv:2407.15754 2024.
Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge. Lu Zhang, Jiawen Zhu, and Huchuan Lu. Streaming video understanding and multi-round interaction with memory--nhanee knowledge. arXiv preprint (rXX::2501./3460 2025.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 2024.
Zhucun Xue, Jiangning Zhang, Xurong Xie, Yuxuan Cai, Yong Liu, Xiangtai Li, and Dacheng Tao. Omni-adavideorrg: Omni-contextual adaptive retrieval-augmmented for efficient long video understanding, arXiv preprint arXiv:2506. 2025.
Shuhang Xun, Sicheng Tao. Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan, Hanqian Li, Linghao Zhang, Shikang Wang, et al. Rtv-bench: Benchmarking milm continuous perception, understanding and reasoning through real-time video. arXiv preprint arXiv:2505.02066 2025.
Linli Yao. Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. arXiv preprint arXiv:2504. 17343, 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01000 2024.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13100 2025.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint wrXiv:2406.08006 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. 2024b.
arXiv preprint arXiv:2406. 2688:22200.6682,, 16852,
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.0422. 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.1044 2025.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen- Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint arXiv:2412.10366 2024.
Hansen-Estruch,
13

---

Published as a conference paper at ICLR 2026
A
ADDITIONAL EXPERIMENTAL SETTINGS AND RESULTS
A.1
MORE RESULTS
To thoroughly showcase the capabilities of Thinking- -eeiin,,--eeeV,,,,. we provide supplementary experi- mental data Table 6& 7& 8, and and attention mask visualization Fig. 10.
Table 6: Performance comparison (accuracy 100%) on OVBench. Subtasks in it are AA: Action Anticipation, GSP: Goal/Step Prediction, MP: Movement Prediction, AP: Action Persistence, SV: Step Verification, OP: Object Presence, AR: Action Retrieval, PR: Procedure Recall, TR: Trajectory Retrieval, AL: Action Location, OP: Object Position, AT: Action Trajectory, OT: Object Trajectory, AS: Action Sequence, SL: Step Localization, OES: Object Existence State.
Task Name
Size
FP
THV
PM
SP
STP
TP
AVG
GSP
MP
SV
OP
AR
PR TR
AL
OP
AT
OT
AS
SL
OES
Proprietary Multimodal
Gemini--..-Flachh
71.4
53.6
21.9
36.7
50.0
Open-source Offline Long Video LLMs
InterVV1.2
7B
52.6
60.2
27.6
57.5
$2.0
58.5
38.8
67.1
58.3
38.1
31.3
87.4
37.0
75.4
31.4
5.9
48.7
InternV1.2
48
57.7
57.0
14.4
59.2
49.4
60.0
30.3
61.8
46.3
30.9
20.1
83.0
92.3
70.7
29.4
3.4
44.1
LLaMA-VID
7B
43.6
50.9
19.6
64.0
47.5
46.8
29.4
48.9
$1.2
31.9
11.2
75.7
24.8
59.
26.0
40,0
41.9
LIaVA-Onevisinn
LongVA
7B
67.4
MiniCPM-V226
7B
33.3
35.9
13.0
592
50.8
55.1
25.0
37.
41.7
26.6
11.8
98.3
36.3
66.
26.4
6.2
39.1
Qwen2-VL.
7B
60.3
66.1
22.1
54.9
51.5
51.1
3788
64.4
69.3
35.3
28.5
97.0
49.4
65
50.8
11.7
49.7
LITA
7B
19.2
24.5
19.9
408
48.9
24.9
3.1
27.3
6.4
6.9
14.6
35.2
23.9
27.4
0.5
3,4
20.4
TimeChal
7B
7.7
15.3
18.7
20.6
15.7
11.7
9.1
14.7
9.8
7.5
19.5
13,9
10.3
9.3
10.1
10.8
12.8
VTimoLLM
25.9
Open-source
VodeoLLM--niim
7B
1.8
20.9
32.6
0
233
26.7
0.6
26.6
0.9
19.9
0.9
1.7
8.3
MovieClat
7B
23.1
27.5
23.6
58.4
43.9
40.3
25.6
51.1
23.9
26.9
39
24.4
9
29 3
25.5
21.9
30.9
Fashh-ssreamm
78
26.9
37.6
23.9
60
41.9
40.0
23.4
35.3
26.1
34.7
28.8
27.0
21 4
29
25.6
26.8
31.2
Thinking QweaVL
7B
27.8
39.6
25.9
622
11.4
25.3
36.3
27.1
30.8
27.6
25.1
30.2
26.5
27.6
35.6
Table 7: Accuracy (100%) on RTVBench. We evaluate without audio; otherwise, all set- tings includicc the frame-sampling method follow RTVBench (Xun et al., 2025) for a fair com- parison. Compared with our baseline model, the overall accuracy of our approach improves from 32.75% to 35.87%, yielding gain of 3.12%. The Subtasks in it are: Temporal Perception (TP), Visual Perception (VP), Scene Perception (SP), Global Understanding (GU), Phenomenological Un- derstanding (PU), Intent Analysis (IA), Future Prediction (FP), and Spatiotemporal Reasoning (SR).
Method
Size
TP
VP
GU
PU
FP
SR
Cloond-Soou Sounce
Gemini 2.0 Flash
40.19
45.19
39.31
35.70
15.65
16.78
44.2
38.16
GPT-4o
48.60
$3.59
52.63
45.02
54.32
48.58
54.67
42.75
Open- Source Offine Video Models
Lad2
7B
39.52
42.49
39.85
37.34
42.21
40.92
41.47
33.50
Vidool.aaMA
37.82
39.24
33.54
38.05
33.84
LLeVsoOneVisionn
32.07
78
34.07
38.97
34.45
20.42
35.69
36.33
39
31.22
VW
7B
32.37
37.18
30.73
29.11
35.69
29,36
35.33
33.67
-5.31
31.86
483t
35.88
A.2 DETAILS ABOUT THE COMPONENTS ANALYSIS OF ATDM
In Fig. 7, we present five sets of ablation experiments on the components of ATDM, conducted on two models. These five sets of experiments are based on the following control conditions:
1) Py: Remove Pi (caption instructions Cly); ) demand P2 give captions directly.
2) P2: Disable P2 Question decomposition; retain single query Q and require P4 to answer Q
each step while still emitting per-step confidence c and progress
3) Py: Remove P3 Streaming captioning to test the value of the textual intermediary; P4 is switched from text-only consumption to multimodal extraction directly retrieving evidence from the current visual stream fill sub-answers.
14

---

Published as a conference paper at ICLR 2026
Table Comparison with current online Video understanding LMMs on OVOBench. The subtasks are: i) Real-Time Visual Perception (OCR: Optical Character Recognition, ACR: Action Recogni- tion, ATR: Attribute Recognition, STU: Spatial Understanding. FPD: Future Prediction, OJR: Object Recognition), ii) Backward Tracing (EPM: Episodic Memory, ASI: Action Sequence Identification, HLD: Hallucination Detection), and iii) Forward Active Responding (REC: Repetition Event Count,
SSR: Sequential Steps Recognition, CRR: Clues Reveal Responding).
Medel
Fraames
Real-Time Visual Percepliin
Backward Trading
Fornard Actire Responding
Oversil
OCR
STU
OJR
Avg.
ASI
HLD
Avg.
REC
SSR
Avg.
Avg.
Agonte
Propristary Multinodal Model
Ganisi Pba
Mppx 64
K7.3
67.0
M0.2
$4.5
68.3
67.4
70.8
68.6
75.7
52.7
62.3
35.5
74.2
61.7
$7.2
65.3
GPT-4o
69.1
65.1
65.5
50.0
68.3
63.7
63.6
49.8
71.0
55.4
58.7
27.6
73.2
99.4
$3.4
$8.6
Open-onrce Oflliww Long Video L.IMs
LLaVA NeXT Vadeo 7B UlaVA-OneVivee
64
09.8
59.6
66.4
50.6
72.3
61.4
63.3
$1.2
64.2
9.7
41.7
24.1
67.6
60.5
54.2
53.1
67.1
66.3
49.4
60.3
Qum2-VL1778
50.6
60.0
44-
66.0
90.1
65.3
IntemVL--V2- VU-78
44.9
25.3
$7.6
18px
48.3
68.3
Open-smnce Ouline Vidvo-LL.MM
Flash- VStram-11
lipx
25.5
12.1
29.3
33.7
39.7
28.8
29.9
36.4
$3.8
5.9
25.4
5.4
67.3
60.0
44,3
33.2
Vilaol 1M-mline-88
Mips
8.1
23.9
12.1
14.0)
45.5
21.2
20.8
22.2
18.8
12.2
13.7
Dispider
Mipe
57.7
40.5
62.1
44.9
61.4
51.6
54.5
48.5
55.4
34.7
4.3
36.1
18.0
37.4
48.8
41.8
TimeChat Online 7B
Hpe (10016)
75.2
46.5
70.7
47.8
69.3
61.4
61.9
55.9
59.5
9.7
41.7
31.5
38.5
40.0
36,7
46.7
Ours Durs
Hipe C1 93.259)
56.4
54.9
60.4
45.0
673
50.4
55.8
41.7
55.9
46.7
81.4
12.9
33.8
40.0
25.6
96.9
1pp (10075)
74.1
57.2
68.1
55.3
75.0
58.3
64.7
46.0
36.3
28.8
44.3
29.1
89.3
40.0
36.1
81.8
4) P4: Replace the graded (p,c) update in P4 (Progressive tracking sub-questions status) with single binary answerable flag (0/1), eliminating accumulated progress and confidence smoothing.
5) Ps:
Remove Ps (self-triggered reflection) to assess the benefit of cross-clip causal revision under low confidence or major semantic shifts.
A.3 SUMMARY OF HYPERPARAMETE SETTINGS
The training process of our Thinking-Qweer Qwen is structured into three distinct phases. 1) In- tegration Pre-training. We pretrain the model on LLAVA-Video-I788 178k (Li et al., 2024a) and ShareGPT4v-40k (Chen et al., 2024b), both containing caption-style data. This stage enables the model to learn how to aggregate and compress visual information into the inserted compress to- kens at specified positions. 2) Integration-Based Time Perception Learning. We fine-tune the model on LimeChat-Online- 139k (Yao et al., 2025), dataset annotated with binary labels indicat- ing whether a question is answerable at a given timestamp. This trains the model to decide whether the compressed visual information is sufficient for answering, relying solely the compress tokens. QA Fine-Tuning. We further fine-tune the model using general QA-style dialog data to enhance its interaction ability and improve alignment with user queries in a stream- ing setting. Throughout all stages, only the intermediate Merge layers and the LLM backbone are fine-tuned, while the visual encoder remains frozen. All experiments are run A100 GPUs. Table 9 provides comprehensive overview of the hyperparameter configurations employed during each training stage.
3) Interaction-Focusee
A.4 POSITION IDS EMBEDDING FOR INTEGRATION
Impact of Positional Encoding. The original QwenVL2.5 model adopts 3D Rotary Position Embedding (3D RoPE) mechanism. When introducing new aggregation tokens, it becomes nec- essary to redefine their positional encoding. To maintain compatibility with the model's dynamic spatial resolution handling, we insert aggregation tokens in multiples of the original frame tokens. In Thinking-Qwer VI., we retain the 3D RoPE format while adjusting the temporal dimension of the inserted aggregation tokens as in Algorithm 1. This ensures the spatial indices are aligned with the original frames while preserving temporal distinction across hierarchical aggregation levels. To evaluate this strategy, replace 3D RoPE with sequential positional encoding and introduce new variant, Offsee Sequential Positional Embedding (OSPR). OSPR explicitly offsets the sequential position IDs of aggregation tokens according to their hierarchy level. On OVOBench, substituting 3D RoPE with OSPR reduces overall accuracy from 46.9% to 43.3% (a drop of 3.6 percentage points), which is also reason we retain 3D RoPE in model.
15

---

Published as a conference paper at ICLR 2026
Table 9: Training hyperparameters of Thinking-QwenV/ VL for all stages.
Configuration
Integration Pre-training
Time Perception Learning Interaction Focuusd QA Tuning
Training Datasets
LLAVA-Video- T7P
TimeChat- MaaeeChh--Oniie-I1 Online-1391
LLLVA-Video-117
Training Datasets Type
Caption
Open-ended QA
Multiple-choice QA
Training Modules
Merge Layer
Merg Layer
MLMg& Layer
Frame Resolution
448 x 448
448 x 448
448 x 448
Max Frames
128
196
128
Optimizer
AdamW
AdamW
AdamW
Learning Rate
2e &
2e &le-
2e 2e~"& &1e=5
Learning Rate Schedule
cosine decay
cosine decay
cosine decay
Weight Decay
0.1
0.1
0.1
Gradient Clip
1.0
1.0
1.0
Warm-up Ratio
0.03
0.03
0.03
Global Batch Size
16
16
16
Numerical Precision
bfloat16
bfloat16
bfloat16
Algorithm The algorithm of Position IDs embedding for aggregation tokens.
Require: X: Input tokens
Require: G: video grid (T, .HH W)
Require: C: compress params (Netips Ncomp)
Require: P: position params (At, T, S)
1: Textended <- T + Netips -v(!)
2: P. + 0.1, ... Textended
x X
3: Py f- 1,
H/S
4: Pw <- [10, I,... W/S]
5: M, + repeat(P,, along spatial dims)
6: MA repeat( Ph. along temporal and width dims)
7: M < rPpeat(P.... along temporal and height dims)
8: POssD <- stack(M,, MA, M,))
9: return PosaD
A.5 EVALUATION METRICS
StreamingBench (Lin et al., 2024) is large-scale online video benchmark spanning 900 videos with 4,500 timestamped multiple-choice QAs, designed to test real-time perception and interaction under realistic stream constraints. Tasks are grouped into three families: Real-Time Visual Un- derstanding, Omni-Source Understanding, and Contextual Understanding. Findings reveal clear gaps: offline long-video MILMS transfer modestly to real-time visual tasks but underperform on omni-source and contextual tasks requiring audio fusion, long-horizon memory, and event-timed ac- tuation; dedicated streaming models remain immature. Each of the 3 types has a split, and since the other two test tasks are non-visual modality-dominant, e.g., the omni-source subset is dominated by the audio modality, we tested on the first Reait-Rell-Time Visual Understanding (2,500 QAs). The subtasks in it are as follows: Object Perception (OP), Causal Reasoning (CR), Clips Summariza- tion (CS), Attribute Perception (ATP), Event Understanding (EU), Text-Rich Rich Understanding (TR), Prospective Reasoning (PR), Spatial Understanding (SU), Action Perception (ACP), and Counting (CT). We use abbreviations for these subtasks in Table 1.
OvOBench (Niu et al., 2025) is a dedicated benchmark designed to evaluate online video under- standing models with tasks of types (real-time visual perception / forward tracking / forward active response). It comprises 644 videos and around 2800 QA pairs, requiring models to with- hold an answer until sufficient future evidence arrives. OVOBench specifically evaluates temporal alignment capabilities by enforcing strict separation between the query timestamp and the earliest timestamp at which the question becomes answerable. This is particularly important for assessing whether model can respond the right moment based on sufficient and relevant visual evidence.
16

---

Published as a conference paper at ICLR 2026
The suite spans 12 tasks grouped into three modes: Backward Tracing- Episodic Memory (EPM), Action Sequence Identification (ASI), Hallucination Detection (HLD); Real-Time Visual Percep- tion-Spatial Understanding (STU), Object/Attriutte/Action Recognition (OJR/ATR/ACR), OCR, and Future Prediction (FPD); and Forward Active Seepondiinn--eeetttiie Event Count (REC), Scene-State Regression (SSR), and Cautious Response Regulation (CRR).
RTVBench (Xun et al., 2025) and OVBench (Huang et al., 2024) jointly offer a complementary yardstick for online video understanding- continuous perception and online spatiotempo- ral reasoning under real-time constraints. RTVBench (552 videos / 4,631 QA pairs) is built around (i) Multi-Timestamp QA and a Hierarchical Question Structure to prevent shortcutting that can be summarized into three Stasks--Prrepption Understanding. and Reasoning (future prediction/s- patiotemporal reasoning). OVBench (5,000 QAs) scales online evaluation across 6 task types with videos ranging from seconds to one hour; it uniquely anchors each query to Pust/Current/Fuuu temporal contexts, requires fine-grainet grounding. Together, the two benchmarks expose persistent limitations of current MLLMs: offline long-video models lose robustness under cluttered, evolving streams and dedicated online models still trail top proprietary wystems-tighlightinng the need for more advanced architectures.
Video.MME, MLVU, LongVideoBence and LVBench (Fu et al., 2024; Zhou et al., 2024; Wu et al., 2024; Wang et al., 2024b) are four long video QA benchmarks. VideoMME (2,700 QA pairs) spans six domains with videos from short clips (< 4 min) to long-form (> h), testing perception, reasoning, and synopsis across temporal scales. MLVU (1,730 videos 2,593 QA pairs) ranges from 3 minutes to 2 hours, providing complementary coverage of long-form video understanding. LVBench probes extreme long-video comprehension with videos up to two hours (68 min on aver- age). Long VideoBench (3, (3,763 videos/ / 6,678 human-authored QA pairs) is a large-scale benchmark for understanding long contexts, which collectively demand granular recall and spatio-tempordl rea- soning under long inputs.
A.6 EVALUATION OF DECISION CLARITY AND RATIONALE CORRECTNESS
To systematically assess whether ATDM genuinely improves users' understanding and trust in the model's 's behaviour, we design four complementary metrics and apply them to the intermediate rea- soning traces produced by each ATDM component as well as to the overall decision process. All four metrics are rated on 1-5 Likert scale (higher better), and are used consistently by human experts, trained non-experts, and strong LLM judges (GPT-4o and Qwen2. 5-VL-72B) Together, they disentangle: 1) Reasoning Readability: how well the full reasoning text is written and struc- tured; 2) Decision Transparency: how clearly the timing of "answer" vs. "wait" is explained; 3) (p, c) Consistency: whether the explicit progress/contidecce signals (p, c) behave in a numerically consis- tent manner; and 4) Rationale Correctness: whether the rationale is factually and causally sufficient to justify the decision. The detailed indicators and the meaning of each corresponding score are in Table 12.
Reasoning Readability evaluates whether each part (e.g., caption, sub-question answers) in the ATDM reasoning trace follows the requested content specification, easy read, and is locally coherent. High scores indicate that each component strictly adheres to the instructions, the content strongly related to the corresponding sub-question, the caption fluent and covers the required aspects, and the sub-question answers align naturally with the caption, yielding a globally coherent and well- strutstr reasoning trajectory.
Decision Transparency measures whether a rater can clearly understand why ATDM decides to answer or to keep waiting at each step, given access to the reasoning trace, the current sub-task states, and the associated progress/confidence scores (p, c). A high transparency score means that the trace explicitly indicates when key evidence appears, how sub-tasks are resolved, how (p, e) are updated, and how these factors jointly trigger the timing of the decision.
(p, c) Consistency focuses specifically on the numerical behaviour of the progress and confidence signals. It assesses whether the magnitudes and step-wise trends of (p,c) are internally consis- tent with the textual description of task progress and uncertainry. High scores correspond to well- calibrated dynamics: p increases as sub-tasks are resolved, c increases when decisive evidence is observed and remains low for ambiguous quesstuss and overall the numbers "make sense" as a faithful quantitative reflection of the explained state.
17

---

Published as a conference paper at ICLR 2026
Reasoning Readability
Decision Transparency
Unreadable
Completely opaque
Mostly hard to read
Mostly opaque
Mostly readable
Partially transparent
Moderately readable
Generally transparent
Clearly readable and well- structured
Highly transparent
?
(0, c) Consistency
Rationale Correctness
Clearly inconsistent
Mostly inconsistent
Incorrect misleading
Mostly incorrect
Weakly aligned
Partially correct but incomplete
Generally well-aligned
Generally correct and sufficient
Well Well-calibrated and tightly consis-
Generally correct and well-justified
Figure 9: The four rubrics for evaluating ATDM's reasoning traces.
Rationale Correctness evaluates whether the reasoning at given step is factually grounded underlying video and question, and whether the cited evidence and causal explanation objectively justify the chosen action ("answer" vs. "wait"). High scores require both factual accuracy (no hal- lucinated events incorrect descriptions) and logically appropriate of evidence, such that the selected segments/events and the corresponding explanation are necessary and sufficient to support the decision.
Table Expert evaluation of the reasoning process ATDM on StreamingBench The data in the table represents the average score for each indicator. The detailed indicators and the meaning of each corresponding score are in Table 12.
Expert
Readable
Transparency
p/c Matching Degree
Correctness
Qwen2.5-VL-722
4.6
4.2
4.3
GPT-4o
4.5
4.2
4.0
3.8
Average
4.55
4.20
4.15
3.90
A.7 SUB-QUESTIONS RELEVANCE AND TYPE CORRECTNESS
Expert Evaluation. To further verify that ATDM generates task-relevant and structurally mean- ingful sub-questions, conduct an additional expert evaluation using the LLM judge (GPT 4o). Given the original question and the corresponding list of ATDM-generated sub-questions, the LLM prompted to assess each sub-question along two dimensions: 1) task relevance to the main ques- tion, and 2) the correctness of its semantic type label, which is required by decomposition prompt of Part-2 in Section F, For each sub-question, the judge returns a scalar task mask-eelatsdnes score in (1. 5), boolean flag indicating whether the declared type is appropriate.
Task relevance is designed to quantify how strongly sub-question contributes to solving the origi- nal main question, assuming that it can be answered correctly. We adopt 1-5 Likert scale with the following rubric: I-Completely unrelated, 2-Mostly unrelated, 3- 3-Partially related, 4-Clearly related, 5-Strongly task-critical.
18

---

Published as a conference paper at ICLR 2026
Type correctness evaluates whether the declared sub-question type matches the semantics of the sub-question content. The LLM judge is instructed to output a binary decision (correct incorrect).
We further compare three prompt variants: (a) the full model with relevance/osservabiiity straints, (b) a no-constraint variant that removes phrases enforcing explicit task relevance, and (c) free-exploration variant that additionally encourages broad, unconstrained decomposition. As ported Table 11, relaxing the constraints consistently degrades both LLM-judged task-relatedness and downstream streaming accuracy (e.g., from 71.60% for the full model to 69.58% and 68. 68.40% for the no-constraint and free-exploration prompts, respectively). These results are aligned with the fail- ure modes discussed in our concurrent analysis (Jang et al., 2025); indiscriminate sub-questioning increases reasoning complexity without improving task utility. whereas ATDM's controlled Part-2 prompt reliably steers the model toward focused, task-critical sub-question..
Table 11: Evaluation for relevance of sub-questions and the prompt's influence in Part-2.
Avg Task relevance
Avg Type correctness
Full Prompt
w.o. Requirement
"Freely"
4.971/5
0.9992/1
71.60
69.58
68.40
B MORE DISCUSSION AND FUTURE WORK.
(1) Depth-as-memory. We encourage treating Transformer depth as staged memory for streaming: assign progressively stronger aggregation to deeper segments and study depth-aware schedules with mask-guided fusion across backbones. This line of inquiry, consistent with hierarchiccl/aggrrea-- tion evidence for long-range spatiotemporal reasoning and robustness, merits systematic, model- agnostic exploration.
(2) Decision timing and control. We advocate explicit, calibrated internal signals in LLMs e.g., progress p and confidence c-to govern when to answer, wait, reflect, moving beyond ad-hoe "longer CoT." This connects naturally to work on model self-calibration and self-reflection. (3) Extensions from video QA. (i) Mulii-stream evidence alignment: equip each modality (vision, audio, motion, text) with its own HPSI-style memory and fuse per-stream p/c via lightweight controller decide when joint evidence is sufficient-usefull under missing frames and hard cuts. (ii) Open benchmarks for decision quality: complement accuracy/latency with ex- plicit scoring of decision timing and evidence alignment (e.g., rewarding on on-time, well-supported answers) for the video understanding task.
19

---

Published as a conference paper at ICLR 2026
Vikon
Totarn
Quary huiriin
20002
15000
20300 25022
Kay Pusiidon
Figure 10: A real example of the attention mask in final 1L/3 layer of the LLM. The ratio between the original video tokens and the 3-level aggregated tokens is depicted. Compared to the original video input tokens, the proportion of aggregated tokens introduce is minimal. As shown in this figure and Fig. 2, our custom attention mask guides the model in hicrarchically allocating attention across different visual regions, fostering progressive focus the visual tokens themselves for improved video understanding in LLMs.
20

---

Published as a conference paper at ICLR 2026
Table 12: Evaluation rubric for ATDM's reasoning traces along four dimensions, each scored from 1 (worst) to 5 (best).
Score
Metric
4
Reasoning Readability
Unreadable
Mostly hard read
Moderately readable
Clearty readable
Highly readable
Severely disorganized largely inrelevnnt to the requested content, with contradictory or unrelated statements forced step; the overall meaning very hard to recover.
Some segments are understandable, but the output is only weakly related the requested cuntent or previous
Fach part broadly follows the instructions; there blatant eff-topic or highly confusing jumpss
Each part strictly follows the
All parts strictly follow the instructions with strong cross-part coherence. forming globally consistent and logically connected reasoning process.
instructions:
contextual relevance is clear, captions are coherent
and captions are mostly coherent, though some text shows weak contextual linkage.
sufficiently detailed, and answers align well with the caption.
same
sub cub-questions; the context foels jumpy or redundant, and cuptions not clearly written.
Decision Transparency
Completely opaque
Mostly opaque
Partially transparent
Generally
Highly transparent
transparent
Fven after reading the full reasoning and inspecting relationstip
The high high-level relation to the main question is intelligible. but the tasks
The
is reasonahly clear when evidence ceusidered sufficient or insffficient
The makes explicit when and why decision triggered, which evideuce makes sufficient to answer now, and how the current sub-task states and (p. jointly justily
basically follow the overall reasoning.
the sub task
between (p. and sub-task states, the rater cannot see how they relate main question nor why answers waits at this step; decisive mi ssingng sub-tasks appear irrelevant.
updates are unclear; if active-thinking steps present, update reasons are opaque and the causal chain is confusing.
updates,
rough causes of (p, ej changes; if reflection steps exist, the causal chain largely
why (p. cj and sub-task states are updated; if
reflection steps
exist, the causal chain before and after the update is clear and coherent.
reasomable, though details remain implicit.
answering versus waiting: g; the timing logic is casy to understand.
(pfe) Consistency
Clearly inconsistent
Mostly inconsistent
Weakly aligned
Generally well-aligned
Well-calibrated and tightly consistent
(p.c) are often odds with the rensoning (e.g., very high progress when the text emphosises strong uncertainty,
Some steps look plausible, but overal (o. c) rarely
The coarse trend of (p. e) roughly
For most steps, the lovel and evolution (o, c) are consistent with the textual description progress and uncertainty; minor mismatches exist but do not substantially undermine trust in the signals.
Step-wise changes in (o, e) closely
follows the narrative (e.g.. both increase as evidence accumulates), but many local steps feel off (e.g. unexplained spikes or plateaus).
track reasoning:
aligu with the described state: the full
rises sub-tasks are completed.
rises when decisive evidence appears and stays low for unresolved or ambiguous sub-questions.
or very low progress when most sub-tasks are stated resolved);
their magnitudes or changes foel
arbitrary or
unirustworthhy.
mismatebes are frequent and severe.
Rationale Correctness
Incorrect or misleading
Mostly incorrect
Partially correct but incomplete
Generally correct and sulficient
Clearly correct and well-justified The rationale is factually accurate. largets the right segments/events, and the sub-question answers are correct
The rationale contains clear factual errors,
Some parts touch on relevant content, but important evidence missing or misinterpreted; the causal story is weak decision is only very weakly supported.
The rationale captures several correct and pertinent aspects, amits critical evidence or leaves key causal links under-specifie;; the decision is somewhut supported, yet the justification is not fully convincing.
The cited evidence is largely accurate and relevant, and the overall reasoning provides a plausibic. reasonable
hallucinates events that
relies on irrelevant contradictory evidence; it fails to justify the chesen action and may even support opposite.
explanation that covers most key factors needed to solve the task.
while highlighting.
from multiple
angles, the key factors needeed solve the question.
21

---

Published as a conference paper at ICLR 2026
Question (at 00:0826): Why does the person shake frying pan after pouring the liquid into
Thirking-Owenl
Stnars
To mix the ingredients.
To cool down the pan.
E To stop the steam.
D. To clean the pan.
Part I: To sesove th quasiion iccuradyy the captoon mod Socus Biee imquincannto: directio the person's hand witlle shaking the fixtag pan
Part 3
dig spare
clal propacee modeers begins seporiing
poerzon the trying per. before and after saxcmng
fills barbs spices
sessered byyioy
ass visble anbing octios
colknng vegetables Usng toegs, flys
any signn cooling clearig
shalkng
legrediezts plates
Part 2 the follwing sub quastons are detiee
Se regetables and tisl everyting togeber
mitedd Finally,
lapel "adionn, "gueston" luv action pefooreed winee youring the leqii into the
frying pas?"
disk, amnrging the fits vegatablie readl. He gicrolla plate eiih freis
tepe": "olgect, "questonn: but poured into the fryngg pas?"
lacts and janas slloss The latzian counter cluztand with vurious ingradanns and utemeiss reflecting aerive cooking prososss
Tepe": "oxual mabac"" "question"; hatis the parpose of stalkng frytng pan after
pouting liqud?"
Outpot upiated questorss
mpnsestative
questoons followe))
actoss
typec object
bype canad miliion
What adion is pefformdd after pouzling the Equadd Lnto
Whatt pourod into the frying paz? valae.
What the purpese of shalcng the frytng pon after pouning lteligudd valte 2 cosfideace: 0.0
frynng pan?
velun
confidince: 3.0
confidence:
Patiruded prigrex
Sype: sbjaet
type elject
bype casad relition
Estarated propess
q: What parforteed gourling the lquid
pound frying pai?
q
pargow dloking the frying pan aftar pouring ligual
fiytng
value:
confidence: frytng pen
Estanated progees 33
typer abject
type: oiject
type: aseaal reletion Wht the purpoa of shoking the frying pas ofter yeuning the lquii?? valae: To mix ingreddenss confidence d.85
Extrrated prigres 1100
@ Watt action is pefoored be pourrig the liquid into dtyang pas?
Whut is pourndl inte the fiying pas? valuz: liqud confitiezre: 0.90
ralte: stake the frytng pmn confidenco 0.55
irggerd during process of progmes continooady tnsmeteg to 100%.
An example illustrating the outputs of each ATDM component in Thinking Qwen in this case, the model's response confidence increases monotonically, so Part-5 (active thinking for reflection) is not triggered.
Figure 1I:
C ADDITIONAL VISUALIZATIONSS
In addition the examples presented in the main text, we provide further srcision-making illustra- tions using ATDM for both Thinking- Qeaegg--eeeVVLe VL and Flash-VStrean in Fig. 11& 12. We also include concise examples of cases that trigger active thinking, to clarify the outputs produced by each ATDM component and to demonstrate their specific roles across the two models.
D ETHICS STATEMENT
This work strictly adheres to the ICLR Code of Ethics. No human-subjects studies or animal ex- perimentation were conducted. All datasets used for training and evaluation were sourced from the open-source community and used in compliance with their licenses and usage guidelines, no personally identifiable information was collected or processed. We took care to assess and mitigate potential biases and discriminatory outcomes, and we performed no experiments that could raise pri- vacy or security concerns. We are committed to transparency and integrity throughout the research process.
E LLM USAGE
Large Language Models (LLMs) were used solely to assist with writing -rriinrilyrrrryyyyy for grammar correction and minor phrasing edits to improve coherence and readability. The LLM did not par- ticipate in ideation, research methodology, experimental design, data analysis, or interpretation of results. All research concepts and analyses were conceived, executed, and validated by the authors. The authors take full responsibility for the content of the manuscript, including any text revised with LLM assistance. We verified that all JLM-assisted text complies with ethical guidelines and does not introduce plagiarism scientific misconduct.
22

---

Published as a conference paper at ICLR 2026
Question (at 00:01:101 What is the man doing while sitting a chair?
A. Reading a book.
B. Polishing his shoes.
C dijusting his trousers and shoes.
D. Writing letter
Part To aeswer Bis queatio accurands. caprioo crant Cocas Coose irotrottnts
Port Tladip dip spaza frees C:00.00.0000 0.01:04
cuact couzt adtaos parforeed by the maa (ea,, sitting staneiin, wilking sto.l spotial relstocstip betweees the man and the chair ditaane, angls, odietation)
The rann is sitting on charrmn room, the senving lighup mom
hands There
sliing with any socadeer spokes ornient Ing, alut the tann leysing e any badkgmaad muiss, stc.)
deoriitaa mesh netions legg
ledly postan,,
pieze firmitur. mon spperrs to forased
apto
znyone aestiing
room.
Part followin sub
detved.
54-0.00
"action. "grestinn: "Whtt
performing while
the
Bl buthoss Another tralnshal réjest ooller slenves short The bockgrward
willl
chain"
"tpp": "spatii rilatoon. quastoa" "Witu the aun sitting is rslation to the chais?
mnd ondes doorrand
"typ"" "horpprna change", "suantion" "When is mon ening the chair
"type" "rannal onteron". "queeton"" "Is the man ening on the chair hocwss of quacitt
meson or event?
Basod
procided
Therdoor,
"Witat while
a winh 'A digucingg me tmuees shows.
The denss for the main gao 0.50
The ougrat jon last the oument citp would be
bgg action
type: spetoi mistion
type: temporri change
type: spoccite zecaus confiiherae 0.90
What
pettonnng while
Wiass son sittlep
0. Wibas sitting daal??
Adyosing confidanci:: 0.50)
confirbence
corficlence 0.90
Estintted progress LCD
Figure 12: An example illustrating the outputs of each ATDM component in Flash VStream. The output of each part is influenced by the model's foundational visual comprehension and its ability to follow instructions.
Ours (with HPSI):
A hand holding round brush labeled 'Sap green Round brush 4' applies green paint canvas. The canvas features blue sky gradient with green foliage at the bottom. The hand moves the brush from right to left, adding details to the green area. A palette with blue paint is vistble nearby, The hand adj usts the brush's 's position and angle to enhance the foliage details. The scene focuses on the meticulous application of paint to create realistic tree outline against the blue sky background.
Qwen2.5-VL:
band holding a round brush no applies sap green paint to canvas, which alroady has blue sky painted at the top. The artist carefully blends the green paint into blue, creating gradient effect. palette with blue and green paint is visible nearby, indicating mixing process. The hand moves methodically, adding texture and depth green area, suggesting an early stage the landscape painting.
Comparison of model-generated captions for the same clip. Our caption explicitly en- codes state changes over time ("moves from right to left", "adjusts position and angle"), which implies that the model is using historical visual memory and new frames to form a coherent, evoly- ing narrative. The baseline, lacking hierarchical integration, mainly describes single static scene.
Figure 13:
23

---

Published as a conference paper at ICLR 2026
F DETAILS OF THINKING-OWENVL' PROMPT
Here, we provide detailed prompts of the five parts as well as their inputs and outputs. Question "What the width of the road right now?" the example.
Part-l:
The prompt for Part-1 giving the instructions for preparing for future steps should be:
Part-1: I: Question-Guided Captioning Instructions
Input:
<TASK DEFINITION>
Your task is to analyze the user's question and define EXACT observation requirements for video captioning from the video in order to help answer
Think carefully: What aspects of the video should caption focus on to make answering this question possible?
<INSTRUCTIONS>
From the given question, generate list of observation requirements: Each requirement should describe an important dimension that a future caption must pay attention to. Some CRITICAL FOCUS: 1. Quantification: Require exact counts when applicable 2. Directionality: Specify spatial relationships, positions and movement vectors 3. Object- anchored Disambiguation of Confusable Concepts: If options include visually similar or easily confused concepts (e.g., "table" vs "counter", "cabinet" vs "shelf"), ensure cap- tions distinguish them clearly through spatial context, object functions, or visual appearance.
For example: exact count of objects someplace or the number of people, actions and their order, hand movements or object manipulation, specific visual details, interactions, gestures, spatial relationships, direction, distance, any sounds or spoken content Such as: "exact count of apples placed in basket", "direction of sword thrust relative to opponent", "distance between white car and pedestrian when braking", "rotation angle of wrench during tightening" <INSTRUCTIONS>
<CONSTRAINTS>
Only generate points that are visually observable. Do not speculate. Focus on ane-grained but relevant aspects. Max 5 points. Return your result in this JSON format:
"question": [What is the width of the road right now?], "cap- tion requirements":
<<lquatiffabbl requirement. 1|>", "< "<|space
observation requirement 21>", "<Jother observation point 3
1}
<CNNSTRIINTS>
<AASK DEFINITION>
Output:
caption_reann requirements:
wiath measurement the road, position of the road within the frame, any obstacles or landmarks for scale reference, any changes to the road width over time
24

---

Published as a conference paper at ICLR 2026
Part-2:
The detailed prompt for Part-2 is shown follows:
Part-2: Question Decomposition
Input:
TASK DEFINITION'>
Your ONLY goal in this step is to read the user's main question below, Break down into a set precise, concrete sub-questions. Each sub-question should focus on a specific, observable aspect of the video (e.g., object, person, action, spatial relation, etc.). These sub-questions represent the key elements that must be visually or aurally verified in the video answer the main question.
<CONSTRAINTS>
Only include attributes that are explicitly required or clearly implied by the question.
Do NOT use background knowledge, commonsense, or speculate.
Do NOT include any explanations or commentary.
Output must be in valid JSON, under the top-level key "required
Do not include trailing commas.
Return your result in this JSON format: ("question": [What is the width of the read right now?], "required subquestions": [["type": <|typel>; "description": < <|Required Subquestion description| |>]}
<CONSTRAINTS>
<TASK DEFINITION>
Output:
required_subeee subquestions:
type: object,
question: Is there road visible in
the video? },
type: temporal_chanee
question: Is the road width
consistent throughout the video, or does it change over time? ) type: spatial relation, question: Is the road width measured from edge to edge, or from center to center? 1, question: 1s there any measurement. tool used te measure the road width?
type: other,
In our sub-question decomposition prompt Part-2, requirements such as "focus on a specific, observable aspect of the video" and "explicitly required or clearly implied by the question" plicitly designed to avoid failure modes where the model generates freely diverging sub-questions with only weak relevance the main query. Instead of allowing free-form decomposition, ATDM tightly controls both the generation and the use of sub-questions. The prompt guides the model to propose only small set (typically 3-5) of sub-questions that are directly derived from the main question and cover it from complementary, semantically aligned perspectives. Beyond this con- strained construction, ATDM incorporates two refinement mechanisms Part and Part 5: Part continuously updates each sub-question's answer new clips arrive, while Part 5 explicitly corrects them using cross-clip evidence. Together, these mechanisms ensure that both the generation and use of sub-questions remain well controlled.
25

---

Published as a conference paper at ICLR 2026
The detailed prompt for Part-3 is shown follows. To convey the overall message, present some content of clip captions before : 0:07:4 :7:45 here:
Part-3:
A bustling city street captured during a rainy day. The road wide, with multiple lanes for traffic. Vehicles, including yellow taxis and various cars, navigate through the wet asphalt. A few pedestrians walk along the sidewalks, while vehicles move steadily despite the rain. The scene transitions from more open area to busier intersection with more traffic and pedestrians. The road remains consistently wide throughout, with clear lane markings and traffic flow.
Part-3: Video Clip Captioning
Input:
<TASK DEFINITION>
Watch the current video clip and generate descriptive caption, you must focus your caption on the following key observation points:
Caption Requir List
Videoo>>
INSTRUCTIONS>
Base your caption only on what clearly visible. Limit to 500 words max.
2. Be specific and concrete: describe actions, hand use, counts, object states, etc.
Use short, factual sentences. No speculation, no vague summaries.
4. Precision first if required: "three apples" NOT "some fruits", "thrusting
downward at 45 angle" NOT "attacking", "2. 3m left of tree" NOT "near tree". etc.
<CONSTRAINTS>
Return the caption in valid JSON format:
clip timestamp: (timestamp),
caption: detailed caption that fulfills the requirements
<CONSTRAINTS>
Output:
clip_timestamp: 0:07:46-0:08:50
caption: A busy urban street scene during rainy weather. The road is wet and reflective, with multiple lanes of traffic moving in both directions. Skyscrapers line both sides of the street, creating dense cityscape. Pedestriens cross the street at a crosswalk, sone holding umbrellas. Vehicles, including yellow taxis and various cars, navigate through the traffic. The road surface shows signs of wear but remains functional. There are visible censtruction or obstacles affecting the road width. The overall atmosphere reflects a bustling city environment despite the rain.
The full prompt used in Part-4 is provided below. For brevity, we omit the complete problem statement and the intermediate outputs referenced in earlier parts.
Part-4:
26

---

Published as a conference paper at ICLR 2026
Part-4: Sub-answer Extraction and Filling Information
Input:
<TASK DEFINITION
Your task is to:
1. Read the main user question and the list required subquestions (from Part-1).
2. Read the caption of the current video clip.
3. For each subquestion, determine whether the caption provides enough information to answer it:
-If yes: provide an appropriate answer ('value") and confidence score between and 1.
no uncertain: "value": "?" and "confidence"; 0.0.
<INPUT>
Main Question:
<I Questioon >>
Required Subquestions (from Part-2 or latest output from Part-4);
<I <|Requires Subquesti ions| >>
Caption of the current clip:
<aaast caption|>
<OUTPUT FORMAT>
Return top-level ISON object with the key "suuquestion.stattu".
Each item must include:
"type": one of ["object", "attribute", "person", "action",
"scene", "event", "temporal change", "spatial relation",
"causal relation", "count", "other"]
"question": the original subquestion (from Part-l)
"value": the answer extracted from the caption (or "?" if not found)
"confidence"; float between 0 and 1
Also include overall "estimmtted.prorress field (e.g., percentage of subquestions with confidence 2 0.85).
<OUTPUT TEMPLATE
subquestion_statuu
type: <attribute_type>,
question: csubquestion_teexe
value: <answer_ar_>>
confidence: 0.xx
,
},
petimmaeed_rrorres::: <int from to 100>
<CNNTTRIINTS>
-Only rely on what explicitly visible or audible in the current caption.
-Do NOT use prior background knowledge or context.
Do NOT speculate fabricate.
Ensure output is valid JSON (no trailing commas). nothing is observed, return all values as "?" with "confidence": 0.0.
Output:
27

---

Published as a conference paper at ICLR 2026
Part-4: Sub-answer Extraction and Filling Information
(Here, we present only single representative intermediate state.)
subqucstion_statun
object, question: Is there road visible in the video?, value: yes,
type:
conficence: 0.95
type: temporal_change
question: Is the road width consistent throughout the video, or does change over time?,
value: consistent, confidence: 0.90
type: renation,
question: Is the road width measured from edge to edge, or from center center?,
value:
confidence: 0.0
},
type: other,
question: Is there any measurement tool used to measure the road width?,
value: no, confidence: 0.85
}
est estimated_progrees 75
Self-triggered reflection deterministic cross-clip reasoning step that revises answers, progress, and confidence when confidence-basse triggers indicate high uncertainty or significant semantic cross-
shifts. In the prompt of Part-5, the "self-trigeered reflection" stage implemented clip reasoning pass built on the confidence signals c. Given the decomposed sub-questions, their current answers, the corresponding (p, e) values, and the recent relative clip captions, reflection (i) restructures these local pieces into globally coherent reasoning chain across clips, and (ii) revises sub-question answers and (p, c) whenever single-clip evidence unreliable or mutually inconsistent. This prevents the controller from being myopically tied to single clip and yields smoother, more globally consistent reasoning trajectory.
Reflection is triggered purely by the quantitative signals, in particular by the confidence [0, 1] and its change |Ac] between adjacent clips. The threshold Rs 0.5 represents the boundary between "clearly relevant" and "not clearly relevant" visual evidence for sub-question. Large |Ac indicates a major semantic shift between clips, while persistently low c suggests that the required evidence distributed across muitiple clips rather than concentrated in the current some single one. In both cases, we trigger reflection to explicitly re-examine and consolidate evidence across clips. For decision commitment, ATDM uses higher threshold e > 0.85: this encodes that the model believes the currently accumulated evidence is both relevant and sufficient, while still leaving room for skep- ticism so that hypotheses formed from single-clip perspective are not over-committed. Together with the progress signal p, these thresholds define a normalized Jo. 1] semantics for "relevance" and "sufficiency". and they directly govern when ATDM keeps waiting, when invokes reflection, and when it finally decides that the visual evidence is first sufficient.
28

---

Published as a conference paper at ICLR 2026
Part-5: The detailed prompt for Part-5 is shown follows. Then, provide two specific examples of the output.
Part-5: Active Thinking for Refining the Reasoning across Clips
Input:
TASK DEFINITION
Cross-clip causal reasoning
Analyze each new clip caption for direct evidence related to each attribute.
Build an explicit, ordered chain only for attributes with relevant evidence. Use arrow notation: "Clip X + [sunports/contaaict//pvoiii
evidence for] [attribute] because [exact caption text]". If clip provides no relevant evidence for any attribute, state: "Clip X - No relevant evidence for current attributes".
Evidence relevance check
For each attribute, explicitly check whether the captions contain relevant information. Mark attributes as "relevant evidence found" or "no relevant evidence".
3. Update the attribute list
Preserve original values and confidences for attributes without relevant evidence. Modify attributes only where direct, explicit evidence is found; quote the exact caption text that supports the change.
<INPUT> Question:
< [luustion] >>
Latest reasoning state (attribute list confidences):
< Past CoT Statel>
Past clip captions when confidence changes (chronological order):
<|Clip Captions|>
<OUTPUT FORMAT>
Return exactly one JSON object (no extra prose).
causal_chaint [
Clip X [reasoning or 'No relevant evidence")
},
attribute_statee starttse__ttuuss__
<Attribute-Type>: value: <value>, confidence: 0.xx,
status: upgraded downgraded. | unchanged |
newly_added,
note: old 0.yy => new 0.xx, [exact caption evidence] | unchanged evidence
estimated progress: <percentage with confidence 0.5>
<CONSTRAINTS>
Remember: NO EVIDENCE NO CHANGE.
No evidence status "unchanged", original confidence preserved.
Quote the exact caption text in notes when updating.
Output:
29

---

Published as a conference paper at ICLR 2026
causai_chailn chain: [
Clip provides evidence forl IIs there black sUv directly in front of the camera right now? because A vehicle directly ahead is visible, appearing to be dark-colored car.]
attribute_status: { Is there a black SUV directly in front of the camera right now?:
value: A vehicle directly ahead visible, appearing to be dark-colored car.,
confidence: 0.75,
status: upgraded,
note: old 0.6 -> new 0.75, [A vehicle directly ahead is visible appearing to be dark-colored car.]
},
estinmatod_rogrres:: 50
causal _cusai_chain:
Clip provides evidence for] [is there painting visible the wall? because] 'A close-up view of artist 's hand meticulously painting details en canvas.',
clip 4 -> (provides evidence for] [is text readable on the painting? because] 'Text 'IT'S IN THE DETAILS' appears prominently over the artwork. Clip -> :provides evidence for] [is the camera focused on the lower left corner of the wall while showing the paintings because ] 'Text 'IT'S THE DETAILS' appears prominently over the artwork."
attribute_stane { is there painting visible on the wall?: value: yes, confidence: 0.95,
status: upgraded,
note: old 0.00 -> new 0.95, 'A close-up view of an artist's 's hand meticulously painting details en a canvas.
},
is text readable the painting?:
value: yes, confidence: 0.95, status: upgraded,
note: old 0.00 -> new 0.95, 'Text 'IT'S IN THE DETAILS' appears prominently over the artwork
the canera focused on the lower left corner of the wall while showing the painting?: value: yes, confidence: 0.95,
status: upgraded,
note: old 0.00 -> new 0.95, 'Text 'IT'S IN THE DETAILS' appears prominently over the artwork."
estimated_ estimatedd_rooress:: progress: 95
30


Reviews:
[{"review_id": "6dUJb4CJJ5", "reviewer": "Reviewer_5jps", "summary": "This paper introduces Thinking-QwenVL, a new framework for online, evidence-aligned video understanding, addressing the key challenge of determining when a model should respond based on visual evidence as it streams. The framework explicitly separates reasoning control from memory integration, leading to two key components: Active Thinking Decision Maker (ATDM) and Hierarchical Progressive Semantic Integration (HPSI). The authors evaluate Thinking-QwenVL across several online and offline benchmarks (StreamingBench, OVOBench, RTVBench, OVBench, MLVU, VideoMME) and report significant improvements, e.g., +3.97% over Dispider on StreamingBench and +4.9% on OVOBench.", "questions": "1. For the reference in Latex, please use \\citep instead of \\cite to fix the reference issue in the main paper so that \"Xun et al. (2025)\" could be \"(Xun et al. 2025)\"\n2. Have you conducted human or expert evaluations on whether ATDM’s progress/confidence signals improve user trust or understanding compared to black-box baselines?\n3. What is the per-frame or per-second latency of Thinking-QwenVL relative to Dispider and Flash-VStream? How does hierarchical aggregation affect token throughput?\n4. How does the model behave when the input stream contains missing frames, abrupt scene transitions, or noisy temporal cues? Does ATDM maintain stability in such conditions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work tackles a largely under-explored but crucial problem — real-time, evidence-aligned video understanding — distinct from traditional offline long-video reasoning.\n2. The idea of decoupling reasoning control from memory integration is conceptually strong and practically motivated. The multi-level aggregation mechanism (HPSI) is an elegant solution for progressive semantic integration, offering a new perspective beyond standard pooling or RAG-based methods.\n3. The paper is clearly written and well-structured, with informative visual diagrams (e.g., Fig. 1–4) illustrating timing, aggregation, and decision flow. Each module (ATDM, HPSI) is described step-by-step, making a complex architecture accessible.", "weaknesses": "1. While empirically strong, the paper could benefit from more conceptual analysis of why ATDM’s quantitative progress and confidence signals yield better evidence alignment. For instance, is the model implicitly learning uncertainty calibration or temporal gating?\n2. The paper claims transparency and interpretability, but does not include user studies or objective interpretability metrics to support these claims (e.g., human evaluation of decision clarity or correctness of rationales).\n3. Although ATDM aims for real-time decision-making, the paper does not explicitly report latency, throughput, or computational overhead compared to simpler baselines. Such results would help assess deployment feasibility for real-world streaming agents.\n4. The ablations treat ATDM and HPSI largely separately, but it would be informative to analyze their synergy — for example, whether ATDM decisions remain robust if memory integration is partially disabled or simplified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psozfTtL1z", "reviewer": "Reviewer_iwc4", "summary": "This paper addresses the problem of online video understanding, focusing on two critical but often overlooked aspects: evidence-aligned response timing and decision transparency. The authors argue that conventional video LLMs, typically evaluated in offline settings, are ill-suited for real-world streaming scenarios where an agent must respond precisely when sufficient evidence becomes available. To tackle this, they propose a novel framework, Thinking-QwenVL, which decouples reasoning control from memory integration. The framework has two core components: The Active Thinking Decision Maker (ATDM): a reasoning controller that externalizes its decision process using explicit progress ($p$) and confidence ($c$) metrics, allowing it to align its response time ($t_r$) with the first-sufficient-evidence timestamp ($t^*$). The Hierarchical Progressive Semantic Integration (HPSI) module: an efficient memory system that uses multi-level aggregation tokens to maintain a global cognitive state under tight computational budgets. The paper demonstrates strong empirical results, achieving a new state-of-the-art on the StreamingBench benchmark.", "questions": "- Could you provide an analysis of the wall-clock latency or computational overhead (e.g., number of forward passes per second of video) introduced by the ATDM module? How does this compare to simpler streaming models like Flash-VStream or Dispider, and how might it impact real-time performance?\n\n- Can you elaborate on the development process for the 5-part ATDM prompt structure? How sensitive is the model's performance to the specific wording and structure of these prompts? Have you experimented with applying ATDM to other base models to test its generalizability?\n\n- Could you provide a more detailed explanation of the \"self-triggered reflection\" mechanism? What are the specific criteria for triggering it (e.g., confidence thresholds, how are \"major semantic shifts\" detected)? What does the model do during reflection (e.g., does it re-process past clips, revise the question decomposition)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly articulates and tackles the critical, practical challenges of response timing and decision transparency in streaming video analysis, which is a major step towards real-world applications.\n\n- The ATDM module provides an effective framework for making the model's decision-making process transparent and quantifiable through progress ($p$) and confidence ($c$) scores. This is a significant strength for user trust and controllability.\n\n- The HPSI module is a technically sound and well-executed solution for maintaining long-term, causally consistent context within a limited token budget, as demonstrated by its strong performance on long-video benchmarks.\n\n- The paper achieves state-of-the-art results on several challenging online video benchmarks, most notably StreamingBench, providing strong evidence for the effectiveness of the proposed Thinking-QwenVL framework. The ablation studies are thorough and convincing.", "weaknesses": "- The primary weakness is the lack of analysis on the computational latency introduced by the ATDM module. The five-part reasoning process (generating instructions, decomposing the question, captioning, extracting answers, and reflecting) seems to require multiple LLM inference steps for each incoming video clip. This could create a significant processing bottleneck in a true real-time scenario, a concern that is not adequately addressed in the paper.\n\n- The 5-part ATDM process, while transparent, appears highly structured and heavily engineered. This raises questions about its robustness and generalizability. How dependent is this structure on the specific base model (Qwen2.5-VL) and extensive prompt engineering? It's unclear if this complex chain-of-thought would transfer effectively to other video LLMs without significant re-tuning.\n\n- The concept of \"Active, Self-triggered Thinking\" (Part 5 of ATDM) is an interesting idea but is not described in sufficient detail. The paper mentions it is triggered by low confidence or major semantic shifts, but the exact trigger conditions (e.g., thresholds, detection mechanisms) and the concrete steps of the \"reflection\" process are not clearly defined. This makes the mechanism less reproducible and its contribution harder to assess.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mKO0s7yeYZ", "reviewer": "Reviewer_arDz", "summary": "The paper introduces Thinking-QwenVL, a framework for streaming video understanding. It consists of two core components: 1) Active Thinking Decision Marker (ATDM) : a module that determines when to provide an answer based on task progress and confidence. 2) Hierarchical Progressive Semantic Integration (HPSI): an efficient memory system for streaming videos that uses multi-level learnable aggregation tokens to capture video content effectively. Thinking-QwenVL demonstrates strong performance across various online and offline video understanding benchmarks.", "questions": "See the Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and of high quality.\n\n- The proposed method is both powerful and elegant. The hierarchical processing of visual signals allows progressive handling of long video features from coarse to fine levels. Moreover, decomposing questions into sub-questions and leveraging confidence scores enhances flexibility while preserving the base model’s capabilities.\n\n- The experiments are extensive and comprehensive; including both online and offline benchmarks effectively demonstrates the framework’s robustness and versatility.", "weaknesses": "Despite its effectiveness, several concerns and missing details remain:\n\n- In HPSI, it is unclear how the authors decided on the number of aggregation levels (i.e., three). Does this configuration balance efficiency and accuracy? From Table 4, the performance gain from levels 2–3 appears marginal, as the third row (only level 1) performs comparably to the full method. Further clarification would help justify the design choice.\n\n- In ATDM, the use of sub-questions and confidence scores is crucial, yet the paper lacks sufficient detail to understand how the module operates. Prior work [1] shows that sub-questioning can increase complexity or even degrade performance if applied indiscriminately. Have authors faced such issues? Furthermore, how were the thresholds (e.g., 0.85 and 0.5) for confidence scores in Parts 4 and 5 determined? A clearer interpretation is needed.\n\n- In Table 1, Thinking-QwenVL performs worse than its backbone model (Qwen2.5-VL), but this issue is not addressed, which contradicts the claim in L410–414.\n\n- The recent SOTA method [2] is missing from the comparison, and Thinking-QwenVL shows inferior performance to [2].\n\n*I would consider revising the rating if I misunderstood any part, and the authors clarify these issues in the rebuttal phase.*\n\n**References**\n\n[1] Confidence-guided Refinement Reasoning for Zero-shot Question Answering, arXiv 2025\n\n[2] StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, arXiv 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bA3HTRqh74", "reviewer": "Reviewer_EPyy", "summary": "This paper introduces a novel framework for online video understanding in streaming scenarios, where visual agents must respond to queries precisely when sufficient evidence first emerges in a video stream. The proposed system, Thinking-QwenVL (built on Qwen2.5-VL-7B), decouples reasoning control from memory integration through two core modules: 1. Active Thinking Decision Maker (ATDM): A transparent reasoning controller that decomposes queries into sub-questions, tracks observable metrics like progress (ρ) and confidence (c), and self-triggers reflections for cross-clip causal updates. 2. Hierarchical Progressive Semantic Integration (HPSI): An efficient memory system using learnable multi-level aggregation tokens inserted at different decoder depths with structured sparse attention. Extensive experiments on online benchmarks (e.g., StreamingBench, OVOBench, OVBench, RTVBench) show significant improvements.", "questions": "What about the computational overhead of the multi-stage ATDM process?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper effectively bridges the gap between offline and real-world streaming video understanding by emphasizing evidence-aligned timing and transparency.\n2. HPSI's hierarchical aggregation reduces token overhead while preserving cross-clip relations and causal consistency, making it suitable for long videos under tight budgets.\n3. Extensive evaluations across multiple benchmarks validate the approach, with clear improvements over state-of-the-art models.", "weaknesses": "1. Built on Qwen2.5-VL-7B, the results may not generalize to smaller or different architectures. What about the performance in smaller-sized models.\n2. Focuses primarily on visual evidences potentially lacks diversity. Have you considered the multi-modal settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)