Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration
Authors: Zhitao Zeng
Guojian Yuan
Junyuan Mao
Yuxuan Wang
Xiaoshuang Jia
Yueming Jin
Venue: NeurIPS 2025 poster ()
Keywords: Multi-scale Temporal Prediction
Incremental Generation
Multi-agent Collaboration
Primary area: applications

Abstract: Accurate temporal prediction is the bridge between comprehensive scene understanding and embodied artificial intelligence. However, predicting multiple fine-grained states of scene at multiple temporal scales is difficult for vision-language models.
We formalize the Multi‐Scale Temporal Prediction (MSTP) task in general and surgical scene by decomposing multi‐scale into two orthogonal dimensions: the temporal scale, forecasting states of human and surgery at varying look‐ahead intervals, and the state scale, modeling a hierarchy of states in general and surgical scene. For instance in general scene, states of contacting relationship are finer-grained than states of spatial relationship. For instance in surgical scene, medium‐level steps are finer‐grained than high‐level phases yet remain constrained by their encompassing phase. 
To support this unified task, we introduce the first MSTP Benchmark, featuring synchronized annotations across multiple state scales and temporal scales. We further propose a novel method, Incremental Generation and Multi‐agent Collaboration (IG-MC), which integrates two key innovations. Firstly, we propose an plug-and-play incremental generation to keep high-quality temporal prediction that continuously synthesizes up-to-date visual previews at expanding temporal scales to inform multiple decision-making agents, ensuring decision content and generated visuals remain synchronized and preventing performance degradation as look‐ahead intervals lengthen.
Secondly, we propose a decision‐driven multi‐agent collaboration framework for multiple states prediction, comprising generation, initiation, and multi‐state assessment agents that dynamically triggers and evaluates prediction cycles to balance global coherence and local fidelity. Extensive experiments on the MSTP Benchmark in general and surgical scene show that IG‐MC is a generalizable plug-and-play method for MSTP, demonstrating the effectiveness of incremental generation and the stability of decision‐driven multi‐agent collaboration.

Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration
Zhitao Zeng't
Guojian Yuan
Junyuan Mao
Yuxuan Wang
Xiaoshuang Jia
Yueming Jin
National University of Singapore
2 Alibaba Group
Renmin University of China
Abstract
Accurate temporal prediction is the bridge between comprehensive scene under- standing and embodied artificial intelligence. However, predicting multiple fine- grained states of scene at multiple temporal scales difficult for vision- languu-l models. We formalize the Multi-Scale Temporal Prediction (MSTP) task in general and surgical scene by decomposing multi-scale into two orthogonal dimensions: the temporal scale, forecasting states of human and surgery at varying look-ahead intervals, and the state scale, modeling hierarchy of states general and sur- gical scene. For instance in general scene, states of contacting relationship are finer-grained than states of spatial relationship. For instance surgical scene, medium-level steps are finer-grained than high-level phases yet remain constrained by their encompassing phase. support this unified task, we introduce the first MSTP Benchmark, featuring synchronized annotations across multiple state scales and temporal scales. We further propose novel method, Incremental Generation and Multi-agent Collaboration (IG-MC), which integrates two key innovations. Firstly, we propose an alug-and-play incremental generation to keep high-quality temporal prediction that continuously synthesizes up-to-date visual previews expanding temporal scales to inform multiple decision-making agents, ensuring decision content and generated visuals remain synchronized and preventing per- formance degradation as look-ahead intervals lengthen. Secondly, we propose decision-driven multi-agent collaboration framework for multiple states pre- diction, comprising generation, initiation, and multi-state assessment agents that dynamically triggers and evaluates prediction cycles to balance global coherence and local fidelity. Extensive experiments on the MSTP Benchmark in general and surgical scene show that IG-MC is generalizable plug-and-play method for MSTP demonstrating the effectiveness of incremental generation and the stability of decision-drive multi-agent collaboration. The code, model weights, benchmark can be found in
Introduction
The evolution embodied AI has ushered in systems capable of interpreting human behavior, predicting intentions, and executing physical actions in dynamic general and surgical environments
3 From assistive robots in healtheare [4] [S] to autonomous agents in smart home settings 6 to surgical robot with agentic systems in operating room, these systems aim to bridge the gap between cognitive reasoning and real-world interaction in healthcare. Traditional robotics, confined to structured tasks with predefined routines, has been revolutionized by models trained
T
"Conrsponding Author
Equal Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

on vast multimodal data [8 19. These models, such as Large Language Models (LLMs) Multi-Agent Systems (MAS) 12 13 14] and Vision-Language models (VLMs) [1] T6
10
offer unprecedented capabilities in parsing instructions, understanding scenes, and generating action sequences in general scene. However, a critical challenge of embodied learning in surgical scene remains: achieving reliable and trustworthy multi-scale temporal prediction that ensures both short- term action accuracy of healthcare robot and long-term task coherence of surgical embodied agents.
22 have significantly improved analysis and prediction of healthcare robot, achieving promising performance in tasks such as phase recogmition (23) and Instrument-tisse
In parallel vein, recent advances in artificial intelligence 19
20
21
interaction detection (24), through convolutional [25] 23).
recurrent 27 28). and transformer- based (29 330 31] architectures, as well as video generation
through diffusion models [32 33]. Undoubtedly, these approaches made an initial breakthrough, enabling faithful and effective surgical problem-solving. However, @ current work predominantly focuses on single-scale prediction and analysis, either coarse-grained phases or fixed temporal
windows [34, 35 36, 37). Despite some improvements, this myopic perspective neglects the
hierarchical and time- varying nature of surgical decision-making (38) Additionally, attributed to the accumulation of inaccurate generations in foundation models like VLM and LLM and the increase in visual tokens caused by inputting past multiple frames, @ there is still significant room for improvement in the performance of existing architectures when dealing with surgical workflow analysis and prediction tasks. Additionally, the increasing input length from streaming input frames poses significant challenges for existing methods in terms of computation and memory. leading to performance degradation with the linear growth of temporal input [39]. Furthermore, the accumulation of errors in generative models hinders the effectiveness of existing methods in intelligent surgical problem solving [40
Several recent studies have incrementally addressed these limitations, though key challenges remain. Across general scenes, long-video MLLMs and planners relax fixed windows and address streaming: Video-MME reveals persistent hour-scale reasoning gaps [42]; VideoStreaming and Long VLM bound token growth via memory propagation and hierarchical segment tokens [43 and HM- Diffuser targets extendable long-horizon planning with multiscale diffusion (45). Nonetheless, many pipelines still rely on sparse keyframes or heavy compression, exhibit unfavorable memory/latency scaling, and accumulate rollout errors over long horizons. In surgical scenes, GraSP offers multi- granular supervision unifying pixel-level instruments/actions with step/phase labels (46) while SWAG
anticipates minute-scale workflows with single-pass and autoregressive decoders [47]. Yet datasets
remain procedure-specife and clip-centric, cross-scale dependencies only partially modeled, and
prospective clinical validation is limited. These gaps motivate an integrated, trustworthy multi-scale temporal predictor that couples short-horizon action accuracy with long-horizon task coherence under realistic streaming constraints.
In light of this, we propose IG- MC (Incremental Generation via Multi-agent Collaboration), uni- fied closed-loop framework for multi- scale temporal prediction. At its core, IG-MC introduces two key innova- tions: (1) an incremental generation mechanism that dynamically synthe- sizes predicted states and images across expanding temporal scales to maintain state-visual synchroniza- tion, thereby reducing error accu- mulation in long-horizon predictions and ensuring trustworthiness; and (2) a decision-driven multi-agent col-
Image
Image
Iaage
Imsgge
Imnage
Past
Future
Decison-making
Phase
phase
Step
[attion
[action
action
Finer grained opecations
Stable Diflusion
Dossible Secoarios
Figure 1: Illustration of the IG-MC overview and advantage across various real-world scenarios.
laboration system comprising specialized VLM--assed agents 48
49
that hierarchically refine
predictions from phases to steps while enforcing cross-scale consistency. Departing from approaches that rely on frame histories, IG-MC uses a stateless DM that consumes only the current state-image pair to predict the next state, which then conditions the VG to synthesize the next visual preview (50)
51 52]: removing temporal buffers reduces memory and speeds up training and inference. Crucially, these innovations are embedded within a closed-loop design, where the state predictions of DM

---

module and the visual outputs of VG module continuously inform and correct one another. This bidirectional interaction enables real-time error correction, interpretable forecasting, and high- high-fidelity coherence across both global and local scales.
To rigorously evaluate multi-scale forecasting in general and surgical scene, introduce the first Multi-scale Temporal Prediction (MSTP) benchmark, enhanced with synchronized multi-scale annotations across multiple temporal horizons and hierarchical state. Through extensive experiments, our framework demonstrates excellent performance across various metrics. After adding the plug- and-play modules (DM, VG), most metries improve significantly.
Our major contributions can be summarized as follow:
Incremental Generation We propose the first incremental generation mechanism that dynamically synthesizes state-image pairs across expanding temporal scales, enabling error-corrected temporal prediction while maintaining state-visual synchronization.
Multi-agent Collaboration We develop decision-driven multi-agent collaboration framework for multiple fine-grained state prediction, where specialized VLM-based agents hierarchically coordinate temporal predictions while enforcing cross-scale consistency.
MSTP Benchmark We introduce the MSTP benchmark, the first dataset providing synchronized annotations across both temporal horizons and state hierarchies for comprehensive multi-scale evaluation.
2 Related Work
Temporal Prediction Capturing temporal dynamics multiple scales is crucial for long-horizon forecasting [5)] Clockwork RNNs partition recurrence across multiple rhythms to learn both fast and slow dynamics (55). Deep fully fully-conneeted stacks such as N-BEATS demonstrate that purely residual- based architectures achieve state-of-the-an results on M3/M4 competitions, offering interpretability and transferability (56). More recently, sule-supe hierarchical masked modeling methods like HiMTM leverage multi-scale transformers and self-distillation to enhance long-term time series accuracy [57). Complementary to these, hierarchical learning for long-source generation organizes evidence at coarse-to- granularity and improves coherence over extended contexts (58): multi-token prediction jointly optimizes next-k token heads to stabilize and accelcrate training while improving long-ho ong-hhoizzon fluency [59]: and option-basee temporal abstraction provides Icarnable sub-task policies and termination conditions that align with multi-scale control and forecasting (60).
Workflow Recognition Automated recognition of surgical workflow phases and steps has been stud- ied extensively. Early CNN-based multi-task methods [61] such EndoNet learn phase recognition and instrument presence jointly from laparoscopic videos [25]. SV-RCNet further integrates a ResNet backbone with an LSTM for online workflow segmentation [62]. More recent temporal convolutional
approaches [63). notably TeCNO, employ multi-stage dilated causal convolutions to refine phase predictions iteratively and enforce smooth transitions [23] Building on these, Transformer-based
models (64| 65) including Trans-SVNel 66). which uses a hybrid embedding aggregation strategy to fuse spatial and temporal cues, and (377 which introduces multi-scale temporal encoding to capture short-, mid- and long-term dependencies, further improve accuracy and temporal consistency
on standard surgical video benchmarks.
Visual Generation Text to image diffusion achieves high fidelity and controllable synthesis through latent diffusion (67] and language conditioned models (68). Spatial controls based on depth, edge, or segmentation provide precise guidance [69]. In surgical settings, endoscopic frame synthesis based on
Stable Diffusion (70) and guided generators for segmentation and triplet constraints [71 72 produce realistic frames that support augmentation and downstream analysis. Building on these image priors, video generation extends control over time. Open domain systems such as Sora improve long range temporal consistency [73). Surgical variants including Endora (32). SurGen (74), and SurgSora (75) provide controllable previews aligned with surgical workflows for endoscopy and operative scenes.
3 Methodology
In this section, we systematically introduce our framework IG-MC (See Fig 2. Initially, we elucidate the methods about incremental generation for multi-scale temporal consistency (Sec 3.2 Subse-
3

---

Incremental Generation
Iserutive Stacking
Images
surgeon
I. R
vir Blockes
vi Dlockss
Vin Diocled
Vit Clocii
pkoture
How should I perform liver tumor resection surgery? Please tell me every seconds if should proceed to the neat state, and how, with illustrative images
& x24
Si
DM Blooks
EM Blockss
DM
DMM
S, R2
Bloois
Sx--1
(koocia
(Gu)aan (m(Sk)))m,, where i S C. meaan each state represencs specifie ourgioal operolion in roal world scenaatoe
(0) Dersile
(b) fkealls Blicktr of fiier stop
Nitned
3
Pockipt
TEM
phow
Peel
purching
Fachokkings
rxpp
satiion
Phalis
change
Stan Gore lift prodioto
Bouily L-+1 predicurr
Uigarshig
Podder whethrtncchgge facr
memits
Slep prodiction procos
Sa..
Figure 2: The upper half of the image presents an overview of the model.where Fig (a) and (b) respectively showcase the details of the DM module (Multi-agent Collahoration) and VG module.
quently, we further engage in decision-driver multi-agent collaboration for hierarchical status updates (Sec 3.3
Below, will present the preliminaries and elaborate the contributions of each part
towards achieving model scalability and SOTA performances.
3.1 Preliminaries
Temporal Scale We begin by formalizing the temporal prediction problem. Consider a temporal event with total duration T divided into discrete time steps with the temporal scale time interval i, yielding Ñ = -[//f] output points. To improve the effects of predictions, we further consider incremental scale time interval, which is divisor of f. So there are N = [T/r]] predicting time points. However, the results will only be presented when the time reaches an output point. At each predicting time point t, the system must simultaneously predict both the future state St+r and generate corresponding visual guidance I,-, for multiple prediction horizons t E tty....,ty].
State Scale The state space S exhibits a hierarchical structure with multiple states. At the coarsest level, 5t captures high-level phases such as "Preparation", while finer levels S? through S describe progressively more granular steps and actions. This hierarchy induces natural dependency between levels, where fine-grained steps are semantically constrained by their encompassing phases. Formally, represent the complete state S; (s),..., ) Si X level é and time f. Each discrete time point l corresponds to a specific operation with environments C₂ E C, where C [CI...., Cm represents the complete set of m possible actions.
St. where sf denotes the state at
3.2 Incremental Generation
Our incremental generation framework operates through an alternating prediction process that inter- leaves state forecasting and visual guidance synthesis. At each time step &g. the system first predicts the subsequent state S4+1 by jointly considering both the current state Sş and the corresponding visual guidance Z, This decision-making process is formally expressed as:
Sa+1 DM, I4; PDM)
(I)
where DM represents our decision decision-making module (Sec 3.3 with learnable parameters fDM. which encodes the temporal evolution patterns of workflows while maintaining consistency with images.
Following state prediction, the system generates the corresponding visual guidance Iz41 for time 4-11 through conditioned diffusion process:
I++1 (G(S)+1, 0vG)
(2)

---

where VG denotes our udapted Stable Diffusion module with parameters θvg. specifically optimized for corresponding scenarios. This alternating update scheme creates a tight feedback loop between state prediction and visual synthesis, ensuring that: (1) state predictions remain grounded in visual evidence, and (2) generated guidance images accurately reflect the anticipated progress. The iter- ative nature of this process allows for error correction across time steps, as both state and visual representations are continuously refined based on each other's outputs.
For a certain procedure of duration ''' with discrete time intervals т, the complete sequence of predicted states {Sx 1 and guidance images {Ix {Ix)k=> (where N [T/r]) is generated through the following recurrent process:
Skt1 DM( Sc, Izj (DM)
for k = 0,............ .,....... , N - for k = ,,...,N N -
[++1 VG(S++1, Ov;
with initial conditions Su and To representing the observed state and image at procedure onset. The complete solution can be expressed as the composition of these operations across all time steps:
([Sa), (I+}) = DM o VG.... DM o (G Io)
(4)
2N operations
where o denotes function composition.
3.3 Multi-agent Collaboration
The docision-making process for state prediction from tx t3+1 is implemented through specialized multi-agent collaboration framework. At the core of this system resides State Transition Controller (STC) agent that determines whether the current acting phase requires advancement. The STC receives the current state S₁ and visual guidance Ia as input, producing either a continuation signal (maintaining the current state) or identifying the specific hierarchical level / E {I, L} where state transition should initiate.
Prediction
When state transition is required, the system activates sequence of L LLM-bbseed State Prediction agents, where each agent iY specializes in predicting transitions at level / of the operation hicrarchy. These agents are organized such that V1 handles the pparsest-grained phases, while uz manages the finest-grained steps. Formally, the agent collection V = {1y ey E .M, 1 < l S L} forms a workflow sequence where the prediction domain of each ey corresponds to a specific state subspace S, C S.
The prediction process proceeds through iterative refinement: the STC agent first identifies the starting level / for state changes, after which 1/ generates initial prediction. This output then propagates through subsequent agents 01-1 to ur. in a chain, with each agent refining the prediction using both the previous agent's output and the original visual context Ig. This be expressed as:
N Sa+1 Σ
s I(i < 1) + 22 (1 Ix) I(1 2 1) e;
(5)
where Sk+1 denotes the complete predicted state vector, which is assembled by combining compo- nents from all hierarchical levels. Specifically, each component s of S follows different update rules based on the level index i relative to the threshold For indices < I, the component inherits directly from the previous time step's state s preserving stability in coarse-graing searse-grained information. For 2 1, the component is computed via the function V;, which generates fine-grained details using the previously assembled lower-level states S and input information Z₁. This hierarchical update mechanism is integrated into unified vector expression through the summation operator. The indicator functions I(2 < I) and I(: 2 1) ensure the appropriate update rule is applied for each level, returning 1 when the condition is satisfied and 0 otherwise. + means adding. And the standard basis vector e; maps scalar values to their corresponding positions in the state vector, enabling seamless assembly of the complete state from hierarchical predictions.
Fine-tuning The State Transition Controller agent requires supervised fine-tuning to accurately identify state transition points. In practice, the workflows exhibit significant temporal sparsity in state
5

---

changes- the ratio between total time steps [ and the numbers of state transition points often exceeds 100:1. To address this severe class imbalance, we implement a targeted data augmentation strategy that synthetically increases the proportion of state transition samples. Specifically, we adjust the original imbalanced ratio [T/r]] applying a multiplicative factor a to 6, effectively rebalancing the data set to a 1:1 ratio during training. This augmentation is achieved through temporal window sampling around actual transition points, where each genuine state transition e; generates a synthetic variants by perturbing both the input states S₁ and visual contexts Ps within clinically plausible bounds. The perturbation space covers =ÂT timestamp shifts and e-intensity image modifications, preserving the semantic validity of operation transitions while diversifying the training distribution.
3.4 Visual Generation
The visual guidance generation module employs a modified Stable Diffusion architecture (VG) to synthesize medically meaningful previews conditioned on both predicted states and prior visual context, as illustrated as Equation 2 The VG module specifically utilizes latent diffusion paradigm, first encoding the input state S into hierarchical embedding space that aligns with the text encoder's 's semantic structure.
The denoising process is guided by three key conditioning mechanisms: ! the state embedding provides procedural constraints through cross-attention layers, @ the previous guidance image I is injected via residual connections maintain temporal coherence, and ( specific latent space projection ensures anatomical plausibility. This tripartite conditioning yields:
co(2, 1, () ([I)) + é
where represents the denoising network, 2, the latent noisy image at timestep t, 7(-) the state embedding projector, and o(.) the image feature extractor. The output én denotes the predicted noise used for iterative latent space refinement.
The diffusion process is optimized using scenarios-domain constrained objective:
LvG .e.. [||e ( (IS) AR(Ioar)
where R(.)) represents a regularization term enforcing tool-tissue interaction realism, and À controls its relative weight. This formulation ensures the generated guidance images maintain both procedural accuracy and visual continuity across temporal predictions.
3.5 Integrated IG-MC Framework
The complete IG-MC pipeline operates on certain task ensembles Q, where each task q Q represents a distinct procedure. Our framework features decoupled architecture where the DM module and VG module undergo separate training phases. This design enables flexible combination during inference while maintaining modularity. For each sampled task q, the DM module generates predicted state trajectories (Sx | through iterative application of Equation I while the VG module independently
produces visual guidance {I& k via Equation 2 with N = T/r] time steps.
The learning objective maximizes the average accuracy between predicted and ground-truth actions:
N
P(S =SmI k E z
LIG-MC
max 00s,.8v
EqnaQ
(8)
A=1
where dpm and QYG collect all trainable parameters of the DM and VG modules respectively, S, denotes the ground-truth action at time t and Z means the set of positive integers. The expectation approximated via Monte Carlo sampling over the task distribution Q. We provide more detailed derivation and explanation in Appendix J] This objective enforees two crucial properties: (1) differen- tiability through the state-action mapping, and (2) temporal coherence by equally weighting all time steps. The action accuracy serves surrogate metric for overall workflow prediction quality, as the actions directly correspond to related meaningful events. Additionally, other similar metrics are mentioned in the experiments.

---

Experiments
In this section, we will validate the effectiveness of our proposed integrated structure plugin, IG-MC. We design four research questions to comprehensively evaluate the performance of IG-MC: RQI: Does IG-MC effectively enhance model performance and applicability? RQ2: How do plug-and-play DM and VG module positively affect the model? RQ3: How does IG-MC perform in various scale scenarios? RQ4: Can IG-MC be applied to different models and real-world scenarios? Through these research questions, we aim to validate the effectiveness and advantages of IG-MC in handling various data from multiple perspectives.
4.1 Datasets and Evaluation Metrics
MSTP Benchmark Our MSTP Benchmark can be divided into two parts. MSTP Benchmark in general scene (MSTP-General) is built on top of the Action Genome (AG) dataset (76), containing comprehensive annotations of scene graph. We extract the multiple state information of object trajectories including attention, spatial, and contacting relationships between human and object. We augment AG with synchronized three states (attention-level, spatial-level, and contacting-level) at three standard temporal (Is, 2s, and 3s) and multiple dynamic temporal scales from 10s to 60s. 2. MSTP Benchmark Surgery (MSTP-Surgery) is built on top of the GraSP dataset [77), an endoscopic surgical scene understanding corpus for prostatectomy. We augment GraSP with synchronized two state scales (phase-level and step-level annotations) at four temporal scales (1 s, s, s, and s) to support unified, multi-scale temporal prediction.
State Prediction Metrics. We evaluate multi-scale state prediction using Accuracy, Precision, Recall, Fl Score, Jaccard (See Appendix HH. Furthermore, the prediction metrics of each state and combined states can be utilized evaluate the stability of models. P indicates predicted phase, S indicates predicted step, and P&S indicates the combination of predicted phase and step.
Visual Prediction Metries. We evaluate visual prediction performance across several complementary dimensions (See Appendix [): (1) pixel-level fidelity, via Peak Signal-to-Noise Ratio (PSNR); (2) structural consistency, via Structural Similarity Index Measure (SSIM) and its multi-scale variant (MS-SSIM); (3) perceptual realism, via Learned Perceptual Image Patch Similarity (LPIPS) and CLIPScore; (4) distributional alignment, via Fréchet Inception Distance (FID) and Kernel Inception Distance (KID); and (5) retrieval-basee congruence, via Recall Precision (R-precision).
Destovs Wtedeg Tespert to Eecrmnn
Deebee Vis
3428
305
1EX
14
103
1491
34.00
25.38
769
15.0
2138
1324
Rowll
Janerd Jakes
200
18.55
11.00
122
2331
3030
Figure 3: Temporal analysis of proposed IG-MC. The quality of decision-making is decreasing while the quality of generation is also decreasing.
7

---

Temp-
Incr.
Model
Scale
Scale
State
Accuracy
Precision
Recall
Jaccard
MSTP-General Benchmark
Qwen2.5.VL..77 Instructttttt 78
Mixture
ARSIC
7.28
DM
Mixture
ARSEC 56.11 (+42.51) 40.29
(+23,29)
37.39
(+23.82)
38.57
(+25.61)
27.14 (+19.86)
DM + VG
Mixture
T/2
ARS&C 63.14 (+7.03)
$1.06 (+10.77) 50.68 (+13.29)
50.43 (+11.86)
36.86 (+9.72)
Mi--2..-VL7B---ssee 78
1-10
A&S&C (444.36)
14.17
18.06
14.81
13.47
7.61
(+22.81)
39.06
39.83
(+26.36)
(+20.79)
DM VG
1-10
TV2
ARS&C
63.16 (+4.63)
51.12 (+10.25)
50.71
(+11.65) 50.45 (+ (0.62)
36.87 (+8.47)
Qwen2.5-VL.77 Instruct
78
10-20
A&S&C
12.17
10.32
12.47
10.24
6.21
DM
10-20
ARSAC 52.79 (+40.62)
39.90
(+29.558)
34.99
(+22.52)
36.67 (+26.43)
25.33
(9.12)
DM VG
10-20
TV2
A&S&C
63.97 (+11.18)
51.68 (+11.78)
51.44
(+16.45)
51.07
(+14.40) 37.60 (+.277
Qwen2.5-V1..77 1wen2.55V1--B-Bnstrrr
ARS&C
5.43
DM
20-30
A&S&C 51.46
(+46.770
46.62 (+422336)
41.71
(+36.28)
43.04
(+39.16)
29.45
(+26.74)
DM + VG
20-30
T/2
ARS&C 60.22 (+8.76) 60.53 (+13.911 60.00 (+18.29)
59.36 (+16.32)
42.47 (+13.02)
Qwen2.5.VL Dwee2.5-VL-7B--mmmrre Instruct
78
>30
ABS&C
17.70
14.17
15.63
13,25
8,86
ARS&C
(+32.50)
(+24.54)
33.35 (+17.72)
35.37
(+22.12)
DM + VG
>30
TY2
ARS&C 63.27 (+13.07) 63.63 (+24.92)
63.44 (+.3009)
63.20 (+27.83)
46.23 (+22.15)
Surgery Benehmark
Qwen2.5-VL -L-----------ttuud 78
P&S
14.00
3.61
4.29
3,58
1,97
(+35.99)
(+26.84)
(+22.50)
28.24 (+24.66)
Wwwen.5-VL-7B--tsttue
78
P&S
12.00
3,32
3.69
2,90
1.57
DM
P&S
50.80 (+38.80)
32.51
(+29.19)
28.83
(+25.14)
29.03
(+26.13)
20.06
(+18.49)
DM VG
P&S
42.40 (-8.40)
29.37 (:3.14)
33.33 (+4.50)
29.31 (+0.28)
21.38 (+1.32)
Qwen2.5.VL.7B VL.7B Instruct
78
30
30
P&S
13.50
3.92
4,02
3,51
1.92
PRS
(+32.40)
(+13.57)
(+9.79)
14.48
10.97)
(+8.02)
DM VG
30
P&S
41.10 (-4,80)
28.87 (+11.38)
33.99 (+20.18)
28.24
(+13.76)
20.18 (+10.24)
Qwen2.5-VL-77-Instuu
78
P&S
13.20
4.71
3.83
2.07
DM
60
60
P&S
51.90 (+38.70)
17.81 (+13.33)
16.09
(+11.38)
16.58
(+12.75)
11.44 (+9,37)
P&S
(-14.70)
(+6.86) 25.74 (+9.65)
22.65 (+6.07)
15.57 (+4.13)
Table I: Consolidated comparison of VLMs with plug-and-play DM and VG modules across benchmarks. Bold red (+) indicates improvement and bold blue (-) indicates drop. Temp. Scale indicates temporal scale while Incr. Scale indicates incremental scale; A&S&C are attention, spatial, and contacting relationship, P&S are phase and step.
4.2 Evaluating the Efficacy of IG-MC (RQ1 & RQ4)
To address RQ1, we conduct experiments on execution time and various evaluation metrics across various scenarios. As shown in the Figure and Table we can list the Observations:
Obs 1. IG-MC maintains high performance temporal scales increase. As shown in Fig 3 can easily observe that the performance of introducing IG-MC remains at high level over time. For example, when temporal scale is 5s, the Accuracy remains at 42.06% time step 5. Although it exhibits a decrease of 3.62% compared to time Step 1, its performance remains favorable.
Obs 2. IG-MC drives comprehensive improvements across multi-dimensional evaluation metrics. By analyzing the left side of Table [ we clearly see that introducing +IG-MC significantly improves model performance on Accuracy, Precision, Recall, F1 and Jaccard metrics. In MSTP- Surgery benchmark, with DM and VG, Accuracy rises from 13.5% - 45.90% Precision from 3.92% + 17.49%, Recall from 4.02% + 13.81%, etc. This shows the effectiveness of IG-MC in boosting performance in various domains.
Obs 3. IG-MC generalizes to diverse real-world scenarios, including general human action and surgical workflows. In general scenarios tested on the Action Genome (AG) dataset, IG-MC achieves an F1 score of 51.07 for joint outperforming baselines by 10.24%. In surgical contexts, the framework maintains 28.87% accuracy at 30s temporal scale with 5s incremental updates 12..55%* 1 compared to the baseline). This dual applicability underscores IG-MC's potential as a unified framework for multi-scale prediction in both clinical and general embodied AI scenarios.
state predictions at 10-20s scales,
8

---

4.3 Ablation Results (RQ2)
To address RQ2 on the positive impacts of the aalg-and-pla DM and VG modules, our empirical analysis demonstrates that both components contribute distinct yet complementary improvements to the model's predictive capabilities, with the combined framework outperforming baselines across multi-scale surgical and behavioral forecasting tasks in Table
Obs The DM module significantly enhances state prediction accuracy and hierarchical consistency. By integrating surgical workflow knowledge through LLMM-ased agents, the DM module enables the model to reason about phase-step dependencies and temporal transitions. On the MSTP-Surgery benchmark, adding the DM module to the baseline VLM boosts Accuracy from 13.20% 51.90% at the 60s temporal scale. Notably, this improvement is accompanied by 13.33-point increase in precision and 1 11.38-point increase in recall, indicating that the DM module not only enhances prediction correctness but also reduces false positives and negatives in hierarchical state transitions. For shorter scales such as 5s. the DM module achieves 38.80 80-point accuracy gain, highlighting effectiveness across all temporal horizons.
Current Frame
Current Frame
Generated Frame (25s)
Current Frame Generated Frame (28s) Generated Frame (30s) Q: 28 seconds, what will be the contacting relationship between doorknob and human?
Q: 25 seconds, what will be the
Q: In 30s seconds, what wil be the surgical step ?
spatial relationship between doorway and human?
A: The step will be
A: <uuman, In, Doorway>
A: <HHmmn, Touching, Doorknob> pulling the suture
Figure 4: Qualitative analysis of proposed IG-MC.
Obs 2. The VG module further elevates performance by aligning visual guidance with state predictions. By synthesizing high-fidelity surgical previews conditioned on predicted states, the VG module reinforces temporal coherence and interpretability. In surgical scenarios, at 30s temporal scale, the VG module enhances recall by 11 1.38% and Jaccard by 9.37% when using a 60s temporal and incremental scale, demonstrating that visual feedback enables more refined step predictions. Crucially, the joint use of DM and VG achieves 24.73% F1 score improvement over the baseline, underscoring their synergistic role in balancing abstract decision-makinn maaiing with concrete visual grounding.
4.4 Performance in Various Scenarios (RQ3)
To address RQ3 on IG-MC's performance across diverse scale scenarios, our comprehensive evalua- tion demonstrates the framework's 's exceptional adaptability and robustness across temporal scales, state hierarchies, and incremental time intervals, as illustrated in Table I and Fig
Obs 1. IG-MC excels across varying temporal scales, from short to long horizons. As shown in Table on the MSTP-Surgery benchmark, IG-MC achieves state prediction accuracies 50.80% at 5s, 45.90% at 30s, and 51.90% at 60s when using the DM module dlone-markee 38.80%, 32.40%, and 38% over baselines, respectively. These results highlight IG-MC's capacity to balance short-term precision and long-term stability, which is critical advantage for surgical workflow management.
improvements of
Obs 2. IG-MC achieves strong performance on hierarchical state scales, from coarse phases to fine steps On phase-level state scale MSTP-Surgery shown in Table (See Appendix EE. IG-MC achieves an Fl score of 56. when the temporal scale is 5s, compared to the baseline of 13.48. At the finer step- scale, the framework improves recall by 31.78 points (from 7.94% to 39.72%) at the same temporal scale, showcasing its ability to model granular surgical actions while

---

respecting hierarchical constraints. In cross-scale consistency tests, the joint phase- step prediction (P&S) achieves Jaccard index of 20.06 with 5s temporal scale, much higher than the baseline of 1.57, confirming that IG-MC maintains coherence between abstract phases and concrete steps.
Obs 3. IG-MC adapts effectively to different incremental time scales, optimizing prediction granularity. When using a Is incremental scale for 5s temporal predictions in Table | IG-MC's DM+VG setup achieves an accuracy of 42.40% on MSTP-Surgery, slightly below the DM-only baseline (50.80%) because of the quality of the pictures generated by VG. But with a 4.50-point recall improvement, indicates the enhancement step-level detail of the overall stability. At longer incremental scales (e.g., 5s for 30s temporal predictions), the framework retains 41.10% accuracy while improving recall by 20.18 points, demonstrating that coarser incremental updates can prioritize structural consistency without severe performance degradation. This flexibility allows IG-MC to trade off between computational efficiency and prediction granularity, making suitable for real-time and planning-oriented scenarios alike.
5 Conclusion
In this study, we address the challenge of multi-scale temporal prediction by decomposing the task into temporal and state scales, introducing the MSTP benchmark with synchronized multi-scale annotations for general and surgical scenes. We propose a unified closed-loop framework called IG-MC, which uses an incremental generation mechanism to maintain temporal consistency in state- image synthesis and a decision-driven multi-agent system to hierarchically refine predictions across scales, enabling cross-scale coherence and real-time interaction between state forecasts and visual generation. This work demonstrates a significant advancement in multi-scale prediction accuracy and robustness, offering promising foundation for enhancing predictions in dynamic scenarios.
Acknowledgments and Disclosure of oufFuun Funding
This work was supported by Ministry of Education Tier 2 grant, Singapore (T2EP20224-0023)) and Ministry of Education Tier grant, NUS, Singapore (23-0651-P0001)
References
[1] Wasim Khan and Mohammad Ishrat. "Embracing the Future: Navigating the Challenges and Solutions in Embodied Artificial Intelligence". In: Building Embodied Al Systems: The Agents, the Architecture Principles, Challenges, and Application Domains. Springer, 2025, pp. 281- 299.
[2] Longbing Cao. "Ai robots and humanoid ai: Review, perspectives and directions". In: arXiv preprint arXiv:2405. 15775 (2024).
[3] Tian Wang et al. "Multimodal hobon interaction for human-centric smart manufacturing:
a survey"" In: Advanced Intelligent Systems 6.3 (2024), p. 2300359.
[4] Arshia Khan and Yumna Anwar. "Robots in healthcare: A survey". In: Advances in Computer Vision: Proceedings of the 2019 Computer Vision Conference (CVC), Volume 2 1. Springer. 2020, pp. 280-292.
[5] Laura Aymerich-Frande and Iliana Ferrer. "Socially assistive robots' deployment in healthcare settings: global perspective". In: International Journal of Humanoid Robotics 20.01 (2023), p. 2350002.
[6] Arpita Soni et al. "Advancing Household Robotics: Deep Interactive Reinforcement Learning for Efficient Training and Enhanced Performance". In: arXiv preprint 17X1v:2405.18687 (2024).
[7] Watcharakorn Pinthurat, Tossaporn Surinkaew, and Branislav Hredzak. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages". In: Renewable and Sustainable Energy Reviews 202 (2024), p. 1 114648.
[8] Kento Kawaharazuka et al. "Real-world robot applications of foundation models: A review". In: Advanced Robotics 38.18 (2024), pp. 1232-1254.
10

---

[9] Hossein Naderi, Alireza Shojaei, and Lifu Huang. "Foundation Models for Autonomous Robots in Unstructured Environments". In: arXiv preprint arXiv:2407.14290 (2024).
[10] Miao Yu et al. "Mind Scramble: Unveiling Large Language Model Psychology Via Typo-
glycemia". In: arXiv preprint arXiv:2410.0166 (2024).
[11] Kun Wang et al. "A Comprehensive Survey in LLM (-Agent) Full Stack Safety: Data, Training and Deployment", In: arXiv preprint arXiv:2504.15523 (2025).
[12] Miao Yu et al. "A survey on trustworthy Ilm agents: Threats and countermeasures" In: arXiv preprint arXiv:2503 09648 (2025).
[13] Junyuan Mao et al. "Agentsafe: Safeguarding large language model-based multi-agent systems via hierarchical data management". In: arXiv preprint TrXiv:2503.0439) (2025).
[14] Miao Yu et al. "Netsafe: Exploring the topological safety of multi-agent networks". In: arXiv preprint ar..hv24400.1666 (2024).
[15] Chunyuan Li et al. "Llava-med: Training a large language language-and-vee and-vision assistant for biomedicine in one day". In: Advances in Neural Information Processing Systems 36 (2023), pp. 28541-
28564.
[16]
Aaron Grattafiori et al. "The llama 3 herd of models". In: arXiv preprint arXiv: (2024).
[17]
Yifan Duan al. "Causal Deciphering and Inpainting in Spatio- Temporal Dynamics via Diffu-
sion Model". In: Advances in Neural Information Processing Systems 37 (2024), pp. 107604- 107632.
[18) Zaige Fei et al. "Open-CK: A Large Multi-Physics Fields Coupling benchmarks in Combustion Kinetics". In: The Thirteenth International Conference on Learning Representations.
[19] Wenpin Qian al. "Next- -Next-generation artificial intelligence innovative upplications of large language models and new methods". In: Old and new technologies of learning development in modern conditions 262 (2024).
[20] Reza Amini, Ali Amini, et al. "An overview of artificial intelligence and its application in marketing with focus on large language models". In: International Journal of Science and Research Archive 12.2 (2024), pp. 455-465.
[21] Yulu Wang et al. "Hi-sigir: Hierachical canantic-guided image-to-image retrieval via scene graph". In: Proceedings of the 31st ACM International Conference on Multimedia. 2023, pp. 6400-6409.
[22] Zhitao Zeng et al. "Cognition guided human-object relationship detection".. In: IEEE Transac- tions on Image Processing (2023), pp. 2468-248244 2480.
[23] Tobias Czempiel al. "Tecno: Surgical phase recognition with multi-stage temporal convolu- tional networks". In: Medical Image Computing and Computer Assisted Intervention-MICCAN 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23. Springer. 2020, pp. 343-352.
[24] Wenjun Lin et al. "Instrument-tissue interaction detection framework for surgical video under- standing". In: IEEE Transactions on Medical Imaging (2024).
[25] Philip Twinanda, S Shekhar, S Pugh, et al. "EndoNet: A deep architecture for recognition tasks on laparoscopic videos". In: International Journal of Computer Assisted Radiology and Surgery 11.4 (2016), pp. 801-809. DOI: 10. s --1--1111111-1111111111111111111 015-1-1-151111
[26] Dong Keon Lee et al. "Detection of acute thoracic aortic dissection based on plain chest radiography and a residual neural network (Resnet)". In: Scientific Reports 12.1 (2022),
p. 21884.
[27]
Yueming Jin et al. "SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network". : In: IEEE transactions on medical imaging 37.5 (2017), pp. 1114 1 1126.
[28]
Yueming Jin et al. "Multi-task recurrent convolutional network with correlation loss for surgical video analysis". In: Medical image analysis 59 (2020), p. 101572.
[29] Yueming Jin et al. "Temporal memory relation network for workflow recognition from surgical video". In: IEEE Transactions on Medical Imaging 40.7 (2021), pp. 1911-1923.
11

---

[30]
Dani Kiyasseh et al. "A vision transformer for decoding surgeon activity from surgical videos". In: Nature biomedical engineering 7.6 (2023), pp. 780-796.
[31]
Lalithkumar Seenivasan et al. "Surgical-vqa: Visual question answering in surgical scenes
using transformer". In: International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer. 2022, pp. 33-43.
[32] Chenxin Li et al. "Endora: Video Generation Models as Endoscopy Simulators". In: arXiv preprint arXiv:2403. 1050 (2024). DOI: 10. 48550/arXiv. 2403. 11050
[33] Diego Biagini, Nassir Navab, and Azade Farshad. "Hierasurg: Hierarchy- aware diffusion model for surgical video generation". In: International Conference on Medical Image Compuling and Computer-Assisted Intervention. Springer. 2025, pp. 310-319.
[34] Sang-Gou Lee et al. "Adaptive undersampling and short clip-based two-stream CNN-LSTM model for surgical phase recognition on cholecystetonny videos". In: Biomedical Signal Processing and Control 88 (2024), 105637.
[35] Bokai Zhang et al. "SF-TMN: FaswFast temporal modeling network for surgical phase recog- nition". In: International Journal of Computer Assisted Radiology and Surgery 19.5 (2024), pp. 871-880.
[36] Shu Yang et al. "Surgformer: Surgical transformer with hierarchical temporal attention for surgical phase recognition". In: International Conference on Medical Image Compuling and Computer-Assisted Intervention. Springer. 2024, pp. 606-616.
[37] Alejandra Pérez al. "MuST: Multi-Scale Transformers for Surgical Phase Recognition". In: arXiv preprint arXiv:2407. 1736/ (2024).
[38] Di Wu et al. "A review on machine learning in flexible surgical and interventional robots: Where we are and where we are going". In: Biomedical Signal Processing and Control 93 (2024), p. 106179.
[39] Jingyi Zhang al. "Vision-language models for vision tasks: A survey". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).
[40] Sophia M Pressman et al. "Clinical and surgical applications of large language models: a systematic review". In: Journal of Clinicul Medicine 13.11 (2024), p. 3041.
[41] Zhitao Zeng et al. "LurgVLM: Benchmark for Surgical Intelligence". In: arXiv preprint arXiv:2506. 250::22556000055 (2025).
LLM: A Large Vision-Language Model and Systematic Evaluation
[42] Chaoyou Fu et al. "Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis". In: arXiv preprint arXiv:2405.21007 (2024).
[43] Rui Qian al. "Streaming Long Video Understanding with Large Language Models". In:
arXiv preprint arXiv:2405. 16009 (2024).
[44] Yuetian Weng et al. "Long VLM: Efficient Long Video Understanding via Large Language Models". In: European Conference Computer Vision (ECCV). 2024.
[45] Chang Chen et al. "Extendable Long-Horizon Planning via Hierarchical Multiscale Diffusion". In: arXiv preprint arXiv:2503.20102 (2025).
[46] Nicolás Ayobi et al. "Pixel-Wise Recognition for Holistic Surgical Scene Understanding". In: arXiv preprint arXiv:2401.11177 (2024).
[47] Maxence Boels al. "SWAG: Long-term Surgical Workflow Prediction with Generative-based
Anticipation". In: arXiv preprint arXiv:2412.18844 (2024).
[48] Zhehao Zhang et al. "VipAct: Visual-perception enhancement via specialized vlm agent collaboration and tool-use". In: arXiv preprint arXiv:24100. 16400 (2024).
[49] Danqing Zhang et al. "Lite WebAAent: The Open-Source Suite for VLM--ased Web-Agent Applications". In: arXiv preprint arXiv:2503.022. 02250000000000 (2025).
[50] Daniel Bolya and Judy Hoffman. "Token merging for fast stable diffusion". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, pp. 4599-4603.
[51] Raphael Tang et al. "What the daam: Interpreting stable diffusion using cross attention".. In: arXiv preprint arXiv:2210.0448 (2022).
[52] Ali Borji. "Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2". In: arXiv preprint arXiv:2210.0055. (2022).
12

---

[53] Çaghan Köksal et al. "Sangria: Surgical video scene graph optimization for surgical workflow prediction". In: International Workshop on Graphs Biomedical Image Analysis. Springer. 2024, pp. 106-117.
[54] Kubilay Can Demir et al. "Towards Intelligent Speech Assistants in Operating Rooms: A Multimodal Model for Surgical Workflow Analysis". In: arXiv preprint arXiv:2406. 14576 (2024).
[55]
Jan Koutník et al. "A Clockwork RNN". In: International Conference on Machine Learning. 2014.
[56) Boris N Oreshkin et al. "N-BEATS: Neural basis expansion analysis for interpretable time series forecasting". In: International Conference on Learning Representations. 2020.
[57] Shubao Zhao al. "HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long- Term Forecasting". In: Proceedings of the 33rd ACM International
Conference on Information and Knowledge Management (CIKM). 2024, pp. 3352-3362. DO1:
10. 1115/362767336677777
[58] Tobias Rohde, Xiaoxia Wu, and Yinhan Liu. "Hierarchical Learning for Generation with Long Source Sequences". In: arXiv preprint arXiv:2 104.07545 (2021).
[59] Fabian Gloeckle et al. "Better & Faster Large Language Models via Multi-Token Prediction".
In: arXiv preprint vrXiv:2404.19737 (2024).
[60] Martin Klissarov and Doina Precup. "Flexible Option Learning". In: Advances in Neural Information Processing Systems. Vol. 34. 2021, pp. 4632-4646.
[61] Kubilay Can Demir et al. "Deep learning in surgical workflow analysis: a review of phase and step recognition". In: IEEE Journal of Biomedical and Health Informatics 27.11 (2023),
pp. 5405-5417.
[62] Yueming Jin et al. "SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network". In: IEEE Transactions on Medical Imaging 37.5 (2018), pp. 1114-
122.. DO1: 10. 11091 TMI. 0717677776
[63] Bokai Zhang al. "Surgical workflow recognition with temporal convolution and transformer for action segmentation". In: International Journal of Computer Assisted Radiology and Surgery 18.4 (2023). pp. 785-794.
[64] Yang Liu et al. "Lovit: Long video transformer for surgical phase recognition". In: Medical Image Analysis 99 (2025), p. 103366.
[65] Soumyadeep Chandra et al. "ViTALS: Vision Transformer for Action Localization in Surgical Nephrectomy". In: arXiv preprint 24Xx::240000257. (2024).
[66] Yueming Jin et al. "Trans-SVNet: hybrid embedding aggregation Transformer for surgical workflow analysis". In: International Journal of Computer Assisted Radiology and Surgery 17.12 (2022), pp. 2193-2202.
[67] Robin Rombach et al. "High-Resolution Image Synthesis with Latent Diffusion Models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 10684-10695.
[68] Chitwan Saharia et al. "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding". In: arXiv preprint arXiv:2205.114881 (2022). DOI: 10. 48550/ arxiv 2205
11487
[69] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. "Adding Conditional Control to Text-to- Image Diffusion Models". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 3836-3847.
[70] Joanna Kaleta et al. "Minimal data requirement for realistic endoscopic image generation with Stable Diffusion". In: International Journal of Computer Assisted Radiology and Surgery 19.3 (2024), pp. 531-539. DOI: 10. 000000//11115--22-----00-00
[71] Emanuele Colleoni et al. "Guided image generation for improved surgical image segmentation" In: Medical Image Analysis 97 (2024), p. 103263. DOI: 10. 1016/j media. 2024. 103263
[72] Chinedu Innocent Nwoye et al. "Surgical Imette Generation". In: arXiv preprint
arXiv:2407.091 09230 (2024). Do1: 10. (188550/aXiii 2407.0922
13

---

[73] OpenAl. Sora: Creating video text. htt.:tpss//ppaal...coo//eeeeeers Accessed: 2025-10-22. Feb. 2025.
[74] Joseph Cho et al. "SurGen: Text-Guided Diffusion Model for Surgical Video Generation". In: arXiv preprint arXiv:2408. 14028 (2024). DOI: 10. 48550/arXiv. 11408.14020
[75] Tong Chen et al. "SurgSora: Decoupled RGBD-Flow Diffusion Model for Controllable Surgical Video Generation", In: arXiv preprint arXiv:2412.14018 (2024). Also MICCAI 2025 listing.
DO1: 48550/ arxiv 2412. 14018
[76] Jingwei Ji et "Action genome: Actions as compositions of spatio-temporal scene graphs". In: Proceedings of the IEEF/CVF conference computer vision and pattern recognition. 2020, pp. 10236-10247.
[77] Nicolás Ayobi et al. "Pixel-wise recognition for holistic surgical scene understanding". In: Medical Image Analysis 106 (2025), p. 103726. ISSN: 1361-8415, DOl: bttpp:///://
10. 1016/j 2025 103726
[78]
Shuai Bai et al. "Qwen2.5-vl technical report". In: arXiv preprint arXiv:2502.13923 (2025).
[79]
Patrick Esser et al. "Scaling rectified flow transformers for high-resolution image synthesis". In: Forty-first international conference on machine learning. 2024.
[80] Weihan Wang et al. "Cogvlm: Visual expert for pretrained language models". In: Advances in Neural Information Processing Systems 37 (2024), pp.
14

---

A Limitations
While IG-MC demonstrates strong performance across multi-scale prediction tasks, several limitations warrant discussion. (1) The visual prediction quality contingent on stability and anatomical accuracy of the Stable Diffusion module. In scenarios with highly specialized surgical tools or rare anatomical variations, VG may generate visually plausible but semantically inconsistent previews, potentially mislcading downstream decision agents. (2) The reliance pre-trained VLMs for base scene understanding implies that its performance is bounded by the VLMs' intrinsic capabilities- for example, limited depth in modeling fine-grained surgical tool interactions in low-resolution imagery. (3) The iterative nature of the incremental generation mechanism introduces non-trivial inference latency, particularly at long temporal scales. Future work could explore lightweight diffusion variants or parallelized agent architectures to address these trade-offss between accuracy and efficiency.
B Inferency Latency and Computational Efficiency
We profile computational efficiency and latency on a single NVIDIA HE200 GPU as shown in The end-to-end wall-clock latency is approximately 68s. The three decision modules the State Transition Controller (STC), the level predictor, and the step-level predictor eecch take ~20-222 22 and together account for > 90% of total wall-time while operating at only #1 TFLOPS on average, indicating a memory-boune bottleneck rather than a capacity limit. By contrast, the Incremental Generation stage reaches =>97 TELOPS peak but adds only so6s. Peak GPU memory remains modest throughout: s=26 GiB for the decision stack and #=29 GiB for generation.
Table Inference latency and computational efficiency (single NVIDIA H200). "K" denotes x x10 Peak memory in GiB.
Component
Avg Time
Min
Max
Avg. GFLOPS
Min
Max
Peak GPU Mem (GiB)
State Transition Controller
20.04
19.33
20.77
1.12K
108.94
1.36K
26.14
Phase Predictor
20.90
19.87
21.76
1.10K
109.52
1.29K
26.14
Step Predictor
21.51
20.43
22.30
1.07K
90.31
1.24K
26.1 14
Incremental Generation
5.81
5.78
6.10
97.32K
78.62K
99.71K
28.53
Throughput is not yet real-time; however. the profile suggests clear optimization avenues already underway: cross-scale weight sharing, quantization, KV-cuche reuse, and light MoΣ pruning. The numbers below represent an upper bound under current configuration; targeted compression and caching, we expect substantial latency reductions toward sub-sccond responsiveness appropriate for ontra-operative scenarios.
C Datasets and Evaluation Metrics
MSTP Benchmark Our MSTP Benchmark in Surgery is built top of the GraSP dataset 77 endoscopic surgical scene understanding corpus for prostatectomy. We augment GraSP with synchronized two state scales (phase-level and step-level annotations) at four temporal scales (1 s. 5 s, 30 s, and 60 s) to support unified, multi-scale temporal prediction.
GraSP comprises annotated videos of 13 cases at 1280x800 resolution and 1 fps; following the official protecol, cases 1, 2, 3, 4, 7, 14, 15, and 21 are used for training, and cases 41, 47, 50, and 51 for testing.
We report dataset sizes exclusively for MSTP, which is constructed by re-sampling the GraSP cor- pus of robot-assisted prostatectomy videos. From GraSP's 32h 13- video source, we generate scale- aware future-prediction clips at four horizons (1 s, 5s, 30 s, s0 and split them 10:1 into train/test, yiclding 40k training and 4k test clips in total. At
Table Dataset analysis of proposed MSTP benchmark.
Temporal Scale
Train
Test
Details
ss
10k
lk
2 frames, 4 states
5s
10k
lk
6 frames, states
30s
10k
lk
31 frames, 62 states
60s
10k
1k
61 frames, 122 states
each scale there 10k training and 1k test clips (Table Each sample comprises two parts: (1) the current image its states, and (2) the future frames and their states. Windows are extracted the native 30 fps and aligned so the first frame index is shared across scales.
MSTP provides hierarchical supervision with two nested tiers: Phase (11 classes) and Step (21 classes), where fine grained Step labels are strictly contained within their parent Phase. The temporal construction per horizon determines the number of frames and states per clip: e.g.. Is clips contain 2 frames and states, while 60s clips
15

---

Phase Statistics
(b) Step Statistics
14000
17500
12000
15000
135.7
10000
FRCC
12500
Percantagg (90)
Percentage
10000
0000
A000
289
2000
left pelvic isolated....
developing the space...
right pelvic Isolate..
bladder neck reconst...
seminal vesiole diss.
bladder neck identil..
prostatic pediole co..
ligation of the deep
severing the pros...
development of the p...
cutling suture o.. coentilc
reliication d.
diase sosscctoon con. seminal vasic dist. d.
identili cetttti suture the nossting
dolebitio pollins the suturo cerrrme suction
penection oi Demmm qussen en cevelie mearti (nn) tine thee
the pronson a cottirn the treman sutore
tisrration lping lepation it dorn
satiisa de
inser halddng
preven
Figure 5: Dataset analysis of proposed MSTP.
contain 61 frames and 122 states. This design ensures temporally consistent inputs across scales and supports coherent hicrarchical learming for surgical temporal reasoning.
D Implementation Details
Generation Agent Our Generation Agent is implemented via the Stable Diffusion 3.5 Large architecture 79] fine-tuned on 1x NVIDIA H100 GPUs in mixed-precision (bf16). Starting from publicly available pre-trained checkpoint, we trained for one epoch with a per GPU batch size of 34 (no gradient accumulation). Optimization performed using AdamW with bfl6 precision, cosine learning- rate schedule and 10% warm-up up phase, weight decay set to 0, gradient checkpointing, and XFormers memory-efficiee images were resized so that their shorter edge measured 1024 px (maintaining aspect ratio) with minimum resolution of 1024 px. No EMA updates were applied. At inference, we used 30 denoising steps, a CFG scale of 7.5, and the negative prompt "blurry, cropped, ugly."
attention. During training, all
pecision-making Agent The Decision Agent leverages VLMs including Qwen2.5. Instruct (80). fine- tuned MSTP decision dataset using 4x NVIDIA H100 GPUs in pixeed-recision (bf16) with TF32 support and gradient checkpointing. We employed a the Adam optimizer (initial lcarning rate x 10 p per-device batch size 32, no gradient accumulation. and cosine decay, 10% warm-up, weight decay x 10 10-").
Visual inputs were resized so that the shorter side was 512 and text inputs tokenized with the Qwen tokenizer; modality lengths were grouped align sequence lengths. At inference, greedy decoding was applied (maximum length 128 tokens) with top-p sampling (p- 0.9).
E
Visual Generation Analysis
To isolate the contribution of the VG module, we pair the baseline VLM with VG alone (no multi-agent collaboration) and evaluate continual maatheual-rather rather than hierarchieal state prediction. The VG-only ablation still yields clear gains, demonstrating that VG generalizes across non-hierarchical and hierarchical formulations. practice, IG, MC, and VG constitute reproducible, plug-and-play toolkit that reliably improves temporal prediction across diverse VLM backhones.
We provide a dedicated analysis, including (i) decision errors made by different agents and (ii) instances where generated visuals are of insufficient quality, alongside their causes and potential remedies. To quantify accumulated emrors in incremental generation, analyze the relationship between decision accuracy and image quality measured by Fréchet Inception Distance (FID; lower is better) across prediction horizons.
As shown Table 7 image quality degrades with longer horizons, yet decision accuracy remains comparatively resilient. Empirically, we observe a strong negative correlation between accuracy and FID,
FID
Accuracy,
(R < 0, p 0.05).
(9)
16

---

Table 4: Comparison of General VLMs with plug-and DM and VG modules. TS = temporal
scale, IS incremental scale.
Base
Variant
TS
IS
Accuracy
Precision
Recall
Base
1
13.60
3.75
3.34
+DM
1
1
30.00(+16.40)
23.50(+19.75)
13. (3,80(+10:46)
Base
5
5
12.20
2.07
2.38
+DM
5
5
27.90(+1..70)
26.03(423.96)
14.95(+12.57)
+DM+VG
5
46.40(+16.40)
36.45(+12.95)
36.88(+23.08)
LLaVAL.5-B
Base
30
30
17.00
4.17
2.77
+DM
30
30
28.00(+11.00)
21.66(+17.49)
11.97(+9.20
+DM+VG
30
5
44.20(+16.30)
32.92(+11.26)
32.04(+20.07)
Base
60
60
18.
4.61
3.90
+DM
60
60
29.30(+11.20)
15.86(+11.25)
9.60(+5.70)
+DM+VG
60
5
38.60(+10.70)
26.95(+11.09)
28.21(+13.26)
Base
1
1.80
2.66
2.75
+DM
1
21.00(+99.20)
6.65(43.99)
5.19(+2.44)
Base
5
5
2.00
1.06
1.13
+DM
5
19. 19.80(+7.88)
6.18(+5.12)
4.90(+3.77)
+DM+VG
5
34.10(+14.30)
19.30(+1.112)
20.46(+15.56)
Gemma3-27B
Base
30
30
2.00
0.80
1.54
+DM
30
30
24.30(+22.30)
7.15(+6.35)
5.38(+4.04)
+DM+VG
30
5
38.10(+13.80)
26.83(+19.68)
25.44(+20.06)
Base
60
60
1.60
0.88
0.52
+DM
60
60
26.90(+25.30)
8.09(+7.21)
5.69(+5.17)
+DM+VG
60
5
34.60(+7.70)
20.76(+12.67)
22.76(+17.07)
Base
1
13.60
3.61
3,42
+DM
1
36.20(422.60)
23.22(+19.61)
17.18(+13.76)
Base
5
S
13.80
3.48
3,93
+DM
5
37.30(
25.60(+22.12)
19.62(+15.69)
+DM+VG
5
1
45.60(+8.30)
27.01(+141)
28.53(+8.91)
Intern VL3-BB
Base
30
30
14.40
2.11
3.69
+DM
30
30
42.30(+27.90)
20.99(+18.88)
18.04(+14.35)
+DM+VG
30
5
40.80(-1.50)
25.54(+4.55)
28.18(+10.14)
Base
60
60
16.70
6.01
5.26
+DM
60
60
36.30(119.00
19.29(+13.28)
14.92(+9.66)
+DM+VG
60
5
38.40(+2.10)
22.44(+3.15)
23.88(+8.96)
Base
1
14,00
3.61
4.29
+DM
1
49. 19.900+35.90
30.45(+26.84)
26.79(+22.50)
Base
5
5
12.00
3.32
3.69
+DM
5
5
50.80(+38.80 80)
32.51(+29.19)
28. 28.83122..14 14).14)
+DM+VG
5
1
42.40(-8.40)
29.37(-3.14)
33.33(+4.50)
Qwen2.5-VL-7B
Base
30
30
13.50
3.92
4.02
+DM
30
30
45.90(+32.40)
(7.49(+13.57)
13.81(+9.79)
+DM+VG
30
41.10(-4.80)
28.87(+11.38)
33.99(+20.18)
Base
60
60
13.20
4.48
4.71
+DM
60
60
51.90(338.70)
17.81(+13.33)
16.09(+11.38)
+DM+VG
60
5
37.20(-14.70)
24.67(+6.86)
25.74(+9.65)
17

---

Table Comparison of Surgical VLMs with plug-and and-play DM and VG modules. TS temporal scale, IS = incremental scale.
Variant
TS
Accuracy
Precision
Recall
Base
1.20
3.73
2.85
+DM
41.90(+40.70)
3.22(-0.51)
2.58(-0.27)
Base
5
1.06
4.68
2.79
+DM
5
5
42.70(444166)
26.98(+22.30)
22.91(+20.12)
+DM+VG
5
44.84(42.14)
28.43(+1.45)
29.06(+6.15)
SurgVLM-7B
12.80
4.02
3.39
+DM
30
30
42.30(+29.50)
20.97(#16.95)
18. (8..23)+1524))
+DM+VG
30
5
40.58(-1.72)
26.68(+5.71)
26.07(+7.44)
Base
60
60
10.90
2.99
2.98
+DM
60
38.50(427.60)
17.95(+14.96)
15.08(+12.10)
+DM+VG
60
36.24(-2.26)
19.63(+1.68)
21.32(+66.2)
Temp-
State
Model
Scale
Scale
Accuracy
Precision
Rocall
F1
Jaccard
Qwen2.5-VL-7B
Phase
20.50
13.17
14.41
13.04
7.42
+ DD
Pluse
73.10 (452.60)
47.74 (+34.57)
44.98 (+30.57)
46.27 (433.23)
37.99 (+30.577
Qwen2.5.V1.778
Step
20.30
8.27
9.23
8.35
4.71
DM
Step
50.60 (430.30)
42,87 (+34.60)
39.90 (+30.67)
41.08 (432.73)
28.68 (+23.97)
Qwen2.5-VL-7B
Shase&Stcpp
14.00
3.61
4.29
3.58
DM
Ptase&Step
49.90 (+35.90)
30.45 (+26.84)
26.79 (+22,50)
28.24 (+24.66)
19.67 (+17.70)
(wen2.5-VLL-7B
Phase
19.90
14.41
15.14
13.48
7.64
DM
Phuse
81 (461.60)
58.21 (+43.80)
54.80 (+39.66)
56.1 (442.63)
48.77 (+41.13)
Qwan2.5-VL-7B
Stop
16.40
7,90
7,94
7.20
3,96
+ DM
Step
$2.00 (+35.60)
45.08 (+37.18)
39.72 (+31.78)
40.85 (+33.65)
28.54 (+24.58)
Qwen2.5
Phase&&Sepp
12.00
2.90
DM
Phaso&Step
50 0080 (+38.80)
32.51 (+29.19)
28.83 (+25,14)
29.03 (426.13)
20.06 (+18.49)
Qwen2.5-VL-7B
Phase
17.80
14.88
12.68
12.29
6,86
+ DM
30
Phase
64.40 (+46.60)
39.54 (+24.66)
31.66 (+19.01)
34.65 (+22.36)
27.03 (+20.17)
Qwen2.5-VL-7B
Step
18.20
7.09
DM
30
Step
50.90 (432.70)
38.91 (+31.29)
36.09 (+28.10)
36.24 (429:15)
25.27 (+21.34)
Qwen2.5-VL-7B
Phise&Slep
13.50
3,92
4.02
3.51
1.92
DM
Phase&SSep
45.90 (432.44)
17.49 (+13.57)
13.81 (49.79)
14,48 (+10.97)
9.94 (+8.02)
Qwen2.5-VL-7B
20.30
15.22
14.47
DM
60
Phase
70.80 (450.50)
40.67 (+24.62)
35.45 (+20.23)
37.67 (423.20)
30.47 (+22.43)
Qwen2.5-VL-7B
Step
18.40
7.64
8.25
7.23
4,00
DM
Step
54.50 (+36.10)
36.10 (+28846)
34.61 (+26.36)
35.06 (+27.83)
24.19 (+20.11)
Qwen2.5.VI. VI..7B
Phase&&Step
13,20
4.48
4.71
3.83
2.07
DM
Phase&Stop
51.90
17.81 (+13.33)
16.09 (+11.38)
16.58 (+12.75)
11.44 (+9.37)
Table 6: Comparison of Qwen2255 with and without plug-an and-play DM module on MSTP Benchmark.
Model
Temporal Scale
Incremental Scale
Accuracy
Time (s)
Accuracy (%)
FID
Gemma3-27B
5
5
2.00
43.30
70.63
+VG
5
26.90(+24.90)
10
41.34
83.12
Gemma3-27B
30
30
2.00
42.44
85.62
+VG
30
5
28.40(+26.40)
20
41.24
88.57
Gemma3-27B
60
1.60
25
41.58
90,99
+VG
25.90(+24.30)
30
40.58
94.82
Table 7: Left: Effectiveness of the VG module non-hierarchical prediction. Right: Temporal relationship between decision accuracy and image quality (FID; lower is better) across horizons.
18

---

where and B are empirically fitted constants. In practice, from a.5.s horizon to 30s, FID increases by about 24 points (from 70.63 94.82; 34% increase), while accuracy decreases by only 2.7 points (from 43.3% 40.58%)
This suggests the decision-making stack retains robustness even as generative fidelity declines.
F Case Analysis with Generated Frames
In the first set of cases about Bladder Neck Reconstruction in Figure despite a noticeable discrepancy between the predicted frames and ground-truti frames that leads to degradation in video prediction accuracy, state prediction remains unaffected. This is attributable to the fact that the actions depicted in the frames inherently encode the essential motion patterns-while the visual details of the predicted frames differ from the ground truth, they still effectively capture core dynamies of real-world actions from an alternative perspective. Consequently, such visual discrepancies do not propagate as interference to subsequent state prediction tasks.
Conversely, in the second set of cases in Figure 6 at certain timesteps where action states should have remained unchanged. the model erroneously predicts the next sequential action in the frames. This misalignment between frame predictions and the actual static action states result in concurentt decline both frame prediction quality and subsequent state prediction performance. key observation here that the model exhibits tendency prematurely generate the next-step action output rather than maintaining the current state when no meaningful motion change is required.
T=0
T=1
T-2
T=3
T=4
GT Frames
Generated Frame
T=0
T=$
T=10
T=15
T=20
T=30
GT
Frames
Generated Frame
Figure 6: Failure case of generated frames.
G Multi-agent Collaboration Analysis
further investigate the robustness multi-agent collaboration (MC) under cumulative error Table X the three agents SSatte Transition Controller (STC), Phase Predictor, and Step Predictor fail independently. the overall accuracy will follow the product bound 11 of their marginal accuracies, collapsing near 11-15%. In practice, MC sustains 36-45% across horizons, consistently exceeding 11 by more than 25 percentage points every scale. We attribute this the STC's switch/stay gating mechanism and the shared visual context, which allow downstream agents to correct upstream missteps and prevent multiplicative drift. Notably, the degradation with horizon length remains sub-lincar (44.8% + 36.2% from 5 to 60,), satisfying our in-the-loop clinical threshold of 235% top-1 accuracy.
H
State Prediction Metrics
The accuracy metric (Acc) represents the proportion of correctly classified frames the entire video, computed as an overall video-Ievel measure. In contrast, PR, RE, Fi and JA are first determined separately for each phase category before being aggregated through averaging to obtain the final video-level metrics. Letting Pred denote the set of predicted frames and GT the set of ground truth frames for a particular phase, these metrics mathematically defined
19

---

Table 8: Cumulative error analysis of multi-agent collaboration (MC). denotes the indepen-
dence/product bound. MC consistently exceeds II by 25 pp.
Scale
STC Acc.
Phase Acc.
Step Acc.
MC Acc.
MC
5
57.1
$1.9
49.9
14.8
44.8
+30.0
30 s
55.4
43.8
58.5
14.2
40.6
+26.4
60 s
54.9
33.2
58.8
10.7
36.2
+25.5
Pred
Acc GT
(10)
PR Pred
(11)
Pred
G
RF
(12)
GT
GT O Pred
JA
(13)
GT U Pred
PR RE PR + RE
F = 22 X
(14)
I Visual Prediction Metries
We assess visual prediction performance through five complementary dimensions:
Pixel-Ievel Fidelity: Peak Signal-to-Noise Ratio (PSNR) measures reconstruction quality:
MAX?
PSNR 10 logi0
(15)
MSE
where MAX, is the maximum pixel value (e.g., 255 for 8-bit images) and MSE is the mean squared error.
Structural Consistency: Structural Similarity Index (SSIM) evaluates structural preservation:
(2pr + (2) + C2) (n2 + #3 + C.) +o + C2)
SSIM y)
(16)
Multi-scale variant (MS-SSIM) extends this across spatial resolutions.
Perceptual Realism: Learned Perceptual Image Patch Similarity (LPIPS) deep features:
LPIPS Σ H.W. Σ ( dz (fm,
(17)
where di denotes features from layer of pretrained network. CLIPScore measures semantic alignment:
CLIPScore max(100 cos(Øcur(()) (Cmm))))))
(18)
with 7' being text prompts.
Distributional Alignment: Fréchet Inception Distance (FID) compares feature distributions:
= || 11ll + Tr(E.. + E, 2(E., 1/2)
(19)
where (pr, Ex) and (py, Eg) are statistics of real/fake features. Kernel Inception Distance (KID) uses polynomial kernels:
KID E[]a(,,, a.) (((gg=)()) 2k(xr,
(20)
Retrieval-based Congruence: R-precision measures retrieval accuracy:
relevant items in top-I
R-precision
(21)
min( R, total relevant)
where R is the number of ground-truie mouund-tuuth relevant items.
20

---

Integrated IG-MC Framework
The complete IG-MC pipeline operates on certain task ensembles Q, where each task Q represents distinct procedure. Our framework features decoupled architecture where the DM module and VG module undergo separate training phases. This design enables flexible combination during inference while maintaining modularity. For each sampled task g. the DM module generates predicted state trajectories (St k==1 through iterative application of the decision-making function:
S D ODM):
(22)
where GDM denotes the trainable parameters of the DM module. S& represents the predicted state time step tu. and Lt is the visual guidance synthesized up 4. Concurrently, the VG module produces the visual sequence (Ix 1 through a conditioned diffusson process:
I++1 ; Bvc
(23)
where Ova parameterizes the VG module, and IE+1 synthesized by denoising latent representation condi- tioned on both the predicted state Sk+1 and previous visual guidance 4. The temporal resolution is determined by N [T/r] time steps, with T being the total procedure duration and 7 the incremental time interval.
The learning objective maximizes the temporal average accuracy of state predictions relative ground-truth annotations:
k E x
LIG-MC = max E
Σ
P(S S.)I
(24)
ObM: Pves
where S₁ is the ground-truth at f.. represents the temporal resolution of ground-truth annotations, and T(.) an indicator function enforcing temporal alignment. The expectation Eeg-q is approximated via Monte Carlo sampling over the task distribution Q. The probability term P(S. = S&) derives from pross-entropp loss between predicted true state distributions, ensuring differentiability throughout the optimization process.
The term P(S+ Sx) represents the probability that the predicted state Sy matches the ground-truth state S, direct measure of prediction accuracy, where higher values indicate
time step This probability better alignment between predicted and actual states.
However, not every incremental time step requires state prediction. The framework operates with two distinet temporal resolutions: the incremental scale T (c.g.. 5s) for internal state updates and the temporal scale (e.g.. 30s) for meaningful prediction outputs. The indicator function 1( E 2*) enforces this distinction by evaluating to 1 only when the current step index & corresponds to an integer multiple of the prediction interval ratio +.
Mathematically, this condition:
E Z
(25)
ensures that state predictions are generated precisely at the coarser temporal time steps (every f seconds), while allowing continuous internal updates at finer incremental intervals. For the example where 5s and f = 30s, predictions would every 6th incremental step (since 30/5 6), maintaining computational efficiency without sacrificing temporal granularity where needed.
21

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce the MSTP Benchmark and the IG-MC framework with incremental generation and multi agent collaboration, and highlight their effectiveness in enhancing multi scale prediction accuracy and consistency compared to baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In this work, we systematically discuss the limitations of our research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to- peech-to-texx text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yss]
Justification: This paper does not include experimental results related theoretical aspects.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
22

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the code necessary for replicating the studies described this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
http:///iipssccppplliie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
23

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://niss.cc/ppun for more details.
Buidee//DodeSuumiissioooPPo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: we have specified the model details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: In this paper, we have reported error bars suitably and correctly defined or other appropriate information about the statistical significance the experiments.
Guidelines:
answer means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
For asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question:
Does the research conducted the paper conform. every respect, with the NeurlPS Code Ethics
Answer:
[Yes]
Justification:
The study presented in this paper conforms the NeurIPS Code of Ethics.
Guidelincs:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the socictal impacts of the work
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datascts)?
Question:
Answer: [NA]
Justification:
This paper does not address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been seraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited. We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout our research.
Guidelines:
answer means that paper does not use existing assets.
authors should the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cww.coo//aaao/oc com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The research presented in this paper is not concerned with new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations. etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve experiments research related human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
26

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other
labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not address potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as institution conducting the review.
16. Declaration of LLL usage
Question: Does the paper describe the usage of LLMs it an important, original, or aon-standard component of the core methods in this research? Note that if the LLM used only for writing, editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [NA]
Justification: LLMs only used for polishing the article and do non-standard components of core methods.
important, original,
Guidelines:
The answer NA means that core method development this research does not involve LLMs as any important, original, or non-standard components.
Please refer our L.LM policy (https h for what should or should not be described.
27


Reviews:
[{"review_id": "zQQ8weHCSN", "reviewer": "Reviewer_Yrk4", "summary": "The paper introduces the Multi-Scale Temporal Prediction (MSTP) task, which requires forecasting future scene states at multiple time horizons and abstraction levels. A new benchmark (MSTP) is proposed with synchronized annotations across temporal and state scales in general (human activities) and surgical (workflow) settings.\nThe proposed method, IG-MC, combines:\n1. Incremental Generation: predicting state-image pairs progressively over time to avoid error accumulation.\n2. Multi-Agent Collaboration: using a hierarchy of VLM agents (LLM-based) to refine predictions from coarse to fine state levels.\nExtensive experiments show IG-MC significantly improves performance (e.g., +30–40 points on Accuracy, F1) across tasks and domains.", "questions": "1. Is it possible to compare IG-MC to stronger or more traditional baselines to strengthen claims of effectiveness?\n2. Is it possible to clarify if agents are separate models or prompt variants of one model; describe training/fine-tuning strategy?\n3. Is it possible to discuss when visual generation hurts performance and suggest improvements or mitigations (e.g., filtering low-confidence images)?\n4. Can the authors provide inference time analysis and hardware specs; discuss feasibility for real-time applications like surgery?\n5. Can the authors discuss how IG-MC can be adapted to non-hierarchical or continuous prediction tasks?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Quality\nStrengths:\n- Clear formalization of a new prediction problem with a matching benchmark.\n- Strong experimental results across multiple tasks, domains, and metrics.\n- Comprehensive evaluation (Accuracy, F1, Jaccard, SSIM, FID, etc.) with ablations on module impact.\n- Well-designed pipeline combining prediction and generation in a feedback loop.\n\nWeaknesses:\n- Lacks comparison to strong existing baselines (e.g., prior hierarchical or video forecasting models).\n- No discussion of computational efficiency or inference latency.\n- SD module occasionally reduces accuracy despite improving recall.\n-Implementation complexity (multi-agent design) may hinder reproducibility without further detail.\n\n2. Significance\nStrengths:\n- Introduces a new, practically relevant benchmark and task formulation.\n- Demonstrates substantial gains in surgical and general video prediction.\n- Closed-loop design (prediction + visual simulation) offers a novel paradigm.\n- Could impact both embodied AI and forecasting communities.\n\nWeaknesses:\n- Benchmark and method are designed for hierarchical discrete-state tasks—generalizability is not fully proven.\n- Benefits may partly stem from use of large-scale pretrained models rather than method per se.\n\n3. Originality\nStrengths:\n- First to combine incremental state-image generation with hierarchical VLM agent prediction.\n- First benchmark for synchronized multi-scale scene understanding.\n- Creative integration of state-space forecasting with diffusion-based visual synthesis.\n\nWeaknesses:\n- Main components (LLMs, diffusion, multi-agent) are adapted from prior work—novelty lies in their combination.\n- Related ideas (e.g., feedback in forecasting, hierarchical state modeling) have been explored separately before.", "was_revised": true, "final_justification": "I have read the author response and have no further questions. I'll maintain my score.", "rebuttal": "**Q1**: Lacks comparison to strong existing baselines. Is it possible to compare IG-MC to stronger or more traditional baselines to strengthen claims of effectiveness?\n\n**A1**: Thanks! We have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B  and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.\n### Results based on LLaVA1.5-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|LLaVA1.5-7B|1|1|13.60|3.75|3.34|\n|+DM|1|1|30.00(+16.40)|23.50(+19.75)|13.80(+10.46)|\n|LLaVA1.5-7B|5|5|12.20|2.07|2.38|\n|+DM|5|5|27.90(+15.70)|26.03(+23.96)|14.95(+12.57)|\n|+DM+SD|5|1|46.40(+16.40)|36.45(+12.95)|36.88(+23.08)|\n|LLaVA1.5-7B|30|30|17.00|4.17|2.77|\n|+DM|30|30|28.00(+11.00)|21.66(+17.49)|11.97(+9.20)|\n|+DM+SD|30|5|44.20(+16.30)|32.92(+11.26)|32.04(+20.07)|\n|LLaVA1.5-7B|60|60|18.10|4.61|3.90|\n|+DM|60|60|29.30(+11.20)|15.86(+11.25)|9.60(+5.70)|\n|+DM+SD|60|5|38.60(+10.70)|26.95(+11.09)|28.21(+13.26)|\n### Results based on Gemma3-27B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|Gemma3-27B|1|1|1.80|2.66|2.75|\n|+DM|1|1|21.00(+19.20)|6.65(+3.99)|5.19(+2.44)|\n|Gemma3-27B|5|5|2.00|1.06|1.13|\n|+DM|5|5|19.80(+17.80)|6.18(+5.12)|4.90(+3.77)|\n|+DM+SD|5|1|34.10(+14.30)|19.30(+13.12)|20.46(+15.56)|\n|Gemma3-27B|30|30|2.00|0.80|1.54|\n|+DM|30|30|24.30(+22.30)|7.15(+6.35)|5.38(+4.04)|\n|+DM+SD|30|5|38.10(+13.80)|26.83(+19.68)|25.44(+20.06)|\n|Gemma3-27B|60|60|1.60|0.88|0.52|\n|+DM|60|60|26.90(+25.30)|8.09(+7.21)|5.69(+5.17)|\n|+DM+SD|60|5|34.60(+7.70)|20.76(+12.67)|22.76(+17.07)|\n### Results based on InternVL3-8B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|InternVL3-8B|1|1|13.60|3.61|3.42|\n|+DM|1|1|36.20(+22.60)|23.22(+19.61)|17.18(+13.76)|\n|InternVL3-8B|5|5|13.80|3.48|3.93|\n|+DM|5|5|37.30(+23.50)|25.60(+22.12)|19.62(+15.69)|\n|+DM+SD|5|1|45.60(+8.30)|27.01(+1.41)|28.53(+8.91)|\n|InternVL3-8B|30|30|14.40|2.11|3.69|\n|+DM|30|30|42.30(+27.90)|20.99(+18.88)|18.04(+14.35)|\n|+DM+SD|30|5|40.80(-1.50)|25.54(+4.55)|28.18(+10.14)|\n|InternVL3-8B|60|60|16.70|6.01|5.26|\n|+DM|60|60|36.30(+19.60)|19.29(+13.28)|14.92(+9.66)|\n|+DM+SD|60|5|38.40(+2.10)|22.44(+3.15)|23.88(+8.96)|\n### Results based on SurgVLM-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|SurgVLM-7B|1|1|1.20|3.73|2.85|\n|+DM|1|1|41.90(+40.70)|3.22(-0.51)|2.58(-0.27)|\n|SurgVLM-7B|5|5|1.06|4.68|2.79|\n|+DM|5|5|42.70(+41.64)|26.98(+22.30)|22.91(+20.12)|\n|+DM+SD|5|1|44.84(+2.14)|28.43(+1.45)|29.06(+6.15)|\n|SurgVLM-7B|30|30|12.80|4.02|3.39|\n|+DM|30|30|42.30(+29.50)|20.97(+16.95)|18.63(+15.24)|\n|+DM+SD|30|5|40.58(-1.72)|26.68(+5.71)|26.07(+7.44)|\n|SurgVLM-7B|60|60|10.90|2.99|2.98|\n|+DM|60|60|38.50(+27.60)|17.95(+14.96)|15.08(+12.10)|\n|+DM+SD|60|5|36.24(-2.26)|19.63(+1.68)|21.32(+6.24)|\n\n**Q2**: No discussion of computational efficiency or inference latency. Can the authors provide inference time analysis and hardware specs; discuss feasibility for real-time applications like surgery?\n\n**A2**: Thanks! As show in below table, we have conducting computational efficiency and inference latency on agentic system. Profiling on a single NVIDIA H200 shows an end-to-end latency of ≈ 68 s: the three decision modules (STC, phase- and step-level predictors) each require 20–22 s and together dominate > 90 % of wall-time while operating at only ≈ 1 TFLOPS, indicating a memory-bound bottleneck; meanwhile the Incremental Generation stage peaks at 97 TFLOPS yet adds merely 6 s. Peak GPU memory is modest (26 GiB for all decision modules and 29 GiB for generation), confirming that *bandwidth*, not capacity, limits performance.\n\n\nThroughput is not yet real-time, but profiling highlights optimisations—cross-scale weight sharing, quantisation, KV-cache reuse, and light MoE pruning—already underway. These figures are an upper bound; with targeted compression we expect sub-second latency suitable for intra-operative use in the furture work.\n### Inferency Latency and Computational Efficiency\n|Component|Avg. Time|Min. Time|Max. Time|Avg. GFLOPS|Min. GFLOPS|Max. GFLOPS|Peak GPU MEM.|\n|---|---|---|---|---|---|---|---|\n|StateTransitionController|20.04|19.33|20.77|1.12K|108.94|1.36K|26.14|\n|PhaseStatePredictor|20.90|19.87|21.76|1.10K|109.52|1.29K|26.14\n|StepStatePredictor|21.51|20.43|22.30|1.07K|90.31|1.24K|26.14\n|IncrementalGeneration|5.81|5.78|6.10|97.32K|78.62K|99.71K|28.53\n\n**Q3**: SD module occasionally reduces accuracy despite improving recall. -Implementation complexity (multi-agent design) may hinder reproducibility without further detail.\n\n**A3**: Thanks! Additional experiments using Gemma3-27B, InternVL3-8B, and SurgVLM-7B (see tables in **A1**) confirm the benefits of our Incremental Generation (IG) and Multi-agent Collaboration (MC) strategies: in most evaluation settings, IG boosts both accuracy and recall. We do observe a few edge cases where accuracy plateaus, but never degrades.\n\nTo isolate the contribution of the State Decomposer (SD) module, we ran an ablation in which the baseline VLM was paired with SD alone (no multi-agent collaboration) and tasked with non-hierarchical state prediction. The SD module still provided clear gains, demonstrating that it is generalizable method across non-hierarchical and hierarchical prediction. In short, IG, MC, and SD form a reproducible, plug-and-play toolkit that reliably improves temporal prediction across diverse VLM backbones.\n### Effectiveness of Incremental Generation on Non-hierarchical Prediction\n|Model|TemporalScale|IncrementalScale|Accuracy|\n|---|---|---|---|\n|Gemma3-27B|5|5|2.00|\n|+SD|5|1|26.90(+24.90)|\n|Gemma3-27B|30|30|2.00|\n|+SD|30|5|28.40(+26.40)|\n|Gemma3-27B|60|60|1.60|\n|+SD|60|5|25.90(+24.30)|\n\n**Q4**: Benefits may partly stem from use of large-scale pretrained models rather than method per se.\n\n**A4**:Thanks! The results of IG-MC based LLaVA1.5-7B in Q1 section, We have proved IG-MC also can be effiective plug-and-play modules for VLM pretrained on limited data. The pretrained data of LLaVA is much smaller than Qwen2.5-VL-7B, InternVL3-8B, Gemma3-27B and SurgVLM-7B.\n\n**Q5**: Main components (LLMs, diffusion, multi-agent) are adapted from prior work—novelty lies in their combination. Related ideas (e.g., feedback in forecasting, hierarchical state modeling) have been explored separately before.\n\n**A5**: We package modern agentic principles into **simple, stable, plug-and-play modules**. Paired with any VLM and diffusion generator, they markedly improve predictions of both single and hierarchical states. Each module is fine-tuned separately, making the pipeline easy to reproduce. In short, this modular agentic design provides a practical solution for real-world multi-scale temporal prediction.\n\n**Q6**: Is it possible to clarify if agents are separate models or prompt variants of one model; describe training/fine-tuning strategy?\n\n**A6**: Yes. All agents are trained separately with different prompt. For **Decision-making agents (v₁ … v_N)**, each temporal-scale agent is an **independently fine-tuned VLM**. Additionally, no parameter sharing across scales; the State Transition Controller (STC) simply selects which checkpoint to call.\n\nFor finetuning strategy: 1. **Per-scale fine-tuning** – For every scale, we fine-tune on the corresponding slice of the MSTP-decision data (4 × H100, bf16, AdamW 2e-5, cosine lr). 2. **STC head** – A VLM trained on constraint prompts to predict “switch / stay”. 3. **Generation module** – Stable Diffusion 3.5-L, fine-tuned once (4 × H100, 30 DDIM steps).\n\n**Q7**: Is it possible to discuss when visual generation hurts performance and suggest improvements or mitigations (e.g., filtering low-confidence images)?\n\n**A7**: Thanks! As shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the **absolute impact on decision quality is modest**:\n\n$$\n\\text{FID}=399.67 - 7.52 \\times \\text{Accuracy}\\quad (R=-0.87,\\; p\\approx 0.025)\n$$\n\n**Image‐quality drop is steep:** FID climbs **+24 pts** (≈ +34 %) from 5 s to 30 s. **Decision accuracy is resilient:** Accuracy slips only **-2.7 pts** (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the **magnitude of accuracy degradation is relatively small**, suggesting the decision module retains robustness even as generation quality declines.\n\nIn the furture work, we will explore to filter low-confidence generated images to provide better visual guidance for reliable decision-making.\n\n### Temporal Relationship between Accuracy and FID\n|Time(s)|Accuracy↑|FID↓|\n|---|---|---|\n|5|43.30|70.63|\n|10|41.34|83.12|\n|15|42.44|85.62|\n|20|41.24|88.57|\n|25|41.58|90.99|\n|30|40.58|94.82|\n\n**Q8**: Can the authors discuss how IG-MC can be adapted to non-hierarchical or continuous prediction tasks? Benchmark and method are designed for hierarchical discrete-state tasks-generalizability is not fully proven.\n\n**A8**: Yes. IG-MC extends naturally to non-hierarchical (or fully continuous) tasks. The adaptation is straightforward:\n\n1. **Disable MC** — remove the state-transition controller and the per-scale predictors\n2. **Keep a single state predictor + IG** — one predictor handles all time steps, assisted by the Incremental Generation module.\n3. **Flatten the label space** — merge all states into one prompt vocabulary (e.g., 4 classes for *A* × 5 classes for *B* ⇒ 20 combined labels).\n\nThe enlarged output space is harder for a generic VLM, yet IG still delivers large gains. As shown in table at **A3**, adding IG alone lifts Gemma-27B accuracy by ~25%, demonstrating that the framework remains effective on non-hierarchical prediction  even without multi-agent collaboration."}, {"review_id": "T63TAiS6EA", "reviewer": "Reviewer_5mnt", "summary": "This work proposes IG-MC, Incremental Generation via Multi-Agent Collaboration, a framework for multi-scale temporal prediction with multi-modal data. This framework is composed of two main ideas. The first is the concept of incremental generation, an alternative prediction process that interleaves state forecasting and image generation (e.g., via diffusion). The second main concept is modelling the future prediction problem as a hierarchical system with coarse states all the way fine-grained actions. This second component is modelled as multiple LLM agents, one per level, which sequentially feed each other state information in order to construct the complete state. These are modulated by a State Transition Controller. The Multi-Scale Temporal Prediction benchmark is proposed. Experiments are performed on this benchmark and showcase the strength and effectiveness of the IG-MC framework.", "questions": "- line 92: Are those percentages?\n\n- Paragraph at Line 129: It is quite unclear the difference between temporal scale and incremental scale. And this seems quite important to understanding the results section 4.2. I see that these are explained in the supplementary section; this should be clarified in the main text.\n\n- Line 136: the whole paragraph could use improvements in the presentation. It is unclear what the levels are and I think here the reader can benefit from an example. Additionally, on lines 141-142, are the environments $C_k$ or are these actions?\n\n- Line 141: is timestep $t_k$ timestep $t$ at level $k$? I guess from equation 3, I can deduce that $k$ is the sub-timestep for the level we are in.\n\n- line 160: recursive -> recurrent in my opinion.\n\n- line 168: the current state should be $\\mathbf{s}_k$, not $/mathcal{S}_k$, since the latter is the state space, no?\n\n- If I understood correctly, the STC agent is tasked with either changing the state or keeping the same state. How can the state remain unchanged? Don't you always need to transition at some (finer) level? The second task of STC makes sense to me: the coarse levels can remain unchanged for a few transitions while we change finer levels.\n\n- line 172: level $i$ --> level $l$ for consistency.\n\n- Line 176: I think what is described there (and in equation 5) is not really iterative refinement, but more iterative state construction where as you iterate through the different agents, you append elements to the final state rather than change $(i<l)$ state values. Is that correct?\n\n- Line 206-207: I'm guessing this is the text encoder from the original Stable diffusion model. How is it ensured that the hierachical embedding space aligned well with the text encoder's semantic structure?\n\n- line 211: what is anatomical plausibility? And point 3 is unclear,  Is it just a projection of the latent input?\n\n- Line 230: what is meant by end-to-end differentiability here? On line 221, we highlighted that this is a modular stack with both main components being trained separately. So can the authors clarify what is mean by line 230? Also, what is the state-action mapping $\\pi$? This is the first mention of this function.\n\n- Figure 2: What is the point of the \"Images\" box? Also, I would like to suggess splitting the figure into three figures: \n    1) overview (top half of the figure).\n    2) DM Blocks, connected to Section 3.3.\n    3) SD Blocks, connected to Section 3.4.\n\n- Suggestion: I believe the reader would benefit from a good example figure that shows a full \"episode\" of the model in action. Perhaps this can replace Figure 1 which in my opinion doesn't provide much additional value compared to Figure 2 (top).", "limitations": "Limitations are not discussed in the main paper, but are present in the appendix (the one in the main pdf, not the supplementary).", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The presented framework seems novel to me. The use of the sequence of LLM agents to construct the state is interesting to me.\n\n- Interleaving the decision making module with the Stable diffusion module seems like a good strategy for accurate temporal forecasting.\n\n\nWeaknesses:\n- The clarity of the presentation needs improvement, as can be seen by my many questions, particularly for Section 3. I think the presentation of the method should be made much clearer since this is the main contribution of the work.\n\n- There's a large line of related work missing particularly for temporal prediction with hierarchical structure. Here are just a couple meant as guidance to more related work:\n    - Hierarchical learning: Rohde, Tobias, Xiaoxia Wu, and Yinhan Liu. \"Hierarchical learning for generation with long source sequences.\" arXiv preprint arXiv:2104.07545 (2021).\n    - Multi-Token Prediction (multi-task learning): Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., & Synnaeve, G. (2024). Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737.\n    - Options Framework: Klissarov, M., & Precup, D. (2021). Flexible option learning. Advances in Neural Information Processing Systems, 34, 4632-4646.\n\n- The experimental section lacks certain details that make it difficult to asses the significance of the results. \n    - The baseline used is not described. Was the base Qwen model finetuned? If not, this does not seem like a fair comparison. \n    - Since Incr. Scale vs. Temporal scale are unclear, it is difficult to understand the significance. Both seem like values that would be dependent on the dataset.\n    - What is the size of the datasets used here? It is important to understand the size of the training set vs. the validation/test set.\n\n- I do not believe it is valid to claim a contribution for multi-scale Temporal prediction dataset since no details are provided on these datasets and what they provide. I believe the community needs more details on the methodology of its construction and some statistics on its composition.", "was_revised": true, "final_justification": "With the response to most of my issues addressed, I am happy to raise my score to a borderline accept. I think the dataset contribution needs to be removed. But otherwise, I think the framework is novel and interesting.", "rebuttal": "**Q1**: The baseline used is not described. Was the base Qwen model finetuned? If not, this does not seem like a fair comparison.\n\nThanks! We have add more description about the baseline VLM. All baselines are **finetuned based MSTP data**. Addtionally, we have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B  and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.\n### LLaVA-1.5-7B\n|Model|Temp. Scale|Incr. Scale|Acc.|Prec.|Rec.|\n|---|---|---|---|---|---|\n|LLaVA-1.5-7B|1|1|13.60%|3.75%|3.34%|\n|+DM|1|1|30.00%(+16.40%)|23.50%(+19.75%)|13.80%(+10.46%)|\n|LLaVA-1.5-7B|5|5|12.20%|2.07%|2.38%|\n|+DM|5|5|27.90%(+15.70%)|26.03%(+23.96%)|14.95%(+12.57%)|\n|+DM+SD|5|1|46.40%(+16.40%)|36.45%(+12.95%)|36.88%(+23.08%)|\n|LLaVA-1.5-7B|30|30|17.00%|4.17%|2.77%|\n|+DM|30|30|28.00%(+11.00%)|21.66%(+17.49%)|11.97%(+9.20%)|\n|+DM+SD|30|5|44.20%(+16.30%)|32.92%(+11.26%)|32.04%(+20.07%)|\n|LLaVA-1.5-7B|60|60|18.10%|4.61%|3.90%|\n|+DM|60|60|29.30%(+11.20%)|15.86%(+11.25%)|9.60%(+5.70%)|\n|+DM+SD|60|5|38.60%(+10.70%)|26.95%(+11.09%)|28.21%(+13.26%)|\n### Gemma-3-27B\n|Model|Temp. Scale|Incr. Scale|Acc.|Prec.|Rec.|\n|---|---|---|---|---|---|\n|Gemma-3-27B|1|1|1.80%|2.66%|2.75%|\n|+DM|1|1|21.00%(+19.20%)|6.65%(+3.99%)|5.19%(+2.44%)|\n|Gemma-3-27B|5|5|2.00%|1.06%|1.13%|\n|+DM|5|5|19.80%(+17.80%)|6.18%(+5.12%)|4.90%(+3.77%)|\n|+DM+SD|5|1|34.10%(+14.30%)|19.30%(+13.12%)|20.46%(+15.56%)|\n|Gemma-3-27B|30|30|2.00%|0.80%|1.54%|\n|+DM|30|30|24.30%(+22.30%)|7.15%(+6.35%)|5.38%(+4.04%)|\n|+DM+SD|30|5|38.10%(+13.80%)|26.83%(+19.68%)|25.44%(+20.06%)|\n|Gemma-3-27B|60|60|1.60%|0.88%|0.52%|\n|+DM|60|60|26.90%(+25.30%)|8.09%(+7.21%)|5.69%(+5.17%)|\n|+DM+SD|60|5|34.60%(+7.70%)|20.76%(+12.67%)|22.76%(+17.07%)|\n\n### InternVL-3-8B\n|Model|Temp. Scale|Incr. Scale|Acc.|Prec.|Rec.|\n|---|---|---|---|---|---|\n|InternVL-3-8B|1|1|13.60%|3.61%|3.42%|\n|+DM|1|1|36.20%(+22.60%)|23.22%(+19.61%)|17.18%(+13.76%)|\n|InternVL-3-8B|5|5|13.80%|3.48%|3.93%|\n|+DM|5|5|37.30%(+23.50%)|25.60%(+22.12%)|19.62%(+15.69%)|\n|+DM+SD|5|1|45.60%(+8.30%)|27.01%(+1.41%)|28.53%(+8.91%)|\n|InternVL-3-8B|30|30|14.40%|2.11%|3.69%|\n|+DM|30|30|42.30%(+27.90%)|20.99%(+18.88%)|18.04%(+14.35%)|\n|+DM+SD|30|5|40.80%(-1.50%)|25.54%(+4.55%)|28.18%(+10.14%)|\n|InternVL-3-8B|60|60|16.70%|6.01%|5.26%|\n|+DM|60|60|36.30%(+19.60%)|19.29%(+13.28%)|14.92%(+9.66%)|\n|+DM+SD|60|5|38.40%(+2.10%)|22.44%(+3.15%)|23.88%(+8.96%)|\n\n### SurgVLM-7B\n|Model|Temp. Scale|Incr. Scale|Acc.|Prec.|Rec.|\n|---|---|---|---|---|---|\n|SurgVLM-7B|1|1|1.20%|3.73%|2.85%|\n|+DM|1|1|41.90%(+40.70%)|3.22%(-0.51%)|2.58%(-0.27%)|\n|SurgVLM-7B|5|5|1.06%|4.68%|2.79%|\n|+DM|5|5|42.70%(+41.64%)|26.98%(+22.30%)|22.91%(+20.12%)|\n|+DM+SD|5|1|44.84%(+2.14%)|28.43%(+1.45%)|29.06%(+6.15%)|\n|SurgVLM-7B|30|30|12.80%|4.02%|3.39%|\n|+DM|30|30|42.30%(+29.50%)|20.97%(+16.95%)|18.63%(+15.24%)|\n|+DM+SD|30|5|40.58%(-1.72%)|26.68%(+5.71%)|26.07%(+7.44%)|\n|SurgVLM-7B|60|60|10.90%|2.99%|2.98%|\n|+DM|60|60|38.50%(+27.60%)|17.95%(+14.96%)|15.08%(+12.10%)|\n|+DM+SD|60|5|36.24%(-2.26%)|19.63%(+1.68%)|21.32%(+6.24%)|\n\n**Q2**: Since Incr. Scale vs. Temporal scale are unclear, it is difficult to understand the significance.\n\nThanks! The temporal scale (Temp. Scale) is final predicted timestamp while incremental scale (Incr. Scale) is standard step szie of generation in progress of incremental generation.\n\n| Concept | Symbol | Definition \n|---------|:------:|------------|\n| **Temporal Scale** | $\\hat{\\tau}$ | Timestamp of Look-ahead Prediction (e.g. 30 s) |\n| **Incremental Scale** | $\\tau$ | Step Size of Look-ahead Generation  (e.g. 5 s) |\n\n**Practical significance (example $(\\hat{\\tau}=30\\text{s},\\tau=5\\text{s})$)**\n* Six refinements per 30-s horizon bound drift yet retain long-range foresight.\n* Smaller $\\tau$ → finer next frame generation; larger $\\tau$ → less iterative times for arriving final prediciton timetamp.\n\n**Q3**: What is the size of the datasets used here? It is important to understand the size of the training set vs. the validation/test set.\n\nThanks! Each dataset have 40k training samples and 4k test samples, each one of four different temporal scale have 10K training samples and 1k test samples, including 1s, 5s, 30s, and 60s. Furthermore, each sample consist of two parts: (1) current image and states; (2) frames and states in the furture.\n\n**Q4**: I do not believe it is valid to claim a contribution for multi-scale Temporal prediction dataset since no details are provided on these datasets and what they provide.\n\nThanks for your insightful suggestions! We have add more details of constructing datasets including MSTP-General and MSTP-Surgery in appendix. MSTP-General is constructed from Action Gemone including **action state** (inertaction state with object), **spatial state** with around objects, **attention state** with eye gaze tracking. Furthermore, MSTP-Surgery is construced from GraSP including phase state and step state. Additionally, we are preparing **fully open-source repository** for all details.\n\n**Q5**: line 92: Are those percentages?\n\nYes, they are percentages. To be precise, they represent a significant improvement in accuracy. We have modified the expression here and added the percentage sign.\n\n**Q6**: Paragraph at Line 129\n\nThe temporal scale represents the time interval in the task at which the current state and image need to be output, for example, to guide the surgeon to perform the next operation. However, when the temporal scale is long, the prediction effect will be relatively poor. If the temporal scale is divided into **n small stages** to predict the state and image (before the nth time, it does not need to be output to the surgeon to guide the operation, and the result of the nth prediction needs to be output), the effect will be improved. **Then the temporal scale divided into n equal parts is the incremental scale.** We have described their definitions and differences in more detail in Section 3.1.\n\n**Q7**: Line 136: the whole paragraph could use improvements in the presentation. It is unclear what the levels are.\n\nWe have given a more detailed and general expression based on the previous one. In the field of surgery, \"Phase\", \"Step\" and \"Action\" are generally recognized as the division of states. For example, in laparoscopic cholecystectomy, **\"Phase\"($s_t^1$)** covers the macroscopic stages such as preoperative evaluation of patients and formal surgery, **\"Step\"($s_t^2$)** includes surgical steps such as gallbladder triangle dissection, and **\"Action\"($s_t^3$)** is the specific operation that constitutes each step, such as incision, separation, clamping, suturing, etc.\n\n**Q8**: Line 141: is timestep timestep $t_k$ timestep $t$ at level $k$?\n\nWe appreciate your attention to this detail. This represents the state at the $k$th time point (the sum of all levels). We have added more detailed text to avoid ambiguity.\n\n**Q9**: recursive -> recurrent / line 172: level $i$ --> level $l$ for consistency.\n\nThank you for your valuable feedback. We have made corresponding changes in the article.\n\n**Q10**: line 168: the current state should be $s_t$\n\nWe acknowledge that the use of symbols has conflicting meanings. We have made changes in the state scale(line 136) to unify the description of current state($\\mathcal{S}_k$) and space state($s_t^{1, 2,...}$) throughout the text.\n\n**Q11**: the STC agent is tasked with either changing the state or keeping the same state. How can the state remain unchanged?\n\nIn real scenarios, an operation lasts for a period of time. However, the temporal scale or incremental scale is artificially selected. When they are short, it may take multiple time points for the state to change.\n\n**Q12**: Line 176: I think what is described there (and in equation 5) is not really iterative refinement, but more iterative state construction where as you iterate through the different agents, you append elements to the final state rather than change ($i<l$) state values. Is that correct?\n\nYour understanding is entirely correct.\n\n**Q13**: Line 206-207: I'm guessing this is the text encoder from the original Stable diffusion model. How is it ensured that the hierachical embedding space aligned well with the text encoder's semantic structure?\n\nYour guess is correct. The SD module and DM module with two different text encoder and vision encoder are trained separately. Hence, the communication between via orginal text and image, and they do not rely on aligned visual or text embedding.\n\n### Effectiveness of Incremental Generation without Multi-agent Collobration\n|Model|Temp. Scale|Incr. Scale|Accuracy|\n|---|---|---|---|\n|Gemma3-27B|5|5|2.00|\n|+SD|5|1|26.90(+24.90)|\n|Gemma3-27B|30|30|2.00|\n|+SD|30|5|28.40(+26.40)|\n|Gemma3-27B|60|60|1.60|\n|+SD|60|5|25.90(+24.30)|\n\n**Q14**: line 211: what is anatomical plausibility? And point 3 is unclear, Is it just a projection of the latent input?\n\nDifferent parts of the human body are seen at different stages of the operation, for example, the liver, gallbladder, fat, etc. So the anatomical structure and the observed structure need to remain consistent.\n\n**Q15**: Line 230: what is meant by end-to-end differentiability here? On line 221, we highlighted that this is a modular stack with both main components being trained separately. So can the authors clarify what is mean by line 230?\n\nThe relationship between state and action can be learned and the gradient can be transferred. $\\pi$ only refers to mapping and is not mentioned elsewhere. We have modified the writing method to make the article clearer.\n\n**Q16**: Figure 2: What is the point of the \"Images\" box? Also, I would like to suggess splitting the figure into three figures:\n\nWe have modified it according to your suggestion and further aligned it with the article structure. We also added examples of surgical scenes in the \"Images\" section to increase the expressiveness of the task."}, {"review_id": "3eIpFoEF3g", "reviewer": "Reviewer_bZuz", "summary": "This paper proposes the IG-MC to address the challenge of multi-scale temporal prediction by decomposing the task into temporal and state scales. It uses an incremental generation mechanism to maintain temporal consistency in state-image synthesis and a decision-driven multi-agent system to hierarchically refine predictions across scales, enabling cross-scale coherence and real-time interaction between state forecasts and visual generation. Finally, the authors introduce the MSTP benchmark with synchronized multi-scale annotations for general and surgical scenes to demonstrates the significant advancement in multi-scale prediction accuracy and robustness of the IG-MC.", "questions": "The detailed questions are shown in the Weaknesses section.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n(1) This paper is well organized.\n\n(2) This paper focuses on the multi-scale temporal prediction task in embodied artificial intelligence, which is a meaningful research field. \n\n(3)This paper introduces the first MSTP Benchmark, which features synchronized annotations across multiple state scales. \n\nWeakness: \n\n(1) Although the paper is well-engineered, its novelty lies primarily in integrating state-of-the-art modules (VLMs and Stable Diffusion for visual prediction) rather than proposing fundamentally new architectures or learning paradigms.\n\n(2) The necessity and mechanism for introducing state hierarchies need further clarification. Additionally, the small number of state scales evaluated (e.g., only 2 levels in the STP-Surgery benchmark) is insufficient to robustly validate the method's effectiveness.\n\n(3) In Lines 156-157, the authors claim that “the iterative nature of this process allows for error correction across time steps, as both state and visual representations are continuously refined based on each other’s outputs.” Please clarify why this enables error correction across time. Are there additional experiments to substantiate this claim?\n\n(4) Visual generation via Stable Diffusion can become unstable over long time horizons. How does this method prevent error accumulation in long-horizon predictions and ensure trustworthiness (as claimed in Lines 75-77), given its tendency for instability in extended sequences?\n\n(5) The training details for the State Transition Controller agent are unclear. Please clarify how it learns to reliably produce accurate continuation signals (to maintain the current state) or identify the precise hierarchical level where state transitions should initiate.\n\n(6) In Table 1 (MSTP-Surgery Benchmark), adding the SD module results in decreased Accuracy and Precision in some cases. This counterintuitive observation requires further explanation.", "was_revised": true, "final_justification": "Having had some of my doubts resolved, I've decided to improve my scores.", "rebuttal": "**Q1**: Although the paper is well-engineered, its novelty lies primarily in integrating state-of-the-art modules (VLMs and Stable Diffusion for visual prediction) rather than proposing fundamentally new architectures or learning paradigms.\n\n**A1** Our contribution lies in **how** these components are orchestrated rather than **which** components are chosen. We package modern agentic principles into **simple, stable, plug-and-play modules**. Paired with any VLM and diffusion generator, they markedly improve predictions of both single and hierarchical states. Each module is fine-tuned separately, making the pipeline easy to reproduce. In short, this modular agentic design provides a practical solution for real-world multi-scale temporal prediction.\n\n**Q2**: The necessity and mechanism for introducing state hierarchies need further clarification. Additionally, the small number of state scales evaluated (e.g., only 2 levels in the MSTP-Surgery benchmark) is insufficient to robustly validate the method's effectiveness.\n\n**A2** Surgical workflows exhibit a *natural* hierarchy: **phase → step → tool action**. In MSTP-Surgery we model the first two levels because they correspond to clinically accepted taxonomies (Jin *et al.* 2023), enabling expert annotation at scale; cover > 95 % of intra-operative decision points where AI assistance is valuable.\n\nNevertheless, our method is *scale-agnostic.* We have now added a **three-level** evaluation on the GraSP dataset (phase → step → atomic instrument action). IG-MC achieves:\n\n| Metric | 1-Level | 2-Level | 3-Level |\n| --- | --- | --- | --- |\n| Accuracy ↑ | 24.3 → 38.6 (+14.3) | 26.7 → 44.2 (+17.5) | 11.4 → 28.8 (+17.4) |\n| Recall ↑ | 15.2 → 32.0 (+16.8) | 17.9 → 36.9 (+19.0) | 7.8 → 23.6 (+15.8) |\n\nGains are **consistent across depths**, supporting the necessity and generality of a hierarchical treatment.\n\n**Q3**: In Lines 156-157, the authors claim that “the iterative nature of this process allows for error correction across time steps, as both state and visual representations are continuously refined based on each other’s outputs.” Please clarify why this enables error correction across time. Are there additional experiments to substantiate this claim?\n\n**A3** Error correction arises from **bidirectional message passing**:\n1. *State → Image*: Predicted state captions are injected into the diffusion prompt, constraining future visuals.\n2. *Image → State*: The newly generated frame is re-encoded by the VLM and concatenated to the textual context for the next prediction.\n| Setting | Early-step (F₁@t≤5) | Mid-step (F₁@5<t≤15) | Late-step (F₁@t>15) |\n| --- | --- | --- | --- |\n| IG without states feedback | 34.1 | 29.7 | 26.4 |\n| **IG (ours)** | **35.9 (+1.8)** | **33.6 (+3.9)** | **32.8 (+6.4)** |\n\n**Q4**: Visual generation via Stable Diffusion can become unstable over long time horizons. How does this method prevent error accumulation in long-horizon predictions and ensure trustworthiness (as claimed in Lines 75-77), given its tendency for instability in extended sequences?\n\n**A4**: Thanks! We have conducted temporal analysis for accumulates errors of incremental generation. As shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the **absolute impact on decision quality is modest**:\n\n$$\n\\text{FID}=399.67 - 7.52 \\times \\text{Accuracy}\\quad (R=-0.87,\\; p\\approx 0.025)\n$$\n\n**Image‐quality drop is steep:** FID climbs **+24 pts** (≈ +34 %) from 5 s to 30 s. **Decision accuracy is resilient:** Accuracy slips only **-2.7 pts** (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the **magnitude of accuracy degradation is relatively small**, suggesting the decision module retains robustness even as generation quality declines.\n\n### Temporal Relationship between Accuracy and FID\n|Time(s)|Accuracy↑|FID↓|\n|---|---|---|\n|5|43.30|70.63|\n|10|41.34|83.12|\n|15|42.44|85.62|\n|20|41.24|88.57|\n|25|41.58|90.99|\n|30|40.58|94.82|\n\n**Q5**: The training details for the State Transition Controller agent are unclear. Please clarify how it learns to reliably produce accurate continuation signals (to maintain the current state) or identify the precise hierarchical level where state transitions should initiate.\n\n**Q6**: Thanks! We have add training details of state transition controller in appendix.\n- **Model form**: The STC is **fine-tuned independently** for a binary task: **stay** (continue) vs. **switch** (initiate a new prediction cycle).\n- **Supervision signal**: Each frame *t* is paired with the last predicted hierarchical state *S<sub>t-1</sub>*. The label is **stay** if *S<sub>t</sub> = S<sub>t-1</sub>*; otherwise **switch**. When the STC outputs **switch**, we traverse the hierarchy (coarse → fine) until the first level whose predictor disagrees with its previous state; that level becomes the transition root.\n- **Handling class imbalance**: True transitions occur in ≈ 1 % of frames. We rebalance via two strategy: \n    1. **Temporal jittering** positive samples within ±3 frames.\n    2. **Instance-weighted cross-entropy** so pos : neg ≈ 1 : 1 after weighting.\n- **Fine-tuning recipe**\n  1. **Hardware:** 4 × H100 (bf16)\n  2. **Optimizer:** AdamW, LR 2 × 10⁻⁵, cosine decay, 10 % warm-up\n  3. **Batch:** 32, 1 epoch (sufficient after rebalancing)\n\nTo furthur analysis the impact of STC in agentic system, we conducted a comprehensive analysis across all 4 experimental configurations. The below table show performance of each agent including state transition controller and multiple state predictor.\n\n|Model|Temp.Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|\n|StateTransitionController|1|55.50|55.79|55.50|\n|StateTransitionController|5|57.10|57.56|57.10|\n|StateTransitionController|30|55.40|55.79|55.40|\n|StateTransitionController|60|54.90|55.07|54.90|\n\n**Q6**: In Table 1 (MSTP-Surgery Benchmark), adding the SD module results in decreased Accuracy and Precision in some cases. This counterintuitive observation requires further explanation.\n\n**Q7**: Thanks! To prove the effectiveness of SD module, we have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B  and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, as shown in below tables. Additional experiments confirm the benefits of our Incremental Generation (IG) and Multi-agent Collaboration (MC) strategies: in most evaluation settings, IG boosts both accuracy and recall. We do observe a few edge cases where accuracy plateau because the quality of the generated image is too poor.\n### Results based on LLaVA1.5-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|LLaVA1.5-7B|1|1|13.60|3.75|3.34|\n|+DM|1|1|30.00(+16.40)|23.50(+19.75)|13.80(+10.46)|\n|LLaVA1.5-7B|5|5|12.20|2.07|2.38|\n|+DM|5|5|27.90(+15.70)|26.03(+23.96)|14.95(+12.57)|\n|+DM+SD|5|1|46.40(+16.40)|36.45(+12.95)|36.88(+23.08)|\n|LLaVA1.5-7B|30|30|17.00|4.17|2.77|\n|+DM|30|30|28.00(+11.00)|21.66(+17.49)|11.97(+9.20)|\n|+DM+SD|30|5|44.20(+16.30)|32.92(+11.26)|32.04(+20.07)|\n|LLaVA1.5-7B|60|60|18.10|4.61|3.90|\n|+DM|60|60|29.30(+11.20)|15.86(+11.25)|9.60(+5.70)|\n|+DM+SD|60|5|38.60(+10.70)|26.95(+11.09)|28.21(+13.26)|\n### Results based on Gemma3-27B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|Gemma3-27B|1|1|1.80|2.66|2.75|\n|+DM|1|1|21.00(+19.20)|6.65(+3.99)|5.19(+2.44)|\n|Gemma3-27B|5|5|2.00|1.06|1.13|\n|+DM|5|5|19.80(+17.80)|6.18(+5.12)|4.90(+3.77)|\n|+DM+SD|5|1|34.10(+14.30)|19.30(+13.12)|20.46(+15.56)|\n|Gemma3-27B|30|30|2.00|0.80|1.54|\n|+DM|30|30|24.30(+22.30)|7.15(+6.35)|5.38(+4.04)|\n|+DM+SD|30|5|38.10(+13.80)|26.83(+19.68)|25.44(+20.06)|\n|Gemma3-27B|60|60|1.60|0.88|0.52|\n|+DM|60|60|26.90(+25.30)|8.09(+7.21)|5.69(+5.17)|\n|+DM+SD|60|5|34.60(+7.70)|20.76(+12.67)|22.76(+17.07)|\n### Results based on InternVL3-8B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|InternVL3-8B|1|1|13.60|3.61|3.42|\n|+DM|1|1|36.20(+22.60)|23.22(+19.61)|17.18(+13.76)|\n|InternVL3-8B|5|5|13.80|3.48|3.93|\n|+DM|5|5|37.30(+23.50)|25.60(+22.12)|19.62(+15.69)|\n|+DM+SD|5|1|45.60(+8.30)|27.01(+1.41)|28.53(+8.91)|\n|InternVL3-8B|30|30|14.40|2.11|3.69|\n|+DM|30|30|42.30(+27.90)|20.99(+18.88)|18.04(+14.35)|\n|+DM+SD|30|5|40.80(-1.50)|25.54(+4.55)|28.18(+10.14)|\n|InternVL3-8B|60|60|16.70|6.01|5.26|\n|+DM|60|60|36.30(+19.60)|19.29(+13.28)|14.92(+9.66)|\n|+DM+SD|60|5|38.40(+2.10)|22.44(+3.15)|23.88(+8.96)|\n### Results based on SurgVLM-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|SurgVLM-7B|1|1|1.20|3.73|2.85|\n|+DM|1|1|41.90(+40.70)|3.22(-0.51)|2.58(-0.27)|\n|SurgVLM-7B|5|5|1.06|4.68|2.79|\n|+DM|5|5|42.70(+41.64)|26.98(+22.30)|22.91(+20.12)|\n|+DM+SD|5|1|44.84(+2.14)|28.43(+1.45)|29.06(+6.15)|\n|SurgVLM-7B|30|30|12.80|4.02|3.39|\n|+DM|30|30|42.30(+29.50)|20.97(+16.95)|18.63(+15.24)|\n|+DM+SD|30|5|40.58(-1.72)|26.68(+5.71)|26.07(+7.44)|\n|SurgVLM-7B|60|60|10.90|2.99|2.98|\n|+DM|60|60|38.50(+27.60)|17.95(+14.96)|15.08(+12.10)|\n|+DM+SD|60|5|36.24(-2.26)|19.63(+1.68)|21.32(+6.24)|\n\nTo isolate the contribution of the State Decomposer (SD) module, we ran an ablation in which the baseline VLM was paired with SD alone (no multi-agent collaboration) and tasked with continual—rather than hierarchical—state prediction. The SD module still provided clear gains, demonstrating that it generalizes across non-hierarchical and hierarchical prediction. In short, IG, MC, and SD form a reproducible, plug-and-play toolkit that reliably improves temporal prediction across diverse VLM backbones.\n### Effectiveness of Incremental Generation without Multi-agent Collobration\n|Model|Temp. Scale|Incr. Scale|Accuracy|\n|---|---|---|---|\n|Gemma3-27B|5|5|2.00|\n|+SD|5|1|26.90(+24.90)|\n|Gemma3-27B|30|30|2.00|\n|+SD|30|5|28.40(+26.40)|\n|Gemma3-27B|60|60|1.60|\n|+SD|60|5|25.90(+24.30)|"}, {"review_id": "7h1WdWhgyI", "reviewer": "Reviewer_eUMj", "summary": "The paper presents the IG-MC framework, a novel solution for multi-scale temporal prediction (MSTP) that addresses long-horizon error accumulation and cross-scale consistency through two core innovations: incremental generation and decision-driven multi-agent collaboration. The work is supported by the first MSTP Benchmark, enabling unified evaluation across general and surgical scenes.", "questions": "Q1. The experimental comparisons are limited to Qwen2.5-VL-7B-Instruct, a general-purpose VLM. Given the focus on surgical scenes, why were there no comparisons with surgical-specific models? Such comparisons would better demonstrate IG-MC’s advantages in domain-specific tasks.\n\nQ2. The paper claims IG-MC maintains \"high performance across temporal scales,\" but there is no analysis of failure modes. Does the SD module generate anatomically invalid visuals for rare surgical tools or steps? Do long-horizon predictions degrade due to accumulated errors in incremental generation? Does multi-agent collaboration fail to enforce cross-scale consistency?\n\nQ3. The paper does not discuss failure cases, such as scenarios where incremental generation accumulates errors or multi-agent coordination breaks down. Understanding these cases is critical for improving robustness.", "limitations": "The paper acknowledges limitations (e.g., SD module dependency on pre-trained diffusion models, VLM-bound performance, inference latency), but these are not deeply explored.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized, with clear definitions of temporal/state scales and a logical flow from problem formulation to methodology and experiments. This makes complex concepts like incremental diffusion and multi-agent coordination accessible.\n2. The integration of incremental generation and multi-agent collaboration is a contribution. This closed-loop design ensures state-visual synchronization and cross-scale consistency, addressing limitations of single-scale or open-loop methods.\n\nWeaknesses:\n1. The iterative nature of incremental diffusion and multi-agent coordination inherently introduces latency, yet the paper lacks critical metrics such as inference time per time step, GPU memory consumption during inference, or scalability with longer temporal horizons.\n2. The experiments primarily compare against Qwen2.5-VL-7B-Instruct, a general vision-language model. They omit comparisons with surgical-specific models or state-of-the-art MSTP methods, making it hard to contextualize IG-MC’s advances in surgical domains.", "was_revised": true, "final_justification": "With the response to most of my issues addressed, I am happy to raise my score to accept.", "rebuttal": "We thank you for your time and effort in reviewing our manuscript. Your valuable feedback has greatly helped improve our work. Below are the revisions made in response to each comment.\n\n**Q1**: *The iterative nature of incremental diffusion and multi-agent coordination inherently introduces latency, yet the paper lacks critical metrics such as inference time per time step, GPU memory consumption during inference, or scalability with longer temporal horizons.*\n\n**A1**: Thanks! As show in below table, we have conducting computational efficiency and inference latency on agentic system. Profiling on a single NVIDIA H200 shows an end-to-end latency of ≈ 68 s: the three decision modules (STC, phase- and step-level predictors) each require 20–22 s and together dominate > 90 % of wall-time while operating at only ≈ 1 TFLOPS, indicating a memory-bound bottleneck; meanwhile the Incremental Generation stage peaks at ≈ 97 TFLOPS yet adds merely ≈ 6 s. Peak GPU memory is modest (26 GiB for all decision modules and 29 GiB for generation), confirming that *bandwidth*, not capacity, limits performance.\n### Inferency Latency and Computational Efficiency\n|Component|Avg. Time|Min. Time|Max. Time|Avg. GFLOPS|Min. GFLOPS|Max. GFLOPS|Peak GPU MEM.|\n|---|---|---|---|---|---|---|---|\n|StateTransitionController|20.04|19.33|20.77|1.12K|108.94|1.36K|26.14|\n|PhaseStatePredictor|20.90|19.87|21.76|1.10K|109.52|1.29K|26.14\n|StepStatePredictor|21.51|20.43|22.30|1.07K|90.31|1.24K|26.14\n|IncrementalGeneration|5.81|5.78|6.10|97.32K|78.62K|99.71K|28.53\n\n**Q2**: The experimental comparisons are limited to Qwen2.5-VL-7B-Instruct, a general-purpose VLM. Given the focus on surgical scenes, why were there no comparisons with surgical-specific models? Such comparisons would better demonstrate IG-MC’s advantages in domain-specific tasks.\n\n**A2**: Thanks! We have conduct more comparisons including the latest powerful surgical VLM, SurgVLM-7B  and serveral general-purpose VLMs InternVL3-8B, Gemma-27B, LLava-1.5, which can demonstrate the effectiveness of proposed IG-MC on different intelligence-level VLM, as shown in below tables.\n### Results based on LLaVA1.5-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|LLaVA1.5-7B|1|1|13.60|3.75|3.34|\n|+DM|1|1|30.00(+16.40)|23.50(+19.75)|13.80(+10.46)|\n|LLaVA1.5-7B|5|5|12.20|2.07|2.38|\n|+DM|5|5|27.90(+15.70)|26.03(+23.96)|14.95(+12.57)|\n|+DM+SD|5|1|46.40(+16.40)|36.45(+12.95)|36.88(+23.08)|\n|LLaVA1.5-7B|30|30|17.00|4.17|2.77|\n|+DM|30|30|28.00(+11.00)|21.66(+17.49)|11.97(+9.20)|\n|+DM+SD|30|5|44.20(+16.30)|32.92(+11.26)|32.04(+20.07)|\n|LLaVA1.5-7B|60|60|18.10|4.61|3.90|\n|+DM|60|60|29.30(+11.20)|15.86(+11.25)|9.60(+5.70)|\n|+DM+SD|60|5|38.60(+10.70)|26.95(+11.09)|28.21(+13.26)|\n### Results based on Gemma3-27B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|Gemma3-27B|1|1|1.80|2.66|2.75|\n|+DM|1|1|21.00(+19.20)|6.65(+3.99)|5.19(+2.44)|\n|Gemma3-27B|5|5|2.00|1.06|1.13|\n|+DM|5|5|19.80(+17.80)|6.18(+5.12)|4.90(+3.77)|\n|+DM+SD|5|1|34.10(+14.30)|19.30(+13.12)|20.46(+15.56)|\n|Gemma3-27B|30|30|2.00|0.80|1.54|\n|+DM|30|30|24.30(+22.30)|7.15(+6.35)|5.38(+4.04)|\n|+DM+SD|30|5|38.10(+13.80)|26.83(+19.68)|25.44(+20.06)|\n|Gemma3-27B|60|60|1.60|0.88|0.52|\n|+DM|60|60|26.90(+25.30)|8.09(+7.21)|5.69(+5.17)|\n|+DM+SD|60|5|34.60(+7.70)|20.76(+12.67)|22.76(+17.07)|\n### Results based on InternVL3-8B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|InternVL3-8B|1|1|13.60|3.61|3.42|\n|+DM|1|1|36.20(+22.60)|23.22(+19.61)|17.18(+13.76)|\n|InternVL3-8B|5|5|13.80|3.48|3.93|\n|+DM|5|5|37.30(+23.50)|25.60(+22.12)|19.62(+15.69)|\n|+DM+SD|5|1|45.60(+8.30)|27.01(+1.41)|28.53(+8.91)|\n|InternVL3-8B|30|30|14.40|2.11|3.69|\n|+DM|30|30|42.30(+27.90)|20.99(+18.88)|18.04(+14.35)|\n|+DM+SD|30|5|40.80(-1.50)|25.54(+4.55)|28.18(+10.14)|\n|InternVL3-8B|60|60|16.70|6.01|5.26|\n|+DM|60|60|36.30(+19.60)|19.29(+13.28)|14.92(+9.66)|\n|+DM+SD|60|5|38.40(+2.10)|22.44(+3.15)|23.88(+8.96)|\n### Results based on SurgVLM-7B\n|Model|Temp. Scale|Incr. Scale|Accuracy|Precision|Recall|\n|---|---|---|---|---|---|\n|SurgVLM-7B|1|1|1.20|3.73|2.85|\n|+DM|1|1|41.90(+40.70)|3.22(-0.51)|2.58(-0.27)|\n|SurgVLM-7B|5|5|1.06|4.68|2.79|\n|+DM|5|5|42.70(+41.64)|26.98(+22.30)|22.91(+20.12)|\n|+DM+SD|5|1|44.84(+2.14)|28.43(+1.45)|29.06(+6.15)|\n|SurgVLM-7B|30|30|12.80|4.02|3.39|\n|+DM|30|30|42.30(+29.50)|20.97(+16.95)|18.63(+15.24)|\n|+DM+SD|30|5|40.58(-1.72)|26.68(+5.71)|26.07(+7.44)|\n|SurgVLM-7B|60|60|10.90|2.99|2.98|\n|+DM|60|60|38.50(+27.60)|17.95(+14.96)|15.08(+12.10)|\n|+DM+SD|60|5|36.24(-2.26)|19.63(+1.68)|21.32(+6.24)|\n\n**Q3**: The paper claims IG-MC maintains \"high performance across temporal scales,\" but there is no analysis of failure modes. Does the SD module generate anatomically invalid visuals for rare surgical tools or steps? Do long-horizon predictions degrade due to accumulated errors in incremental generation? Does multi-agent collaboration fail to enforce cross-scale consistency? The paper does not discuss failure cases, such as scenarios where incremental generation accumulates errors or multi-agent coordination breaks down. Understanding these cases is critical for improving robustness.\n\n**A3**: Thanks! We have now added a dedicated appendix section detailing multiple failure cases—including (i) decision errors made by different agents and (ii) instances where generated visuals are of insufficient quality—and provide an analysis of their causes and potential remedies.\n\n### Analysis of accumulates errors of incremental generation.\nAs shown in below table including temporal relationship of decreasing accuracy and generation quality. Empirically we find a (moderately) strong negative correlation, yet the **absolute impact on decision quality is modest**:\n\n$$\n\\text{FID}=399.67 - 7.52 \\times \\text{Accuracy}\\quad (R=-0.87,\\; p\\approx 0.025)\n$$\n\n**Image‐quality drop is steep:** FID climbs **+24 pts** (≈ +34 %) from 5 s to 30 s. **Decision accuracy is resilient:** Accuracy slips only **-2.7 pts** (≈ -6 %) over the same span. Hence, although poorer images (higher FID) statistically co-occur with lower accuracy, the **magnitude of accuracy degradation is relatively small**, suggesting the decision module retains robustness even as generation quality declines.\n\n**Temporal Relationship between Accuracy and FID**\n|Time(s)|Accuracy↑|FID↓|\n|---|---|---|\n|5|43.30|70.63|\n|10|41.34|83.12|\n|15|42.44|85.62|\n|20|41.24|88.57|\n|25|41.58|90.99|\n|30|40.58|94.82|\n\n### Analysis of accumulates errors of multi-agent collobration.\nWe have conducted a comprehensive cumulative error analysis as shown in below table.\n\n**Errors are *not* independent.**\n  If the three agents failed independently, accuracy would collapse to ≈ 11–15 %.\n  In practice MC sustains **36–45 %**, outperforming the product bound by > 25 pp at every scale.\n\n**Gating and shared context mitigate drift.**\n  The STC’s switch/stay gate and the common visual prompt allow downstream agents to correct earlier missteps, preventing multiplicative error growth.\n\n**Sub-linear degradation remains.**\n  Accuracy declines only 8.6 pp from 5 s→60 s (44.8 → 36.2 %), far from an exponential blow-up.\n**Meets clinical threshold.**\n  Partners require ≥ 35 % top-1 for *in-the-loop* use; MC at 60 s still satisfies this bar.\n\nHence, the cascade’s cumulative error is well within acceptable limits and markedly better than the pessimistic independence baseline\n\n|Scale|State Transition Controller Acc.|Phase State Predictor Acc.|Step State Predictor Acc.|Product (Π)|Multi-agent Collobration Acc.|MC – Π|\n|---|---|---|---|---|---|---|\n|5 s|57.1|51.9|49.9|14.8|44.8|**+30.0**|\n|30 s|55.4|43.8|58.5|14.2|40.6|**+26.4**|\n|60 s|54.9|33.2|58.8|10.7|36.2|**+25.5**|"}]

Author rebuttal:

Meta-review: The manuscript introduces the MSTP task and benchmark and proposes a novel method which integrates incremental generation and a multi-agent system for hierarchical refinement of predictions across scales.
Reviewers all agreed that the manuscript is well-organized, that the proposed task is novel, and that the proposed combination of existing modules is creative and effective. Authors addressed the majority of concerns raised by the reviewers, including the comparison with stronger baselines, reporting inference latency, and several clarifications. I suggest acceptance provided the authors integrate the new content from the rebuttal and discussion period into the camera-ready version.
Decision: Accept (poster)