Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control
Authors: Xiao Fu
Xintao Wang
Xian Liu
Jianhong Bai
Runsen Xu
Pengfei Wan
Di ZHANG
Dahua Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Robotic Manipulation
Trajectory Control
Video-to-Policy
Primary area: generative models
TL;DR: RoboMaster learns an interactive world simulator for robotic manipulation, based on a video diffusion model conditioned on a collaborative trajectory, which controls the motions of both the robotic arm and the manipulated object.

Abstract: Recent advances in video diffusion models shows promise for generating robotic decision-making data, with trajectory conditions further enabling fine-grained control. However, existing methods primarily focus on individual object motion and struggle to capture multi-object interaction crucial in complex manipulation. This limitation arises from entangled features in overlapping regions, leading to degraded visual fidelity. To address this, we present RoboMaster, a novel framework that models inter-object dynamics via a collaborative trajectory formulation. Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction, and models each phase using the dominant object, specifically the robotic arm in the pre- and post-interaction phases and the manipulated object during interaction. This design effectively alleviates the multi-object feature fusion issue in prior work. To further ensure subject semantic consistency across the video, we incorporate appearance- and shape-aware latent representations for objects. Extensive experiments on the challenging Bridge dataset, as well as RLBench and SIMPLER benchmarks, demonstrate that our method establishs new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation. Project Page: https://fuxiao0719.github.io/projects/robomaster/

Published as a conference paper at ICLR 2026
LEARNING VIDEO GENERATION FOR ROBOTIC MANIP- ULATION WITH COLLABORATIVE TRAJECTORY CON-
TROL
Xiao Fu- Xintao Wang Xian Liu²
Jianhong Bai
Runsen Xu
Pengfei Wan Di Zhang
Dahua Lin
'The Chinese University of Hong Kong
Kuuiishu Technology *Zhejiang University
Input Image
Annotation
Generated Frames
piek up spoon and place on the left the pan
move the lemon the front of the bottle
pick up the coke bottle
Skills
close
fold
open
topple
wipe
Pre-Interaction Traj.
Interaction Traj.
Post-Interaction Traj.
Manipulated Object Mask
Rosterster synthesizes realistic robotic manipulation video given initial frame, prompt, user-defined object mask, and collaborative trajectory deseribing the motion of both robotic arm and manipulated object in decomposed interaction phases. supports diverse manipula- tion skills and can generalize to in-the-wild scenarios. Please check more on our website
Figure 1:
ABSTRACT
Recent advances in video diffusion models shows promise for generating robotic decision-making data, with trajectory conditions further enabling fine-grained con- trol. However, existing methods primarily focus on individual object motion and struggle to capture multi-object interaction crucial in complex manipulation. This limitation arises from entangled features in overlapping regions, leading de- graded visual fidelity. To address this, we present RoboMaster, novel framework that models inter-object dynamics via collaborative trajectory formulation. Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction, and models each phase using the dominant object, specifically the robotic arm in
Corresponding Authors.

---

Published as a conference paper at ICLR 2026
the pre- and post-interaction phases and the manipulated object during interaction. This design effectively alleviates the multi-object feature fusion issue in prior work. To further ensure subject semantic consistency across the video. we incorporate appearance- and shape-aware latent representations for objects. Extensive exper- iments the challenging Bridge dataset, well as RLBench and SIMPLER benchmarks, demonstrate that our method establishs new state-of-the-ar perfor- mance in trajectory-coontoolee video generation for robotic manipulation. Project Page: httpp:: gig....hb..... to/iio//oooo jects/ robomaster/ /obpaaooctt///bbbaatttt//e//
INTRODUCTION
Embodied AI has achieved remarkable progress in recent years (Brohan et al. 2022; 2023 Cheang
et al. 2024, Bjorck et al. 2025 Lynch et al. 2023 O'Neill et al. 2024, Li et al. 2023). holding
promise to replace human labor performing diverse tasks. Scalable robot learning plays crucial
role in empowering embodied intelligent agents to fulfill diverse and generalizable skills in unseen
environments. However, major bottleneck remains: data scarcity eu al. 2024a Liu et al. 2024b).
Collecting large-scale data using real robots is costly and requires human supervision to ensure safety.
Recently, video generation (Peebles & Xie 2023 Yang al. 2024b Agarwal al.. 2025 Wang
itt al. 2025 Kong et al. [024| Bai et al., 2025) has emerged as promising approach for simulating realistic environments, offering visually plausible content that closely resembles the real world. Lever- aging this, several works have explored generating robotic deciisionmmaiing data from multimodal
inputs, e.g., instruction (Du et al., (2023] Yang et al. 2023 Team 2024 Ko al.) 2023), sketch (Zhou
et al. 2024). and trajectory (Zhu et al. 20224] Team 2024). Among these,
generation enables one-grained control over robot planning by structuring the motions of both the robotic arm and the manipulated object. However, previous works, such Tora (Zhang al. 2025) and Anythnig (Wu et al.) 2024). simply focus on driving individual object motion with separate trajectories (see Tab. DT This design leads to feature entanglement in overlapping regions during interaction (highlighted by the red box in Fig. 2). which impairs the model's 's ability to capture physi- cally plausible interactions and impairs visual hideitty. In robot learning, high-quality demonstrations from video world models (Ali et al., 20255 Agarwal et al. 2025) can derive executable action labels via inverse dynamics models for downstream action planning. However, if the synthesized video fails to accurately capture the interaction phase, the inverse dynamics model may extract unreliable actions, potential limiting the effectiveness of the learned robotic policy.
Interaction
Inceraction
Decompose Interaction
Generate
Robor Tryj
pick up apple and place on the right
Tora
Object Thai.
Entengled
Decompose Objects
Figure 2: Collaborative Trajectory (Ours) vs Separated Trajectories (Previous, e.g. Tora). Unlike Tora (Zhang et al.] 2025) that decomposes objects and model the motion of robot arm and manipulated object separately, we decompose the interaction phase and unify their joint motions into a single collaborative trajectory with fine-grained object awareness. This integration alleviates the feature fusion issue in overlapping regions (see the missing apple in Tora), and improves visual quality.
To address these limitations, propose RoboMaster, which models robotic manipulation with novel collaborative trajectory. Unlike Tora, which decomposes multi-object motion using separate trajectories, RoboMaster captures interactive dynamics within unified trajectory representation. Specifically, we decompose the interaction process into three sub-phases pre-interaction, interaction, and post-interaction. Each phase is guided by the dominant agent, namely the robotic arm in the pre-
2

---

Published as a conference paper at ICLR 2026
and post-interaction phases, and the manipulated object during interaction. This design is motivated by the observation that the robotic arm initiates and concludes the motion, while the object remains largely static; during interaction, the object's 's motion reflects the physical response to manipulation, implicitly synchronizing with the robotic arm's 's trajectory. By explicitly modeling the driving subjects and their features across interaction phases, RoboMaster mitigates the feature entanglement issues and facilitates learning plausible interactions, rather than strictly following trajectory by compromising interaction fidelity. Furthermore, ensure semantic consistency of the manipulated object throughout the video sequence, we leverage the user-defined object mask to sample encoded RGB latents. These latents are then associated with the object's shape to construct circular volumetrie representation, preserving both appearance and shape across frames.
Table 1: Comparison of Ours with Previous Trajectory-Controllee Methods.
Interaction Gramularity
Object Awareness
Trajectory
Decomposed?
Format
Appearance
Shape
IRAsim (huu et al. 2024
Single (Isolated)
N/A
Arythhng (Wu et al. 2024
Multiple (Isolated)
X
Mask
<
Tora (Zhang et al. 2025
Multiple (Isolated)
x
Point
x
RoboMaster (Ours)
Single (Collaborative)
V
Mask
Beyond improved manipulation modeling, our design also facilitates user interaction: 1) user can easily annotate the interaction phase instead of full arm-object trajectories, simplifying trajectory correction under annotation errors 2) user can flexibly specify object regions with a brush tool. Our experiments demonstrate that RoboMaster remains robust even with imprecise user input.
We conduct extensive experiments on the challenging Bridge dataset (Walke et al. 2023) and
demonstrate that RoboMaster outperforms prior gegeectory-cotrrllle video generation methods in both visual quality and trajectory accuracy. Further evaluations on RLBench (James et al. (020) and SIMPLER (Li et al. 2024) benchmarks validate its effectiveness for robotic action planning. Our
contributions are as follows:
1) We introduce a collaborative trajectory framework that models robotic manipulation by decompos- ing the interaction phase into sub-phases, enabling video generation as an interactive simulator for high-quality robotic data. We also contribute high-quality 21k video-trajectory dataset from Bridge,
2) Our design combines collaborative trajectories with mask- bassed object embeddings, allowing for more intuitive user annotation and significantly enhancing user interactivity.
3) Extensive experiments show that our approach achieves state-of-the-arr results on both visual and robotic action planning benchmarks, outperforming existing trajectory-cooddtioned methods.
2 RELATED WORK
Sinjectory-Controln Video Generation for Object Movement. Early works (Wang et al. 2024d
Ma al. 2024] Mou et al.) [024 Yin et al., [2023] Zhang al. (025) leverage point-based control to enhance adaptability and user interactivity. Subsequent methods adopt mask-based representations (including bounding boxes) (Yang et al.
2024a| Quu et al. 2024 Wang et al. 2024b Wu et al.) 2024
Dai et al. 2023) or optical flow (Geng et al. 2024a Shi et 2024) to improve robustness over point-
based alternatives. Beyond 2D representations, 3asrajMaster (Fu et al. 2025) and ObjCtrl-2.5 WWan et al. 2024c) model object motion using 6-DoF trajectories, while LeviTor (Wang et al. 2024a) incorporates depth to enable 3D-aware object manipulation. However, existing works overlook interaction scenarios and treat object motion as independently controlled, which degrades visual quality in overlapping regions. In contrast, RoboMaster introduces collaborative trajectory control, unifying interactive features and decomposed trajectories to model interaction effectively.
Video Generation as World Simulator for Robotic Manipulation. Scalable robot learning (Brohan et al. 2022: 2023 Cheang et al. 2024 Bjorck et al. 2025 Lynch et al.) 2023) relies heavily on
large-scale realistic data, but collecting real-world robot trajectories from human demonstrations remains tine-consuming and
limiting public accessibility. To address this, generative video models (Wu et al. 2023 Agarwal et al. 2225)) offer a cost- alternative for synthesizing
realistic data for policy learning. UniPi (Du et al. 2023) and AVDC (Ko et al. 2023) frame robot.
3

---

Published as a conference paper at ICLR 2026
planning as text-to-video generation, with AVDC further incorporating inverse dynamic estimation via a pretrained flow network. UniSim (Yang et al. 2023) learns à unified real-world simulator with diverse conditions (text and control inputs). TesserAct (Zhen et al., 2025) learns 4D robotic manipulation by additional modeling video depth&normal. IRASim (Zhu et al. (024) also employs
wajectory-conditioee video generation but only models robot arm motion. In contrast, our method jointly models both robot and object trajectories with fine-grained object awareness, enabling higher visual quality and greater user interactivity in unseen scenarios.
3
METHOD
Our goal is to enable fine-grained and user-friendly control in image-to-video generation for robotic
manipulation. To this end, we present RoboMaster (see Fig- 3) framework built upon collaborative trajectory mechanism. We begin by reviewing the prior trajectory control paradigm and outlining our
task formulation (Sec. 3.1). We then introduce the key components required for control: (1) object
embeddings that encode appearance and shape to maintain identity consistency (Sec. 3.2 and (2)
collaborative trajectory that models the interactive dynamies (Sec. (.3). These are integrated via motion injector (Sec. 3.4) that effectively guides motion generation.
Prompt move the carrot on the left the plate
e 3d
X
Motion Injector
Motion Injertor
Motion Injector
DV Block
(Sec 3.4)
DIT Block
Dil
R
XxxxXXx
Input Image E R 1 HxW
Fraines
User Action
End Effector Mask Mr (0, 1)
Embred. Subjects
Annotation Order
2 Object Mask
M, (0,1) How A
Pre-Interaction "Daj. ((z, P) K Interaction Trai.
Object Latenes Va. v. 2
((e, slele-m 4 Post- Interaction Traj. (1A, (04.)))
Assorinne 3.3)
Cullaberntive &
X
Figure 3: RoboMaster Framework. Given an input image and a prompt c, it generates a desired robotic manipulation video X with the collaborative trajectory design. Specifically, it first encodes the object masks, including robotic arm Ma and submissive object M, (acquired either from I) Grounded-SAM Ren al. (2024) or user-defined brush mask) with the awareness of appearance and shape to obtain Vd, V. for maintaining identity consistency in the video. To precisely model the manipulation process, the controlled trajectory is decomposed into sub-interaction phases: pre-interaction C1, interaction C2, and post-interaction C3, associating each phase with object-specific latents Vd. V.. and va. respectively. The collaborative trajectory latent V is then injected into plug-and-plan motion injectors, enabling the reasoning of video dynamics during generation.
3.1 PRELIMINARY: VIDEO DITS WITH DECENTRALIZED TRAJECTORY CONTROL
Video diffusion transformers (DiTs) Peebles & Xie (2033) Lin al. 2024 Yang al. (0044b)
Agarwal et al. 2025); Wang et al. (025): Kong et al. (2024) with trajectory condition C learns the of the compressed video data
patchify(E(X)) where N is the trajectory number, E-) is a 3D VAE encoder and X E Fx3xx#x is clean video. It involves forward process q to progressively inject noise e on Xg to the desired Gaussian distribution in Markov chain: {x,, (1, x; axXo + ofE, ~~ (0, I)}. and reverse process pe to remove noise via a noise estimator ég, trained by minimizing:
conditional distribution p(x J
(xx 1, (C, (x ||
min B
(1)
4

---

Published as a conference paper at ICLR 2026
Task Formulation Given initial frame I containing interaction subjects, dominant subject Od and a submissive subject os, along with user-defined text prompt C, binary object masks Ma
and M, (where M E {0, 1 ") and a collaborative trajectory = {Ix, )) our objective is synthesize a plausible manipulation video X. The trajectory f
structured into three temporal phases: pre-interaction C₁ = {(x, )) interaction C2 = {(x, 1) ) and post-interaction
C3 ((.,,))) We define the general formulation fo(-) of the generative model
fo(:): IE I2 3xXXW ,ce yd, Ma, M, {0, [0,1) #xW ,C ((,,y)e +XER xx3xHXW
where y is the alphabet, L is the token length, and X s D(umpatchify(xx)))
3.2
SUBJECT REPRESENTATION VIA COUPLED APPEARANCE AND SHAPE EMBEDDING
radius
K
3D VAR
Sample (Alg. 3)
(ReXÁXM
Shape-Awaare Exponding (Aly
Input Image (blended with mask)
I R3x
Downsample
Object Mask (K Pieels)
(0,1)
Object Toleens E R
Figure 4: Subject Embedding Illustration. The object mask M is interpolated to align with the encoded RGB latents Then samples z with valid pixels and applies an average pooling operator to generate the embedding V. To enhance spatial awareness, it expands the object token by radius which is proportional to the area of the valid mask region, and obtains the circular volume v.
As shown Fig, the initial frame I is first projected into latent features z via the VAE encoder E(-) R W
Rexhxw with spatial compression factors C.. The object masks are subsequently downsampled using an interpolation operator Fa(.) Ma, M, E (0, 1] H m., m, {0, 1] /xw match the spatial resolution of the latent feature map. We then extract the latent subject features by applying the corresponding masks to z, followed by pooling operator, resulting in Va. V, E R, defined as:
W
V. [i]
zd li, x.
for i= 0,1,....
Ei En m [i.i
2d [i, x. y] 2/6, x. v] if ma, [r, y] =1 otherwise 2d, x, vr =0
At each timestep within the latent video length f. which the temporal-comppessed length of F, we represent the subjects as circular volume v. Rexhxw centered at the trajectory point (z, y). and confined to the corresponding valid mask region. This volume is constructed as:
Li, k] [ if ((-z)) x (k- y <= rd.a otherwise -
Va,e
Vi,j. (4)
where the radius rd,s is proportional to the mask area, i.e., ΣL - m(i,j]. Incorporat- ing both object appearance and spatial shape into this latent representation accelerates training convergence and improves identity consistency across subsequent frames the video sequence.
3.3 COLLABORATIVE TRAJECTORY REPRESENTATION
Decentralized modeling of multiple trajectories, represented ( c, ( ) is appropriate for scenarios where objects follow independent motion patterns without mutual intervention. How- ever, when applied interactive scenarios, e.g., picking up or moving objects, it exhibits several
'We omit the robotic arm mask in Fig. as 1) for better illustration 2) user is not required to additionally input it and only need the pre-defined mask instead.
5

---

Published as a conference paper at ICLR 2026
limitations: 1) Feature overlap: Interaction phase dominates the overall motion, and it introduces feature ambiguity in such overlapping regions as models are primarily trained on independently moving objects, leading to degraded synthesized quality. 2) Trajectory precision during interaction: Accurately specifying the trajectory of the dominant subject 0g during the interaction phase is challenging. Users may find it difficult define precise temporal boundaries (e.g., start and end timestamps) and relative spatial positioning with respect to the submissive object O,.
To address these limitations, we propose learning unified distribution ( I, c, Va. Va: with collaborative trajectory C, which is further temporally decomposed into three subsets:
Pre-& & Post-Interaction During these phases, the dominant subject og serves as the sole moving agent, while the submissive subject o, remains static or exhibits minor motion due to inertia. Accordingly, we leverage the dominant subject's 's trajectory C₁ - {(Tas Ba)c) C3 = {(Ea, M2) ) F=++1 along
with its circular volume Va to model the distribution (x I, C, Va. Ci) and (x I, Va. C3 P
Interaction At this stage, the interactive agents Og and o, collaborate to carry out the instruction c. We incorporate submissive subject's 's trajectory C2 = {(x,, 3=) and its corresponding circular feature v. to model the conditional distribution po (x2 C. C2). Our intuition is twofold: 1) the motion of the submissive subject implicitly guide the dominant subject, owing to the typically constrained relative dynamics between interacting entities during this phase 2) temporal variations in the feature representation (i.e., Va > V, + Va) can provide valuable for modeling behavioral changes (sole object movement >> interactive objects movement) over time.
Causal Representation. Given the causal nature of the 3D VAE encoder E-), we incorporate latent feature map from previous frames into subsequent ones to enhance smoother transitions. Specifically, 1 is propagated forward, and the current
at each timestep latent feature map from timestep object feature (Va or V.)) is overwritten onto it. Consequently, the interaction and post-interaction distributions updated po (X2 I, c, Va, Va, C1, C2) and (x I, C, Vd, Vs, C2, Ca)
In general, our collaborative design factorizes the vanilla distribution po c, C., Ca) into multiple object-aware sub-distributions, thereby alleviating feature confusion and improving interaction:
(x1 I, c, Va, C1) (x
c, Va. Vo C1, (x I, Vd, Vs, C1, C2,
(5)
pre-interaction
interaction
post-interaction
User Interaction Our design offers several key advantages for user-friendly access to generalizable experiments: 1) Robustness in object extraction: Due to mask-based representation, users can flexibly specify interaction object using a simple brush tool. Our experiments show that object identity remains generated
well-preserved, even with a coarse imput brush-based mask, in contrast complete with SAM Ravi et al. 2024). 2) Flexibility in impul trajectory: instead of requiring two full-length trajectories, users can deñne decomposed sub-trajectories within single motion path. This not only simplifies the input process but also enhances adaptability for iterative refinement.
3.4 MOTION INJECTION MODULE
The collaborative trajectory latent e [exxxxx which associates Va. V. with latent frame length f. is patchified and sequentially encoded by zero-initialized 2D spatial convolutional layer and zero-initialized ID temporal convolutional layer. This produces a compact representation V T x x The output hidden state from the previous DiT block, denoted as h E T x x)xx is then combined with the trajectory latents (V and its group normalized output) before being forwarded to remaining DiT blocks: h norm V) VV, V Conv Conv (V)))
Loss Function To learn the desired motion patterns, we optimize the parameters 0, including both the DiT blocks and the motion injector, as follows:
[[|c éo, (x,, c, M, C
(()) Ex
(6)
We decompose the causal latent video x three semporall-partttionnn segments: X1, X2, and x3, corre- sponding to the pre-interaction, interaction, and post-interaction phases, respectively.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We implement our conditional video diffusion model based on the pre-trained Cog Video X-5B
architecture Yang et al. 2024b). We conduct experiments on the Bridge dataset (Walke al. 2023) (Please refer to Sec. A.Ipp we adopt resolution of 480 x 640 and video length of 37 frames during both training and inference. The model is trained using AdamW (Loshchilov & Hutter 2017) on
8 NVIDDA A800 GPUs, with learning rate of 2 x 10-5 for the DIT blocks and x 10 for the motion injector, and total batch size of 16. Training conducted for 30,000 steps. At inference, we employ 50 DDIM steps and set the CFG scale to 6.0.
4.2 BASELINES
We compare RoboMaster with existing state-of-ther ofrte--rrr trajectory baselines: Tora (Zhang et al. 2025). MotionCtrl (Wang et al. 2024d). DragAnything (Wu et al. 2024) and IRASim (Zhu et al. 2024). For fair comparison, all baselines are retrained on the same dataset based on Cog VideoX-5B with their respective optimal training configurations. We further compare with SOTA 4ID-grounded
method, TesserAct (Zhen et al. (2025).
4.3 EVALUATION METRICS
We perform evaluation on 214 test samples in Bridge, covering diverse manipulation skill based on:
1) Trajectory Accuracy: We report the Trajectory Error (TrajError), which computes the average L1 distance between the input and generated trajectories of both the robot arm and the manipulated object.
2) Video Quality: We adopt standard metries, including Frechét Video Distance (FVD) (Unterthiner ct al. 2018), PSNR (Hore & Ziou 2010), and SSIM Wang et al. 2004).
4.4 QUANTITATIVE& QUALITATIVE COMPARISON
Table 2: Quantative Comparison. Note that all the baselines are retrained on our curated dataset.
Video Quality
Trajectory Accuracy
User Study
Method
FVD
PSNR 1
SSIM
TrajError, wa
TrajErrrr,
Preference 1 (%)
TesserAct Zhen et al. IRASim
2025
261.84
18.99
0.778
37.34
54.64
8.01
a..
159.04
20.88
0.782
19.25
34.39
6.45
MotionCuri
Wang et 2024d
170.79
19.89
0.761
21.17
28.52
9.68
DragAnything [Wü et al. 2024
158.42
21.13
0.792
18.97
27.41
12.90
Tora Zhang al. 2025
152.28
21.24
0.788
18.14
26.43
17.74
RoboMaster (Ours)
147.31
21.55
0.803
16.47
24.16
45.16
As shown in Fig. aland Tab. 2 RoboMaster consistently outperforms prior state-of-the-ard methods in quantitative metrics of visual quality and trajectory accuracy, as well in qualitative visual performance. Our strengths lie in two aspects: 1) Interaction-aware Trajectory Design: We explicitly decompose interaction phases and integrate object features into unified trajectory. In contrast, baseline methods struggle with feature entanglement in regions where the robotic arm and object interact. IRAsim only controls the robot trajectory, resulting in coarse object control and increased trajectory error (24.16 + 34.39). 2) Object Representation: We use mask-based representations rather than shape-ambiguous point representations (as Tora and MotionCtrl), leading to improved object identity consistency across frames. See the white-box region in Fig. 5 for a comparison of object identity preservation. RoboMaster further exhibits enhanced robustness on in-the-wild image collections, outperforming baseline methods as shown in Fig. 6
'Generate video based on an initial frame, a prompt, robot and object trajectories, and an optional object mask (for Tora and Ours), and then compare it with GT video. "Skills: move, pick, open, close, upright, topple, pour, wipe, and fold
7

---

Published as a conference paper at ICLR 2026
Image
Tora
DragAnything
Prompt: wipe pot with sponge
(2): severe appearance distortion on sponge
Prompt: olose close brown box flap
&1)) severe appearance distortion on box flap
Prompt:move the spoon on the oloth
shape deformation on spoon head (1) / handle (2)
Prompt: upright pot in sink
&1)& erroneous shape deformation on pot
Prompt: close oven
appearance distortion robotic (1) and glass surface (2)
Figure 5: Qualitative Comparison. RoboMaster demonstrates superior performance across range of manipulation skills (e.g., move, pick, close, upright, close), exhibiting improved visual consistency of the manipulated subject compared to prior baselines.
4.5 EMBODIED ACTION PLANNING FOR ROBOTIC POLICY
For robotic planning simulation, we adopt five challenging tasks from RLBench (James al. 2020 ("pick up cup", "put knife", "put plate". "open microwave", and "close box") and four tasks from the SIMPLER benchmark (Li al. 2024) ("pick coke can". "close drawer". "move near". and "pick object"). For baselines, We adopt Tora (Zhang et al.] 2025), TesserAct (Zhen et al.; (025) and VLe (Kim et al., 2024) and also finetune them on the same curated dataset fair comparison.
The general pipeline for applying RoboMaster in robot learning is as follows: 1) Generate demonstra- tion videos given the first frame and the robotic task prompt using RoboMaster. 2) Extract executable action labels from the generated demonstrations, 3) Simulate robotic planning and evaluate the task success rate.
For the first stage, collect small set of 1,300 video-trajectory pairs from both benchmarks, ensuring that none of the corresponding tasks appear in the test set. Here we fix the camera location in each task. We then post-train RoboMaster this limited dataset to adapt the model to the benchmark robot morphologies (Franka in RLBench and Google Robot in SIMPLER). The model architecture is kept unchanged and training is performed NVIDIA A800 GPUs for 6 hours. After this
8

---

Published as a conference paper at ICLR 2026
adaptation, RoboMaster is able generate demonstrations conditioned on the benchmark robot morphologies.
the second stage, collect 300 vidco-action samples for each task in testset to train the in- verse dynamic model. Following AVDC (Ko et al. 2023), the inverse model trained to regress executable action labels from synthesized videos. To further validate its effectiveness,
ad- model from Cosmos2.5 (Ali al.) 2025) on the curated training data. The model takes as input the first frame and a se- quence of 7-DoF actions ( Az, Ay, Az, A0r, A0p. A0,, GripperWidth), and outputs the pre- dicted video. We perform fully fine-tuning, resizing videos to 320 x 256 with 37 frames, us- ing a learning rate of 2e-5, training for 2,500 iterations with a batch size of 16. Then we
ditionally post-train the Cosmos-Predict2. 2iteddicc..222//boooaatttooooom
feed actions predicted by our inverse dy- namic model into this model to generate oetion-conditioned videos. Quantitative evaluation against GT videos, shown Tab. 3. demonstrates that actions pre- dicted by inverse dynamics model produce comparable video quality, fur- ther validating its high performance.
Table 3: Evaluation of Action-conditioned Videos
Action Type
PSNR
SSIM f
Latent L2
FVD 1
Training Set Predicted
25.48
0.87
0.31
132
25.12
0.84
0.34
127
For the final evaluation stage, we generate 100 videos per task using the RoboMaster adapted in Stage 1, conditioned the task prompt and the initial frame. We then apply the inverse dynamics model to infer executable action labels from these generated videos and deploy the resulting action sequences on simulated robots. The success rates over 100 trials are summarized in Tab. RoboMaster consistently improves embodied action planning over existing baselines. The clear margin by both Tora and RoboMaster over Open VLA demonstrates that high-quality video generations serve as effective demonstrations for downstream robotic policy extraction. Moreover, RoboMaster outperforms Tora on out the 10 evaluated tasks, suggesting that RoboMaster produces more reliable interaction videos, enabling the inverse dynamics model to obtain more accurate action labels for execution. This further validates our core motivation: more accurate modeling of robot-object interactions leads to higher-quality execution demonstrations for robot learning.
Table Action Planning Comparision on RLBench and SIMPLER. We report the success rate averaged over 100 episodes for each task. Best is bolded and second best is underlined.
RLBench
SIMPLER
Method
pick up cup
put knife
put plate
open microwave
close box
pick coke
close drawer
pick object
Open VLA (Kim 2024
0.55
0.46
0.56
0.35
0.45
0.59
0.41
0.53
0.59
TesserAct Zhen et 2025
0.76
0.79
0.82
0.43
0.67
0.85
0.56
0.62
0.79
Tora Zhang et at. 2025
0.79
0.82
0.81
0.61
0.72
0.89
0.61
0.61
0.74
RoboMaster
0.83
0.76
0.85
0.54
0.79
0.91
0.63
0.67
0.81
4.6
ABLATION STUDY
Input Frame
Ours
DragAnything
Figure Generalizable Comparison with Input Prompt: 'Pick up the bee.'
We perform ablation on the full evaluation benchmark to validate model component effectiveness.
Subject Representation. Removing the causal embedding for latent control (w/o Causal Embedding in Tab. 5) leads to a decline both visual quality and trajectory accuracy, as evidenced by the misplacement of the can in Fig. highlighting the necessity of conditioning causal visual latents

---

Published as a conference paper at ICLR 2026
Inpuc Frame
Full Model
Causal Embedding
Separate Trajectories
Figure Ablation on Generalizable Sample: 'Move the can the right place of the eggplant.'
Table 5: Ablation Study on Bridge Full Benchmark.
Table 6: Ablation on Mask Sparsity
Method
FVD 4
PSNR 1
SSIM
TrajError
TrajEnor
w/o Causal Embedding
151.62
21.30
0.797
18.32
27.15
Sparsity (%)
PSNR (%)
w/ Points Representation
157.49
20.87
0.779
19.71
31.41
99.81
Separate Trajectories
17.24
25.84 84
98.12
w/ Cross Attention
163.56
19.38
0.761
21.52
29.16
98.02
Full Model
147.31
21.55
0.803
16.47
24.16
97.89
causal control signals. Moreover, replacing the mask-based representation with a point-based one (w/ Point Representation), as in Tora, significantly increases the subject trajectory error (24.16 + 31.41), indicating that mask provides more effective object representation. The mask-basee approach also exhibits greater robustness to input sparsity, as shown Tab. 6 where PSNR is reported relative to the full-mask baseline-aa an important property for real-world user input that is often incomplete.
Trajectory Injection. Replacing the collaborative trajectory with sep- arate ones (w/ Separate Trajectories) introduces feature fusion issues in overlapping regions, leading to reduced visual quality (see can distortion in Fig. (7) and lower trajectory accuracy. This supports the effectiveness of our decomposed trajectory design. Moreover, our model remains simple and effective, as alternative designs such as cross-attention--aseed trajectory injection (w/ Cross Attention) result in degradation. When randomly deviating partial subset (~ 15%) of sampled points from the original trajectory, the generated video remains robust under such disturbances shown in Tab.
Table 7: Ablation Tra- jectory Perturbation
Deviration (%)
PSNR (%)
5
99.17
10
98.25
15
97.68
20
97.15
Imperfect Prompts. We further analyze the sensitivity of our model to inaccurate prompts. Specifically, we randomly replace the subject prompt with prompt of similar or entirely different semantics (c.g., yel- low sponge" yellow block" "cotton ball"; "a spoon" spoon"->"l > branch") using GPT-4. Experiments conducted on the full Bridge benchmark produce the results shown in Tab. g where PSNR is reported relative to the accurate- eecurate-proopt baseline. Even with 40% of the prompts replaced by imperfect descriptions, RoboMaster retains 96% of its full-prompt performance, demonstrating strong robustness to prompt inaccuracies.
Table 8: Ablation on Im- perfect Prompt Input
Erronenus Portion (%)
PSNR (%)
10
98.42
20
97.53
30
97.13
40
96.54
CONCLUSION
In this work, we present RoboMaster, a trajectory-controle video generation framework with collaborative interaction design tailored for robotic manipulation. By decomposing interactions into sub-interaction phases, our method achieves superior visual quality and trajectory accuracy over prior approaches. Coupled with shape- and appearance-aware object encoding, RoboMaster enables more intuitive user annotation and enhances overall interactivity.
Limitations& Work: (I) RoboMaster may produce incomplete or distorted objects during manipulation when applied to out-of-domain inputs. This could be mitigated by training on more diverse object categories with richer semantic and geometric variations. (2) The current framework
operates purely in 2D pixel space; integrating depth cues (Fu et al. 2024 Chen et al., 2025| Hu ctt al.) 2024) may enable more accurate 3D control. (3) Generalization to varied robotic embodiments remains challenge and requires expanding training data encompass broader robot configurations.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint
000:250110337555 2025.
Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao. Elizabeth Cha, Wei Chao, al. World simulation with video foundation models for physical ai. arXiv preprint wrXiv2251.00006 2025.
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers. arXiv preprint urXiv:2411.18677 2024a.
Sherwin Bahmani. Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Ying Lee, Chaoyang Wang. Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407.1278/ 2024b.
Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, and Di Zhang. Syncammaster: Synchronizing multi-camera video generation from diverse viewpoints. arXiv preprint arXiv:2412.07760, 2024.
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Cameraa-cottolll generative rendering from a single video. arXiv preprint arXiv:2503. /647, 2025.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da. Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068177 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et Rt-2: lision-language-ace models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. (5..::330../888.. 2023.
Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang, Yong Zhang, Ying Shan, and Xiangyu Yue. Ditetrl: Exploring attention control in multi-modal diffusion transformer for tuning-fiee free multi-prompt longer video generation. arXiv preprint arXiv:2412.18859 2024.
Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language--e knowledge for robot manipulation. arXiv preprint arXiv:24/0.06 10.006658, 2024.
model with web-scale
Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv preprint arXiv:2501. 12375, 2025.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao. Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine-grained open domain image animation with motion guidance. arXiv preprint arXiv:2311. 12886, 2023.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- ext-ggiidd video generation. Advances in neural information processing systems, 36:9156 9172, 2023.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference on Computer Vision, pp. 241-258. Springer, 2024.
Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. In The Thirleenth International Conference Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Qijun Gan, Yi Ren, Chen Zhang, Zhenhui Ye, Pan Xie, Xiang Yin, Zehuan Yuan, Bingyue Peng, and Jianke Zhu. Humandit: Pose- gase-guided diffusion transformer for long-form human motion video generation. arXiv preprint arXiv:2502.00 04847, 2025.
Daniel Geng. Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Ta- tiana Lopez-Guevara, Carl Doersch, Yusuf Aytar. Michael Rubinstein, al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:24/2.02700) 2024a.
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Ta- tiana Jopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint arXiv:2412.02700, 2024b.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models, In European Conference on Computer Vision, pp. 330-348. Springer, 2024.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cam- eractrl: Enabling camera control for text-to-video generation. arXiv preprint arXivv2404.02101, 2024.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369. IEEE, 2010.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8163, 2024.
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-lahelling real videos. arXiv preprint arXiv:2410.11833 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09244 2024.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.08577 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.030000 2024.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu. Hongsheng Li, Leonidas J Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi video generation with camera control. Advances in Neural Information Processing Systems, 37:16240-16271, 2024.
12

---

Published as a conference paper at ICLR 2026
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- lk: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80-93. PMLR, 2023.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 2rX1v:2405.05941, 2024.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang. Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.001.. 2024.
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767, 2024a.
Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning eyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint wwXiv:2407.06886, 2024b.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv: 2017.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023.
Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake video with motion and content control. Advances in Neural Information Processing Systems, 37:18481-18505 2024.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903. IEEE, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4195-4205 2023.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang. Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning- free trajectory control in video diffusion models. arXiv preprint arXiv:2406. 16863, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 24t.::200000071111 2024.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world world models for diverse visual tasks, 2024.
Xuanchi Ren, Tianchang Shen, Jiahui Huang. Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3e: 3d-informed consisten video generation with precise camera control. arXiv preprint ar arXiv:2503.0375/ 2025.
Xiaoyu Shi, Zhaoyang Huang, Fu- Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2y: Consistent and controllable image-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
13

---

Published as a conference paper at ICLR 2026
Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, and Ming Yang. Animate-x: Universal character image animation with enhanced motion representation. arXiv preprint arXiv:2410.10306 2024.
1X World Model Team. Ix world model challenge. http:://tthhbbboomm//ccccthooo/p
2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:/8/2.01717, 2018.
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503 203 2025.
Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Qifeng Chen, Yujun Shen, and Limin Wang. Levitor: 3d trajectory oriented image-to-video synthesis. arXiv preprint arXiv:2412.15214. 2024a.
Jawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng. Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024b.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36:7594-7611, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Zhouxia Wang, Yushi Lan, Shangchen Zhou, and Chen Change Loy. Objetrl-2.5 d: Training-free object control with camera poses. arXiv preprint arXiv:2412.07721 2024c.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024d.
Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi, and David J Fleet. Controlling space and time with diffusion models. In The Thirteenth International Conference on Learning Representations, 2024.
Hongtao WU, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312. 13119,9.,,1,,, 2023.
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pp. 331-348. Springer, 2024.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-fre motion interpreter and controller. Advances in Neural Information Processing Systems, 37:76115- 76138, 2024.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbecl. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6, 2023.
Shiyuan Yang. Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. airect-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024a.
14

---

Published as a conference paper at ICLR 2026
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint 2tt1:24000000072,, 2024b.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08000 08089,0089... 2023.
Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint (xXi:2240.153364 2024.
Albert Yu, Adeline Foote, Raymond Mooney, and Roberto bridge the sim2real gap. arXiv preprint arXiv:2405.1002) 2024a
Natural language can help
Mark YU, Wenbo Hu, Jinbo Xing, and Ying Shan. Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models. arXiv preprint arXiv:2503.00 056688 2025.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409. 02048, 2024b.
Zhenghao Zhang. Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectery-oriented oriented diffusion transformer for video generation. In CVPR, 2025.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pp. 273-290. Springer, 2024.
Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: learning 4d embodied world models. In ICCV, 2025.
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint 12377 2024.
Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators. arXiv preprint arXiv:2406. /4540, 2024.
15


Reviews:
[{"review_id": "Vf6Wb8F7l8", "reviewer": "Reviewer_ydrE", "summary": "The method tackles the task of trajectory-conditioned robotic manipulation video synthesis through a novel demonstration trajectory decomposition scheme that separates trajectories into a pre-interaction, interaction and post-interaction phase. This novel decomposition helps alleviate feature confusion issues observed in previous works. The method is evaluated against several baselines on the tasks of video synthesis and downstream robotic manipulation (through inverse dynamics), and manages to outperform them.", "questions": "Can the method handle multi-step interactions, such as grasping a sponge, rotating it on a plate several times, then moving the robotic manipulator away?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written and easy to understand. The method outperforms the baselines against which it is compared. The design choices are sensibly ablated. The work contributes a dataset of 21.000 human-annotated 2D robot manipulator trajectories. The work includes an honest discussion of its limitations.", "weaknesses": "The proposed method operates purely in image space: the generated trajectories require postprocessing by an inverse kinematics model and are not guaranteed to be realistic or executable.\nUnlike its baselines, the method requires a segmentation of the provided trajectory into multiple stages by the user.\nThe manual masking of the interacted object could be replaced by an automatic grounding and segmentation.\nA purely 2D trajectory input is very limiting, yet this is somewhat alleviated by the ability to describe the desired trajectory more specifically through a textual input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7cA1TS4rH", "reviewer": "Reviewer_fCEc", "summary": "This paper proposes RoboMaster to model interactions between a robotic arm and objects, dividing the interaction process into three stages: before and after the interaction it mainly controls the arm’s motion, while during the interaction it controls the object’s motion. The effectiveness of the proposed method is validated through visual results and simulation.", "questions": "Could you explain in detail how the proposed model is applied to robot learning, and on that basis clarify the motivation for decomposing the action/control signals? At present, the stated motivation seems to be driven by generating better visual effects.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper argues that interaction should originate from multiple entities, including the arm and the object—this is a novel viewpoint.\n\n2. RoboMaster exhibits impressive OOD generalization.", "weaknesses": "1. The motivation for decoupling the control signals is unclear; it is not explained how 2D trajectories help the robot learn, and the paper does not discuss the overall design in detail.\n\n2. Section 4.5 is too brief, making it difficult to verify the method’s effectiveness for the robot; visual quality is not the core of the research—the core is whether the designed method can effectively aid robot learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MirAODUw7r", "reviewer": "Reviewer_1NRP", "summary": "Summary :\n\nRoboMaster is a trajectory‑conditioned video diffusion framework for robotic manipulation that jointly controls the robot arm and the object with a single collaborative trajectory. By decomposing each task into pre‑interaction, interaction, and post‑interaction phases and using mask‑based, appearance‑ and shape‑aware embeddings, it avoids the feature entanglement seen when controlling objects separately. It achieves state‑of‑the‑art visual fidelity and trajectory accuracy and improves downstream action planning.\n\n\nContributions:\n\nCollaborative trajectory control for interaction: A unified trajectory that models pre-interaction → interaction → post-interaction phases, switching the dominant controller (arm → object → arm) to capture inter-object dynamics and avoid feature fusion during contact.\n\nAppearance and shape aware subject embeddings: Mask-based tokens drawn from the initial frame’s VAE latents, expanded into shape-aware circular volumes along the trajectory to preserve object identity across frames.\n\nCausal latent propagation: Frame-to-frame latent carryover (with overwrite at trajectory points) for smoother, temporally consistent motion during generation.", "questions": "Question 1: How does the model determine the precise transition points between these phases in practice, especially when the temporal boundaries of “interaction” are ambiguous or vary across tasks?\n\n\nQuestion 2: How well would RoboMaster generalize to unseen robot morphologies or entirely different kinematic structures, and what adaptations (e.g., in trajectory representation or latent space) would be needed for that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel collaborative trajectory design\n\nIntroduces a new way to model robot–object interactions using a single collaborative trajectory split into pre-interaction, interaction, and post-interaction phases.\n\nAvoids the feature entanglement issues (e.g., missing or distorted objects) that plague prior methods like Tora and DragAnything.\n\n2. High visual and physical realism\n\nProduces smoother, more physically plausible manipulation videos with consistent object identities across frames.\n\nQuantitatively achieves better FVD, PSNR, and SSIM, and lower trajectory errors on the Bridge benchmark.\n\n3. Generalization and robustness\n\nHandles diverse manipulation skills and in-the-wild scenarios.\n\nRobust to imperfect user input—works with coarse or partial object masks and noisy trajectories.", "weaknesses": "1. Restricted to 2D pixel space\n\nThe system does not yet model depth or 3D geometry; this limits physical accuracy and makes 3D control (e.g., precise grasping) difficult.\n\n2. Possible failure on out-of-domain inputs\n\nCan produce incomplete or distorted objects when encountering unseen categories or backgrounds.\n\nStill relies on training data diversity to generalize effectively.\n\n3. Semantic dependency on user input\n\nRelies on accurate prompts and roughly correct masks. Misleading text or poor masks may still degrade quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)