Learning Video Generation for Robotic Manipulation with Collaborative
  Trajectory Control
Authors: name: Xiao Fu; name: Xintao Wang; name: Xian Liu; name: Jianhong Bai; name: Runsen Xu; name: Pengfei Wan; name: Di Zhang; name: Dahua Lin
Published: 2025-06-02 17:57:06

Abstract: Recent advances in video diffusion models have demonstrated strong potential
for generating robotic decision-making data, with trajectory conditions further
enabling fine-grained control. However, existing trajectory-based methods
primarily focus on individual object motion and struggle to capture
multi-object interaction crucial in complex robotic manipulation. This
limitation arises from multi-feature entanglement in overlapping regions, which
leads to degraded visual fidelity. To address this, we present RoboMaster, a
novel framework that models inter-object dynamics through a collaborative
trajectory formulation. Unlike prior methods that decompose objects, our core
is to decompose the interaction process into three sub-stages: pre-interaction,
interaction, and post-interaction. Each stage is modeled using the feature of
the dominant object, specifically the robotic arm in the pre- and
post-interaction phases and the manipulated object during interaction, thereby
mitigating the drawback of multi-object feature fusion present during
interaction in prior work. To further ensure subject semantic consistency
throughout the video, we incorporate appearance- and shape-aware latent
representations for objects. Extensive experiments on the challenging Bridge V2
dataset, as well as in-the-wild evaluation, demonstrate that our method
outperforms existing approaches, establishing new state-of-the-art performance
in trajectory-controlled video generation for robotic manipulation.