GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Authors: name: Tianyi Xie, name: Haotian Zhang, name: Jinhyung Park, name: Zi Wang, name: Bowen Wen, name: Jiefeng Li, name: Xueting Li, name: Qingwei Ben, name: Haoyang Weng, name: Yufei Ye, name: David Minor, name: Tingwu Wang, name: Chenfanfu Jiang, name: Sanja Fidler, name: Jan Kautz, name: Linxi Fan, name: Yuke Zhu, name: Zhengyi Luo, name: Umar Iqbal, name: Ye Yuan
Organization: NVIDIA

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84\% real-world success on diverse object pick-up and 90\% success on stair-climbing.