X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
Authors: name: Prithwish Dan, name: Kushal Kedia, name: Angela Chao, name: Edward Weiyi Duan, name: Maximus Adrian Pace, name: Wei-Chiu Ma, name: Sanjiban Choudhury

Human videos offer a scalable way to train robot manipulation policies, but
lack the action labels needed by standard imitation learning algorithms.
Existing cross-embodiment approaches try to map human motion to robot actions,
but often fail when the embodiments differ significantly. We propose X-Sim, a
real-to-sim-to-real framework that uses object motion as a dense and
transferable signal for learning robot policies. X-Sim starts by reconstructing
a photorealistic simulation from an RGBD human video and tracking object
trajectories to define object-centric rewards. These rewards are used to train
a reinforcement learning (RL) policy in simulation. The learned policy is then
distilled into an image-conditioned diffusion policy using synthetic rollouts
rendered with varied viewpoints and lighting. To transfer to the real world,
X-Sim introduces an online domain adaptation technique that aligns real and
simulated observations during deployment. Importantly, X-Sim does not require
any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2
environments and show that it: (1) improves task progress by 30% on average
over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with
10x less data collection time, and (3) generalizes to new camera viewpoints and
test-time changes. Code and videos are available at
https://portal-cornell.github.io/X-Sim/.