Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot
  Data
Authors: name: Ben Moran, name: Mauro Comi, name: Steven Bohez, name: Tom Erez, name: Zhibin Li, name: Leonard Hasenclever

Creating accurate, physical simulations directly from real-world robot motion
holds great value for safe, scalable, and affordable robot learning, yet
remains exceptionally challenging. Real robot data suffers from occlusions,
noisy camera poses, dynamic scene elements, which hinder the creation of
geometrically accurate and photorealistic digital twins of unseen objects. We
introduce a novel real-to-sim framework tackling all these challenges at once.
Our key insight is a hybrid scene representation merging the photorealistic
rendering of 3D Gaussian Splatting with explicit object meshes suitable for
physics simulation within a single representation. We propose an end-to-end
optimization pipeline that leverages differentiable rendering and
differentiable physics within MuJoCo to jointly refine all scene components -
from object geometry and appearance to robot poses and physical parameters -
directly from raw and imprecise robot trajectories. This unified optimization
allows us to simultaneously achieve high-fidelity object mesh reconstruction,
generate photorealistic novel views, and perform annotation-free robot pose
calibration. We demonstrate the effectiveness of our approach both in
simulation and on challenging real-world sequences using an ALOHA 2 bi-manual
manipulator, enabling more practical and robust real-to-simulation pipelines.