Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
  for Real-World Robot Policy Evaluation
Authors: name: Jad Abou-Chakra; name: Lingfeng Sun; name: Krishan Rana; name: Brandon May; name: Karl Schmeckpeper; name: Maria Vittoria Minniti; name: Laura Herlant
Published: 2025-04-04 17:05:56

Abstract: Recent advancements in behavior cloning have enabled robots to perform
complex manipulation tasks. However, accurately assessing training performance
remains challenging, particularly for real-world applications, as behavior
cloning losses often correlate poorly with actual task success. Consequently,
researchers resort to success rate metrics derived from costly and
time-consuming real-world evaluations, making the identification of optimal
policies and detection of overfitting or underfitting impractical. To address
these issues, we propose real-is-sim, a novel behavior cloning framework that
incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
the entire policy development pipeline: data collection, training, and
deployment. By continuously aligning the simulated world with the physical
world, demonstrations can be collected in the real world with states extracted
from the simulator. The simulator enables flexible state representations by
rendering image inputs from any viewpoint or extracting low-level state
information from objects embodied within the scene. During training, policies
can be directly evaluated within the simulator in an offline and highly
parallelizable manner. Finally, during deployment, policies are run within the
simulator where the real robot directly tracks the simulated robot's joints,
effectively decoupling policy execution from real hardware and mitigating
traditional domain-transfer challenges. We validate real-is-sim on the PushT
manipulation task, demonstrating strong correlation between success rates
obtained in the simulator and real-world evaluations. Videos of our system can
be found at https://realissim.rai-inst.com.