Title: GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
Primary area: datasets and benchmarks
Keywords: simulation, photoreal, robotics, real2sim, sim2real

Abstract: We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed—exceeding 100,000 steps per second on consumer GPUs—while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://gauss-gym.com.

TL;DR: We integrate 3D Gaussian Splatting into fast vectorized simulators, achieving photorealism at over 1M FPS on consumer GPUs and reducing the sim-to-real gap across diverse tasks.