HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for
  Autonomous Driving
Authors: name: Hongyu Zhou, name: Longzhong Lin, name: Jiabao Wang, name: Yichong Lu, name: Dongfeng Bai, name: Bingbing Liu, name: Yue Wang, name: Andreas Geiger, name: Yiyi Liao

In the past few decades, autonomous driving algorithms have made significant
progress in perception, planning, and control. However, evaluating individual
components does not fully reflect the performance of entire systems,
highlighting the need for more holistic assessment methods. This motivates the
development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator
for evaluating autonomous driving algorithms. We achieve this by lifting
captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving
the rendering quality for closed-loop scenarios, and building the closed-loop
environment. In terms of rendering, We tackle challenges of novel view
synthesis in closed-loop scenarios, including viewpoint extrapolation and
360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further
enables the full closed simulation loop, dynamically updating the ego and actor
states and observations based on control commands. Moreover, HUGSIM offers a
comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo,
nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair
and realistic evaluation platform for existing autonomous driving algorithms.
HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks
the potential for fine-tuning autonomous driving algorithms in a photorealistic
closed-loop setting.