Title: PointARSim: Point-cloud–enhanced Generative Auto-Regressive Simulation for Closed-loop End-to-end Autonomous Driving Evaluation
Primary area: applications to robotics, autonomy, planning
Keywords: generative simulator, autonomous driving

Abstract: End-to-end autonomous driving (E2E-AD) has garnered increasing attention in recent years. However, exist evaluation fails to simultaneously meet the requirements for both interactivity (e.g., CARLA) and realism (e.g., nuScenes). In this work, we introduce \textbf{\emph{PointARSim}}, an autoregressive generative simulation framework that leverages point cloud skeleton representation accumulated from offline datasets as conditions for diffusion models to synthesize multi-view images. Specifically, we separately aggregate foreground-background point cloud, facilitating utilization of visual geometry. To enable accurate control of the generated driving scene, we design a multi-conditioning ControlNet that integrates point cloud priors, previous-frame condition and optional backgrounds injection, 
 guided by a geometry-aware cycle loss. 
 Furthermore, to assess generative models’ fidelity when the ego-vehicle’s trajectory differs greatly from recorded offline logs, we introduce a new evaluation protocol based on the nuPlan dataset, called \textbf{\emph{nuPlan-SimGenEval}} benchmark.
 Extensive experiments on both nuScenes and nuPlan-SimGenEval demonstrate that the proposed approach significantly outperforms existing methods, highlighting its potential for next-generation closed-loop autonomous driving simulation.