GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS

Authors: Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, Eugene Vinitsky
Primary area: datasets and benchmarks
Keywords: Simulation, benchmark, multi-agent reinforcement learning, autonomous vehicles, planning

Abstract:
Multi-agent learning algorithms have been successful at generating superhuman planning in various games but have had limited impact on the design of deployed multi-agent planners. A key bottleneck in applying these techniques to multi-agent planning is that they require billions of steps of experience. To enable the study of multi-agent planning at scale, we present GPUDrive, a GPU-accelerated, multi-agent simulator built on top of the Madrona Game Engine capable of generating over a million simulation steps per second. Observation, reward, and dynamics functions are written directly in C++, allowing users to define complex, heterogeneous agent behaviors that are lowered to high-performance CUDA. Despite these low-level optimizations, GPUDrive is fully accessible through Python, offering a seamless and efficient workflow for multi-agent, closed-loop simulation. Using GPUDrive, we train reinforcement learning agents on the Waymo Open Motion Dataset, achieving efficient goal-reaching in minutes and scaling to thousands of scenarios in hours. We open-source the code and pre-trained agents at \url{www.github.com/Emerge-Lab/gpudrive}.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 3

Reviews and discussion:
<thread>
creation date:
1730041683221

thread_type:
official_review

<thread_object>
creator: reviewer_dlhs

<content>
summary:
This paper presents GPUDrive, a GPU-based simulator for autonomous driving. The simulator is compatible with existing datasets and allows parallel simulations. The experiments show that it can train a policy with 25-40x training speedup against the baseline.

strengths:
1. The GPU-based simulation is important in facilitating the training for complex real-world applications, highlighted by recent advances in robotics.
2. The experiment result shows significant wall-clock time speedup again baselines.

weaknesses:
1. The paper claims compatibility with existing datasets but only demonstrates map loading, leaving other functionalities unclear. For example, the imitation learning experiment or mixing agent behaviors—some from datasets and others from RL agents during training.
2. I believe one major advantage of parallel environments is that it allows you to do randomization across different environments (worlds). However, the paper lacks detail on whether GPUDrive supports this capability.
3. While GPUDrive offers a Python interface, I am curious how easy to customize those key elements in the environment given that the observation, reward, dynamic functions are written in C++. 
4. Experiments only evaluate IPPO, despite the paper claims that it targets at mixed motive setting.

questions:
1. In Figure 3, the speedup appears nearly linear. However, it would be helpful to examine scaling performance by adding more environments to identify saturation points and gain insights into system limitations.. 
2. What is the scaling of speedup with respect to the number of agents in the environment? e.g., fix the number of environments and scales the number of agents?
3. In Figure 5, do you use the CPU-parallel version of Nocturne?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582174695

thread_type:
official_review

<thread_object>
creator: reviewer_gedv

<content>
summary:
This paper proposes GPUDrive, a GPU-accelerated multi-agent driving simulator designed to increase efficiency of learning-based systems. The simulator allows for loading expert trajectories from real-world driving datasets, can support multiple observation spaces (including e.g. LiDAR), and displays favorable throughput compared to other openly accessible simulators. The simulator, together with pre-trained goal-conditioned policies, is made openly available with accessible pythonic interfaces.

strengths:
- The proposed simulator improves over alternatives in terms of sample efficiency. All of the design choices appear reasonable, while the underlying source code with pre-trained driving baselines will be released.
- The ability to load real-world driving datasets is extremely useful, while providing a variety of observation spaces is a great feature.
- Transparency about current limitation of the benchmark are very helpful for user adaptation.

weaknesses:
- While the focus of this paper is on providing a novel simulator, it would be very interesting to see some more complex behavior over longer time-horizons to fully capture the capabilities unlocked by the simulator (e.g. training a single agent policy with higher velocity limit to weave through a simulated traffic scene, etc.)
- Showcasing such behavior would likely require addressing the “Absence of a map” limitation raised in the paper, in order to formulate more sophisticated reward function. An important question would then be how easily this could be integrated, and how much the absence of such a feature could hurt adaptation of the simulator.
- The discussion of batched simulators could be extended to include references [1-3], where [1] has driven many results in single-agent robot learning, while [3] considers heterogenous multi-agent settings
- Figure 3 mentions performance on an RTX 4080, while line 711 states RTX 8000
- Line 308: “number valid number”

**References**

[1] V. Makoviychuk, L. Wawrzyniak, Y, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. “Isaac gym: High performance gpu-based physics simulation for robot learning.” NeurIPS, 2021.

[2] J. Panerati, H. Zheng, S. Zhou, J. Xu, A. Prorok, and A. P. Schoellig. "Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control.” IROS, 2021.

[3] M. Lechner, L. Yin, T. Seyde, T.-H. Johnson Wang, W. Xiao, R. Hasani, J. Rountree, and D. Rus. “Gigastep - one billion steps per second multi-agent reinforcement learning." NeurIPS, 2024.

questions:
- How easily can the simulator be updated to efficiently provide map-like utilities that allow for lane-keeping rewards (re mentioned limitations)?
- Do you support loading multiple different polices for individual agents? Could they have different sampling rates? How would these aspects affect efficiency?
- How do traffic jams affect throughput (re BVH)? This could be an interesting experiment to add.
- In video scene_53.mp4, agent 4 displays rather jerky behavior when moving towards its goal - could you elaborate on the underlying reasons?
- In video scene_43.mp4, agents 1 and 10 seemingly disappear without reaching their goals - could you elaborate on this behavior?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698747049

thread_type:
official_review

<thread_object>
creator: reviewer_y9u7

<content>
summary:
GPU drive introduces a fast multi agent simulator build using C++ that helps you run complex scenarios especially related to self driving cars at scale built on top of the Waymo Open Motion dataset. This allows iterating on these scenarios quicker reaching greater than a million FPS thus allowing more experimentation runs and iterating/trying out different scenarios even on desktop grade GPU's.

strengths:
1. A multi agent simulator accelerated on the GPU iteration of over a million steps per second.
2. Very well written and structured code to run any experiment easily with a lot of easy experimentation code readily available. 
3. Extensive results analyzing the sampling frequency of the simulation.

weaknesses:
1. Figure 2 needs a better caption and an explanation
2. Designed to fit one exact dataset. A section explaining the effort required to integrate other datasets is desirable.

questions:
1. Benchmarks consist of limitations because of the dataset. Can it be addressed by using another dataset ?
2. Stable baselines is not known that well for speed. Could other implementations of PPO have been used ?
3. Is there support for multi GPUs ? And if they do exist, an ablation or benchmark would be great for that
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722835317

thread_type:
official_review

<thread_object>
creator: reviewer_auke

<content>
summary:
The paper presents a GPU accelerated simulator that can generate millions of simulation steps samples per second that can be used to train multi-agent reinforcement learning (RL) algorithms. The simulator is claimed to simulate hundreds to thousands of scenarios/scenes in parallel with each scene containing thousands of agents.

The simulator is built on top of the Madrona Game Engine and is written in C++. The C++ simulator engine can also be interfaced with learning environments written in JAX and Torch.
The authors have released implementations of RL algorithms capable of processing millions of agent steps per second and some baseline agents trained on these algorithms that achieve 95% of their goals. The simulator claims to provide both recorded logs and RL agents for the environment.

The authors introduced certain metrics to evaluate the simulation speed of GPUDrive in terms of agent steps per second (ASPS), controllable agent steps per second (CASPS) and scene completion time. Compared against other sim engines like Nocturne GPUDrive achieved 25-40x training speedup solving 10 scenarios in less than 15 minutes.

strengths:
- The proposed simulator has the flexibility to handle multiple modalities of sensor data. 

- The authors have implemented ways to reduce the memory footprint due to the large number of agents and observation space using algorithms like Bounding Volume Hierarchy (to exclude certain agent pairs for collision checking) and polyline decimation to approximate the straight polylines.

- The trained agents are claimed to be useful for out-of-distribution tests for the driving agents.

- The authors presented the different simulator features in a comprehensive way.

- The paper shows that the simulator gets the scaling benefits in terms of increased amortized sample efficiency with increasing dataset size. This can be beneficial when dealing with large scale datasets with limited compute.

weaknesses:
- The paper does not provide simple IDM (intelligent driving models) agents that can be sometimes practical to have basic reactivity to the ego-agent.
- The authors mention that the current work is limited in properly utilizing the generated samples for optimal training.

- Just a thought: The implementation is in C++ and it provides a binding interface with Python environments. It would have been nice to have a mono-language (primarily Python based) tool as the model training and other related pipelines are mostly in Python.

questions:
- Were other agents in the scenes like pedestrians and cyclists also controlled? If so, what were the dynamics used to model their behavior if they were not logged?
- Nit: Ethical statement was missing?
- Nit: Can the x-axis in the center plot in Fig 5 be made to a log scale?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731633004074

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Main response

comment:
We would like to thank all the reviewers for their time and constructive feedback. We address each comment below and highlight the $\color{green}{\text{feedback that is directly incorporated in green}}$ and items we are $\color{orange}{\text{actively working on in orange}}$. 

We hope that the reviewers feel we have addressed their questions and welcome further discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose GPUDrive, a novel GPU-accelerated high fidelity simulator designed for multi-agent reinforcement learning in autonomous driving. This high-performance simulation environment allows for efficient training of reinforcement learning agents in complex, multi-agent scenarios. Furthermore, the authors highlight the simulator's ability to enable agents to navigate thousands of scenarios within hours. 

### Strengths
- Novelty: GPUDrive introduces a new approach to autonomous driving simulation that leverages the power of GPUs for high-performance and scalability.
- Practical relevance: The simulator provides a realistic and efficient training environment for autonomous driving, whose efficiacy is clear from experiments on real-world data from Waymo OMD.
- Scalability: GPUDrive can handle a large number of agents and complex scenarios, making it suitable for studying intricate interactions in traffic. 
- Open-source: The availability of the code base and pre-trained agents promotes further research and development in the field.

### Weaknesses

- The evaluation of GPUDrive seems limited to Waymo Open Motion dataset and IPPO algorithm. Further evaluation on a wider range of scenarios and different types of algorithms could strengthen the paper's claims. Additionally, improving the diversity of simulation agents seem to be critical. 

- Empirical comparisons with state-of-the-art simulators other than Nocturne (e.g., Waymax) in terms of simulation speed, realism, and scalability would provide a better context for GPUDrive's performance and capabilities.

 - The computational cost of running high-fidelity simulations in GPUDrive with a large number of agents could be significant. This could limit the accessibility of the simulator for researchers with limited computational resources.

- While the authors provide the code base, providing detailed documentation on how to install, configure, and use the simulator, along with clear explanations of the code and its functionalities, would facilitate wider adoption of GPUDrive.

Decisions to accept / reject: This simulator can accelerate the development and testing of autonomous driving algorithms, as it allows researchers to evaluate the performance of their agents in a wide range of situations efficiently. Despite weaknesses pointed above, this work would be a useful addition to the conference. As such, I'm recommending conditional acceptance assuming the authors would address the above weaknesses.