GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS

## Abstract

Abstract Multi-agent learning algorithms have been successful at generating superhuman planning in various games but have had limited impact on the design of deployed multi-agent planners. A key bottleneck in applying these techniques to multi-agent planning is that they require billions of steps of experience. To enable the study of multi-agent planning at scale, we present GPUDrive, a GPU-accelerated, multi-agent simulator built on top of the Madrona Game Engine that can generate over a million simulation steps per second. Observation, reward, and dynamics functions are written directly in C++, allowing users to define complex, heterogeneous agent behaviors that are lowered to high-performance CUDA. We show that using GPUDrive we can effectively train reinforcement learning agents over many scenes in the Waymo Open Motion Dataset, yielding highly effective goal-reaching agents in minutes for individual scenes and enabling agents to navigate thousands of scenarios within hours. The code base with pre-trained agents is available at https://github.com/Emerge-Lab/gpudrive .

## 1 Introduction

Multi-agent learning has been impactful across a wide range of fully cooperative and zero-sum games (Cui et al., ; Wurman et al., ; Pérolat et al., ; Silver et al., ; Jaderberg et al., ; Bakhtin et al., ). However, its impact on multi-agent planning for settings that mix humans and robots has been muted. In contrast to the ubiquity of multi-agent learning-based agents in zero-sum games, multi-agent planners for most practical robotic systems are not derived from the output of game-theoretically sound learning algorithms. While it is hard to characterize the space of deployed planners since many of them are proprietary, the majority likely use a mixture of collected data for the prediction of human motion and hand-tuned costs. These are then fed into a cost-based trajectory optimizer or into a planner based on imitation learning (Bronstein et al., ; Lu et al., ). This approach has been highly effective in scaling up real-world autonomy but can struggle with reasoning about long-term behavior, contingency planning, and interaction with humans in rare, complex scenarios.

The divergence in preferred technique between these two domains is partially the outcome of two distinct, challenging components of real-world multi-agent planning. First, unlike zero-sum games, it is necessary to play a human-compatible strategy that is difficult to identify without data. Second, generating the billions of samples needed for multi-agent learning algorithms is difficult with existing simulators. The former challenge is difficult for multi-agent learning since there is not a clear equilibrium concept that algorithms should be pursuing. The latter problem is a challenge for simulators since it is difficult to simulate embodied multi-agent environments at appropriately high rates.

To address these challenges and unlock multi-agent learning as a tool for generating capable self-driving planners, we introduce GPUDrive. GPUDrive is a simulator intended to mix real-world driving data with simulation speeds that enable the application of sample-inefficient but effective RL algorithms to planner design. GPUDrive runs at over a million steps per second on both consumer-grade and datacenter-class GPUs and has a sufficiently light memory footprint to support hundreds to thousands of simultaneous worlds (environments) with hundreds of agents per world. GPUDrive supports the simulation of a variety of sensor modalities, from LIDAR to a human-like view cone, enabling GPUDrive to be used for studying the effects of different sensor types on resultant agent characteristics. Finally, GPUDrive takes in driving logs and maps from existing self-driving datasets, enabling the mixing of tools from imitation learning with reinforcement learning algorithms. This enables the study of both the development of autonomous vehicles and the learning of models of human driving, cycling, and walking behavior.

Our contributions are:

- •
We provide a multi-agent, GPU-accelerated, and data-driven simulator that runs at over a million steps per second (Section 4.1). Our simulator provides a testbed for:
1.
Investigating the capability of learning algorithms to solve challenges related to self-play or autonomous coordination.
2.
Researching the effects of limited or human-like perception on agent behavior.
- •
We provide gymnasium (Towers et al., ) environments in both torch and jax that can be easily configured with standard open-source multi-agent RL and imitation learning libraries. Additionally, we have open-sourced a basic policy-gradient training loop that can be readily used to develop agents.
- •
We release implementations of tuned RL algorithms that can process $30$ million steps of experience per hour on consumer-grade GPUs. These can be used to train $95$% goal-reaching agents across 1000 different multi-agent scenarios in 15 hours on relatively accessible hardware.
- •
We open-source these strong driving baseline agents that achieve 95% of their goals on a subset of scenes. These are integrated into the simulator so that the simulator comes with default, reactive agents.

## 2 Related work

#### Frameworks for batched simulators.

There are various open-source frameworks available that support hardware-accelerated reinforcement learning environments. These environments are generally written directly in an acceleration framework such as Numpy (Harris et al., ), Jax (Bradbury et al., ), or Pytorch (Ansel et al., ).
In terms of multi-agent accelerated environments, standard benchmarks include JaxMARL(Rutherford et al., ), Jumanji (Bonnet et al., ), and VMAS (Bettini et al., ) which primarily feature fully cooperative or fully competitive tasks. Each benchmark requires the design of custom accelerated structures per environment.
In contrast, GPUDrive focuses on a mixed motive setting and is built atop Madrona, an extensible ECS-based framework in C++, enabling GPU acceleration and parallelization across environments (Shacklett et al., ).
Madrona comes with vectorization of key components of embodied simulation such as collision checking and sensors such as LIDAR. GPUDrive can support hundreds of controllable agents in more than 100,000 distinct scenarios, offering a distinct generalization challenge and scale relative to existing benchmarks. Moreover, GPUDrive includes a large dataset of human demonstrations, enabling imitation learning, inverse RL, and combined IL-RL approaches.

#### Simulators for autonomous driving research and development.

Table 1
shows an overview of current simulators used in autonomous driving research. The purpose of GPUDrive is to facilitate the systematic study of behavioral, coordination, and control aspects of autonomous driving and multi-agent learning more broadly. As such, visual complexity is reduced, which differs from several existing simulators, which (partially) focus on perception challenges in driving (Dosovitskiy et al., ; Cai et al., ). Driving simulators close to GPUDrive in terms of either features or speed include MetaDrive (Li et al., ), nuPlan (Caesar et al., ), Nocturne (Vinitsky et al., ), and Waymax (Gulino et al., ) which all utilize real-world data. Unlike MetaDrive and nuPlan, our simulator is GPU-accelerated. Like GPUDrive, Waymax is a JAX-based GPU-accelerated simulator that achieves high throughput through JIT compilation and efficient use of accelerators. With respect to Waymax, our simulator supports a wider range of possible sensor modalities (Section  3.2) including LIDAR and human-like views, can scale to nearly thirty times more worlds (see Section  4.1), and comes with performant reinforcement learning baselines. However, it does not currently come with reactive IDM agents like Waymax though it does come with pre-trained simulated agents based on RL policies.

#### Driving agents in simulators and algorithms.

Existing simulators often feature baseline agents for interaction, such as low-dimensional car following models that describe vehicle dynamics through a limited set of variables or parameters (Kreutz & Eggert, ; Kesting et al., ; Treiber et al., ). Rule-based agents exhibit predetermined behaviors, like car-following agents (Gulino et al., ; Caesar et al., ; Lopez et al., ; Casas et al., ) such as the IDM model, or parameterized behavior agents like CARLA’s TrafficManager (Dosovitskiy et al., ). Some simulators offer recorded human driving logs for interaction through replaying the human driving logs (Lu et al., ; Vinitsky et al., ; Gulino et al., ; Caesar et al., ). Additionally, certain simulators provide learning-based agents, leveraging reinforcement learning techniques (Li et al., ). In GPUDrive, we provide both human driving logs and high-performing reinforcement learning agents.

**Table 1: Comparison of GPUDrive to related driving simulators. Columns represent whether the simulator supports multi-agent simulation, GPU acceleration, simulation of sensors such as LIDAR or human views, is built atop data, comes with existing driver models, and whether the agents are provided explicit goal points or waypoints along the way to the goal.**
| Simulator | Multi-agent | GPU-Accel | Sensor Sim | Expert Data | Sim-agents | Routes / Goals |
| --- | --- | --- | --- | --- | --- | --- |
| TORCS (Wymann et al., ) |  |  | ✓ |  | ✓ | - |
| GTA V (Martinez et al., ) |  |  | ✓ |  |  | - |
| CARLA (Dosovitskiy et al., ) |  |  | ✓ |  | ✓ | Waypoints |
| Highway-env (Leurent et al., ) |  |  |  |  |  | - |
| Sim4CV (Müller et al., ) |  |  | ✓ |  |  | Directions |
| SUMMIT (Cai et al., ) | ✓  ($\geq 400$) |  | ✓ |  | ✓ | - |
| MACAD (Palanisamy, ) | ✓ |  | ✓ |  | ✓ | Goal point |
| SMARTS (Zhou et al., ) | ✓ |  |  |  |  | Waypoints |
| MADRaS (Santara et al., ) | ✓  ($\geq 10$) |  | ✓ |  |  | Goal point |
| DriverGym (Kothari et al., ) |  |  |  | ✓ | ✓ | - |
| VISTA (Amini et al., ) | ✓ |  | ✓ | ✓ |  | - |
| nuPlan (Caesar et al., ) |  |  | ✓ | ✓ | ✓ | Waypoints |
| Nocturne (Vinitsky et al., ) | ✓  ($\geq 128$) |  |  | ✓ | ✓ | Goal point |
| MetaDrive (Li et al., ) | ✓ |  | ✓ | ✓ | ✓ | - |
| InterSim (Sun et al., ) | ✓ |  |  | ✓ | ✓ | Goal point |
| TorchDriveSim (Ścibior et al., ) | ✓ | ✓ |  |  | ✓ | - |
| BITS (Xu et al., ) | ✓ |  |  | ✓ | ✓ | Goal point |
| Waymax (Gulino et al., ) | ✓  ($\geq 128$) | ✓ |  | ✓ | ✓ | Waypoints |
| GPUDrive (ours) | ✓  ($\geq 128$) | ✓ | ✓ | ✓ | ✓ | Goal point |

## 3 Simulation Design

### 3.1 Simulation Engine

Learning to safely navigate complex scenarios in a multi-agent setting requires generating many billions of environment samples. To feed sample-hungry learning algorithms, GPUDrive is built on top of Madrona (Shacklett et al., ), an Entity-Component-State system designed for high-throughput reinforcement learning environments. In the Madrona framework, multiple independent worlds (each containing an independent number of agents(^†^††In the Waymo Open Motion Dataset, an agent constitutes a vehicle, cyclist, or pedestrian.) ) are executed in parallel on an accelerator via a shared engine.

However, the simulation of driving poses a unique set of challenges which require careful technical design choices to solve. First, road objects, such as road edges and lane lines are frequently represented as polylines (i.e. connected sets of points). These polylines can consist of hundreds of points as they are sampled at every $0.1$ meters, leading to even small maps having upwards of tens of thousands of points. This can blow up the memory requirements of each world as well as lead to significant redundancy in agent observations. Second, the large numbers of agents and road objects can make collision checking a throughput bottleneck. Finally, there is immense variability in the number of agents and road objects in a particular scene. Each world allocates memory to data structures that track its state and accelerate simulation code. Though independent, each world incurs a memory footprint proportional to the maximum number of agents across all worlds. In this way, the performance of GPUDrive is sensitive to the variation in agent counts across all the worlds in a batch.

These challenges are partially resolved via the following mechanisms. First, a primary acceleration data structure leveraged by GPUDrive is a Bounding Volume Hierarchy (BVH). The BVH keeps track of all physics entities and is used to easily exclude candidate pairs for collisions. This allows us to then run a reduced-size collision check on potential collision candidate pairs. The use of a BVH avoids invoking a collision check that would otherwise always be quadratic in the number of agents in a world. Secondly, we observed that a lot of the lines in the geometry of the roads are straight. This allows us to omit many intermediate points while only suffering a minor hit in the quality of the curves. We apply a polyline decimation algorithm (Visvalingham-Whyatt Algorithm) (Visvalingam & Whyatt, ) to approximate straight lines and filter out low-importance points in the polylines. With this modification, we can reduce the number of points by 10-15 times and significantly improve the step times while decreasing memory usage. Finally, rather than allocate memory for the maximum number of agents in a scene (as is likely necessary in frameworks like Jax), we only allocate memory equal to the actual number of instantiated agents.

### 3.2 Simulator features

We provide an overview of some of the pertinent simulator features as well as sharp edges and limitations of the simulator as a guide to potential users.

Figure: Figure 1: Four example scenarios from the Waymo Open Motion Dataset rendered in GPUDrive. The blue circles indicate vehicle destinations.
Refer to caption: x1.png

#### Dataset.

GPUDrive represents its map as a series of polylines and does not require a connectivity map of the lanes. As such, it can be made compatible with most driving datasets given the pre-processing of the roads into the polyline format. Currently, GPUDrive supports the Waymo Open Motion Dataset (WOMD)  (Ettinger et al., ) which is available under a non-commercial license. We show four representative example scenarios from the WOMD in Figure 1. The WOMD consists of a set of over 100,000 multi-agent traffic scenarios, each of which contains the following key elements: 1) Road map - the layout and structure of a road, such as a highway or parking garage. 2) Logged human trajectories from vehicles, cyclists, and pedestrians. 3) Road objects, such as stop signs and crosswalks. Figure 2 depicts an example of an intersection traffic scenario as rendered in GPUDrive.

#### Sensor modalities.

GPUDrive supports a variety of observation spaces intended to enable heterogeneous types of agents. Fig. 2 depicts the three types of supported state spaces. The first mode is somewhat unphysical in which all agents and road objects within a fixed radius are observable to the agent. This mode is intended primarily for debugging and quick testing, enabling a user to minimize the amount of partial observability in the environment. The other two modes are based on a GPU-accelerated LIDAR scan, representing what an autonomous vehicle would be able to see and what a human would likely be able to see respectively. Both modes are based on casting LIDAR rays; to model human vision we simply restrict the LIDAR rays to emanate in a smaller, controllable-sized cone that can be rotated through an action corresponding to head rotation. Note that since all objects are represented as bounding boxes of fixed height, the LIDAR observations are over-conservative as in reality it is frequently possible to see over the hoods of cars as their height is lower than the body of the rest of the car.

Figure: Figure 2: Visualization of different observation spaces available in GPUDrive. The top scene is an example scenario from the dataset, rendered from the ego-centric perspective of the red vehicle. Grey cars are parked cars while white cars are other controlled agents. From left to right: the Radius Filter returns all objects within 100 meters, the LIDAR observation with 3000 rays spread around 360 degrees, and a view cone consisting of 3000 rays emanating in a 120-degree view cone.
Refer to caption: extracted/5899962/Figures/obs_fig_v2.png

#### Agent dynamics.

By default, agents are stepped using a standard Ackermann bicycle model (Details in Appendix B) with actions corresponding to steering and acceleration. This model enables the dynamics of objects to be affected by their length, creating different dynamics for small cars as opposed to large cars like trucks. However, this model is not fully invertible which can make it challenging to use as a model for imitation learning. To enable full invertibility for imitation learning, we also support the simplified bicycle model, taken from Waymax (Gulino et al., ), which is a double-integrator in the position and velocity and updates its yaw as:

$$ $\theta_{t+1}=\theta+s_{t}(v_{t}\Delta t+\frac{1}{2}a_{t}\Delta t^{2})$ $$

where $\theta$ is the yaw, $s$ is the steering command, $v$ is the velocity, and $a$ is the acceleration at time $t$ respectively. $\Delta t$ is the timestep.
This model is always invertible given an unbounded set of steering and acceleration actions but is independent of the vehicle length. See the appendix for full details on the models.

Note that this model does not factor in the length of the car, causing both long and short objects to have identical dynamics. However, computing the expert actions and then using them to mimic the expert trajectory under this model leads to lower tracking error than the default bicycle model.

#### Rewards.

All agents are given a target goal to reach; this goal is selected by taking the last recorded position in the vehicle’s logged trajectory. A goal is reached when agents are within some configurable distance $\delta$ of the goal. By default, agents in GPUDrive receive a reward of $1$ for achieving their goal and otherwise receive a reward of 0 0. There are additional configurable collision penalties or other rewards based on agent-vehicle distances or agent-road distances though these are not used in the experiments reported in this work.

#### Termination conditions.

We terminate an agent’s episode when they achieve their goal position or collide. As it is not clear where an agent should go next after it reaches its goal, we simply remove it from the scene afterwards (we note that an alternative might be to generate a new goal for the agent to drive to). Car and cyclist agents are considered to be in collision when they drive through a road edge while pedestrian agents are allowed to cross road edges.

#### Environment Interface.

As GPUDrive is implemented in C++, we provide a Pythonic interface through nanobind (Jakob, ). We create environments for both torch and jax that conform to the Gymnasium API (Towers et al., ) so users can use the simulator entirely through Python if they prefer.

#### Available driving simulation agents.

We use reinforcement learning to train a set of agents that reach their goals 95 % of the time on a subset of $1000$ training scenes. While this number is far below the capability of human drivers, these agents are reactive in a distinct fashion from parametrized driver models in other simulators. In particular, many logged-data simulators construct reactivity by having the driver follow along its logged trajectory but decelerate if an agent passes in front of it. In contrast, these agents can maneuver and negotiate without remaining constrained to a logged trajectory. These trained agents are extremely aggressive about reaching their goals and can be used as an out-of-distribution test for proposed driving agents. The training procedure and more details can be found in Section  4.2.

#### Simulator sharp-edges.

We note the following limitations of the benchmark:

- •
Absence of a map. The current version of the simulator does not have a well-defined notion of lanes or a higher-level road map which makes it challenging for algorithmic approaches that require maps. The absence of this feature also makes it challenging to define rewards such as ”stay lane-centered.”
- •
Convex objects only. Collision checking relies on the objects being represented as convex objects.
- •
Unsolvable goals. Due to incorrect labels of some road points in the Waymo dataset, such as an exit to a parking lot being labelled as an impassable road edge, some agent goals (roughly 2%) are unreachable. For these agents, we default them to simply replaying their logged trajectory and do not treat them as agents.
- •
Variance in controllable agents per scenario. In most scenes, a significant fraction (25-75%) of the agents are not controllable because they do not move (e.g. parked cars). This is because the dataset is collected from an autonomous vehicle’s sensors, so some agents’ initial states are recorded after the first time-step of the simulator. These agents are not included to avoid collisions with other agents.

## 4 Simulator performance

The following Sections describe the simulator speed. Section 4.1 first shows the raw simulator speed and peak goodput (throughput achieved by the number valid number of agents in a scene). Section 4.2 then investigates the impact on reinforcement learning workflows by evaluating the time it takes to train reinforcement learning agents through Independent PPO (IPPO) (Yu et al., ), a widely used multi-agent learning algorithm. IPPO is implemented atop Stable Baselines 3 (Hill et al., ).

### 4.1 Simulation speed

Since scenarios contain a variable number of agents, we introduce a metric called Agent Steps Per Second (ASPS) to measure the sample throughput of the simulator. We define the ASPS as the total number of agents across all worlds in a batch that can be fully stepped in a second:

$$ $\displaystyle\text{ASPS}=\frac{S\times\sum_{k=1}^{N}\left|A_{k}\right|}{\Delta T}$ (1) $$

where $A_{k}$ is the set of agents in the $k^{th}$ world, $S$ is the number of steps taken, and $\Delta T$ is the number of seconds elapsed. Figure 3 examines the scaling of the simulator as the number of simulated worlds, which represents the amount of parallelism, increases. To measure performance, we sample random batches of scenarios of size equal to the number of worlds, so that every world is a unique scenario with $K$ agents. On the left-hand side of Figure 3, we compare the performance of GPUDrive to Nocturne (Vinitsky et al., ) (CPU, no parallelism), a CPU-accelerated version of Nocturne via Pufferlib (16 CPU cores) (Suarez, ) and Waymax Gulino et al. (). Empirically, the maximum achievable ASPS of Nocturne is 15,000 (blue dotted line) though we note that additional speedups may be possible. GPUDrive achieves a peak ASPS of 2.3 million steps, which is 2 to 3 orders of magnitude faster compared to Nocturne. This performance also surpasses that of Waymax (Gulino et al., ), a JAX-based simulator, where we could not run more than 16 environments in parallel due to Out of Memory (OOM) issues.

Figure: Figure 3: Peak goodput of GPUDrive on a consumer-grade and datacenter-class GPU compared to original, CPU and GPU-based, implementations using the radial filter observation. Left: The total number of agent steps per second (ASPS) is the number of objects for which our system computes observations at each time step. To ensure a fair comparison, we align the conditions with those used in (Gulino et al., ), where all cars, bicyclists, and pedestrians are considered valid experience-generating agents. Center: The distribution of controllable agents across 512 scenarios in the Waymo Open Motion Dataset $\mu\approx 10.8\,(\text{red line}),\sigma\approx 9.3$). Right: The total number of controllable agent steps per second (CASPS) as we increase the number of worlds (parallelism).
Refer to caption: x2.png

In addition to the ASPS, we report another metric to indicate the number of controllable agents that are stepped per second. In the Waymo Open Motion Dataset, each scenario contains a varying number of moving agents (Examples in Figure 1). By default, our system only classifies something as a controllable agent if its movement is necessary to achieve the goal. Therefore, parked cars throughout the episode are not considered controllable agents. To illustrate what this looks like, we plot the distribution of controllable agents across a subset of scenes in Figure 3.

Considering the variance of controllable agents per scene, the right-hand side of Figure 3 depicts the Controlled Agent Steps Per Second (CASPS). CASPS reflects the expected performance of our system when utilizing the Waymo Open Motion Dataset with a randomly sampled subset of scenarios. Due to the significant variability in agents throughout the dataset, the highest CASPS is notably lower than the ASPS, at around 200,000. Note that this can be improved by strategically selecting scenes, such as filtering for dense scenes with a high number of controllable agents.

Figure: Figure 4: ASPS and CASPS comparison between Radial Filter and LiDAR observation types. The radial filter is slower than lidar due to its linear scan of nearby objects. In contrast, the LiDAR observation type is GPU-accelerated, delivering significantly enhanced performance. As demonstrated in the plots, depending on the scene selection, the LiDAR can achieve a speedup of approximately 3x over the radial filter.
Refer to caption: x3.png

Lastly, we demonstrate the speed of different observation spaces in Figure 4. We observe that using LiDAR is approximately three times faster than using the radial filter observation (Details about the supported observation spaces are found in Section 3.2).

### 4.2 End-to-end speed and performance

The purpose of GPUDrive is to facilitate research and development in multi-agent algorithms by 1) reducing the completion time of experiments, and 2) enabling academic research labs to achieve scale on a limited computing budget. Ultimately, we are interested in the rate at which a machine learning researcher or practitioner can iterate on ideas using GPUDrive. This section highlights what our simulator enables in this regard by studying the end-to-end process of learning policies in our simulator.

Figure 5 contrasts the number of steps (experience) and the corresponding time required to solve 10 scenarios from the WOMD between Nocturne (Vinitsky et al., ) and GPUDrive. We compare Nocturne to GPUDrive, as it is the closest simulator in functionality, except that it is implemented on the CPU. For benchmarking purposes, we say a scene is solved when agents can navigate to their designated target position $95\%$ of the time without colliding or going off-road. Ceteris paribus (Details in Appendix D), GPUDrive achieves a 30-40x training speedup, solving 10 scenarios in less than 15 minutes compared to approximately 10 hours in Nocturne.

Figure: Figure 5: From hours to minutes. Left: Training performance (goal-reaching rate) as a function of the global step (AFPS). Center: Training performance as a function of wall-clock time. Right: Comparison of the total time to solve the same 10 scenarios while replicating environmental and experimental conditions as closely as possible. Runs are averaged across three seeds, see Appendix D for the hyperparameters and training details. The green dotted line indicates optimal performance.
Refer to caption: x4.png

As shown in Fig. 5, GPUDrive allows us to solve scenes in minutes. Next, we investigate how the individual scene completion time, the time it takes to solve a single scenario, changes as we increase the total number of scenarios we train on. In practice, it may be desirable to train agents on thousands of scenarios. Therefore, we ask whether it is feasible to fully leverage the simulator’s capabilities with a single GPU.

Interestingly, we find that the amortized sample efficiency increases with the size dataset of scenes we train in. Figure 6 shows the average completion time per scenario as we increase the dataset. For instance, using IPPO with 32 scenarios takes 2 minutes per scenario. In contrast, solving 1024 unique scenarios takes about 200 minutes, which amounts to only 15 seconds per scenario. We expect that these scaling benefits will continue as we further increase the size of the training dataset. This suggests that GPUDrive should enable effective utilization of the large WOMD dataset comprising 100,000 diverse traffic scenarios, even with limited computational resources.

Figure: Figure 6: Scale reduces individual scene completion time. Left: Total time required to solve a fixed number of scenarios to a goal-reaching rate of 95%. Note that time-to-completion is sub-linear concerning the number of scenes. Right: Each additional scenario costs less to solve than the previous scenario. At 1024 scenes, the per-scene cost of solving an additional scene is on the order of 15 seconds.
Refer to caption: x5.png

### 4.3 Measuring remaining unsolved tasks

The Waymo Open Motion Dataset contains a total of 103,354 traffic scenarios (Ettinger et al., ). In this paper, we demonstrate that an agent can achieve 95% performance on a subset of 1000 scenes after 15 hours of training. Additionally, we show that the time required to solve scenarios decreases as the dataset size increases.
Our analysis of the failure rates in the current best-performing policy suggests that the goal-reaching limit is around 98 %, due to mislabeled road edges in the dataset, which render some goals unreachable because they are located beyond uncrossable roads. An important direction for future work is developing agents that achieve near-perfect performance, with a goal-reaching rate approaching 100% and a 0% collision rate while addressing the limitations posed by data inaccuracies in the benchmark.

## 5 Conclusion

In this work, we present GPUDrive, a GPU-accelerated, data-driven simulator. GPUDrive is intended to help generate the billions of samples that are likely needed to achieve effective reinforcement learning for multi-agent driving planners. By building atop the Madrona Engine (Shacklett et al., ), we can scale GPUDrive to hundreds of worlds with potentially thousands of agents leading to throughput of millions of steps per second. This throughput occurs while synthesizing complex observations such as LiDAR. We show that this throughput has consequent implications for training reinforcement learning agents, leading to the ability to train agents to solve any particular scene in minutes and in seconds when amortized across many scenes. We release the simulator and integrated trained agents to enable further research.

This paper is a first step in scaling up reinforcement learning for multi-agent planning in safety-critical, mixed human-autonomous settings. However, several important challenges remain for future work. Firstly, we have not yet identified the optimal hyperparameter settings to effectively utilize the collected data, resulting in training being the bottleneck instead of data collection time. Secondly, while the simulator is fast, collecting data for reinforcement learning leads to frequent reset calls that significantly impact throughput. Lastly, training fully human-level drivers in the simulator to navigate without crashing in any scenario remains an unresolved challenge. We have shown that our simulator allows training of driving agents to achieve a 95% success rate in a few hours. However, this still does not match human capabilities or the safety standards we strive for. We hope GPUDrive will help the community tackle this open challenge.

## Appendix A Reproducibility

### A.1 Code Reproducibility

All code required to reproduce the paper is open-sourced at
[https://github.com/Emerge-Lab/gpudrive](https://github.com/Emerge-Lab/gpudrive) under release number v0.2.0.

### A.2 Computational Resources

All RL experiments in this paper were run on an NVIDIA RTX 8000 or A100. Total resources for the paper correspond to less than 2 GPU days.

## Appendix B Dynamics Model

Agents are driven by a kinematic bicycle model (Rajamani, ) which uses the center of gravity as reference point. The dynamics are as follows. Here $(x_{t},y_{t})$ stands for the coordinate of the vehicle’s position at time $t$, $\theta_{t}$ stands for the vehicle’s heading at time $t$, $v_{t}$ stands for the vehicle’s speed at time $t$, $a$ stands for the vehicle’s acceleration and $\delta$ stands for the vehicle’s steering angle. $L$ is the distance from the front axle to the rear axle (in this case, just the length of the car) and $l_{r}$ is the distance from the center of gravity to the rear axle. Here we assume $l_{r}=0.5L$.

$$ $\displaystyle\dot{v}$ $\displaystyle=a$ $\displaystyle\bar{v}$ $\displaystyle=\text{clip}(v_{t}+0.5\ \dot{v}\ \Delta t,-v_{\text{max}},v_{ \text{max}})$ $\displaystyle\beta$ $\displaystyle=\tan^{-1}\left(\frac{l_{r}\ \tan(\delta)}{L}\right)$ $\displaystyle=\tan^{-1}(0.5\ \tan(\delta))$ $\displaystyle\dot{x}$ $\displaystyle=\bar{v}\ cos(\theta+\beta)$ $\displaystyle\dot{y}$ $\displaystyle=\bar{v}\ sin(\theta+\beta)$ $\displaystyle\dot{\theta}$ $\displaystyle=\frac{\bar{v}\ \cos(\beta)\ \tan(\delta)}{L}$ $$

We then step the dynamics as follows:

$$ $\displaystyle x_{t+1}$ $\displaystyle=x_{t}+\dot{x}\ \Delta t$ $\displaystyle y_{t+1}$ $\displaystyle=y_{t}+\dot{y}\ \Delta t$ $\displaystyle\theta_{t+1}$ $\displaystyle=\theta_{t}+\dot{\theta}\ \Delta t$ $\displaystyle v_{t+1}$ $\displaystyle=\text{clip}(v_{t}+\dot{v}\ \Delta t,-v_{\text{max}},v_{\text{max }})$ $$

## Appendix C License Details and Accessibility

Our code is released under an MIT License. The Waymo Motion dataset is released under a Apache License 2.0. The code is available at [<ANONYMIZED-DURING-REVIEW>](<ANONYMIZED-DURING-REVIEW>).

## Appendix D Training details

### D.1 End-to-end performance

The Table below depicts the hyperparameters used to produce the results in Section 4.2.

**Table 2: Experiment hyperparameters used for comparing the training runs between Nocturne and GPUDrive in Figure 5. The environment configurations are aligned as closely as possible, using the same observations and field of view. The dataset includes the same 10 scenarios. It’s important to note that the length of the GPUDrive rollout is approximately equal to the number of worlds multiplied by the rollout length and then multiplied by the number of controllable agents. We have set this value to be $92\times 50\approx 4600$ to approximately match the rollout length in Nocturne.**
| Parameter | IPPO GPUDrive | IPPO Nocturne |
| --- | --- | --- |
| $\gamma$ | 0.99 | 0.99 |
| $\lambda_{\text{GAE}}$ | 0.95 | 0.95 |
| PPO rollout length | 92 | 4096 |
| PPO epochs | 5 | 5 |
| PPO mini-batch size | 2048 | 2048 |
| PPO clip range | 0.2 | 0.2 |
| Adam learning rate | 3e-4 | 3e-4 |
| Adam $\epsilon$ | 1e-5 | 1e-5 |
| normalize advantage | yes | yes |
| entropy bonus coefficient | 0.001 | 0.001 |
| value loss coefficient | 0.5 | 0.5 |
| seeds | 42, 12, 67 | 42, 12, 67 |
| number of worlds | 50 | 1 |

Figure: Goal achievement rate
Refer to caption: extracted/5899962/Figures/perc_goal_rate.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Were other agents in the scenes like pedestrians and cyclists also controlled? If so, what were the dynamics used to model their behavior if they were not logged?
- Nit: Ethical statement was missing?
- Nit: Can the x-axis in the center plot in Fig 5 be made to a log scale?
soundness: 3
strengths: - The proposed simulator has the flexibility to handle multiple modalities of sensor data. 

- The authors have implemented ways to reduce the memory footprint due to the large number of agents and observation space using algorithms like Bounding Volume Hierarchy (to exclude certain agent pairs for collision checking) and polyline decimation to approximate the straight polylines.

- The trained agents are claimed to be useful for out-of-distribution tests for the driving agents.

- The authors presented the different simulator features in a comprehensive way.

- The paper shows that the simulator gets the scaling benefits in terms of increased amortized sample efficiency with increasing dataset size. This can be beneficial when dealing with large scale datasets with limited compute.
summary: The paper presents a GPU accelerated simulator that can generate millions of simulation steps samples per second that can be used to train multi-agent reinforcement learning (RL) algorithms. The simulator is claimed to simulate hundreds to thousands of scenarios/scenes in parallel with each scene containing thousands of agents.

The simulator is built on top of the Madrona Game Engine and is written in C++. The C++ simulator engine can also be interfaced with learning environments written in JAX and Torch.
The authors have released implementations of RL algorithms capable of processing millions of agent steps per second and some baseline agents trained on these algorithms that achieve 95% of their goals. The simulator claims to provide both recorded logs and RL agents for the environment.

The authors introduced certain metrics to evaluate the simulation speed of GPUDrive in terms of agent steps per second (ASPS), controllable agent steps per second (CASPS) and scene completion time. Compared against other sim engines like Nocturne GPUDrive achieved 25-40x training speedup solving 10 scenarios in less than 15 minutes.
weaknesses: - The paper does not provide simple IDM (intelligent driving models) agents that can be sometimes practical to have basic reactivity to the ego-agent.
- The authors mention that the current work is limited in properly utilizing the generated samples for optimal training.

- Just a thought: The implementation is in C++ and it provides a binding interface with Python environments. It would have been nice to have a mono-language (primarily Python based) tool as the model training and other related pipelines are mostly in Python.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Benchmarks consist of limitations because of the dataset. Can it be addressed by using another dataset ?
2. Stable baselines is not known that well for speed. Could other implementations of PPO have been used ?
3. Is there support for multi GPUs ? And if they do exist, an ablation or benchmark would be great for that
soundness: 4
strengths: 1. A multi agent simulator accelerated on the GPU iteration of over a million steps per second.
2. Very well written and structured code to run any experiment easily with a lot of easy experimentation code readily available. 
3. Extensive results analyzing the sampling frequency of the simulation.
summary: GPU drive introduces a fast multi agent simulator build using C++ that helps you run complex scenarios especially related to self driving cars at scale built on top of the Waymo Open Motion dataset. This allows iterating on these scenarios quicker reaching greater than a million FPS thus allowing more experimentation runs and iterating/trying out different scenarios even on desktop grade GPU's.
weaknesses: 1. Figure 2 needs a better caption and an explanation
2. Designed to fit one exact dataset. A section explaining the effort required to integrate other datasets is desirable.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - How easily can the simulator be updated to efficiently provide map-like utilities that allow for lane-keeping rewards (re mentioned limitations)?
- Do you support loading multiple different polices for individual agents? Could they have different sampling rates? How would these aspects affect efficiency?
- How do traffic jams affect throughput (re BVH)? This could be an interesting experiment to add.
- In video scene_53.mp4, agent 4 displays rather jerky behavior when moving towards its goal - could you elaborate on the underlying reasons?
- In video scene_43.mp4, agents 1 and 10 seemingly disappear without reaching their goals - could you elaborate on this behavior?
soundness: 3
strengths: - The proposed simulator improves over alternatives in terms of sample efficiency. All of the design choices appear reasonable, while the underlying source code with pre-trained driving baselines will be released.
- The ability to load real-world driving datasets is extremely useful, while providing a variety of observation spaces is a great feature.
- Transparency about current limitation of the benchmark are very helpful for user adaptation.
summary: This paper proposes GPUDrive, a GPU-accelerated multi-agent driving simulator designed to increase efficiency of learning-based systems. The simulator allows for loading expert trajectories from real-world driving datasets, can support multiple observation spaces (including e.g. LiDAR), and displays favorable throughput compared to other openly accessible simulators. The simulator, together with pre-trained goal-conditioned policies, is made openly available with accessible pythonic interfaces.
weaknesses: - While the focus of this paper is on providing a novel simulator, it would be very interesting to see some more complex behavior over longer time-horizons to fully capture the capabilities unlocked by the simulator (e.g. training a single agent policy with higher velocity limit to weave through a simulated traffic scene, etc.)
- Showcasing such behavior would likely require addressing the “Absence of a map” limitation raised in the paper, in order to formulate more sophisticated reward function. An important question would then be how easily this could be integrated, and how much the absence of such a feature could hurt adaptation of the simulator.
- The discussion of batched simulators could be extended to include references [1-3], where [1] has driven many results in single-agent robot learning, while [3] considers heterogenous multi-agent settings
- Figure 3 mentions performance on an RTX 4080, while line 711 states RTX 8000
- Line 308: “number valid number”

**References**

[1] V. Makoviychuk, L. Wawrzyniak, Y, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. “Isaac gym: High performance gpu-based physics simulation for robot learning.” NeurIPS, 2021.

[2] J. Panerati, H. Zheng, S. Zhou, J. Xu, A. Prorok, and A. P. Schoellig. "Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control.” IROS, 2021.

[3] M. Lechner, L. Yin, T. Seyde, T.-H. Johnson Wang, W. Xiao, R. Hasani, J. Rountree, and D. Rus. “Gigastep - one billion steps per second multi-agent reinforcement learning." NeurIPS, 2024.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In Figure 3, the speedup appears nearly linear. However, it would be helpful to examine scaling performance by adding more environments to identify saturation points and gain insights into system limitations.. 
2. What is the scaling of speedup with respect to the number of agents in the environment? e.g., fix the number of environments and scales the number of agents?
3. In Figure 5, do you use the CPU-parallel version of Nocturne?
soundness: 3
strengths: 1. The GPU-based simulation is important in facilitating the training for complex real-world applications, highlighted by recent advances in robotics.
2. The experiment result shows significant wall-clock time speedup again baselines.
summary: This paper presents GPUDrive, a GPU-based simulator for autonomous driving. The simulator is compatible with existing datasets and allows parallel simulations. The experiments show that it can train a policy with 25-40x training speedup against the baseline.
weaknesses: 1. The paper claims compatibility with existing datasets but only demonstrates map loading, leaving other functionalities unclear. For example, the imitation learning experiment or mixing agent behaviors—some from datasets and others from RL agents during training.
2. I believe one major advantage of parallel environments is that it allows you to do randomization across different environments (worlds). However, the paper lacks detail on whether GPUDrive supports this capability.
3. While GPUDrive offers a Python interface, I am curious how easy to customize those key elements in the environment given that the observation, reward, dynamic functions are written in C++. 
4. Experiments only evaluate IPPO, despite the paper claims that it targets at mixed motive setting.

Meta-review: The authors propose GPUDrive, a novel GPU-accelerated high fidelity simulator designed for multi-agent reinforcement learning in autonomous driving. This high-performance simulation environment allows for efficient training of reinforcement learning agents in complex, multi-agent scenarios. Furthermore, the authors highlight the simulator's ability to enable agents to navigate thousands of scenarios within hours. 

### Strengths
- Novelty: GPUDrive introduces a new approach to autonomous driving simulation that leverages the power of GPUs for high-performance and scalability.
- Practical relevance: The simulator provides a realistic and efficient training environment for autonomous driving, whose efficiacy is clear from experiments on real-world data from Waymo OMD.
- Scalability: GPUDrive can handle a large number of agents and complex scenarios, making it suitable for studying intricate interactions in traffic. 
- Open-source: The availability of the code base and pre-trained agents promotes further research and development in the field.

### Weaknesses

- The evaluation of GPUDrive seems limited to Waymo Open Motion dataset and IPPO algorithm. Further evaluation on a wider range of scenarios and different types of algorithms could strengthen the paper's claims. Additionally, improving the diversity of simulation agents seem to be critical. 

- Empirical comparisons with state-of-the-art simulators other than Nocturne (e.g., Waymax) in terms of simulation speed, realism, and scalability would provide a better context for GPUDrive's performance and capabilities.

 - The computational cost of running high-fidelity simulations in GPUDrive with a large number of agents could be significant. This could limit the accessibility of the simulator for researchers with limited computational resources.

- While the authors provide the code base, providing detailed documentation on how to install, configure, and use the simulator, along with clear explanations of the code and its functionalities, would facilitate wider adoption of GPUDrive.

Decisions to accept / reject: This simulator can accelerate the development and testing of autonomous driving algorithms, as it allows researchers to evaluate the performance of their agents in a wide range of situations efficiently. Despite weaknesses pointed above, this work would be a useful addition to the conference. As such, I'm recommending conditional acceptance assuming the authors would address the above weaknesses.

Decision: Accept (Poster)