GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS
Authors: Saman Kazemkhani
Aarav Pandya
Daphne Cornelisse
Brennan Shacklett
Eugene Vinitsky
Venue: ICLR 2025 Poster (2025)
Keywords: Simulation
benchmark
multi-agent reinforcement learning
autonomous vehicles
planning
Primary area: datasets and benchmarks
TL;DR: We propose a GPU-accelerated simulator for extremely-fast multi-agent planning and simulation in autonomous driving.

Abstract: Multi-agent learning algorithms have been successful at generating superhuman planning in various games but have had limited impact on the design of deployed multi-agent planners. A key bottleneck in applying these techniques to multi-agent planning is that they require billions of steps of experience. To enable the study of multi-agent planning at scale, we present GPUDrive, a GPU-accelerated, multi-agent simulator built on top of the Madrona Game Engine capable of generating over a million simulation steps per second. Observation, reward, and dynamics functions are written directly in C++, allowing users to define complex, heterogeneous agent behaviors that are lowered to high-performance CUDA. Despite these low-level optimizations, GPUDrive is fully accessible through Python, offering a seamless and efficient workflow for multi-agent, closed-loop simulation. Using GPUDrive, we train reinforcement learning agents on the Waymo Open Motion Dataset, achieving efficient goal-reaching in minutes and scaling to thousands of scenarios in hours. We open-source the code and pre-trained agents at \url{www.github.com/Emerge-Lab/gpudrive}.

Published as a conference paper at ICLR 2025
GPUDRIVE: DATA-DRIVEN, MULTI-AGENT DRIVING SIMULATION AT MILLION FPS
Saman Kazemkhani"
Aarav Pandya'
Daphne Cornelisse
Brennan Shacklett
Eugene Vinitsky!
ABSTRACT
Multi-agent learning algorithms have been successful at generating superhuman planning in various games but have had limited impact on the design of deployed multi-agent planners.
key bottleneck in applying these techniques to multi- agent planning is that they require billions of steps of experience. To enable the study of multi-agent planning at scale, we present GPUDrive. GPUDrive is a GPU-accelerated, multi-agent simulator built top of the Madrona Game Engine capable of generating over a million simulation steps per second. Ob- servation, reward, and dynamics functions are written directly in C++, allowing users to define complex, heterogeneous agent behaviors that are lowered to high- performance CUDA. Despite these low-level optimizations, GPUDrive is fully accessible through Python, offering a seamless and efficient workflow for multi- agent, closed-loop simulation. Using GPUDrive, we train reinforcement learning agents on the Waymo Open Motion Dataset, achieving efficient goal-reaching in minutes and scaling to thousands of scenarios in hours. We open-source the code and pre-trained agents at
www. gi github...........
com/ wtt.ccomm/e/ec..e Bewergeeaabggpuueeeeeeeeeeeeeeee
INTRODUCTION
Multi-agent learning has been impactful across wide range of fully cooperative and zero-sum games (Cui et al., 2023; Wurman et al., 2022: Pérolat et al., 2022; Silver et al., 2017: Jaderberg et al., 2018; Bakhtin et al., 2023). However, its impact mix humans and robots has been muted. In contrast to the ubiquity of multi-agent learning-bass agents in zero-sum games, multi-agent planners for most practical robotic systems from the output of game-theoretically sound learning algorithms. While it is hard to characterize the space of deployed planners since many of them are proprietary, the majority likely collected data for the prediction of human motion and hand-tuned costs. These are then fed into a cost-based trajectory optimizer or into a planner based on imitation learning (Bronstein al., 2022; Lu et al., 2023). This approach has been highly effective in scaling up real-world autonomy but can struggle with reasoning about long-term behavior, contingency planning, and interaction with humans in rare, complex scenarios.
multi-agent planning for settings that
not derived
mixture of
The divergence in preferred technique between these two domains is partially the outcome of two distinct, challenging components of real-world multi-agent planning. First, unlike zero-sum games, necessary to play a human-compatible strategy that is difficult to identify without data. Sec- ond, generating the billions of samples needed for multi-agent learning algorithms is difficult with existing simulators. The former challenge is difficult for multi-agent learning since there is not a clear equilibrium concept that algorithms should be pursuing. The latter problem is challenge for simulators since it is difficult to simulate embodied multi-agent environments at appropriately high rates.
NYU Tandon School of Engincering
Sttanford University
"These authors contributed equally to this work
Corresponding authors: skazemkhanid gmail.com, pandya.narav.977 @mgaill.ommmmmmmmmmmmmmmm cornelisse @nu.eddu

---

Published as a conference paper at ICLR 2025
Simulator state
Agent view
8
U
Figure 1: Extremely fast multi-agent simulation with GPUDrive. Top: Bird's-eye view Waymo Open Motion Dataset scenarios GPUDrive, with boxes marking controlled agents and circles denoting their goals. Botion: Corresponding agent views, centered on one agent. Observations can be easily configured based on the user's objectives. Here, agents are provided with a scene view through a relative coordinate frame. Shown are nearby road points within configurable radius (set 50 meters) and the relative positions of other agents in the scene.
To address these challenges and unlock multi-agent learning tool for generating capable self- driving planners, we introduce GPUDrive. GPUDrive a simulator intended to mix real-world driving data with simulation speeds that enable the application of sample-inefficient but effective RL. algorithms to the design of autonomous planners. GPUDrive runs at over a million steps per second on both consumer-grade and datacenter-class GPUs and has a sufficiently light memory footprint to support hundreds to thousands of simultaneous worlds (environments) with hundreds of agents per world. GPUDrive supports the simulation of variety sensor modalities, from LIDAR human- like view cone, enabling GPUDrive to be used for studying the effects of different sensor types on resultant agent characteristics. Finally, GPUDriie takes in driving logs and maps from existing self- driving datasets, enabling the mixing of tools from imitation learning with reinforcement learning algorithms. This enables the study both the development of autonomous vehicles and the learning of models of human driving, cycling, and walking behavior.
Our contributions are:
We provide multi-agent, GPU-accelerated, and data-driven simulator that runs at over million steps per second (Section 4.1). Our simulator provides a testbed for:
1. Investigating the capability of learning algorithms solve challenges related self- play or autonomous coordination.
Researching the effects of limited or human-like perception on agent behavior.
We provide gymnas i um (Towers al., 2024) environments in both torch and jax that can be easily configured with standard open-source multi-agent RI and imitation learning libraries. Additionally, we open-sourced two policy-gradien training loops that can be readily used to develop agents.
We release implementations of tuned RL algorithms that can process 30 million steps of ex- perience per hour consumer-grade GPUs. These be used train 95% goal-reaching
2

---

Published as a conference paper at ICLR 2025
agents across 1000 different multi-agent scenarios in 15 hours relatively accessible hard- ware.
We open-source these strong driving baseline agents that achieve 95% of their goals subset of scenes. These are integrated into the simulator so that the simulator comes with default, reactive agents.
RELATED WORK
Frameworks for batched simulators. There are various open-source frameworks available that support hardware-accelerate reinforcement learning environments. These environments are gener- ally written directly in an acceleration framework such as Numpy (Harris et al., 2020), Jax (Bradbury al., 2018), or Pytorch (Ansel al., 2024; Makoviychuk al., 2021; Panerati et al., 2021). In terms of multi-agent accelerated environments, standard benchmarks include LaMMARL.(uuherrfo 2023), Jumanji (Bonnet et al., 2024), VMAS (Bettini al., 2022) Gigastep (Lechner et al., 2024) which primarily feature fully cooperative fully competitive tasks. Each benchmark requires the design of custom accelerated structures per environment. contrast, GPUDrive focuses on a mixed motive setting and is built atop Madrona, extensible ECS-based framework in C++, enabling GPU acceleration and parallelization across environments (Shacklett et al., 2023). Madrona comes with vectorization of key components of embodied simulation such collision checking and sen- sors such as LIDAR. GPUDrive can support hundreds of controllable agents in more than 100,000 distinct scenarios, offering distinct generalization challenge and scale relative existing bench- marks. Moreover, GPUDrive includes large dataset of human demonstrations, enabling imitation learning, inverse RL, and combined IL-RL approaches.
et al.,
Simulators for autonomous driving research and development. Table shows an overview of current simulators used in autonomous driving research. The purpose of GPUDrive is to facilitate the systematic study of behavioral, coordination, and control aspects of autonomous driving and multi- agent learning more broadly. As such, visual complexity is reduced, which differs from several existing simulators, which (partially) focus on perception challenges driving (Dosovitskiy al., 2017; Cai et al., 2020). Driving simulators close to GPUDrive in terms of either features or speed include MetaDrive (Li et al., 2022), nuPlan (Caesar et al., 2021), Nocturne (Vinitsky et al., 2022), and Waymax (Gulino et al., 2024) which all utilize real-world data. Unlike MetaDrive and nuPlan, our simulator is GPU-accelerated. Like GPUDrive, Waymax is a JAX-based GPU-accelerated sim- ulator that achieves high throughput through JIT compilation and efficient of accelerators. With respect to Waymax, our simulator supports a wider range of possible sensor modalities (Section 3.2) including LIDAR and human-like views, can scale nearly thirty times more worlds (see Section 4.1,, and comes with performant reinforcement learning baselines. However, it does not currently come with reactive IDM agents like Waymax though does come with pre-trained simulated agents based on RL policies.
Driving agents in simulators and algorithms. Existing simulators often feature baseline agents for interaction, such as low-dimensional car following models that describe vehicle dynamics through a limited set of variables or parameters (Kreutz & Eggert, 2021; Kesting et al., 2007; Treiber al., 2000). Rule-based agents exhibit predetermined behaviors, like car-following agents (Gulino et al., 2024; Caesar et al., 2021; Lopez et al., 2018; Casas et al., 2010) such as the IDM model, or parameterized behavior agents like CARLA'S TrafficManager (Dosovitskiy et al., 2017). Some simulators offer recorded human driving logs for interaction through replaying the human driving logs (Lu et al., 2023; Vinitsky et al., 2022; Gulino et al., 2024; Caesar et al., 2021). Additionally, certain simulators provide learning saaning-based based agents, leveraging reinforcement learning techniques (Li et al., 2022). In GPUDrive, we provide both human driving logs and tegh-performig reinforcement learning agents.
3

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of GPUDrive to related driving simulators. Columns represent whether the simulator supports multi-agent simulation, GPU acceleration, simulation of sensors such as LIDAR or human views, is built atop data, comes with existing driver models, and whether the agents are provided explicit goal points waypoints along the way to the goal.
Simulator
Multi-agent
GPU-Accel
Sensor Sim
Expert Data
Sim-agents
Routes / Goals
TORCS (Wymann 2000)
SSS <<<
GTA V (Martinez et al., 2017)
CARLA (Dosovilskiy et al,, 2017)
Waypoints
Highway-env (Leurent et 2018)
Sim4CV (Müller et al., 2018)
Directions
SUMMIT (Cai et al., 2020)
( 400)
MACAD (Palanisamy, 2020)
Goal point
SMARTS (Zhou al., 2021)
Waypoints
MADRaS (Santara et al., 2021)
V (2 10)
Goal point
DriverCiym (Kothari et al., 2021)
SSS SESSSS
VISTA (Amini et al., 2022)
nuPlan (Caesar et al. 2021)
Waypoints
Nocturne (Vinitsky al., 2022)
V 2 128)
Goal point
MetaDrive (Li et al., 2022)
InterSim (Sun et al., 2022)
Goal point
TorehDriveSin (Scibior et al., 2021)
BITS et al., 2023)
Goal point
Waymax (Gulino al., 2024)
2 128) (2 128)
Waypoints
GPUDrive (ours)
Goal point
3 SIMULATION DESIGN
3.1 SIMULATION ENGINE
Learning to safely navigate complex scenarios in a multi-agent setting requires generating many billions of environment samples. To feed sample-hungry learning algorithms, GPUDrive is built on top of Madrona (Shacklett et al., 2023), an throughput reinforcement learning environments. In the Madrona framework, multiple independent worlds (each containing an independent number of agents' ) are executed in parallel on an acceler- ator via a shared engine.
system designed for high-
However, the simulation of driving poses a unique set of challenges which require careful techni- cal design choices to solve. First, road objects, such as road edges and lane lines are frequently represented as polylines (i.e. connected sets of points). These polylines can consist of hundreds of points as they are sampled at every 0.1 meters, leading to even small maps having upwards of tens of thousands of points. This can blow up the memory requirements of each world as well as lead to significant redundancy agent observations. Second, the large numbers of agents and road objects make collision checking throughput bottleneck. Finally, there is immense variability in the number of agents and road objects a particular scene. Each world allocates memory to data structures that track its state and accelerate simulation code. Though independent, each world incurs a memory footprint proportional to the maximum number of agents across all worlds. this way, the performance of GPUDrive sensitive the variation in agent counts across all the worlds in batch.
These challenges partially resolved via the following mechanisms. First, primary acceleration data structure leveraged by GPUDrive is a Bounding Volume Hierarchy (BVH). The BVH keeps track of all physics entities and is used easily exclude candidate pairs for collisions. This allows us to then run a reduced-sizz collision check on potential collision candidate pairs. The use of a BVH avoids invoking collision check that would otherwise always be quadratic in the number of agents in world. Secondly, we observed that lot of the lines in the geometry of the roads are straight. This allows us to omit many intermediate points while only suffering a minor hit in the quality the curves. We apply polyline decimation algorithm (Visvalinghan-Wii & Whyatt, 1993) to approximate straight lines and filter out low now-importance points in the polylines. With this modification, the step times while decreasing memory usage. Finally, rather than allocate memory for the maxi-
Algorithm) (Visvalingam
reduce the number of points by 10- 15 times and significantly improve
In the Waymo Open Motion Dataset, agent constitutes vehicle, eyclist, pedestrian.
4

---

Published as a conference paper at ICLR 2025
mum number of agents in scene (as is likely necessary in frameworks like Jax), only allocate memory equal to the actual number of instantiated agents.
3.2 SIMULATOR FEATURES
We provide overview of some of the pertinent simulator features as well sharp edges and limitations of the simulator as a guide to potential users.
Figure 2:
Example scenarios from the Waymo Open Motion Dataset rendered in GPUDrive. The blue boxes and circles indicate agents and their respective destinations.
Dataset. GPUDrive represents its map series of polylines and does not require a connectivity map of the lanes. As such, it can be made compatible with most driving datasets given the pre- processing of the roads into the polyline format. Currently, GPUDrive supports the Waymo Open Motion Dataset (WOMD) (Ettinger al., 2021) which is available under non-commercial license. We show four representative example scenarios from the WOMD in Figure 2. The WOMD consists of a set of over 100,000 multi-agent traffic scenarios, each which contains the following key elements: 1) Road map the layout and structure of road, such as highway parking garage. 2) Logged human trajectories from vehicles, cyclists, and pedestrians. 3) Road objects, such as stop signs and crosswalks. Figure depicts example of an intersection traffic scenario rendered in GPUDrive.
Sensor modalities. GPUDrive supports variety of observation spaces intended to enable hetero- geneous types of agents. Fig. 7 depicts the three types of supported state spaces. The first mode somewhat unphysical in which all agents and road objects within fixed radius are observable the agent. This mode intended primarily for debugging and quick testing, enabling user to minimize the amount of partial observability in the environment. The other modes based accelerated LIDAR scan, representing what an autonomous vehicle would be able to see and what respectively. Both modes are based casting LIDAR rays;
GPU-
human would likely be able model human vision we simply restrict the LIDAR rays to emanate in smaller, controllable-sizer cone that can be rotated through action corresponding to head rotation. Note that since all objects are represented as bounding boxes of fixed height, the LIDAR observations are coeeevnssevvatvee as, in reality, it is frequently possible see over the hoods of cars as their height is lower than the body of the rest of the car.
Agent dynamics. By default, agents are stepped using standard Ackermann bicycle model (De- tails in Appendix B) with actions corresponding to steering and acceleration. This model enables the dynamics of objects to be affected by their length, creating different dynamics for small cars as opposed to large cars like trucks. However, this model is not fully invertible which can make it chal- enable full invertibility for imitation learning, we also support the simplified bicycle model, taken from Waymax (Gulino et al., 2024), which is double-integrator in the position and velocity and updates its yaw as:
lenging to use as a model for imitation learning.
0.1 = 0 se + 2 a, At
5

---

Published as a conference paper at ICLR 2025
where 0 is the yaw, is the steering command, is the velocity, and is the acceleration time t respectively. At is the timestep. This model is always invertible given an unbounded set of steering and acceleration actions but is independent of the vehicle length. See the appendix for full details on the models.
Note that this model does not factor in the length of the car, causing both long and short objects to have identical dynamics. However, we have observed that computing the expert actions and then using them to mimic the expert trajectory under this model leads lower tracking error than the default bicycle model.
Rewards. All agents are given a target goal to reach; this goal is selected by taking the last recorded position in the vehicle's logged trajectory. A goal reached when agents are within some configurable distance 5 of the goal. By default, agents in GPUDrive receive reward of for achieving their goal and otherwise receive a reward of O. There are additional configurable col- lision penalties or other rewards based on agent-vehicle distances or agent-road distances though these are not used the experiments reported this work.
Termination conditions. We terminate an agent's episode when they achieve their goal position and support an option to also terminate the episode if the agent collides. As it is not clear where an agent should go next after it reaches its goal, we simply remove it from the scene afterwards (we note that alternative might be to generate new goal for the agent to drive to). Car and cyclist agents are considered to be in collision when they drive through a road edge while pedestrian agents allowed cross road edges.
Environment interface. As GPUDrive is implemented in C++, we provide a Pythonic interface through nanobind (Jakob, 2022). We create environments for both torch and jax that conform to the Gymnasium API (Towers et al., 2024) so users can use the simulator entirely through Python they prefer.
Available driving simulation agents. We use reinforcement learning to train a set of agents that reach their goals 95% of the time on a subset of 1000 training scenes. While this number far below the capability of human drivers, these agents are reactive in distinct fashion from parametrized driver models in other simulators. In particular, many logged-data simulators construct reactivity by having the driver follow along its logged trajectory but decelerate if agent passes in front of it. In contrast, these agents can maneuver and negotiate without remaining constrained to a logged trajectory. These trained agents extremely aggressive about reaching their goals and can be used as an distr--iiitiitttio test for proposed driving agents. The training procedure and more details can be found in Section 4.2.
Simulator sharp-edges. We note the following limitations of the benchmark:
Absence of map. The current version of the simulator does not have well-defined notion of lanes or a higher-level road map which makes it challenging for algorithmic approaches that require maps. The absence of this feature also makes challenging define rewards such as "stay lane-centered."
Convex objects only. Collision checking relies on the objects being represented as convex objects.
Unsolvable goals. Due to incorrect labels of some road points in the Waymo dataset, such exit parking lot being labeled as impassable road edge, some agent goals (roughly 2%) are unreachable. For these agents, we default them to simply replaying their logged trajectory and do not treat them as agents.
Initialization modes: In many scenarios, a significant portion of agents (25-75%) are al- ready at very close to their goal positions, such as parked cars. To highlight the difficulty of controlling agents, GPUDrive supports different initialization modes. By default, the ini- tialization mode is set to "all nontrivial" meaning that only agents more than 2 meters away from their target positions are considered controllable. Note that while this sharply reduces the number of agents in the scene, it more accurately represents actual driving challenges.

---

Published as a conference paper at ICLR 2025
4 SIMULATOR PERFORMANCE
The following Sections describe the simulator speed. Section 4.1 first shows the raw simulator speed and peak goodput (throughput achieved by the valid number of agents in scene). Section 4.2 then investigates the impact on reinforcement learning workflows by evaluating the time it takes to train reinforcement learning agents through Independent PPO (IPPO) (Yu et al., 2022), widely used multi-agent learning algorithm. The performance results here are from IPPO implemented with PufferLib (Suarez, 2024). slower IPPO implementation based on Stable Baselines (Hill et al., 2018) is also available in the repo.
4.1 SIMULATION SPEED
Since scenarios contain variable number agents, we introduce a metric called Agent Steps Per Second (ASPS) measure the sample throughput of the simulator. We define the ASPS as the total number agents across all worlds in a batch that can be fully stepped in a second:
ASPS SX Ek= |
(1)
AT
where Aş is the set of agents in the 5thh world, S is the number of steps taken, and AT' is the number of seconds elapsed. Figure examines the scaling of the simulator as the number of simu- lated worlds, which represents the amount of parallelism, increases. To measure performance, we sample random batches of scenarios of size equal to the number of worlds, so that every world unique scenario with K agents. On the left-hand side of Figure 3, we compare the performance of GPUDrive to Nocturne (Vinitsky et al., 2022) (CPU, no parallelism), a -PU-accelee version of Nocturne via Pufferlib (16 CPU cores) (Suarez, 2024) and Waymax Gulino et al. (2024). Em- pirically, the maximum achievable ASPS of Nocturne is 15,000 (blue dotted line) though we note that additional speedups may be possible. GPUDrive achieves peak ASPS of 2.3 million steps, which is 2 to 3 orders of magnitude faster compared to Nocturne. This performance also surpasses that of Waymax (Gulino al., 2024), a JAX-based simulator, where we could run more than 16 environments in parallel due to Out of Memory (OOM) issues.
Dctributionn of controlabic agents across traftic scenes
Agent steps per second
agent steps second
10"
1.0
50
ASPS
Count
Simuleter
CPJ
10-
101
Device
10a
Nactume For ard 125 (PUO)
Peralled environments
Controlled
Parallel environments
Figure 3: Peak goodput of GPUDrive on a consumer-grade and datacenter-class GPU com- pared to original, CPU and GaU-based, implementations using the radial filter observation. Left: The total number of agent steps per second (ASPS) is the number of objects for which our sys- tem computes observations each time step. To ensure fair comparison, we align the conditions with those used in (Gulino et al., 2024), where all cars, bicyclists, and pedestrians are considered valid sverrincee-enerattin
agents. Center: The distribution of controllable agents across 512 sce- narios in the Waymo Open Motion Dataset A 10.8 (red line), a = 9.3). These numbers are obtained using the "nontrivial" initialization mode in GPUDrive, which initializes only the agents that are more than 2 meters away from their final position. Right: The total number of controllable agent steps per second (CASPS) as we increase the number of worlds (parallelism).
In addition to the ASPS, report another metric to indicate the number of controllable agents that are stepped per second. In the Waymo Open Motion Dataset, each scenario contains a varying number of moving agents (Fxamples in Figure 2). By default, system only classifies something
7

---

Published as a conference paper at ICLR 2025
controllable agent if its movement is necessary to achieve the goal. Therefore, parked cars throughout the episode are not considered controllable agents. To illustrate what this looks like, we plot the distribution of controllable agents across subset of scenes in Figure 3.
Considering the variance controllable agents per scene, the right-hand side of Figure depicts the Controlled Agent Steps Per Second (CASPS). CASPS reflects the expected performance of our system when utilizing the Waymo Open Motion Dataset with a randomly sampled subset of scenarios. Due to the significant variability in agents throughout the dataset, the highest CASPS is notably lower than the ASPS, at around 200,000. Note that this can be improved by strategically selecting scenes, such as filtering for dense scenes with a high number of controllable agents.
Agent steps per second
Controlled agent steps per second
Observation space
Radial Filter
LIDAR
10a
10a
Device
ASPS
CASPS
NVIDIA A100-5XK4 8OGB
NVIDIA
10a
10
10a
10a
10:
10a
10a
Number parallel environments
Number paraliel environments
Figure 4: ASPS and CASPS comparison between Radial Filter and LiDAR observation types. The radial filter is slower than lidar due to its linear scan of nearby objects. In contrast, the LiDAR observation type is GPU-accelerated, delivering significantly enhanced performance. As demon- strated in the plots, depending the scene selection, the LiDAR can achieve a speedup of approxi- mately over the radial filter.
Lastly, we demonstrate the speed of different observation spaces in Figure 4. We observe that using LiDAR is approximately three times faster than using the radial filter observation (Details about the supported observation spaces are found in Section 3.2).
4.2 END-TO-END SPEED AND PERFORMANCE
The purpose of GPUDrive is to facilitate research and development in multi-agent algorithms by 1) reducing the completion time of experiments, and enabling academic research labs achieve scale on a limited computing budget. Ultimately, we are interested in the rate at which a machine learning researcher or practitioner can iterate on ideas using GPUDrive. This section highlights what our simulator enables in this regard by studying the end-to-end process of learning policies in our simulator.
Figure contrasts the number of steps (experience) and the corresponding time required to solve 10 scenarios from the WOMD between Nocturne (Vinitsky al., 2022) and GPUDrive. We compare Nocturne to GPUDrive, as it is the closest simulator in functionality, except that it is implemented on the CPU. For benchmarking purposes, we mark scene as solved when agents can navigate to their designated target position 95% of the time without colliding going off-road. Ceteris paribus (Details Appendix D), GPUDrive achieves a 200 300x training speedup, solving 10 scenarios in less than 3 minutes compared to approximately 10 hours in Nocturne.
In absolute terms, the number of controlled agent steps per second for the Pufferlib PPO implemen- tation ranges from 200K to 500K, depending the hardware used and factors such the number of visible road points per agent.
As shown in Fig. 5, GPUDrive allows us to solve scenes in minutes. Next, we investigate how the individual scene completion time, the time it takes to solve a single scenario, changes as we increase the total number of scenarios train In practice, it may be desirable to train agents on thousands of scenarios. Therefore, we ask whether it is feasible to fully leverage the simulator's capabilities with single GPU.
8

---

Published as a conference paper at ICLR 2025
Training steps
Training
Training scenaries (log axis)
100
100
Traiting perforance (9)
Traininng (1g)
Training performance (KC
60-
40-
40-
Simulator Nortume GPLDrive
Simulator
Simulator
Nect urne GPUDrvv
Nachurne
GPUDrive
0.0
0.2
0.4
0.5
0.8
5000
10000
15000
20000
10'
10'
Global stop
167
cllk time (9)
Wall clock time (s)
Figure 5: From hours to seconds. Left: Training performance (goal-reaching rate) as a function of the global step of the controlled agents (CASPS). Center: Training performance function of wall-clock time in seconds. Right: Training performance as a function of wall-clock time where the x-axis experimental conditions as closely as possible. See Appendix D for the hyperparameters and train- ing details. The green dotted line marks optimal performance (all agents reach their goal without collisions).
log scale. Runs averaged across three seeds, replicating environmental and
Interestingly, we find that the amortized sample efficiency increases with the size dataset of scenes increase the dataset. train in. Figure shows the average completion time per scenario contrast, solving 1024 For instance, using IPPO with 32 scenarios takes minutes per scenario. unique scenarios takes about 200 minutes, which amounts to only 15 seconds per scenario. We ex- pect that these scaling benefits will continue as we further increase the size of the training dataset. This suggests that GPUDrive should enable effective utilization of the large WOMD dataset com- prising 100,000 diverse traffic scenarios, even with limited computational resources.
Total time to solve multiple scenarios
Amortized time to solve multiple scenarios
Time to salve (minutes)
200
Time to salve (minutes)
2.0
150
1.5
100
1.0
50
0.5
32
128
256
512
1024
32
128
256
512
1024
Number of unique scenarlos
Number unique scenarios
Figure 6: Scale reduces individual scene completion time. Left: Total time required to solve fixed number scenarios to goal-reaching rate of 95%. Note that tim-to-completion linear concerning the number of scenes. Right: Each additional scenario costs less to solve than the previous scenario. At 1024 scenes, the per-scene cost of solving an additional scene is on the order of 15 seconds.
sub-
4.3 MEASURING REMAINING UNSOLVED TASKS
The Waymo Open Motion Dataset contains a total of 103.354 traffic scenarios (Ettinger et al., 2021). In this paper, we demonstrate that an agent can achieve 95% performance on a subset 1000 scenes after 15 hours of training. Additionally, we show that the time required to solve scenarios decreases as the dataset size increases. Our analysis of the failure rates the current best performing policy suggests that the goal-reaching limit is around 98%, due to mislabeled road edges in the dataset, which render some goals unreachable because they are located beyond uncrossable roads. An important direction for future work is developing agents that achieve near- near-perfect performance,

---

Published as a conference paper at ICLR 2025
with goal-reaching rate approaching 100% and 0% collision rate while addressing the limitations posed by data inaccuracies in the benchmark.
5 CONCLUSION
In this work, we present GPUDrive, a GPU-accelorated, multi-agent, and data-driven simulator. GPUDrive is intended to help generate the billions of samples that are likely needed to achieve effective reinforcement learning for multi-agent driving planners. By building atop the Madrona Engine (Shacklett al., 2023), we scale GPUDrive hundreds of worlds with potentially thou- sands of agents leading to throughput of millions of steps per second. This throughput occurs while synthesizing complex observations such as LiDAR. We show that this throughput has consequent implications for training reinforcement learning agents, leading to the ability to train agents to solve any particular scene in minutes and in seconds when amortized across many scenes, We release the simulator and integrated trained agents to enable further research.
Future work and simulator extensions. This paper represents an initial step toward scaling re- inforcement learning for multi-agent planning in safety-critical, mixed human-autonna autooomous settings. Several important opportunities remain for future work:
Agent performance: Training agents to navigate without crashing in any scenario, matching human capabilities, remains an unresolved challenge. Currently, agents trained GPU- Drive achieve 95% success rate in a few hours, but this still falls short of the standards we aim for. Getting to 100% goal reaching performance is left for future work.
Integrating multiple datasets: Future work will focus on integrating various datasets, such NuPlan NuScenes (Caesar et al., 2020). This integration will enable training data from multiple geographic locations, including Boston and Singapore. To achieve this, we need to establish data-proee data-processing pipeline that ensures the datasets are converted into format compatible with the simulator.
Diverse realistic sim agents: We plan to extend GPUDrive by introducing a variety of simulation agents that represent broad range of human-like behaviors. This will improve sim realism.
6 REPRODUCIBILITY STATEMENT
The simulator's 's source code and data are available at ltttps://iituu. coom//merge-Lbb gpudri vve. To ease reproducibilityy we provide Dockerfiles to simplify setup. The experiments in the paper can be reproduced using a single file on a single A100 in 16 hours or slightly longer for less performant hardware. The data has been pulled from the Waymo Motion dataset and parsed into JSON files that are used to initialize the simulator; these are all available from Huggingface datasets.
7 ETHICS STATEMENT
This paper presents GPUDrive, GPU-accelerated simulator for multi-agent learning in autonomous driving. We use publicly available datasets, such as the Waymo Open Motion Dataset (Ettinger et al., 2021), which are anonymized to protect privacy. GPUDrive is intended for research purposes and not for real-worrdd deployment without further validation. We recognize the risks of autonomous systems and emphasize the importance of safety and fairness in their application. Although this work does not involve human participants directly, it leverages real-world data that may reflect human behavior, and we strive to avoid harm or discrimination. The codebase, pre-tras pre-trained agents, and evaluation scripts are openly available to promote transparency. There sponsorships affecting this research. We are committed to ethical practices in data handling, model deployment, and research integrity.
conflicts of interest external
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is funded by the C2SMARTER Center through grant from the U.S. DOT's University Transportation Center Program. The contents of this report reflect the views of the authors, who are responsible for the facts and the accuracy of the information presented herein. The U.S. Government assumes no liability for the contents or use thereof. This work was also supported in part through the NYU High- Righ-Peerfomance Computing resources, services, and staff expertise.
REFERENCES
Alexander Amini, T'sun-ssun Wang, Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus. Vista 2.0: An open, data-driven simulator for multimodal sens- ing and policy learning for autonomous vehicles. In 2022 International Conference Robotics and Automation (ICRA), pp. 2419-2426. IEEE, 2022.
Jason Ansel, Edward Z. Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesen- sky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vetito, Elias Ellison, Will Feng. Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael La- 208, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch. Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk. Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang. Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compi- lation. In Proceedings of the 29th ACM International Conference
Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024, pp. 929 947. ACM, 2024. doi: 10.1145/362066533646338 URL 10. 1145/ 00055/3520660 0206666666650 3640366.
https:
Anton Bakhtin, David J. Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H. Miller, and Noam Brown. Mastering the game of no-press diplomacy via human- regularized reinforcement learning and planning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. VMAS: A vectorized multi-agent simulator for collective robot learning. In Distributed Autonomous Robotic Systems 16th International Symposium, DARS 2022, Montbéliard, France, 28-30 November 2022, vol- ume 28 of Springer Proceedings in Advanced Robotics, pp. 42 42-56.255556 Springer, 2022. doi: 10.1007/ 031 51497-4474 -- 5_4.
978-3-031-51497-5-44 URL https: /oii wwt.doi..ogooo.oooooo wttp:o...00////./....... 10000
Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos- Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: diverse suite of scalable reinforcement learning environments in jax, 2024. wethtttp:///aaat org/ abs /2306. 09884.
James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github. com/ google/ge////oooooooooooooo jax.
Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, Justin Fu, Austin Abrams, Punit Shah, Evan Racah, Benjamin Frenkel, Shimon Whiteson, and Dragomir Anguelov. Hierarchical model-based imitation learning for planning in autonomous driving. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pp. 8652 8659. IEEE, 2022, doi: 10. 1091000097616222000000 9998999.9.......... URL https: : / doii org / 10. 1109/ TROS4761 2, 2022.0222222222222..222 9981 695.

---

Published as a conference paper at ICLR 2025
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 1161--1663,, 2020.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 11810, 2021.
Panpan Cai, Yiyuan Lee, Yuanfu Luo, and David Hsu. Summit: A simulator for urban driving in massive mixed traffic. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4023 4029. IEEE, 2020.
Jordi Casas, Jaime L Ferrer, David Garcia, Josep Perarnau, and Alex Torday. Traffic simulation with aimsun. Fundamentals of traffic simulation, pp. 173-232, 2010.
Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J. Wu, and Jakob Nicolaus Fo- erster. Adversarial diversity in hanabi. In The Eleventh International Conference Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023. URL https: / /owpoo/evee.neet/e/eeeee 3i==LEEEE H_..
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving The waymo open motion dataset. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp. 9690-9699, IEEE, 2021. doi: 10. (0L110//CCC44992222..00000 https: / doii org/ 10. 1109 0wA.009/ICC49990 2021 00957.
URL
Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems, 36, 2024.
Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array pro- gramming with numpy. Nat., 585:357-362, 2020. doi: 10.1038/541188-020226666 org/10.groo.oooooooo /0/. 038 s41586 020---49-222222222222
URL
https:/ /ww.//doi.// .ow.i.org////
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. htttps://githh 2018.
com/ him/hill-aa aall-a//e mill-a/stable aassaalle wetrrabeebbaeeeinn
Max Jaderberg, Wojeiech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio García Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. CoRR, abs/1807 002.01281, 2018. URL http://arae orggabss.......oo.....e..ooo..be /1 807, 01281.
Wenzel Jakob. nanobind: https://gthub..co//wjkkbbbbin
tiny and efficient c++/python
bindings, 2022.
Ame Kesting, Martin Treiber, and Dirk Helbing. General lane-changing model mobil for car-
following models. Transportation Research Record, (999(1):86-88) 2007.
12

---

Published as a conference paper at ICLR 2025
Parth Kothari, Christian Perone, Luca Bergamini, Alexandre Alahi, and Peter Ondruska. Drivergym: Democratising reinforcement learning for autonomous driving. arXiv preprint arXiv:2111.06889, 2021.
Karsten Kreutz and Julian Eggert. Analysis of the generalized intelligent driver model (gidm) for uncontrolled intersections. In 2021 IEEE International Intelligent Transportation Systems Con- ference (ITSC), pp. 3223-3230. IEEE, 2021.
Mathias Lechner, Tim Seyde, Tsun-Hsuan Johnson Wang, Wei Xiao, Ramin Hasani, Joshua Roun- tree, Daniela Rus, et al. Gigastep-one billion steps per second multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Edouard Leurent et al. An environment for autonomous driving crrision-making.. 2018.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461-3475. 2022.
Pablo Alvarez. Lope7, Michael Behrisch, Laura Bieker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Mi- croscopic traffic simulation using sumo. In The st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL htt http//llbbbbllle /tt.:://e//a........dd/////9999
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rehecca Roelofs, Benjamin Sapp. Brandyn White, Aleksandra Faust, Shimon Whiteson, al. Imitation is not enough: Robustify- ing imitation with reinforcement learning for challenging driving scenarios. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7553-7560. IEEE, 2023.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 104700 2021.
Mark Martinez, Chawin Sitawarin, Kevin Finch, Lennart Meincke, Alex Yablonski, and Alain Korn- hauser. Beyond grand theft auto for training, testing and enhancing deep learning in self driving cars. arXiv preprint arXiv: 171.017712.01997, 2017.
Matthias Müller, Vincent Casser, Jean Lahoud, Neil Smith, and Bernard Ghanem. Sim4ev: A photo- realistic simulator for computer vision applications. International Journal of Computer Vision, 126:902-919, 2018.
Praveen Palanisamy. Multi- agent connected autonomous driving using deep reinforcement learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2020.
Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning fly-a a gym environment with pybullet physics for reinforcement learning of multi- agent quadcopter control. In 2021 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pp. 7512-7519. IEEE, 2021.
Julien Pérolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas W. Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers. Toby Pohlen, Tom Eccles, Mark Rowland, Mare Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguer- lange, Rémi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of stratego with model-free multiagent reinforcement learning. CoRR, abs/2206.15378 2022. doi: 10.45550/ARXIV222066 15378. URL https: org 10, 4855088550000 2206. 15378.
Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint arXiv:2311.10090 2023.
13

---

Published as a conference paper at ICLR 2025
Anirban Santara, Sohan Rudra, Sree Aditya Buridi, Meha Kaushik, Abhishek Naik, Bharat Kaul, and Balaraman Ravindran, Madras: Multi agent driving simulator. Journal of Artificial Intelli- gence Research, 70:1517-1555, 2021.
Adam Scibior. Vasileios Lioutas, Daniele Reda, Peyman Bateni, and Frank Wood. Imagining the road ahead: Multi-agent trajectory prediction via differentiable simulation. In 2021 IEEE Inter- national Intelligent Transportation Systems Conference (ITSC), pp. 720-725, IEEE, 2021.
Brennan Shacklett, Luc Guy Rosenzweig, Zhiqiang Xie, Bidipta Sarkar, Andrew Szot, Erik Wij- mans, Vladlen Koltun, Dhruv Batra, and Kayvon Fatahalian. An extensible, data-oriented archi- tecture for perforreffrm,,ce, many-world simulation. ACM Transactions on Graphics (TOG), 42 (4):1-13, 2023.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lilli- crap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat., 550(7676):354-39) 2017. doi: 10.1038/NATURE22200 URL https: /do. wwwp/................. 11.11.0....11 http//110003///tur22222 ntt///uuuure222..
Joseph Suarez. Pufferlib: Making reinforcement learning libraries and environments play nice.
arXiv preprini arXiv:2406. S0TT::2406.29055.5 /2905, 2024.
Qiao Sun, Xin Huang. Brian C Williams, and Hang Zhao. Intersim: Interactive traffic simulation via explicit relation modeling. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1 11416-1110 11423. IEEE, 2022.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17022 2024.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 62(2):1805, 2000.
Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, and Jakob Foerster. Nocturne: scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Advances in Neural Information Processing Systems, 35:3962-3974 2022.
M. Visvalingam and J. D. Whyatt. Line generalisation by repeated climination of points. The Cartographic Journal, 30:46-51, 1993. doi: 1011199000870443786666666
Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi. Leon Barrett, Rory Douglas, Dion Whitchead, Peter Dürr, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion gran turismo drivers with deep reinforcement learning. Nat., 602(7896): doi: 10. 111101//44158-001--44444 223-228. 2022.
URL https: //c.. orrg worg/10..110.
Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Torcs, the open racing car simulator. Software available at http://torcs. source- forge. net, 4(6):2, 2000.
Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2939.2 IEEE, 2023.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances Neural Information Processing Systems, 35:2461 1-24624, 2022.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, al. Smarts: An open-source scalable multi-agent rl training school for autonomous driving. In Conference on Robot Learning, pp. 264-285. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY
A. EXTRA FIGURES
Radial Filter
LiDAR
View Cone
Figure 7: Visualization of different observation spaces available GPUDrive. The top scene an example scenario from the dataset, rendered from the ego-centric perspective of the red vehicle. Grey cars are parked cars while white cars are other controlled agents. From left to right: the Radius Filter returns all objects within 100 meters, the LIDAR observation with 3000 rays spread around 360 degrees, and a view cone consisting of 3000 rays emanating in a 120-degree view cone.
A.2 CODE REPRODUCIBILITY
All code required to reproduce the paper open-sourced at htttps://ithhut com/ Emerge-Lab/ wwwpgppeedttreeeeeeeeeeeeeeeeeee n under release number v0.4.0
A.3 COMPUTATIONAL RESOURCES
All RL experiments in this paper the paper correspond to less than GPU days.
NVIDIA RTX 4080 A100. Total resources for
B DYNAMICS MODEL
Agents are driven by a kinematic bicycle model (Rajamani, 2011) which uses the center of gravity as reference point. The dynamics are as follows. Here (I,, W/) stands for the coordinate of the vehicle's position at time t, 6, stands for the vehicle's heading at time t, i stands for the vehicle's 's speed at time t, stands for the vehicle's 's acceleration and 8 stands for the vehicle's steering angle. is the distance from the front axle to the rear axle (in this case, just the length the car) and I, is the distance from the center of gravity to the rear axle. Here assume 0.5T.
15

---

Published as a conference paper at ICLR 2025
clip(Ur ( 0.5 i At,
Pmax: Umax
lr Lan
tan-
tan 1 (0.5 tan(
4 cos ( + ()
sin + 3)
ú cos(B) (a)
L
We then step the dynamics as follows:
It+1 = It + i At
Ut+1 Yt + At
O++1 + A
Ut+1 clip(v; + ē At, Umax, Umax
C LICENSE DETAILS AND ACCESSIBILITY
Our code is released under MIT License. The Waymo Motion dataset is released under a Apache License 2.0. The code is available at https: /ttl:///gtttane hgithub.......//.gg com /emeegeg--eeee Lab haad/guuriive
16

---

Published as a conference paper at ICLR 2025
D TRAINING DETAILS
D.1 END-TO-END PERFORMANCE
The Table below depicts the hyperparameters used produce the results in Section 4.2.
Table 2: Experiment hyperparameters used for comparing the training runs between Nocturne and GPUDrive in Figure 5. The environment configurations are aligned as closely possible, using same observations and field of view. The dataset includes the same scenarios. It's important to note that the length of the GPUDrive rollout is approximately equal to the number of worlds multiplied by the rollout length and then multiplied by the number of controllable agents. We have set this value to be 92 50 x 4600 to approximately match the rollout length in Nocturne.
Parameter
IPPO GPUDrive
IPPO Nocturne
0.99
0.99
AGAE
0.95
0.95
PPO rollout length
92
4096
PPO epochs
5
PPO mini-batch size
2048
2048
PPO clip range
0.2
0.2
Adam learning rate
3e-4
3e-4
Adam
le-5
le-5
normalize advantage entropy bonus coefficient value loss coefficient seeds
yes
yes
0.001
0.001
0.5
0.5
42, 12, 67
42, 12, 67
number of worlds
50
Perc. goal achieved
50
1004
125
40
30
10
50
100
150
200
250
150
150
200
280
102
150
200
250
Figure 8: Key performance metries as function of training time grouped by the number of unique scenes in batch reported in Figure 6. Left: The aggregate percentage of agents that achieved their goal. Center: The aggregate percentage of agents that collided with another vehicle. Right: The aggregate number of vehicles that crossed road edge.
17


Reviews:
[{"review_id": "nZRunJ9KI1", "reviewer": "Reviewer_AUkE", "summary": "The paper presents a GPU accelerated simulator that can generate millions of simulation steps samples per second that can be used to train multi-agent reinforcement learning (RL) algorithms. The simulator is claimed to simulate hundreds to thousands of scenarios/scenes in parallel with each scene containing thousands of agents.\n\nThe simulator is built on top of the Madrona Game Engine and is written in C++. The C++ simulator engine can also be interfaced with learning environments written in JAX and Torch.\nThe authors have released implementations of RL algorithms capable of processing millions of agent steps per second and some baseline agents trained on these algorithms that achieve 95% of their goals. The simulator claims to provide both recorded logs and RL agents for the environment.\n\nThe authors introduced certain metrics to evaluate the simulation speed of GPUDrive in terms of agent steps per second (ASPS), controllable agent steps per second (CASPS) and scene completion time. Compared against other sim engines like Nocturne GPUDrive achieved 25-40x training speedup solving 10 scenarios in less than 15 minutes.", "questions": "- Were other agents in the scenes like pedestrians and cyclists also controlled? If so, what were the dynamics used to model their behavior if they were not logged?\n- Nit: Ethical statement was missing?\n- Nit: Can the x-axis in the center plot in Fig 5 be made to a log scale?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed simulator has the flexibility to handle multiple modalities of sensor data. \n\n- The authors have implemented ways to reduce the memory footprint due to the large number of agents and observation space using algorithms like Bounding Volume Hierarchy (to exclude certain agent pairs for collision checking) and polyline decimation to approximate the straight polylines.\n\n- The trained agents are claimed to be useful for out-of-distribution tests for the driving agents.\n\n- The authors presented the different simulator features in a comprehensive way.\n\n- The paper shows that the simulator gets the scaling benefits in terms of increased amortized sample efficiency with increasing dataset size. This can be beneficial when dealing with large scale datasets with limited compute.", "weaknesses": "- The paper does not provide simple IDM (intelligent driving models) agents that can be sometimes practical to have basic reactivity to the ego-agent.\n- The authors mention that the current work is limited in properly utilizing the generated samples for optimal training.\n\n- Just a thought: The implementation is in C++ and it provides a binding interface with Python environments. It would have been nice to have a mono-language (primarily Python based) tool as the model training and other related pipelines are mostly in Python.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lgpouiCW9G", "reviewer": "Reviewer_Y9u7", "summary": "GPU drive introduces a fast multi agent simulator build using C++ that helps you run complex scenarios especially related to self driving cars at scale built on top of the Waymo Open Motion dataset. This allows iterating on these scenarios quicker reaching greater than a million FPS thus allowing more experimentation runs and iterating/trying out different scenarios even on desktop grade GPU's.", "questions": "1. Benchmarks consist of limitations because of the dataset. Can it be addressed by using another dataset ?\n2. Stable baselines is not known that well for speed. Could other implementations of PPO have been used ?\n3. Is there support for multi GPUs ? And if they do exist, an ablation or benchmark would be great for that", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A multi agent simulator accelerated on the GPU iteration of over a million steps per second.\n2. Very well written and structured code to run any experiment easily with a lot of easy experimentation code readily available. \n3. Extensive results analyzing the sampling frequency of the simulation.", "weaknesses": "1. Figure 2 needs a better caption and an explanation\n2. Designed to fit one exact dataset. A section explaining the effort required to integrate other datasets is desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fnsMDazfyC", "reviewer": "Reviewer_geDV", "summary": "This paper proposes GPUDrive, a GPU-accelerated multi-agent driving simulator designed to increase efficiency of learning-based systems. The simulator allows for loading expert trajectories from real-world driving datasets, can support multiple observation spaces (including e.g. LiDAR), and displays favorable throughput compared to other openly accessible simulators. The simulator, together with pre-trained goal-conditioned policies, is made openly available with accessible pythonic interfaces.", "questions": "- How easily can the simulator be updated to efficiently provide map-like utilities that allow for lane-keeping rewards (re mentioned limitations)?\n- Do you support loading multiple different polices for individual agents? Could they have different sampling rates? How would these aspects affect efficiency?\n- How do traffic jams affect throughput (re BVH)? This could be an interesting experiment to add.\n- In video scene_53.mp4, agent 4 displays rather jerky behavior when moving towards its goal - could you elaborate on the underlying reasons?\n- In video scene_43.mp4, agents 1 and 10 seemingly disappear without reaching their goals - could you elaborate on this behavior?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed simulator improves over alternatives in terms of sample efficiency. All of the design choices appear reasonable, while the underlying source code with pre-trained driving baselines will be released.\n- The ability to load real-world driving datasets is extremely useful, while providing a variety of observation spaces is a great feature.\n- Transparency about current limitation of the benchmark are very helpful for user adaptation.", "weaknesses": "- While the focus of this paper is on providing a novel simulator, it would be very interesting to see some more complex behavior over longer time-horizons to fully capture the capabilities unlocked by the simulator (e.g. training a single agent policy with higher velocity limit to weave through a simulated traffic scene, etc.)\n- Showcasing such behavior would likely require addressing the “Absence of a map” limitation raised in the paper, in order to formulate more sophisticated reward function. An important question would then be how easily this could be integrated, and how much the absence of such a feature could hurt adaptation of the simulator.\n- The discussion of batched simulators could be extended to include references [1-3], where [1] has driven many results in single-agent robot learning, while [3] considers heterogenous multi-agent settings\n- Figure 3 mentions performance on an RTX 4080, while line 711 states RTX 8000\n- Line 308: “number valid number”\n\n**References**\n\n[1] V. Makoviychuk, L. Wawrzyniak, Y, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. “Isaac gym: High performance gpu-based physics simulation for robot learning.” NeurIPS, 2021.\n\n[2] J. Panerati, H. Zheng, S. Zhou, J. Xu, A. Prorok, and A. P. Schoellig. \"Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control.” IROS, 2021.\n\n[3] M. Lechner, L. Yin, T. Seyde, T.-H. Johnson Wang, W. Xiao, R. Hasani, J. Rountree, and D. Rus. “Gigastep - one billion steps per second multi-agent reinforcement learning.\" NeurIPS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gvlsgx9aeL", "reviewer": "Reviewer_dLhs", "summary": "This paper presents GPUDrive, a GPU-based simulator for autonomous driving. The simulator is compatible with existing datasets and allows parallel simulations. The experiments show that it can train a policy with 25-40x training speedup against the baseline.", "questions": "1. In Figure 3, the speedup appears nearly linear. However, it would be helpful to examine scaling performance by adding more environments to identify saturation points and gain insights into system limitations.. \n2. What is the scaling of speedup with respect to the number of agents in the environment? e.g., fix the number of environments and scales the number of agents?\n3. In Figure 5, do you use the CPU-parallel version of Nocturne?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The GPU-based simulation is important in facilitating the training for complex real-world applications, highlighted by recent advances in robotics.\n2. The experiment result shows significant wall-clock time speedup again baselines.", "weaknesses": "1. The paper claims compatibility with existing datasets but only demonstrates map loading, leaving other functionalities unclear. For example, the imitation learning experiment or mixing agent behaviors—some from datasets and others from RL agents during training.\n2. I believe one major advantage of parallel environments is that it allows you to do randomization across different environments (worlds). However, the paper lacks detail on whether GPUDrive supports this capability.\n3. While GPUDrive offers a Python interface, I am curious how easy to customize those key elements in the environment given that the observation, reward, dynamic functions are written in C++. \n4. Experiments only evaluate IPPO, despite the paper claims that it targets at mixed motive setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)