Robust Autonomy Emerges from Self-Play
Authors: Marco Francis Cusumano-Towner
David Hafner
Alexander Hertzberg
Brody Huval
Aleksei Petrenko
Eugene Vinitsky
Erik Wijmans
Taylor W. Killian
Stuart Bowers
Ozan Sener
Philipp Kraehenbuehl
Vladlen Koltun
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Autonomy
Simulation
Driving
Self-play
Primary area: reinforcement_learning->deep_rl
TL;DR: Robust and naturalistic driving emerges from self-play in simulation at unprecedented scale

Abstract: Self-play has powered breakthroughs in two-player and multi-player games. Here we show that self-play is a surprisingly effective strategy in another domain. We show that robust and naturalistic driving emerges entirely from self-play in simulation at unprecedented scale -- 1.6 billion km of driving. This is enabled by Gigaflow, a batched simulator that can synthesize and train on 42 years of subjective driving experience per hour on a single 8-GPU node. The resulting policy achieves state-of-the-art performance on three independent autonomous driving benchmarks. The policy outperforms the prior state of the art when tested on recorded real-world scenarios, amidst human drivers, without ever seeing human data during training. The policy is realistic when assessed against human references and achieves unprecedented robustness, averaging 17.5 years of continuous driving between incidents in simulation.

Robust Autonomy Emerges from Self-Play
Marco Sesumano-Towneer David Hafner"! Alex Hertzberg" Brody Huval 11
Aleksei Petrenko'
Eugene Vinitsky"
Erik Wijmans" 1 Taylor Killian
Stuart Bowers' Ozan Sener'
Philipp Krähenbühl
Vladlen Koltun
Abstract
and robustness emerge when self-play training is taken to unprecedented scale orders of magnitude beyond prior experiments (Feng et al., 2023; Zhang et al., 2023). This discovery is enabled by GIGAFLOW, a batched simulator architected from the ground up for self-play reinforcement learning on a massive scale. GIGAFLOW is capable of sim- ulating and learning from 4.4 billion state transitions (7.2 million km of driving, or 42 years of continuous driving experience) per hour on a single 8-GPU node. It simulates urban environments with up to 150 densely interacting traf- fic participants 360 000 times faster than real time at a cost of under $5 per million km driven (based on public cloud rates). A full training run simulates over one trillion state transitions, 1.6 billion km driven, or 9500 years of subjec- tive driving experience, and completes in under 10 days on one 8-GPU node.
Self-play has powered breakthroughs in two- player and multi-player games. Here we show that self-play is a surprisingly effective strategy in another domain. We show that robust and nat- uralistic driving emerges entirely from self-play in simulation at unprecedented scale 1.6 bil- lion km of driving. This is enabled by GIGAFLOW. batched simulator that can synthesize and train on 42 years of subjective driving experience per hour on single 8-GPU node. The resulting policy achieves state-of-the-an art performance on three in- dependent autonomous driving benchmarks. The policy outperforms the prior state of the art when tested on recorded real-world scenarios, amidst human drivers, without ever seeing human data during training. The policy is realistic when as- sessed against human references and achieves un- precedented robustness, averaging 17.5 years of continuous driving between incidents in simula- tion.
We use GIGAFLOW to train a parameterized family of driv- ing policies. The parameters specify the type of traffic par- ticipant controlled by the policy (passenger vehicle, large truck, bicyclist, or even a pedestrian) and the driving style (e.g., aggressive vs. cautious). These parameters can be modified at test time with no additional training (Dosovit- skiy & Koltun, 2017), such that single trained policy can be used to control variety of traffic participants, with a vari- cty of behavioral styles. During training, this parameterized policy architecture enables all simulated traffic participants to be collecting experience in parallel, all flowing through a single neural network. This supports self-play simulations where more than hundred agents are all controlled by single neural network, which is learning from all of their experiences, yet the agents exhibit diverse outward manifes- tations (truck vs. bicycle), functional characteristics (turning radius), and behavioral styles (adherence to traffic laws).
1. Introduction
Self-play has been an effective strategy for training policies for board games, card games, 3D multiplayer games, real- time strategy games, robotic manipulation, and even bio- engincering (Silver al., 2017: 2018; Jaderberg et al., 2019; Berner et al., 2019; Brown & Sandholm, 2019; Vinyals et al., 2019; Plappert et al., 2021; Perolat et al., 2022; Wang et al., 2023a). In this work, we demonstrate the effectiveness of self-play in another domain. We show that simulated self- play yields naturalistic and robust driving policies, while using only a minimalistic reward function and never seeing human data during training.
The result is robust and naturalistic driving policy that achieves state-of-the-art performance when tested in recorded real-world scenarios, amidst recorded human drivers, without ever seeing human data during training. We test the GIGAFLOW policy in three leading independent third-party benchmarks: CARLA (Dosovitskiy et al., 2017), nuPlan (Caesar et al., 2022), and the Waymo Open Motion Dataset (Ettinger al., 2021) (through the Waymax sim- ulator (Gulino et al., 2023)). State-of-the-ar performance
We demonstrate that qualitatively new levels of realism
"Equal contribution, alphabetical order. 'Apple. Correspon- dence to: Philipp Krähhnbiihl sphilkr@apple.com:
Proceedings of the 4ddd International Conference Machine Learning. Vancouver. Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robust Autonomy Emerges from Self-Play
Waymax Benchmark (Waymo Open Dalaset)
CARLA Benehmark
nuPlan Benchmark
dk
94.7
92.1
92.2
93.8
99.16
SOTA
94.3
88.3
SOTA
89
SOTA
Prior state of the art (CARLA-speciles
Prior state of the art (nuPlan-specilic)
Prior state of the art (Maymo-speeilic)
Gigaflow (zero-shot) (Ore generalitt policy for all benchmarks)
b
Argoverse Waymo
Prior best
CARLA
Waymax
CARLA
nuPlen
Waymax
Platosu
nuPlan
T
Carla
Gigallow
0.05
100 -100
800
1200
1600
Million km
Figure 1: Self-play reinforcement learning yields a generalist policy. a. A single GIGAFLOW agent outperforms the best dataset-specific specialists across leading benchmarks. GIGAFLOW is evaluated zero-shot without training single benchmark, while dataset-specific specialists train on benchmark-speciffc datasets. Each benchmark includes different maps, scenarios, and evaluation metrics. b, GIGAFLOW enables cost-effective training of policies via self-play on massive scale. Our largest policies drive over 1.6 billion km during training, more than the distance from the Sun to Saturn and orders of magnitude farther than prior datasets or simulations. At this scale, self-play yields a generalist policy. Dashed lines indicate points which the performance of our single generalist policy passes the prior state of the on each benchmark ('prior best') and points at which the performance on each benchmark plateaus ('plateau").
each benchmark previously achieved by specialist agents that were trained specifically for that benchmark, commonly using srrchmark-speciffe datasets. In contrast, we outperform the prior state of the art on all benchmarks with a single policy (Fig. 1) that was trained entirely via self- self-play, using none of the provided datasets for training.
2. GIGAFLOW
The goal of GIGAFLOW is train generalist policy r[a S, A, C) in simulation (Fig. 2d). The policy ob- serves the static world W, its state S, and other dy- namic agents A to produce action (Fig. 2c). A condi- tioming parameter C' modulates the policy's 's behavior. Learn- ing this generalist policy requires careful modeling two core concepts: uncertainty and other-agent behaviors.
The behaviors exhibited the GIGAFLOW policy natu- ralistic despite never seeing human data during training. The trained policy exhibits long-horizon planning without any dedicated planning or search modules, can deal with heavily contentious traffic scenarios, is quantitatively realistic when assessed against human references (Montali et al., 2023), and exhibits unprecedented robustness, averaging over 3 million km (or 17.5 years of continuous driving) between incidents in simulation.
Unvertainty in driving stems from partial or incomplete observations. The driver is generally unaware of the goals and intentions of other agents, even their exact location, speed, or acceleration. Objects parts of the static world may be hidden or occluded. Real-world sensors often intro- duce noise. GIGAFLOW models uncertainty directly through noise on the state S, noise in the state transitions, stochas- ticity in the dynamic agents, and partial observability on dynamic agents A and the static world W. GIGAFLOW agents observe the positions and speeds of nearby agents
2

---

Robust Autonomy Emerges from Self-Play
Batched Gigaflow simulation
b Gigaflow world
4800 worlds per GPU
Up to 150 agent
per world
Road boundary
W boundary
MaxPool
one poincy agents
Lane points Waoer
MLP
MaxPool
Stop lines Weop
MLP
MaxPool
action or value
MLP
V S
Other agents
MLP
MaxPool
Condnioning and goals C
MLP
Gigaflow policy
Gigaflow agent
Figure 2: Overview of GIGAFLOW. a, GIGAFLOW simulates tens of thousands of worlds with millions of agents in a massively parallel self-play reinforcement learning setup. b, Each world tasks agents to navigate to goals on a map without collisions. c, Each agent optimizes its own performance, given a set of local observations. d, All agents use compact shared policy network.
but their acceleration, and crucially, neither their goals conditioning.
formly over the map. This creates world in which agents drive for long distances before reaching their destinations (Fig. 2b). Agents tasked with visiting variable number of intermediate waypoints, requiring the ability to follow complex routes (see Appendix B for details).
Modeling agent behaviors is particularly impactful and complex aspect of driving. Prior work approached behav- ior models through hand-designed agents (Kesting et al., 2010; 2007; Gulino et al., 2023), recorded and replayed data (Gulino al., 2023; Li et al., 2023; Vinitsky et al., 2022), or models of driving learned from data (Ivanovic & Pavone, 2019; Nayakanti al., 2023; Wang al., 2023b; Suo et al., 2021; Shi et al., 2022: Xu et al., 2023; Zhong et al., 2023). In contrast, in GIGAFLOW, realistic and gen- eral driving behaviors for all traffic participants emerge via self-play reinforcement learning on massive scale.
Dense traffic flows with diverse interactions emerge as agents navigate their destinations. As training progresses, we can observe agents executing zipper merges and tight maneuvers in traffic jams, managing congested roundabouts and uncontrolled intersections, resolving occasional grid- locks, and performing multi-poin turns to reroute around accidents or obstructions.
GIGAFLOW agents train fully in self-play. All dynamic agents vehicles, pedestrians, and eyclists use the same single reactive parametric policy æ; their behaviors ied through conditioning C (Fig. 2d). The policy is aware of the dynamics of the agent it controls part of the con- ditioning Coynamies- The agent reward is a mixture of incen- tives to reach its goal, avoid collisions, drive centered and lane aligned, as well as penalties for running red lights or stop signs, and exceeding acceleration and jerk limits. The weights on each of these reward components are randomized per agent and provided conditioning Creward the agent (see Appendix B for details). This allows a single reactive policy π to exhibit a wide range of behaviors. The result
2.1. GIGAFLOW world
The GIGAFLOW world is simple: we do not script scenarios, use human driving traces, or design delicate reward terms. We show that simulation at massive scale makes up for much of this simplicity (Fig. 2).
of eight maps, randomly perturbed with rescaling, shears, flips and reflections. Total drivable lanes per map range from four to 40 km for a total of 136 km of road across the eight maps (Fig. 2a). In each map, we spawn one to N, agents at random locations and orientations on the road and ask them to reach goal points sampled uni-
Agents train
3

---

Robust Autonomy Emerges from Self-Play
is a diverse training world where agents learn a continuum of driving styles: some drive cautiously, others are likely to run traffic lights, while rare few are willing to drive against the flow of traffic. Because the policy only observes the conditioning C' of the agent it controls, it must learn to be robust to the unpredictable behaviors of other drivers.
during experience collection at a batch size of 2.6 million, and eight gradient updates per second in the training phase with batch size of 256 000. See Appendix D for more details.
GIGAFLOW training. We train the GIGAFLOW policy us- ing Proximal Policy Optimization (PPO) (Schulman et al., 2017). One of the main challenges associated with au- tonomous driving is the inherent imbalance in the data distribution. As training progresses, the on-policy data is dominated by ordinary traffic configurations, such as orderly driving in a straight line between intersections. The critic is often able to accurately predict the returns for such trajecto- ries, resulting in a large portion of samples with near-zero advantage (Greensmith et al., 2004) that consequently yield vanishingly small gradients.
2.2. GIGAFLOW simulation and training
The GIGAFLOW simulation and training framework is de- signed to optimize driving data collection and training throughput per unit of computation. We simulate and learn from 4.4 billion state transitions per hour on single 8-GH GPGP node, rolling out urban commute simulations 360 000 times faster than real time at the cost of under $5 per million kilo- meters driven (based on public cloud rates). These training rates require three core ingredients: fast batched simula- tor (Shacklett et al., 2021; Petrenko et al., 2021; Shacklett et al., 2023), a compact and expressive policy for fast infer- ence and backpropagation, and -hhhhhhouuhput training algorithm.
We use variant of Prioritized Experience Replay (Schaul et al., 2016) that filters samples that have minimal impact on learning. The filtering is based on the absolute value of the estimated advantage. We filter up to 80% of samples with low absolute advantage, which significantly increases learning throughput without sacrificing sample efficiency. Our approach, which we refer to as advantage filtering, focuses training the most informative state transitions, prioritizing learning from the underexplored tails of the data distribution where selected actions are measurably better or worse, and makes more efficient use of the data we generate. See Appendix C for more details.
GIGAFLOW simulation. GIGAFLOW simulates 38 400 en- vironments in parallel across 8 GPUs with up to N₂ = 150 vehicles each (Fig. 2a). Basic operations, such as policy in- ference and dynamics updates are batched across all agents. Agent localization, collision checking, and observation con- struction rely on dedicated optimized data structures. Due to the large map sizes, we precompute and cache all map observations in spatial hash and perform fast, GPU-based runtime lookup and retrieval. Agents perceive the map by observing sets of points sampled sparsely along drivable lanes Wime. and densely along the nearby road edges for precise mancuvering Wocundary (Fig. 2c).
3. Zero-shot evaluation on driving benchmarks
We evaluate a trained GIGAFLOW policy on the leading closed-loop driving benchmarks: CARLA (Dosovitskiy et al., 2017). nuPlan (Caesar et al., 2022), and the Waymo Open Motion Dataset (Ettinger et al., 2021) through the Waymax simulator (Gulino et al., 2023). These benchmarks encompass wide range of actor behaviors, driving scenar- ios, maps, traffic densities, durations, and scoring method- ologies. The CARLA benchmark consists of routes with hand-designed scenarios based on the NHTSA pre-crash topology (Najm et al., 2007). It evaluates long distance driv- ing (several minutes per 1-3 km route). nuPlan and Waymax evaluate short distance driving (8-14 seconds per scenario, < 100 m) in scenarios derived from recorded real-world driving with the associated sensor data.
Beyond map features, agents get observations of nearby traffic participants A containing nearby vehicles' sizes, locations, orientations, and velocities and nearby stop lines and traffic lights W Wstop- GIGAFLOW models static obstacles as immobile vehicles Agatic- To reduce memory, we do not store these observations in the rollout buffer, but calculate them on demand from stored world states. See Appendix A for a detailed description of the simulator.
GIGAFLOW policy. GIGAFLOW can simulate diverse actor types, from pedestrians to heavy trucks, by parameterizing single unified feed- forward policy (Fig. 2d). The decision to use the same underlying neural network policy for all traffic participants significantly impacts the overall throughput: we need only a single (batched) forward pass per simulation step to calculate actions for all agents. The policy resembles Deep Sets architecture (Zaheer et al., 2017) and is invariant to permutation w.r.t. each observation type. Critically, the entire trainable artifact is relatively compact at six million parameters. On an 8-GPU A100 node, the policy allows inference throughput of 7.4 million decisions per second
A generalist GIGAFLOW policy outperforms state-of-the- art specialists. For each benchmark, we compare to spe- cialist state-of-the-art policies that are either trained (Gulino et al., 2023) or carefully band-designed (Chitta et al., 2023; Jaeger et al., 2023; Dauner et al., 2023) to perform well on that specific benchmark. In contrast, we use a single policy across all benchmarks. Our policy is trained purely in self-play and is evaluated zero-shot in each benchmark
4

---

Robust Autonomy Emerges from Self-Play
environment. Without any fine-tuning, our policy surpasses the state of the art in CARLA, nuPlan, and Waymax (Fig. with details in Tables A5 to A7 and Appendix E). This demonstrates robust driving with strong generalization. Our self-play policy outperforms the state of the art on real driving traces with human traffic participants, without ever seeing human data during training.
variations, the single generalist GIGAFLOW driver outper- forms specialist policies optimized for individual benchmark scores.
The GIGAFLOW policy approaches the ceiling of bench- mark performance. The vast majority of the infractions sustained by the GIGAFLOW policy during testing on the benchmarks can be attributed to limitations of the bench- marks. For instance, 20% of the reported infractions in CARLA are caused by pedestrians or eyclists darting from the sidewalk into the roadway without reacting to the evasive maneuver of the driver or other traffic participants. Prevent- ing such collisions would require drastic overfitting to this type of scenario (Jaeger et al., 2023). Other exemplary limi- tations are gridlocks caused by CARI.A-controlled traffic (33% of all infractions) or fuzzy stop sign and red light checks (16% of all infractions).
GIGAFLOW policy generalizes to diverse actor behaviors. The benchmarks implement a diverse set of environment actors. CARLA uses reactive rules-based vehicles with lane- changing capabilities, combined with events triggered by the driver's ' behavior (e.g., a pedestrian that darts suddenly in front of the driver). The actors in nuPlan and Waymax are controlled by different variants of the Intelligent Driver Model (Treiber et al., 2000). Vehicles in nuPlan follow the lane center line, whereas vehicles in Waymax follow the paths of logged human drivers. The GIGAFLOW policy exhibits robust driving amongst all of these actor types.
In nuPlan our policy sustains 15 collisions in 1118 scenar- ios, We analyzed each of them. Nine are unavoidable due to invalid initialization or sensor noise (agents appearing inside the vehicle's bounding box). Four are caused by non- reactive pedestrian agents walking into the vehicle while the vehicle was stopped or in an evasive maneuver. Two collisions are due to traffic light violations of other agents.
GIGAFLOW policy generalizes to diverse maps and driv- ing situations. GIGAFLOW trains on variants of synthetic maps with closed road networks (Dosovitskiy et al., 2017), but generalizes
the real-world maps in nuPlan (Cae- sar et al., 2022) and in the Waymo Open Motion Dataset (WOMD) (Ettinger et al., 2021). The WOMD maps are small, with incomplete road networks constructed from logs of instrumented vehicles in several US cities. The nuPlan benchmark is based on driving logs of human drivers in locales with both right-handed and left-handed driving (Cae- sar et al., 2020); it contains larger maps that encompass the entire testing area of the vehicle. Both nuPlan and WOMD scenarios include merges, unprotected turns, and interac- tions with pedestrians and cyclists (Ettinger et al., 2021). The GIGAFL.OW policy achieves state-of-the- results in these benchmarks without any training on recorded driving logs or any human-designed scenarios.
In Waymax our policy sustains 187 collisions in 44 097 scenarios. We again analyzed each of them. 55.6% were caused by unavoidable IDM agent behavior (Treiber et al., 2000) of the traffic participants controlled by the benchmark, such as swerving directly into the ego vehicle. 41.7% were caused by initialization in a state of collision, typically with pedestrian. 2.7% (i.e. five scenarios) were considered at- fault and avoidable by the GIGAFLOW policy. Of the at-fault collisions, there were additional contributing factors such as perception issues or aggressive and spurious IDM behaviors. One example is when the GIGAFLOW policy seeks to avoid a rear-end collision with an IDM agent approaching from behind at high speed.
GIGAFLOW policy generalizes to real-world observation noise. Both Waymax and nuPlan construct observations, maps, and other actors with auto-labeling tools from real- world perception data. This brings occlusion. incorrect or missing traffic-light states, and obstacles revealed at the last moment. Despite the minimalistic noise modeling in GIGAFLOW, the GIGAFLOW policy generalizes zero-shot to these conditions.
We include videos of all reported infractions in the supple- mentary material.
4. Analysis
GIGAFLOW training employs two neural networks: the pol- icy (actor) that chooses actions and the value function ap- proximator (critic) that estimates the expected cost-to-go from given state. We examine how the policy's driving be- havior changes over the course of training (Fig. 3) and how the policy and value networks respond to targeted changes to their inputs in various scenarios (Fig. 4).
GIGAFLOW policy is ofate-of-the-art according to multi- ple scoring methodologies. Each benchmark brings its own definition of "oood driving'. Those definitions are distinct and sometimes contradictory. For example, running a red light in CARLA incurs nearly the same penalty as colliding with another vehicle. Yet the same action can be advanta- geous in nuPlan, where red light violations are ignored by the scoring criteria, hard braking causes comfort penalties, and forward progress is strongly rewarded. Despite such
Reinforcement learning at scale yields mastery of com- plex skills. The scale of GIGAFLOW training enables the policy to handle complex scenarios despite never seeing real-
5

---

Robust Autonomy Emerges from Self-Plas
Weymax
Rottlancck
K
K 10
K=6x
107
K- 10
Scow
10"
Roudicosuu couue
K- 10
10
10
12
Figure 3: Evolution of the policy during training. a, Zero-shot performance on example benchmark scenarios with increasing number of state transitions (K) seen during training. From top to bottom: K 10 (3 million km), K 10" (90 million km), and K 1012 (1600 million km). The color of the controlled vehicle and its trajectories indicate the benchmark score. b, Performance of the GIGAFLOW policy diagnostic scenario where a static obstruction (red bounding boxes) forces merge of several highway lanes into one. All agents are controlled by the same policy. Trajectories from 100 independent rollouts are shown. Agent colors indicate probability of success (moving past the bottleneck). Binary matrices on the right indicate the success of each actor each rollout (white indicates success). After K = 10a state transitions, the agents drive out of their lanes and crash. After K = 10 the agents drive forward, avoid collision, and sometimes successfully change lanes, but do not successfully merge across multiple lanes (only agents on the right succeed). After K 10", merging ability emerges but agents in the leftmost lane usually fail. After K 1000 agents sometimes merge from the left lane, but not reliably. After K 10 all agents reliably succeed with no incidents. e, Performance of the GIGAFLOW policy in diagnostic scenario where road closures require three traffic flows to merge into one without the aid of traffic lights. As training progresses, the policy goes from only succeeding at the right turn to successfully completing unprotected left turns and u-turns. After K = 10 steps of training, all agents reliably succeed. Binary matrices show the worst 100 samples from 1000.
world or hand-designed driving scenarios during training. The policy learns to execute unprotected left turns, drive in crowded roads used by both pedestrians and vehicles, and handle vehicles dangerously merging into the driver's lane (Fig. 3a). In diagnostic tests designed for analysis, GIGAFLOW vehicles are able safely negotiate through narrow bottleneck into single lane when the other lanes are blocked by an accident (Fig. 3b) and quickly merge three traffic flows into a single one due to road closures (Fig. 3c). Many these skills are mastered only after 10 to 10 steps of training experience (90 1600 million km driven).
Value network detects dangerous states. To examine the value network's 's ability to detect dangerous states, we evaluate on a set observations generated by densely sampling all possible positions and orientations of the driver a fixed region of the map. We find that the network appropriately assigns low value to states where the driver is taking corner too fast, and where collision with another vehicle is imminent due to high relative velocity (Fig. 4a).
Policy and value networks attend to salient scene fea- tures. Driving requires attending to the most consequential traffic participants at any given time, among hundreds of
6

---

Robust Autonomy Emerges from Belf-Play
Value Function
Long-horicon decision-mak
Obsruction (aawaa sc the veicco denuat
(Wlavve)
[widker)
20 kinh
valcory onownc dostrctton
Atlention
Change In addenan Wedtotuution
Goel and reward conditorngg
Pollyy Networs Motiiion (fut Informatoon
Recd woard
livinng
Fancorized woard
Nescling 10 ofher actors
conditoning
Eakoy makes
Paicy
Polky roon ppar whar croaning
Uan
Figure 4: Analysis of the learned policy and value networks. a, Value network estimates danger fine granularity over the surface of the road, Heatmaps show state-value estimates for a driver placed at every possible location, oriented in parallel to the nearest lane. Left: The network assigns lower value to rounding a sharp corner at higher speed. Right: Value estimates for driving at various speeds in the vicinity of vehicle (gray) moving 36 km/h to the right. Being placed at rest (0 km/h) in front of the moving vehicle is likely to result collision. As velocity increases, the danger zone shifts behind the other vehicle. The estimated danger becomes acute as the relative speed increases. b, Attention of the policy and value networks of the controlled vehicle (blue) to surrounding actors in two scenarios from the Waymo Open Motion Dataset. Top row: Mutual information between the policy network's action distribution and actor presence. Bottom row: Decrease in the value network's 's vtatev estimate due to actor presence. c, Scenarios where scripted nearby vehicle (orange) either continues moving forward at constant velocity (solid orange line) or executes a different behavior (dashed orange line). The policy (blue vehicle) executes maneuvers (solid blue line, dashed blue line) contingent on the other vehicle's behavior. Final poses are translucent. d, Policy and value networks perform morghooizon prediction and decision- decision-making. Top: Policy drives to its goal (20 second trajectory in blue) when the road is clear. When an obstruction (black) is placed 156 m away, the policy re-routes (black trajectory). Bottom: Heatmap shows change in value estimate over the surface of the road due introduction of the obstruction. Top: Diversity of trajectories due goal conditioning (three goals shown with associated trajectories in matching color). Bottom: Additional variability due to randomized reward conditioning parameters. f, Changing vehicle dimensions with all else fixed yields different behaviors: A slender vehicle can squeeze around an obstacle; compact vehicle with a tight turning radius can make u-turn; larger vehicle resorts to three-poing turn.
actors who may be present in the environment. We assess the attention our policy and value networks by analyzing the change in action distribution (via mutual information) and change value estimate when each actor is individually removed (Fig, 4b). As expected, the networks sometimes attend to different actors: For example, the value estimate is affected by all actors that make the scene more dangerous over the long term (for example a speeding car approaching
line of vehicles queued at red traffic light), whereas the policy's action might not change due to such an actor there is no way for the policy to mitigate the danger (Fig. 4b).
Policy executes maneuvers contingent on nearby traf- fie behavior. We evaluate the policy in scenarios where nearby vehicle either behaves predictably (continuing to move at constant velocity) or unpredictably, with all else

---

Robust Autonomy Emerges from Self-Play
fixed. We find that the policy executes appropriate discrete maneuvers contingent on the nearby vehicle behavior, like changing lanes to avoid collision with vehicle that is cut- ting into its lane and passing vehicle that unexpectedly stops in the road. The policy executes contingent longer- term routing maneuvers, like turning around by cireling the block instead of making a three-point turn, depending on traffic (Fig. 4c).
(WOSAC), which measures the ability reproduce real- world driving behaviors for simulation purposes (Montali et al., 2023). Despite not using any human data for train- ing. the GIGAFLOW policy exhibits many characteristics of human driving, achieving a score of 0.62 in zero-shot eval- uation on the realism meta-metric, outperforming several approaches based on supervised autoregressive prediction. (Details in Section E.4.)
Policy reacts to potential events far in the future. Both networks are also able to react to salient scene features even when they are distant, like an obstruction 150 m down the road (Fig. 4d), and can ignore actors that are near but irrelevant (e.g.. a car parked few meters from the driver in a parking lot). While considering potential events beyond the planning horizon is often a challenge for trajectory- based planners (Casas et al., 2021), the GIGAFLOW policy optimizes long-term return directly without the limitations of a short time horizon.
5. Discussion
Many questions remain to fully understand the long-term role of self-play in delivering broad-competence robust au- tonomy. First, our work has been conducted entirely in simulation. Techniques for transferring policies from simu- lation to reality will have to be brought bear before claims can be made regarding the efficacy of self-play policies in the physical world (Müller et al., 2018; Lee et al., 2020; Kaufmann et al., 2023).
One policy learns a continuum of driving styles. Reward function coefficients Creward vehicle dimensions, and vehi- cle goals are all randomized during training. As result, the policy learns a parameterized family of driving styles (Fig. 4c.f; Fig. A1); different styles can be elicited from a single trained policy by setting the parameters accord- ingly. without any retraining or fine-tuning. For example, the policy squeezes through narrow passages or performs tight turns and only if the vehicle dimensions permit this (Fig. 4f). Likewise, reducing the conditioning parameter that controls sensitivity to red lights makes the policy more willing run red lights in order to accomplish other goals.
Second, our work has focused on planning and decision- making, largely abstracting the perception stack. To in- tegrate the presented findings into an operational system, sensing and perception will have be modeled much more closely. An exciting possibility is to combine large-scale self-play training with data-driven simulation of the asso- ciated perceptual inputs (e.g., camera images) (Ost et al., 2021; Yang et al., 2023; Hu et al., 2023). It is likely fea- sible in the coming years due to ongoing improvements in simulation methodology (Shacklett et al., 2021; Petrenko et al., 2021; Shacklett et al., 2023), computing hardware, and system architectures. Combining self-play with pho- torealistic sensor simulation would substantially increase the computational footprint of each experience, but the wall- clock training time can be maintained by scaling out over a commensurate number of compute nodes (Dubey et al., 2024).
GIGAFLOW yields a highly efficient, capable, and re- alistic simulation environment. The GIGAFLOW train- ing configuration features substantial dynamics noise and diverse reward conditioning parameters Creward (see Sec- tion 2). We can configure the same simulation infrastructure for long-form evaluation of trained policies. In this config- uration, reduce the injected dynamics noise, increase control frequency, and set conditioning parameters Creward that prioritize safety for all actors This yields a fast, cost- effective, and highly robust traffic simulator. In this regime, a fully trained GIGAFLOW agent experiences on average 17.5 years of driving and travels over million km before encountering an incident. (For reference, human drivers average approximately 829 000 vehicle kilometers traveled per police-reported traffic crash in the United States (Stew- art, 2023), or as much as I crash per 24 800 km in narrower domains such as San Francisco (Flannagan et al., 2023).) To our knowledge this is the first demonstration of long- term robust traffic simulation based on independent agents traversing a diverse urban road network.
Third, our work has demonstrated that training without real- world driving traces can yield policies that are surprisingly human- like (Montali et al., 2023) and highly robust when tested in recorded real-world scenarios with human partici- pants (Caesar et al., 2022: Gulino et al., 2023). By contrast, autonomous driv-
common perspectives on learning- ing hold that recorded datasets will play a key role in training driving policies (Jain et al., 2021; Hawke et al., 2021; Chen et al., 2023). How do we reconcile our findings with these views? One possibility is to combine large-scale self-play training with training on recorded scenarios, perhaps via a combination of reinforcement learning and imitation learn- ing (Lu et al., 2023; Zhang et al., 2023). This can further increase robustness and help bridge simulation and reality.
We evaluate the realism of the driving behaviors learned via GIGAFLOW on the Waymo Open Sim Agents Challenge
8

---

Robust Autonomy Emerges from Self-Play
Impact Statement
Chen, L.., Wu, P., Chitta, K., Jaeger, B., Geiger, A., and Li, H. End-to-end autonomous driving: Challenges and fron- tiers, 2023. URL atttp://ate http::://axxvvvooooe org/ harrg/ab///t abbs/306. 16927.
Our findings may inspire broader application of self-play in training agents that act in the presence of (and in close coordination with) humans in physical and digital environ- ments. Such coordinated action may be called for in mobile robotics, in both consumer and industrial settings, and in digital domains such as online games. We have shown that policies that function effectively in the presence of human actors in complex dynamic environments can be trained without utilizing human data. Broader application of this methodology may substantially reduce the cost and com- plexity of training autonomous policies by meaningfully reducing the need for human data collection.
Chitta, K., Prakash, A., Jaeger, B., Yu, Z.,, Renz, K., and Geiger, A. TransFuser: Imitation with transformer-basee sensor fusion for autonomous driving. IEEF Transactions on Pattern Analysis and Machine Intelligence, 45(11); 12878-12895, 2023.
Dauner, D., Hallgarten, M., Geiger, A., and Chitta, K. Part- ing with misconceptions about rearning-basee vehicle mo- tion planning. In Conference on Robot Learning (CoRL.), 2023.
References
Dijkstra, E. W. A note on two problems in connexion with graphs. Numerische Mathematik, 1(1):269-271, 1959.
Ansel, J., Yang, E., He, H.,, Gimelshein, N., Jain, A., Voz- nesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., et al. PyTorch 2: Faster machine learning through dynamic python bytecode transformation and graph com- pilation. In International Conference on Architectural Support for Programming Languages and Operating Sys- tems, 2024.
Dosovitskiy, A. and Koltun, V. Learning to act by predict- ing the future. International Conference on Learning Representations (ICLR), 2017.
Dosovitskiy, A.,, Ros, G., Codevilla, F., López, A. M., and Koltun, V. CARLA: An open urban driving simulator. Conference on Robot Learning (CoRL), 78:1-16, 2017.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q. Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. Dota 2 with large scale deep reinforcement learning, arXiv:1912.06680, 2019.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models, 2024. URL https: /tt////avvvvr////.
Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Prad- han, S., Chai, Y., Sapp, B., Qi, C. R., Zhou, Y., Yang, Z., Chouard, A., Sun, P., Ngiam, J., Vasudevan, V., Mc- Cauley, A., Shlens, J., and Anguelov, D. Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset. In International Con- ference on Computer Vision (ICCV), 2021.
Brown, N. and Sandholm, T. Superhuman AI for multiplayer poker. Science, 365(6456):885-88)) 890, 2019. doi: 10.1126/ science.aay2400
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuScenes: A multimodal dataset for autonomous driving. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., and Liu, H. X. Dense reinforcement learning for safety valida- tion of autonomous vehicles. Nature, 615(7953))620-6277 2023. doi:
Flannagan, C., Leslie, A., Kiefer, R., Bogard, S., Chi- Johnston, G., Freeman, L., Huang, R., Walsh, D., and Anthony, J. Establishing crash rate benchmark us- ing large-scale naturalistic human ridehail data. Tech- nical Report UMTRI-2023-18, University of Michigan Transportation Research Institute, 2023. URL https: //dx. .rggrr///// 10 0 7302/8636.
Caesar, H., Kabzan, J.. Tan, K. S., Fong, W. K., Wolff, E., Lang, A., Fletcher, L., Beijbom, O., and Omari, S. nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles, 2022. URL https://arxx org/abs/21is 11810.
Casas, S., Sadat, A., and Urtasun, R. MP3; A unified model to map, perceive, predict and plan. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mor- datch, I., and Bachem, O. Brax - A differentiable physics engine for large scale rigid body simulation. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Chen, D. and Krähenbühh, P. Learning from all vehicles. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

---

Robust Autonomy Emerges from Self-Play
Greensmith, E.,, Bartlett, P. L.,, and Baxter, J. Variance reduc- tion techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 2004.
T., Deason, L.,, Leibo, J. Z., Silver, D., Hassabis, D., Kavukeuoglu, K., and Graepel, T. Human-level perfor- mance in 3D multiplayer games with population-based reinforcement learning. Science, 364(6443):859-866 2019. doi: 10. 26/science..au6249
Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., Co-Reyes, J. D., Agarwal, R., Roclofs, R., Lu, Y., Montali, N., Mougin, P.. Yang, Z., White, B., Faust, A., McAllister, R., Anguelov, D., and Sapp, B. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. In Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Jaeger, B., Chitta, K., and Geiger, A. Hidden biases of end-to-end driving models. In International Conference on Computer Vision (ICCV), 2023.
Jain, A., Pero, L. D., Grimmett, H., and Ondruska, P. Auton- omy 2.0: Why is self-driving always years away?, 2021. URL https::t//// /ttp/viiiv////////a/ wtg.:/iiiv../aaaoa Latgg//////// 2210. 0w112111001111101 08142.
Guo, Z., Gao, X., Zhou, J., Cai, X., and Shi, B. Sc Scene-level auett trajectory generation with con- sistent diffission models, 2023. URL hattps://arx:t
Kaufmann, E.,, Bauersfeld, I.., Loquercio, A., Müller, M.,
Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976):982-4 987, 2023.
org/abs 2211. 15736.
Hallgarten, M., Stoll, M., and Zell, A. From prediction to planning with goal conditioned lane graph traversals. In International Conference on Intelligent Transportation Systems, 2023. doi: 10.1 1011019/557777 202.17..222....2285.
Kesting, A., Treiber, M., and Helbing, D. General lane- changing model mobil for car-following models. Trans- portation Research Record, 1999(1) (999(1))866999)) )::6-94, 94, 2007.
Kesting, A., Treiber, M., and Helbing, D. Enhanced in- telligent driver model to access the impact of driving strategies on traffic capacity. Philosophical Transactions of the Royal Society A, 2010.
Hansen, E. A., Bernstein, D. S., and Zilberstein, S. Dynamic programming for partially observable stochastic games. In Conference Artificial Intelligence. AAAI Press/ / The MIT Press, 2004. URL http://www awaaaarorgg...... Library/ ///.A/AT//////// 20a.///44///aaaa/// 00eaaa00011122 php.
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: / / arriv. org/ abs 111//1111111 6980.
Hawke, J., E, H., Badrinarayanan, V., and Kendall, A. Reimagining an autonomous vehicle, 2021. URL https: http//xxxvvvvr/////// htrg/abs/2100 05805.
Hu, A., Russell, L.. Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. GAIA-1: A generative world model for autonomous driving, 2023. URL https: http:///wxxvvv..aa/aa////
Lee, J., Hwangbo, J., Wellhausen, I.., Koltun, V., and Hutter, M. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47):5986, 2020.
17080.
Huang, Z., Zhang, Z., Vaidya, A., Chen, Y., Lv, C.,, and Fisac, J. F. Versatile seene-consistent traffic scenario generation as optimization with diffusion, 2024. URL atttps://rrxe org/ abs//bb//aaa 220424444000 32524.
Li, Q., Peng, 2., Feng, L.,, Zhang, Q., Xue, Z., and Zhou, B. Metadrive:
Composing diverse driving scenarios for gen- eralizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45633:4661- 3475, 2023. doi: 10. 10.109/TPAM1.2202.31994
Ivanovic, B. and Pavone, M. The trajectron: Proba- bilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. International Conference on Computer Vision (ICCV), pp. 2375-2384, 2019. doi: 10.1109/CCC2.2219.00222 09.046.....
Lu, Y., Fu, J., Tucker, G., Pan, X., Bronstein, E., Roelofs, R., Sapp, B., White, B., Faust, A., Whiteson, S., Anguelov, D., and Levine, S. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. International Conference on Intelligent Robots and Systems (IROS), pp. 7553-7560, 2023. doi: 10.1 10911000555552222000 10342038.
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O.,, Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. Population based training of neural networks, 2017. URL http:// /arviv. org/ abs/ /1111/111000 09846.
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., and State, G. Isaac gym: High performance GPU based physics simulation for robot learning. In Neu- ral Information Processing Systems Track on Datasets and Benchmarks, 2021.
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz,
N. C., Morcos, A. S., Ruderman, A., Sonnerat, N., Green,
10

---

Robust Autonomy Emerges from Self-Play
Montali, N., Lambert, J., Mougin, P., Kuefler, A., Rhinehart, N., Li, M., Gulino, C.,, Emrich, T., Yang, Z. Z., Whiteson, S., White, B., and Anguelov, D. The Waymo open sim agents challenge. Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Zaremba, W. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv:2101.04880 2021.
Qian, C., Xiu, D., and Tian, M. A simple yet effective method for simulating realistic multi-agent behaviors.
Müller, M., Dosovitskiy, A., Ghanem, B., and Koltun, V. Driving policy transfer via modularity and abstraction. In Conference on Robot Learning (CoRL), pp. 1-15. PMLR, 2018.
Technical report, 2023. URL https://storaage
recearrchl/0223222reeetta2222222 SA G (1) .pdf.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 2021.
Najm, W. Smith, D., and Yanagisawa, M. Pre-crash sce- nario typology for crash avoidance research. Technical re- port, John A. Volpe National Transportation Systems Cen- ter (U.S.), 2007. URL https:: http://rsapppppppppp ntl sbt.
gov/view /ot. 6281.
Renz, K., Chitta, K., Mercea, O.-B., Koepke, A. S., Akata, Z., and Geiger. A. PlanT: Explainable planning trans- formers via object-level representations. In Conference on Robot Learning (CoRL), 2022.
Nayakanti, N., AI-Rfou, R., Zhou, A., Goel, K., Refaat, K. S., and Sapp, B. Wayformer: Motion forecasting via simple & efficient attention networks. In Interna- tional Conference on Robotics and Automation (ICRA), pp. 2980-2987, 2023.
Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learn ing to walk in minutes using massively parallel deep reinforcement learning. In Faust, A., Hsu, D., and Neu- mann, G. (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceed- ings of Machine Learning Reseurch, pp. 91-100. PMLR, 2021. URL ht.ts:: /ttees//prrooeee http//poeeddngggeeoee ngs ml press/ v164 v164 0044/uuii222.. html.
Ost, J., Mannan, F., Thuerey, N., Knodt, J., and Heide, F. Neural scene graphs for dynamic scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2856-2865, June 2021.
Perolat, J., Vylder, B. D., Hennes, D., Tarassov, E., Strub, F,, de Boer, V., Muller, P., Connor, J. T., Burch, N., An- thony, T., McAleer, S., Elie, R., Cen, S. H., Wang, Z., Gruslys, A.. Malysheva, A., Khan, M., Ozair, S., Tim- bers, F. Pohlen, T., Eccles, T., Rowland, M., Lanetot, M.. Lespiau, J.-B.. Piot, B., Omidshafiei, S., Lockhart, E., Sifre, I.., Beauguerlange, N., Munos, R., Silver, D., Singh, S.. Hassabis, D., and Tuyls, K. Mastering the game of Stratego with model-free multiagent reinforce- ment learning. Science, 378(6623):990 -969999999999999 2022. doi: 10.1 id6/
Schaul, T., Quan, J., Antonogiou, I., and Silver, D. Priori- tized experience replay. In International Conference Learning Representations, 2016.
Scheel, O., Bergamini, L., Wolczyk, M.. Osinski, B., and Ondruska, P. Urban driver: Learning to drive from real- world demonstrations using policy gradients. In Confer- ence on Robot Learning (CoRL.), 2021.
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In International Con- ference Learning Representations, 2016.
Petrenko, A., Wijmans, E., Shacklett, B., and Koltun, V. Simulating embodied agents at one million experiences per second. In International Conference on Machine Learning, 2021.
Megaverse:
Schulman, J., Wolski, F, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization al- gorithms, 2017. URL http://arxiv...r abs// 1 06347.
Petrenko, A., Allshire, A., State, G., Handa, A., and Makoviychuk, V. PDexPBT: Scaling up dexterous ma- nipulation for hand-arm systems with population based training, In Robotics: Science and Systems, 2023.
Shacklett, B., Wijmans, E., Petrenko, A., Savva, M., Batra, D., Koltun, V., and Fatahalian, K. Large batch simula- tion for deep reinforcement learning. In International Conference on Learning Representations, 2021.
Philion, J., Peng. X. B., and Fidler, Trajeglish: Learn- ing the language of driving scenarios. In International Conference on Learning Representations (ICLR), 2023.
Shacklett, B., Rosenzweig, L. G., Xie, Z., Sarkar, B., Szot, A., Wijmans, E., Koltun, V., Batra, D., and Fa- tahalian, K. An extensible, data-oriented architecture for high- performance,, mory-world simulation. ACM Trans. Graph., 42(4), 2023.
Plappert, M., Sampedro, R., Xu, T., Akkaya, I., Kosaraju, V., Welinder, P., D'Sa, R., Petron, A., de Oliveira Pinto, H. P., Paino, A., Noh, H., Weng, I.., Yuan, Q., Chu, C., and

---

Robust Autonomy Emerges from Self-Play
Shi, S., Jiang, L., Dai, D., and Schiele, B. Motion trans- former with global intention localization and local move- ment refinement. Neural Information Processing Systems, pp. 6531-6543,
Wang, W. and Zhen, H. Joint-multppthh+ for simulation agents. Technical report, 2023. URL https://
com wwwpwwymowwww///.oo...oooooo.. waayoo--ulodd tLLes//esearcch/222330000ee 20Reports hm_ htmps//ssshs_m_sootrrrpp pdf.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., Chen, Y., Lillicrap, T. P., Hui, F., Sifre,, L., van den Driessche, G., Graepel, T., and Hassabis, D. Mas- tering the game of Go without human knowledge. Nature, 550(7676):354 2017. doi: 10. 0.108//ATURRE24270)
Wang. Y., Tang, H., Huang. L.,, Pan, L.., Yang, I.., Yang, H., Mu, F., and Yang, M. Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, (5):845-860, 2023a.
Wang, Y., Zhao, T., and Yi, Multiverse transformer: 1st place solution for Waymo open sim agents chal- lenge 2023, 2023b. URL https://arxiv.. abs/2306.11866
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I. Lai, M., Guez, A., Lanctot, M., Sifre, L.,, Kumaran, D., Grae- pel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362 (6419); 1000:1140-11110 144, 2018. doi: 10. aar4444
Wijmans, E., Kadian, A., Morcos, A., Lee, S., Essa, I., Parikh, D., Savva, M., and Batra, D. DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations ICLR, 2020.
Stewart, T. Overview of motor vehicle traffic crashes in 2021. Report No. DOT HS 813 435, 2023. URL
wtt.nhtt......o// gov/ Api htpp//ublicc//Leewubllcccc http://wwww.aaaaaaa//a///1/1 tional Highway Traffic Safety Administration.
Xu, D., Chen, Y., Ivanovic, B., and Pavone, M. Bits: Bi- level imitation for traffic simulation. In International Con- ference on Robotics and Automation (ICRA), pp. 2929- 2936, 2023.
Na-
Suo, S., Regalado, S., Casas, S., and Urtasun, R. TrafficSim: Learning to simulate realistic aulti-agent behaviors. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10400-10409, 2021.
Yang, B., Su, H., Gkanatsios, N., Ke, T.-W., Jain, A., Schnei- der, J., and Fragkiadaki, K. Diffusion-ES: Gradient- free planning with diffusion for autonomous driving and zero-shot instruction following, 2024. URL https:
Tao, Y., Gene, S., Chung, J., Sun, T., and Mallya, S. Repaint: Knowledge transfer in deep reinforcement learning. In International Conference on Machine Learning (ICML), pp. 10141-10152, 2021.
/ / araii.vv.co org/ abs/ ht///bb//224000 2402.
Yang, Z., Chen, Y., Wang, J., Manivasagam, S., Ma, W.-C. Yang, A. J., and Urtasun. R. UniSim: A neural closed- loop sensor simulator. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1389-1399, June 2023.
Treiber, M., Hennecke, A.. and Helbing, D. Congested traf- fic states in empirical observations and microscopic sim- ulations. Physical Review E. (2(2):1805-11 1824, August doi: 10. 103/physreve.62. 1805.
2000. ISSN 1095-3787.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Póczos, B., Salakhutdinov, R., and Smola, A. J. Deep sets. In Neural Information Processing Systems, 2017.
Vinitsky, E., Lichtlé, N., Yang. X., Amos, B., and Foerster, Nocturne a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. Neural Information Processing Systems, 35, 2022.
Zhang, C., Tu, J., Zhang, L.. Wong, K., Suo, S., and Urtasun, R. Learning realistic traffic agents in closed-loop. In Conference on Robot Learning (CoRL), 2023.
Vinyals, O., Babuschkin, I., Czamecki, W. M., Mathieu, M.,
Dudzik, A.. Chung, J., Choi, D. H., Powell, R., Ewalds, T.,
Zhong, Z., Rempe, D., Xu, D., Chen, Y., Veer, S., Che, T., Ray, B.. and Pavone, M. Guided conditional diffusion for controllable traffic simulation. In International Confer- ence on Robotics and Automation (ICRA), pp. 3560-3566. 2023.
Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I.,
Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M.,
Vezhnevets, A. S., Leblond, R. Pohlen, T., Dalibard, V.,
Budden, D., Sulsky, Y., Molloy, J., Paine, L., Gülçehre, Ç., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level StarCraft II using multi agent reinforcement learning. Nature, 575(7782): 350- 354,,4,,240 2019. doi: 10.
12

---

Robust Autonomy Emerges from Self-Play
A. Simulator Design
at once and then consume that buffer until it is empty as new initializations requested, e.g., when episodes reset. This allows for better cost amortization when generating initial states. Second, we use sequential rejection sampling to produce collision-free initializations. We add one agent to the world, then add a second agent that is not in collision with existing agents (via rejection sampling), and so on until we reach the desired number of agents. This approach was necessary because the probability that an independent sample of over one hundred vehicles contains no collisions is extremely low for small maps.
GIGAFLOW is a batched simulator (Makoviychuk et al., 2021; Freeman et al., 2021; Petrenko al., 2021; Shacklett et al., 2021), implemented in PyTorch (Ansel et al., 2024) and designed for GPU acceleration. A single instance of GIGAFLOW simulates thousands of worlds, enabling it to leverage the parallelism of modern GPUs. Implementing GIGAFLOW efficiently required the development of custom batched operators for the kinematics, collision checking, initialization of urban driving environments, and many other features. Below we outline the overall flow of GIGAFLOW and describe the acceleration techniques used.
A.2. Dynamics model update
At every timestep, GIGAFLOW takes the current state of the i- th world s and vehicle controls a.
The dynamics model (described below) produces g(++1) given s(t) and a This is set of element wise operations (trigonometric functions, multiplications. divisions, etc.) that are parallelized on a GPU using their respective PyTorch implementations.
(c) to produce the state at the next timestep 8 (t+1), Instead of running multiple copies of the environment to simulate N = 38 400 worlds, a single instance of GIGAFLOW simulates N worlds in parallel. The state and vehicle controls of all N worlds are denoted s(() and (f).
A.3. Road localization
The next step is to localize (1++1) on the road surface.
A.I. World initialization
Road representation. GIGAFLOW represents the road sur- face as a set of potentially overlapping polygons, exclusively using convex quadrilaterals. We find quadrilaterals offer a good balance between expressivity and the simplicity of op- erations. A given lane on the road surface is approximated by quadrilaterals that have the same width as the lane and are m in length. We find this resolution of polygons to be a good trade-off between the accuracy of road geometry approximation and the total number of primitives.
The simulation process begins with the initialization of ur- ban driving environments by placing up to N₂ 150 vehi- cles at random positions on the map. We first draw a random sample over map locations, vehicle headings, and vehicle bounding box dimensions and reject states that are off-road (off-road checking is detailed below). A naive application of this process results in a marginal distribution of vehicle locations that is biased towards wider road sections. To cor- rect for this, we estimate this marginal distribution from an initial sample and use adjust the proposal distribution used in subsequent rejection sampling. Given the set of valid vehicle states, we then select a collision-free subset with the desired number of agents (collision detection is detailed below). This subset becomes the initial traffic configuration at f 0.
It should be noted that the number of geometric primitives could be minimized further by merging polygons in the re- gions with simple geometry, such as straight road segments. However, we retain the uniform polygon density irrespective of the geometric complexity because this polygonal subdi- vision additionally serves as a spatial hash map for certain types of queries (c.g., map observations are pre-computed for all polygon center points).
For each retained vehicle, we select sequence of its way- points (goals). The first waypoint is sampled uniformly over the map and additional waypoints are sampled such that given the jihh waypoint, the (j + 1 )th waypoint is at least 20 m away and no more than 200 m away, and has lane heading that is within 60 degrees of the jth waypoint's 's lane heading. There are cases where the jth waypoint is in a loca- tion where these constraints cannot be met (e.g., a dead end). In these cases, we gradually relax the constraints as we try to sample points that fit them. The intent of this sampling procedure is to generate waypoint sequences that resemble realistic driving routes where intermediate destinations are reached in a natural succession.
Frenet coordinates. Let represent the distance along a lane, d be the distance from lane center, and poly I be a unique identifier of the polygon approximating the road geometry at the current location. The Frenet coordinate for position (æ. y) is then (g. d, polyId).
We convert world-frame state s()) to Prenet-frame state (() as this information is useful for constructing actor obser- vations and covers off-road checking in the majority of cases. We construct the Frenet-frame state by first finding the polygon that contains (x., y). Given this polygon, we can compute (g, d) by transforming (a, 3) into the coordinate frame defined by the polygon's 's heading and center point then adding the distance from the start of the lane to the
We use two acceleration techniques specific to initialization. First, we draw a large buffer of vehicle states and goals all
13

---

check Robust Autonomy Emerges from Self-Play
polygon's center q. If multiple polygons contain (ar y), we use the polygon where the agent is most lane centered (where d is the smallest).
We again use our spatial hashing data structure acceler- ate the check between vehicle bounding box and the OOB points, Specifically, each OOB point is first associated with its hash bucket. Then, we lookup which buckets the vehicle overlaps with and perform a point-in-boudding with that set of points.
Naively implementing this would require performing O(N x A x P) point-in-polygon checks, where N is the number of worlds, is the number of agents, and P is the number of polygons. This is prohibitively computationally expensive.
We associate the vehicle bounding box with appropriate hash buckets as follows: first, we set the bucket size to be 2x the maximum possible vehicle length. Then, we approxi- mate the vehicle's 's oriented bounding box by its axis-aligned bounding box (AABB). Due to our choice of spatial hash bucket size, the AABB overlaps with at most 4 buckets, each containing (at least) one of the AABB's ' corners. We retrieve the OOB points that correspond to S 4 buckets overlapped by the vehicle and perform a noint-in-bounding-- with each. We find that this fairly simple approximation is appropriate since it enables very fast lookup which is crucial as it is performed at every simulation step.
Spatial hashing. We accelerate this via spatial hashing on the GPU. Specifically, we first construct 2-D grid of non-overlapping axis-aligned boxes with fixed width and height. This scheme allows us to trivially map a point (as, 3) to its given bucket in the hash map. We then assign each polygon in the road representation to the (potentially multiple) buckets that it overlaps. Note that this step only needs to be performed once at startup.
check
We localize (I. y) by first retrieving the polygons that are in its bucket and then performing a point-in-polygon check for each matching quadrilateral. One challenge of implement- ing this efficiently is that each hash bucket has a different number of corresponding polygons. We maintain efficiency by designing APIs to expect these "ragged" variable-size tensors and only perform padding on operations that require it (e.g., finding the polygon that minimizes d).
Gaps in the road surface. Due to the extremely low off- road rates achieved by GIGAFLOW policies, we found that most remaining off-road events are characterized by thin gaps in the road surface (sometimes under 1 mm), often located between neighboring lanes. These thin gaps are not actual geometrical features but numerical inaccuracies in the underlying map files.
This spatial hash additionally allows for the support of mul- tiple maps via augmentation of vehicle coordinates (a, y) with the map Id: (z., y, ma) Essentially, the map an agent is driving on functions as a "third" dimension, allow- ing us to filter out polygons from other maps at the hash bucket retrieval step.
During training, we allow any of the 5 points used for off- road checking to be in one of the spurious gaps by measuring the distance from the point to the closest road polygon and considering points within ā 15 cm be still the road surface. We again use spatial hashing to accelcrate this lookup. We take edges of all road polygons and assign them to any bucket that they overlap with or pass within the distance 5 of, so we can quickly obtain a list of candidate edges.
A.4. Off-road checking
Given s(1), we localize the vehicle's bounding box center and corners the road surface using the spatial hash proce- dure described above. In the majority of cases, vehicle is off-road if any of these 5 points could not be localized onto the road surface. However, there are two edge cases that are important to handle.
A.5. Collision detection
Given s(!) and s(t-1), we detect collisions as follows. Let sa and sa, (I+1) be the successive states of agent j. We perform two Saz
(t+1) be successive states of agent i, and s(t) and
Curved roads and islands. These are various situations where all 5 of these points can be the road surface but the vehicle should still be considered off-road. Two such examples are curved roads (where part of the bounding box overhangs the edge of the road) and pedestrian safety islands (where the vehicle can straddle the islands). To handle both these cases, we generate a set of out-of-bounde (OOB) points by taking the mid-point of each polygon edge, nudging slightly in the outwards direction, and keeping all points that do not lie within any other polygon. A vehicle is classified as roa- if any of these OOB points are found to be within its bounding box.
(t)
checks to see if a collision occurred. First we transform se, (t+1) into the coor- into the coordinate frame of sa. and sa; dinate frame of sa. lines defined by the movement of agent j's corners intersect with agent i's bounding box (the bounding box is centered at the origin). We then swap the roles of agents and j and perform this check again. A collision occurred between agents and j if either check is positive. Note that we only perform collision detection, not collision simulation.
(t+1), Then we check see if any of the
We accelerate this using our spatial hash by constructing, for all agents, the axis-aligned bounding box (AABB) that
14

---

Robust Autonomy Emerges from Self-Play
contains the vehicle's s bounding box at both states s(t) and g(++1). We assign the vehicle all buckets that overlap with this bounding box and perform the collision detection procedure described above for all pairs of vehicles that share at least one bucket. We use the world ID of each agent as an additional dimension in the spatial hash so that vehicles can only collide with other vehicles in the same world. This hashing method is fast to compute and greatly reduces the number of candidate collision pairs. Therefore hashing represents a large improvement over a naive pairwise check.
We additionally developed multiple simplified versions of GIGAFLOW to speed up diagnostics and troubleshooting. Using the minimal single-agent configuration we could train policies to convergence in under 20 minutes, allowing us to get immediate feedback on the core functionality. We established infrastructure for continuous testing, repeatedly retraining policies from scratch using these streamlined sim- ulator configurations. This system has significantly acceler- ated the identification and isolation of regressions, thereby conserving numerous development cycles.
The check described above only works when (at least) one vehicle is in motion. This is acceptable during simulation, as at least some movement is required to transition from collision-free to a colliding state. For detecting collisions see if the two bounding boxes overlap at state s We use the spatial hash in similar manner as before, except the AABB corresponds to a single (initial) state.
B. Defining the partially observed stochastic game
We model the environment that our agents learn in a partially observed stochastic game (POSG) (Hansen et al., 2004); an extension of POMDPs to the multi-agent setting in which there are multiple agents with conflicting goals. We define each of the components of the POSG below.
= 0, we simply check to
A.6. 2.5-D simulation
B.1. Observations
Simulating driving can be largely approximated without errors as a two-dimensional problem. This approximation enables performance improvements and reduces code com- plexity (thereby limiting the surface area for coding errors). However, certain cases cannot be accurately approximated in two dimensions, like overpasses. GIGAFLOW handles this with "2.5-D" simulation. We simulate the world as if it were 2-D and then correct for these errors. For example, we per- form collision detection in two dimensions and then remove collisions that would not have occurred in 3-D. We maintain the vehicle's s 2-coordinate by applying the dynamics model purely in 2-D and then looking up new (1+1) for all vehi- cles from the map that correspond to new locations (t++1).
We render the world state s(t) into a relatively low dimen- sional vector representation. In order to drive safely, a GIGAFLOW agent utilizes information about vehicle's 's dy- namics and its position w.r.t. the lane Stt), approximate map of the surrounding area, including roads and traffic lights (withh w-(t) lane boundary wo(op) observations of other traf- fic participants A(1,, desired destination and intermediate waypoints C(O,, as well as the agent's conditioning Creward (Table A2).
Local observation 5(() can be further broken down as fol- lows (time indices omitted for clarity):
We use z (0, s(t), and the maximum slope of given map to filter out conflicting values in cases like overpasses.
c, 0: the distance from the current lane center and the angle relative to lane heading.
A.7. Hardening
k: local road curvature.
Given the extremely low collision and oof-road rates seen in GIGAFLOW training we found that extremely rare bugs would dominate the collisions and off-road events when present. We iterated extensively; training numerous policies to very high fidelity, watching videos of collisions and off- road events, and sieving those caused by coding errors and numerical inaccuracies rather than the agent's poor decision making. Each iteration would yield new bug fixes, improv- ing both training and downstream benchmark performance.
current speed of the vehicle.
Vimm maximum allowed speed.
d: current steering angle.
Gjong, Oaai current longitudinal and lateral acceleration.
Driver's acceleration limits Cace-
To address these bugs, we built numerous visualization tools, as merely knowing that rare bug existed was not enough to fix it it- we had to find the exact steps to reproduce it as well. The visualization, diagnostic, and recording tools we developed were instrumental to the overall success of the project.
Cinnutee Cueer: randomized coefficients determining the vehicle's responsiveness to throttle and steering inputs.
I, we driver's vehicle's length and width.
15

---

Robust Autonomy Emerges from Self-Play
Note that all of these observations are normalized to [-1,1] range and provided in egocentric frame.
tions, the Cartesian product of the sets of available values of
longitudinal 0gong E (-15, 4, 0, 4) ms and lateral jerk Alat E {-4, 0, 4) ms
Map observations outline the surrounding area at two lev- els of resolution, including the high-level overview of the nearby road network W lane and more detailed set of fea- boundary that represent the precise shape of the local driveable area.
We compute longitudinal and lateral accelerations using numerical integration (At = 0.3 s during training):
tures W(())
(8) (t-1) + Cihroute Atong At
The coarse map view Wr( provides a set of positional features sampled along driveable lanes at 40 m intervals, containing information about lane widths and heading di- rections. These features also convey higher-level routing in- formation similar to that provided by OPS-based navigation applications. Each observation within wr(t) is augmented with relative (w.r.t. other observations) and absolute normal- ized distances to the next goal, which allows GIGAFLOW agents to make informed routing decisions at complex junc- tions. We pre-calculate all routing information by running the Dijsktra's 's algorithm (Dijkstra, 1959) offline, computing pairwise distances between all locations of interest.
Goog
Gong
(1)
(1) (t-1) + Cstcer Ulat At
Diat
(2)
where coefficients Cinnsule Csteer E CCyyaaiccs are sampled from a mixed uniform distribution X (1.25), defined as:
((a) = 0.5 0.5 (a- (a= (1, a a>1
This distribution generates equal number of samples smaller and greater than one, thereby allowing for a bal- anced randomization of dynamics properties.
We apply a small modification to the Eqs. (1) and (2), setting (t) (8)
values Gong Ciat to exactly 0 when acceleration changes sign (i.e. when (t-1) (t)
For more fine-grained road geometry representation wiit)
doog "iong < 0). We found that this modification
houndary we provide midpoints of nearby polygon edges spaced roughly 1 m apart which delineate the out-of-bounds areas closest to the driver (Fig. 2c). For both wrl0 and writ)
makes it easier for the agent to wait in place or drive at constant velocity, producing smoother trajectories.
The acceleration components are then clipped ensur- ing the g g-forces stay within the specified limits (here Cuce x(1.5)):
boundary we provide the 80 features closest to the driver, additionally limited to 200 m viewing horizon for the coarse map. We pre-calculate Wime and W boundary for each map polygon which enables extremely fast retrieval runtime, only requiring a simple conversion to driver's egocentric frame to obtain (:(1)
(1)
(t)
Goog clip
(iong -5, 2.5 Cace
lane and W((t)
boundary"
(t) clip
1 4, 4
(t)
For observations of other agents we return the No nearest agents within δmax = 200 m of the driver. We pad the array of observations in case there are fewer than N, nearby actors. For each agent, we observe its position, orientation, velocity, dimensions (i.e. width and length), and z-coordinate, all transformed into driver's egocentric frame. The goals, mo- mentary accelerations, dynamics properties, or conditioning parameters of other agents are not observed.
We update the velocity magnitude using the trapezoidal rule (averaging previous and current accelerations):
(+) = (-(-1) + 0.5
(Soog
(1-1) At "
Just as for the accelerations, set ((() to exactly 0 when its value changes sign. We then clip u(() to stay within the randomized speed limit (Cvel ~ X(1.5)):
While we do not explicitly model occlusions, we keep the maximum number of observed agents during train- ing relatively low: N,- = 20 < No- Together with the distance-basee based visibility threshold 8mmx this allows us to train the agent well adjusted to limited observability. Our model architecture allows the value No be trivially increased at test time to all available information (see the model architecture details below).
1 + -ciip (1'0, -2, 20 200
To reach the acceleration (t) low the are with radius |p] and signed curvature p-1 apply- ing the steering angle ø (here lub is the vehicle's wheelbase and e = 10-5 ensures numerical stability):
0 the vehicle would have to fol-
permutation-invarite
B.2. Actions and dynamics
Glat
max ( c))
Our agents discrete set of actions to control the vehi- cle's change in acceleration using jerk-actuated bicycle dynamics model. The action space includes 12 total ac-
p sign (p-1) max (1e)
arctan (p-1lwb) (ub)
16

---

Robust Autonomy Emerges from Self-Play
We calculate the change in the steering angle, dg. and update the effective steering angle, s ensuring that both values remain within conservatively defined limits (6mux = 0.6
Roomton is penalty for exceeding the comfortable lim- its of acceleration and jerk. Randomized comforr cre- ates distribution of driving styles, from relatively smooth to practically unconcerned with comfort.
0max = 0.55rad):
d clip(p ( -dmax At, 6max At) ( = clip(p + 6ø. -max, pmnx)
(3)
alggi incorporates the preferences for driving in the designated driving direction and staying parallel to road lanes. We randomize this term in a wide range, which occasionally exposes our agent to erratic actors driving against traffic.
(4)
The clipping in Eqs. (3) and (4) prevent excessive changes in the steering angle, limiting the maximum lateral accel- eration at low speed. To account for the limited steering actuation, we update the effective signed curvature p=1 and (f) acceleration dja accordingly:
Ri-eenter rewards the agent for staying centered in the lane. We randomize Okeentr-hhias for additional behavioral diversity.
tan ())
Rvelocity encourages forward progress and motivates the agent to prefer routes with consistent traffic flow over traffic jams. We found that this term accelerates convergence in the early stages of training and helps us train policies that are better at avoiding gridlocks in self-play.
Lwb 2
(t) lat
f- (0(1)²
p-1
Finally, the resulting movement of the vehicle is updated using the bicycle dynamics model:
Reeverse penalizes the agents for reversing. Due to the randomization of Oreverse some agents perform multi- point turns reach the goals behind them, while others prefer to drive forward and wrap around the block or perform a U-turn instead.
d 0.5 (0()) + ol(-11) At
0- p
Az p sin (0)
Rato-line penalizes the agent for crossing a stop line at a red light.
Ay cos (0)
where d is the displacement along the are and @ is the angular displacement.
Ramestep a small penalty applied at each simulation step. This serves function similar to the discount % except disable Rümestep when ego is stationary which creates agents more willing to patiently wait at traffic lights and intersections when necessary.
B.3. Reward
Our agents optimize a scalar reward function consisting of multiple terms weighted by their respective coefficients 0(-) which are randomly sampled at the beginning of each episode (see Table A2). This reward function is defined by:
B.4. Additional randomized components
In addition to the reward function coefficients we randomize a number of simulator's components to generate diverse embodiments and behavioral patterns. Agents have access to the randomized parameters as a part of their conditioning.
R- Rgost + Reollision + Rott-road + Reomfort Riane
+ Rvelocity Breverse Rstop-lime + Ramestep
Vehicle size. We randomize vehicle's dimensions as follows (measured in meters):
where the individual terms can be described as follows:
Ryoal rewards the agent for reaching its intermediate goals (waypoints) and the final goal.
Vehicle length and wheelbase: ~ U 7), Lub 0.6
Raollision penalizes agents for colliding with vehicles or pedestrians with an additional penalty for collid- ing high speed. Randomizing Occoliision allows us to populate the training environments with agents of varying risk tolerance, from very aggressive to very conservative.
Vehicle width: us ~ 3().8,3).
The vehicle width is clipped: w + min(w, L).
Size randomization allows us to generate a variety of road users, from very maneuverable agents with a footprint of a pedestrian to relatively big vehicles. We represent occa- sional bigger vehicles at test time using a set of smaller
Roll nnd penalizes agents for leaving the road.
17

---

Robust Autonomy Emerges from Self-Play
bounding boxes (e.g., a semi-truck with a trailer can be split into two bounding boxes).
eration of other traffic participants to ensure safety. Ideally, even clearly erratic maneuvers by other vehicles should not lead to collisions or comfort violations on behalf of GIGAFLOW agent. In order to achieve this, we degrade a fraction of agents during training by introducing two types of modifications:
Goals. A final goal is sampled for each agent. Additionally we sample Nup ~ U',,,3, 3} intermediate waypoints (see "World Initialization" for details). We randomize the goal collection radius dgoal to control how precisely our driver is required to adhere to the designated route (see Table A2).
1. Up to 5% of agents occasionally do not see other vehi- cles. This models inattentive drivers and drivers with blind spots.
Vehicle dynamics. For each agent in each episode we ran- domly sample maximum permitted acceleration, velocity, throttle and steering coefficients (refer to "Actions and dy- namics" " for specific randomization ranges).
2. Up to 10% of agents sharply apply their brakes arbitrary moments for short duration before resuming normal driving. This models drivers that stop without warning or remain stationary when traffic light turns green.
Traffic lights. Real driving situations include a diverse set of traffic light systems, from synchronized lights at 3-way and 4-way intersections to occasional faulty, disabled, or miscalibrated signals. Instead of explicitly modeling this distribution in GIGAFLOWW we simply pick random dura- tions of green, yellow, and red signals for each traffic light independently. We start with default signal durations used in CARLA (typically fred = 2, yeelow = 3, and fgneen = 10 seconds) and then randomize them for each episode within the following ranges:
Between random applications of these modifications, agents are still controlled by GIGAFLOW policy and are completely oblivious to their modification, which makes it impossible for regular drivers to predict their erratic behavior. We exclude trajectories from these agents from the rollouts used for training.
Tred ~ (0.15 Fred, 5.0 Fred)
C. Training algorithm
Tyellow ~ (0.5 (0.5ffyllow:
fyellow; 0.75 fyellow)
Agents are trained using version of Proximal Policy Op- timization (PPO) (Schulman et al., 2017) derived from the Stable Baselines codebase (Raffin et al., 2021). PPO trains a policy that outputs actions, and a critic that estimates dis- counted returns. In our implementation, we do not share any parameters between the policy and the critic, which empirically produces comparatively more robust low en- tropy policies convergence. Additionally, we add the terminal value estimate to reward at the end of all truncated episodes, thus emulating infinite-horizo learning, similar to "Reset Handling" in Rudin et al. (2021). We use Adam optimizer (Kingma & Ba, 2015) with annealed cosine-rate learning schedule:
Tgreen ~ (0.1 - Tgreen; *green)
We additionally remove 20% of individual lights, 20% of traffic light groups (c.g., all lights at an intersection), and in 20% of all episodes we disable traffic lights entirely, making all intersections unregulated. We also set 5% of all remaining traffic lights to be constantly green.
Even though we train only on 128 variants of CARLA maps (8 maps modified with affine transformations), this aggres- sive traffic light randomization allows us to combinatorially increase the total number of unique training environments and traffic patterns, preventing overfitting. Our policy learns to handle arbitrary traffic light configurations which helps significantly in benchmarks like nuPlan where traffic signal states are derived from the sense stack and often have errors.
(0)
π k K
]
a (k)
1 cos
π
2
where k is the current training iteration and K is the maxi- mum number of iterations. We use in DD-PPO (Wijmans et al., 2020) to train with mutliple GPUs. Specifically, each GPU collects experience independently and then gradients are synchronized before the model update.
The maximum red light duration in our simulator is delib- erately limited (Trea < 10s). This allows us to mitigate the impatient nature of discounted reward maximization. When durations of required stops are short during training, agents with Qiso--line >> O are never incentivized to run red lights to maximize their return by getting to the goal faster. We find that our agents, due to their reactive nature, generalize to much longer red light durations occasionally encountered in benchmarks.
During development, we optimized a subset of hyperparam- eters of PPO and GIGAFLOW simulator using Population Based Training (PBT) (Jaderberg et al., 2017: Petrenko et al., 2023). We conducted this optimization using a simplified version of GIGAFLOW training setup (single map, reduced randomization) which enabled faster iteration. Our PBT ex- periments informed a number of non-trivial hyperparameter
Modeling erratic drivers. We aim to train an attentive, rational, and defensive driver which does not require coop-
18

---

Robust Autonomy Emerges from Self-Play
Algorithm 1 Advantage filtering
D. Neural network architecture
Require: Initial model parameters O RL environment & EWMA
Our actor and critic are parameterized by virtually iden- tical compact neural networks with 3 million parameters each (6 million trainable parameters combined). At the high level, the architecture consists of fully-connected back- bone (1024 x 1024 1024) MLP) which receives concate- nated feature embeddings and outputs either a distribution over actions or scalar value estimate (see Fig, 2d).
decay / 0.25. 1: for to K - do
b Training iteration k 2: Besp + COLLECTROLLOUTS((,, Experience buffer
3: GAE
GAE(Besp: 7,
GAF advantages
4: Amax
max BeBp
A(t)
GAE
Max abs. advantage
5: Amna < 1x=0Amx + 1550 (3Ama + (1 3)Am
Observations represented by simple feature vectors (S()) G Crewand. etc.) are trivially encoded by smaller fully- connected networks, e.g.. 15 = MLP (S!O).
6: Filtering threshold: 7y +- 0.01 Amux
7: Baihared f- FILTER (Bep: AOXE <n)
Observations containing multiple features per agent (W/C0) w:(0) lane boundary: W A(t)) are represented by sets of feature vectors and require permutation-invant invarrann en- coders (Zaheer et al., 2017). For each feature type we employ a small fully-connected network to encode each element in set (e.g., an individual map feature in w followed by channel-wis maxpooling layer. Outputs of all maxpooling encoders are then concatenated.
8: θ 4- PPO PPO(Bntens, Buitened θ)
9:
end for
choices, from the learning rate schedule to comparatively large integration step At = 0.3 Table A3 provides our final list of training hyperparameters.
Advantage filtering. To make use of the vast scale of syn- thesized data, implement technique termed advantage filtering which can be viewed as variant of Prioritized Experience Replay (Schaul et al., 2016) where the majority of transitions are sampled exactly zero times.
Observation sets ((1) and ( represent some of the largest data structures in our computation graph containing over 10 individual feature vectors per inference step across all agents. Storing all of these observations in the PPO's rollout buffer would be prohibitively expensive. Instead, we store only the world states s(t) and reconstruct most of the observations as needed for each training minibatch.
Specifically, for each transition we calculate the advantage estimate GAE (Schulman et al., 2016) and then simply dis- card all transitions that satisfy (
GAB < The adaptive filtering threshold 1 is set to 1% of the moving average esti- mate of the maximum advantage magnitude at the current stage of training, thus rendering the method insensitive to the absolute scale of rewards (see Algorithm 1).
In addition to that, for the largest feature sets Wane and W boundary we omit a fraction of input features (akin to the dropout regularization technique). We randomly drop 40% of W boundary and 50% of Wime features during training for each agent. This feature dropout approach allows us to fit the full GIGAFLOW training system on 40 GB A100 GPUs, while additionally modelling potential sensor noise, increas- ing robustness of the policy and preventing overfitting.
Empirically, we filter out on average 80% of all samples throughout training (over 90% in the early epochs). This significantly increases learning throughput as we avoid com- puting gradients that contribute minimally to the overall parameter update. With the proposed adaptive threshold ") observe 2.3-fold increase in training throughput, from 0.53 to 1.2 million steps per second. Our experiments suggest that application of this technique not only acceler- ates learning, but also yields more robust policies (c.g., see Carla LAV results in Fig. A2). We hypothesize that advan- tage filtering could be beneficial across various RL setups, especially where data collection is cheaper than gradient calculation (c.g., training LLMs on synthetic data).
E. Benchmark evaluation
For each benchmark, we use the denchmark-provide simu- lation infrastructure, translate observations into GIGAFLOW, and run GIGAFLOW as a co-simulator. For Waymax and CARLA, we co-simulate a single GIGAFLOW step for each evaluation step. nuPlan additionally requires trajectory pre- dictions at each step. At each time-step, we roll out an entire scenario using GIGAFLOW policies for all agents for 0.3 sec- onds, and use the subsequent observed trajectory from the driver as its trajectory.
Previously Tao et al. (2021) explored experience filtering approach based on the fixed advantage threshold, selecting a subset of maximally informative samples from the teacher policy in the context of transfer learning. We further develop this idea and propose a general adaptive method compatible with broad class of reinforcement learning applications.
E.1. nuPlan benchmark evaluation
The nuPlan benchmark consists of training, validation and held out test set built from thousands of hours of data col-
19

---

Robust Autonomy Emerges from Self-Play
lected from Las Vegas, Boston, Pittsburgh, and Singapore. However, in our work we never use data from the training set. The benchmark provides three different challenges, open-loop (OL) Challenge 1, closed-loop non-reactive (CL- NR) Challenge 2, and closed-loop reactive (CI.-R) Chal- lenge 3. We focus only on challenge because it repre- sents the most realistic setting with both the driver and the traffic vehicles being closed loop. The reactive traffic ve- hicle agents are controlled by the Intelligent Driver Model (IDM) (Kesting et al., 2010) with their initial placement taken from real traffic distributions. However pedestrians are non-reactive and follow their logged trajectory. After initialization, the IDM agent will select a series of lanes (independent of the log vehicles trajectory) and follow a centerline lane path provide by nuPlan's 's map. Longitudinal acceleration control comes from the IDM equation.
tracks do not provide such information, Jaeger al. adapt the leaderboard code accordingly. We adopt this procedure and extract this information in a similar way. Moreover, we identified a bug in CARLA that can lead to faulty pedestrian states when a pedestrian collides with other traffic partici- pants. This bug can result in erroneous collision infractions; see video in the supplementary material. We obviate this bug by removing a pedestrian actor from the scene in case of an event. To ensure the same test bed for all competing approaches, we run all of them in our setup and report the re- sulting scores along with the scores reported by the authors. In addition to the expert in Jaeger et al. (2023), we compare to the CARLA autopilot and another expert from Chitta et al. (2023). Since simulations CARLA contain large amount of randomness, Table A6 presents the mean and standard deviations of the evaluation results for 3 different runs.
du di 1 (1 -
E.3. Waymax benchmark evaluation
To run the Waymax simulator, wrote distributed sce- nario runner to evaluate on the full Waymo Open Mo- tion Dataset (WOMD) 1.2.0 validation set consisting of 44 097 scenarios each 8s long running at 10 Hz. To match the results from Table 3 in Gulino et al. (2023), for our Dataset we use max_numLobject.. and use the default IDMRoutePolicy settings. We also applied this patch, https://githe: com/ laymo-researee http///wwaxx/ppll////e agent issues on top of commit 720f9214a in the pub- lic Waymax Github repo. For goals used for the GIGAFLOW policy we use the final location of the expert logged tra- jectory as an intermediate goal, as well as two more goals projected out at 50 and 100 m and snapped to the nearest lane center.
With acceleration limit a, desired speed vo. current speed v, distance to lead agent s, safety margin s", and exponent ö is usually set to 4. The agent defaults to acceleration unless it's close to desired Up or close to an object in its path. In this challenge, there is less diversity in behavior, but it has more realistic vehicle interactions when the driver deviates from the logged vehicle trajectory.
to fix IDM
Due to the nuPlan online leaderboard evaluation servers no longer being accessible, we evaluate GIGAFLOW on the Val14 benchmark, which has been shown to be a good proxy for the lcaderboard evaluation and has other publicly reported results (Dauner et al., 2023).
The GIGAFLOW policy achieves state of the art results zero- shot on the Challenge 3 closed loop score. The closed- loop score is measured from O to 100 and is weighted evaluation composed of ft-fa collisions, drive-able area compliance, driving direction compliance, progress towards goal, time-to-collssion bounds, speed limit compliance, and comfort. Full results are shown in Table A5. We are able to compute the component scores for the PDM-Hybrid entrant and simply present the total score for the other entrants. Finally, we categorized all videos of nuPlan's collisions and we provide them in the supplementary material.
We use the provided agent overlap, off road rate, log diver- gence, and SDC kinematic infeasibility metrics provided by Waymax to calculate our results. However, due to the Waymax route information not being released at the time of this writing, we report a modified metric for Route P Progress Ratio. We calculate it by dividing the dis- tance the GIGAFLOW policy drove by the expert logged trajectory per scenario, and clamp to 100% if the expert drove less than 1 m. We also report an additional progress metric, Total Distance Driven Ratio, which is simply the total distance the GIGAFLOW policy drove over all scenarios divided by the total the expert log drove.
E.2. CARLA benchmark evaluation
Jaeger et al. (2023) implemented a CARLA expert driver that serves as teacher for trained perception- based model. To our best knowledge, the presented expert manifests the best scores on CARLA benchmarks ever reported in the literature. This expert makes use of privileged information such as exact positions of all traffic participants and perfect map information. Since the default CARLA leaderboard
Unlike Carla and nuPlan, Waymax does not have an ag- gregate score which allows for easier comparison and visu- alization of results. Because of this, we propose sim- ple aggregate score composed of a progress score (i.e. a clamped Total Distance Driven Ratic), colli- sion rates, and off-road rates show in Eq. (5) below. This
20

---

Robust Autonomy Emerges from Self-Play
metric is what is reported in our figures when referring single Waymax score.
until a collision is longer imminent. To account for sensor noise in recorded data, add random Gaussian noise to all generated actor positions, We achieved a 2-point improvement in the overall metametric using o = 0.0125 when compared to rollouts without the added noise.
Σdg d expert
progress = min
1
Σ dg SES
Results of our zero-shot evaluation of GIGAFLOW to WOSAC are reported Table A8, using the Waymo-defined validation set to allow for detailed analysis in Table A9. In addition to GIGAFLOW we include comparison to a pol- icy where all actors are fixed to be stationary, establishing a lower bound on generated behaviors by the GIGAFLOW policy. We also include the reported performance for the top methods included on the public 2023 WOSAC leaderboard as well as a set of baseline approaches reported in Montali et al. (2023), nearly all based around supervised autoregres- sive modeling.
Σ
collissoo
poll-ood
SES
success =1 1
S
score progress X success
Where S is the set of scenarios, dg is the distance the policy under test drove in scenario 8, d, is the distance the expert policy drove in scenario 1 is the indicator off-rood function for if scenario 8 contained a collision, and g is the indicator function for if scenario s contained an off- road event.
We see that the GIGAFLOWW policy approaches the perfor- mance of expert models, despite having never been exposed to any map or human data recorded in WOMD during train- ing. Notably, GIGAFLOW offers strong performance for col- lision and off-road metries, out-performing most approaches. GIGAFLOW also maintains respectable performance among acceleration metrics, a consequence of producing kinemati- cally feasbile and smooth rollouts following the dynamics model the policy was trained with. With GIGAFLOW, we are able to control multiple actors within real-world scenarios without colliding or driving off-road, producing natural- istic driving. The scoring metrics used for WOSAC are likelihood measures of the ground truth behavior within the distribution induced by generated actor trajectories. To achieve idealized performance for the likelihood of colli- sion and off-road behavior, it is expected that the generated trajectories collide or drive off-road whenever ground truth actors are assessed to do so. Because GIGAFLOW explicitly penalizes these outcomes during training we cannot expect our policy to substantively improve over the reported scores in Table A8 for these metrics. We characterize the collision and off-road behavior of GIGAFLOW within WOSAC in Table A9, conditioned on actor type and whether the ground truth trajectories collide or go off--rodd
Table A7 presents Waymax results.
E.4. Waymo Open Sim-Agents Challenge
Waymo introduced their Open Sim Agents Challenge (WOSAC) in 2023 to promote the development of intel- ligent, interactive simulation agents capable of exhibiting diversity of behaviors in response to decisions made by the autonomous vehicle (Montali et al., 2023). In WOSAC, driv- ing scenarios are characterized by 9.1 second sequences of recorded tracks of road users derived from the Waymo Open Motion Dataset (WOMD) (Ettinger et al., 2021), stored at 10 Hz, where the first 1.1s s of history is used to form an initial context. Models are evaluated by their ability to accu- rately reproduce the remaining 8s of the scenario for up to 128 agents.
Despite being designed for imitative traffic modeling, we use WOSAC to evaluate GIGAFLOW in real-world multi- agent settings and demonstrate the policy's 's ability to control diversity of road users while avoiding undesired behavior. Following the evaluation of other benchmarks, we evaluate fully trained GIGAFLOW policy zero-shot on WOSAC without fine-tuning. We use the provided 1.1s s history to conditionally sample from the map distribution of possible goal locations, filtering out positions that are either unreach- able or are trivially reachable. Proposed goal locations are retained proportional to the GIGAFLOW agent's value esti- mate from the provided initial position of each actor.
F. Analysis
F.I. Reward conditioning analysis
To capture a continuum of driving styles in GIGAFLOW we vary reward parameters for our agents and then allow the policy to condition on them. In total we condition 12 different parameters used in our reward shown in Table A2.
Behavior of all vehicles and bicycles is modeled using GIGAFLOW policy, while pedestrians are controlled with an IDM-like policy based on the provided initial velocity and heading. If a collision with a vehicle is forecast within a 0.5s horizon, the pedestrian actor stops moving forward
To better understand how these reward parameter conditions affect GIGAFLOW policy behavior, we attempt to find which parameters can explain the most variation in trajectories. We do this by sampling 200 random reward parameters
21

---

Robust Autonomy Emerges from Self-Play
within the training distribution range, roll out each of them on various scenarios in nuPlan, then cluster trajectories to find which conditioned reward parameter provides the most mutual information on the clusters. Because single reward parameter will typically dominate the variation seen in the trajectories, we repeat the analysis again but with the previ- ous dominant parameter fixed. We repeat this procedure for the top 6 parameters with the results shown in Fig. Al.
Fig. A2 demonstrates the effect of the advantage filtering technique described earlier. Using this method we filter out samples that have near zero contribution to PPO's policy and critic losses, thus focusing on rare events at the tails of data distribution. While we expect this technique to improve the wall time performance of the algorithm, to our surprise we also see qualitative difference at convergence. The version of GIGAFLOW without filtering plateaus substantially lower the Carla LAV benchmark and is unable to reach up to 5% of goals in time leading to timed out episodes in the self-play evaluation mode.
The order of parameters with highest mutual information with the clusters is Okcenter-bias followed by ågaal, Oll-centers Occomfort Ollaalinn and Ovel-align- As seen in Fig. A1, the vari- ation is interpretable and relatable to human driver. For Orenter-bias the policy will pick the left lane if it's biased to- wards the right (so that can be further from the boundary) and vice versa for the right lane. For ōgeat, the policy will change lanes get closer to the goal when the max distance is small enough. For ON-ceater. if this value is high enough, the policy prefers to stay centered in its lane to not incur the extra cost of a lane change, despite the additional progress could make. For Occomfont the policy cares less about high accelerations and jerks when this penalty is closer to zero, so will take more aggressive turns to stay within its lane and receive centering rewards. For anggn the policy incurs less reward for staying oriented with its lane if this value is low, and therefore if this value is low enough the driver is willing to U-turn to get to its goal behind it. Finally for Ovel-align we are rewarded for staying centered in our lane at higher speeds, and therefore when this value is low the policy takes a wider turn to maximize comfort.
Fig. A3 demonstrates the influence of various algorithmic features on zero-shot benchmark performance. Here, we quantify the error rate across multiple benchmarks; specifi- cally, we measure the additional percentage points needed to achieve perfect score on each benchmark.
F.3. Long-form evaluation in self-play
To evaluate robustness of our agent in self-play, we devise a version of GIGAFLOW training environment tuned for long- form evaluation. We remove dynamics noise, set the number of drivers to N₂ = 50, and slightly modify the initialization procedure to ensure minimal clearance between all vehicles at f = 0. We remove all traffic light randomization and simply use CARLA's default traffic light system to promote conventional interactions at intersections. We additionally modify drivers' observations to increase their perceived vehicle size by 10 em on each side.
We observe that at decision-making frequency of 15 Hz, GIGAFLOW agent conditioned for safe and conservative driving (see Table A4) exhibits nearly accident-fre driving, covering on average million km before encountering any collision or going off-road (Fig. A4).
By allowing our policy to condition on range of reward parameters, we solve a few problems at once while keep- ing the simplicity of a single policy. First, it introduces a wide variety of agent behaviors that makes our policy more robust. One reason is because the policy now must handle both the hidden goals and hidden behaviors of other agents. Due to this, the GIGAFLOW policy learns to handle vehi- cles that favor different positioning within the lane, drive aggressively, lane change more often, plus combinations of the above. The second benefit is that the optimal driving style could be captured in our conditioning range. So as an additional alignment step, the reward conditions can be tuned to reduce the error between the GIGAFLOW policy and an ideal logged driver. This method would then allow us create both a chaotic self-play training environment to increase robustness, and a safe and comfortable driving controller, all with the same policy.
F.2. Ablation studies
We perform series of experiments to understand how dif- ferent algorithmic choices influence the final performance of the GIGAFLOW agent using a reduced compute budget of 660 GPU-hours (~~ 30% of the final training run's 's budget).
22

---

Robust Autonomy Emerges from Self-Play
Parameter
Value
Total num, of maps w/ affine transformations
128
Number of agents per sim.
U [1, 150)
Number sims per GPU
4800
Max number of agents per GPU
720 000
GPUs used
8x Nvidia A100 (40 GB)
Total GPU hours used in training
1900
Total number sims
400
Total number of agents
5 760 000
Throughput: steps per second
200 000
Throughput: km driven second Training cost per million km driven Total simulated distance driven Total number of simulated state transitions
1850
$5 USD
000 000
1.000 000 000 000
Table Al: Setup and scale of GIGAFLOW training.
Reward
Training distribution
fool U(2, 12) U 3
Rgoal=
[( ==g (sc6gol A (Haagean (o)))))))))
Rooltison
(Ocoilision + 0p1))l)
collicion
Occollisoon U(0,3)
Rottwot== Ohoundary boundaey
Oboudday U(0,3)
Reoetor Occomfont [1 Insea [3-3 + (ow 233 + 1 Li 125 15
Occomfon: 0(0.0.0.1)
18y Ryalign Al-align&d (in (cos (8)),0) + Ovvel-alinn min (cos (8y) v. 0) + 0.0025 (1
Olaaliaa~> U(2.5 x 10-1, 10-
+/2
Oryel align U(U,1)
Ol-eemer U(2.5 10-4, 4,7.7.. 10
Recenter At At (coos )x0.s lay Occenter-iia]
0.95
exp | Quanen -0.5)
Ocemarrbia
U(-0.5, 0.5.
Reetocty Oryeteiity max (cos (0y), 0.00 m/22.5
Olvelocity = 2.5 10
Rrvee Creverss At 1
Orevena U(2.5 x 10 7.5 x 10-3)
Rsup line Okstup line stop line violation
Olsopp li U(0,1)
Rimestop (Orimescy At) 1 1 Ia|ps0
Orinesoop 2.5 x 10
Table A2: Training distribution for each of the reward components.
23

---

Robust Autonomy Emerges from Self-Play
Setting/Parameter
Value
Vehicle length
1~ (2,
Parameter
Value
Vehicle width
w~ (
Maximum speed
m0
Training batch size Batch size per GPU
256 000
Timestep At
0.066
32 000
Episode length
9000 steps (600s) 50
Rollout length
128
Agents per sim No.
Num. PPO epochs
3
Observed agents N.
up to 40 closest
Discount factor y
0.999
AGAE
0.95
6goll
10 m
Max, episode length
1200 steps (360s) 0.2
1goal
3 mss"
PPO clipping ratio
Occollision
3.0
Value function clipping
None
Oboundary
3.0
Initial L. (0)
5 10
Occondort
0.05
LR schedule
Cosine
O1-align
2.5 x 10-2
Entropy coefficient
0.01
Ovell align
1.0
Value loss coefficient
0.5
Ol-senter
3.8 x 10-3 0.0
Max grad. norm
0.5
Ocenter-bias
Advantage normalization
Enabled
Qvelocity
2.5 10-3
Adv. filtering threshold v
0.01 Amax (Alg. 1)
Oreverse
5.0 X 10-3
Inference & training precision
16-bit AMP
Ostop-line -ine
1.0
Model weights initialization
Orthogonal, zero bias
Otimestop
2.5 x 10-
Table A3: RL algorithm settings and hyperparameters used during training.
Table A4: GIGAFLOW settings for long-form evaluation in self-play.
Mathel
Scome
Fgo Progresst No AF-Clllisinns
Comfort
TTC in bounds
Driving Dirvction
Speed Spedliie limit
Drivable Aren 1
Ege Making Progros 1
Varl4 Benchmark
Urban Deiver (Scheel et al. 2021)
50
GPP Ollllganonn al., 2023)
PlanCNN (Rene all. .02)
72
IDM (Trcher at al,, 2000)
77
PDM-Hyted (Danner et ol., 2023)
92.1
90.2
19.1
94.8
93.5
99.9
99.8
99.5
99.1
Diffion-FS (Ying 2024)
92.3
93.4
90.7
GIGAIL oW (murs)
938-+011
93.6-0.66
98.4+0.99
96:4-0.27
93.8-0333
99.5-0.05
99.9-0.00
99.7-0.04
99.0-0.07
Table A5: Results on Val14 (Dauner al., 2023) nuPlan benchmark.
Mathod
DS1
RC*
IP1
Pod|
Vih.j
1ay
Red]
Stopl
(M))
Dey
TO1
Black
CI.1 Testing Router| [leadeeboeedd rdf papee/...r////o////
stareed vi)
CARLA Agent (our run)
10
0.73
10.03
0.00 10000
0.36
10.11
0.00 1.:00
0.28
10.02
0.00 10.02
0.00 1.0.00
0:64 10.02
0.01 10.00
0.15 10.00
Expert from larger et al. (2023) (our run)
90 10
96
11
0.94
10.01
0.00
:0.00
0.06
1001
00000
0.06
10:03
0.00
:0000
0.00
100.00
0.00
:0.00
0.08
10.00
0.05 10.00
GIGAFLOW (ouns)
11
0.95
10.01
0.00
10.00
0.07
10011
0.00
10:00
001
10.01
0,00 10,00
0.64
10044
10.01
10.02
0.00
10.00
LAY Benobmark (Chos Krähenbabl, 2022) (with odgpted scenorios from Jneger (202.))
CARLA Agent (our run)
912
1.0
0.15 10.04
0.22
1.000
0.74
10.43
0.00
10,00
3.99 10.10
0.58 10.21
0.00 100000
0.87 10000
0.00 10.00
0.00
10.00
Expert from Jaeger al. (2013)
94
95
0.99
0.00
U102
0.00
0:02
0.00
0.00
0.00
0.06
Expert From largor
(2023) (our
+9
+0.02
0.00
0.04
+0.05
0.00
0.02
0.00
0.00
+0.00
+0.07
=0.07
GIGAILOW (ours)
#11
=0.00
0.00
#0.00
0.00
#0.00
8.00
#0:00
0.00
#0.00
0:00
#0.00
0.00
+:0000
0:00
=0.00
40.00
0.00
=0.00
Lungeu5 Reachwark (Clitto ol, 2023))
CARLA Agent (our nin)
7:t
54 ::0
0.25 +0.01
0.31 +:000
1.50 #0.21
0.00
00000
1.84 1:0 06
0.10 =0002
0.00 110000
0,43 0..00
0.02 +0000
0.04 =0,00
Expert From
(3023)
37
0.02
0.28
0.33
Expert from laeger et al. (2023)
1.3
90 11
0.91 10.04
0.01
0.21
0.00
0.01
0.00
0.07
0.09
Expert from larger et al. (2023) (pur run)
83 #11
94 +22
0.88
=0.01
0.00
00.00
0.20
=003
0.00
00000
0.04
+000
0.02
00.01
0.00
+0:00
0.00
=0.00
0.00
10.00
0.06 =0.02
GIGAFLOW (ouns)
12
11
0.93
10.01
0.02
10.00
0.08
10.01
0.00
10:00
0.04
100002
0.03
10.01
0.24
10.00
0:03
10.02
0.05
1.0.00
0,00 10.00
Table A6: Results on CARLA benchmarks.
Mithod
(Iff-bred) Rate (1))
Cullision (1k)
Kinematic Inleschiltyy (9) |
Lay (Dl
Boste Prograss Rarke (4)
Tocal Diiesece Driven (93
Scceo (99
Expert Demonstration: (Gulno et al., 2023)
0.32
0.61
4.33
0.00
100.00
100.00
999077
Wiyfoomer (Gulino et 2023)
7.89
10:68
5.40
2.38
123,54
DQN (Gillne ai, 2023)
6 6.50111.11
0,00:000
9,83-014 ex
177.91255 677
89 10
BC (Gulao al, 2023)
0.11:022
4 4.99.0.06
0.00:0.0
2.26-:022
129 84-00.8
04.3.0221
GRGAR ow (ours)
0.41 +0/00
0-45-00005
14 H000
5.87
146 23 10.00
106 29 10.05
9-16 00000
Table A7: Results on Waymax benchmarks when evaluated with IDM agents. For the Score of other agents, we assumed 100% progress and mutually exclusive collision and offroad events.
24

---

Robust Autonomy Emerges from Self-
Method
Lincar Speed
Linear Accel.
Ang. Speed
Ang. Accel.
Collision
TTC
Road
Off-road
Composite Metrie
Exgent Demo." (Montali 2023)
0,5610
0,3300
0,5630
0.4890
0,4850
1,0000
0.3510
0,2130
15000
0,7220
Agem!
0.0440
0,0740
0.1200
0,0000
0.7340
0.1780
0.2570
0.1550
Extmpdaation
0.1190
0.0190
0.0350
0,7750
0.5020
0.4630
0.3240
Stationary Policy
0,0408
0.612
0,4984
0:0553
0.9466
0.736
0.2408
0.8575
0.5007
Joint Multiputh+= (Wang & Zhen, 2023)
0,4340
0.2300
0,5150
0,4520
0,3450
0.5670
0.8120
0,6390
0.6820
0,5330
PredSina"
0,4051
0.2208
0.4558
0.4653
0.3441
0.7193
0.8027
0.6167
0.7519
0,3663
Wayforrer* (Nayakamti at ol., 2023)
0,3310
0.0980
0.1130
0.4060
0,2970
0,5700
0,7820
0.5920
0.5660
0,5750
MoenaDMFF (Gino ol., 2023)
0.4355
0.2767
0,5230
0.4666
0,3678
0.7447
0.8128
0.6215
0.7392
0.5821
MIK++= (Qun et al., 2022)
0,4119
0.1066
0,4838
0.4365
0,3457
0.8630
0.7969
0.6545
0.8954
0.6077
CIGAFLOW (ouns)
0.2613
0.2534
0.5060
0.4756
0.3161
0.9470
0.8077
0.5445
0.9065
0,6190
VPD-BP" (Hunag 20224)
0.4751
0.2161
0,5358
0.4775
0,8162
0.8234
0.6628
0,9010
0.6315
MIR-E* (Qian 2023)
0.2353
0.5335
0.4753
0.7983
0.6541
0.9143
0.6348
MVTE (Wang 20236)
0,4476
0.2218
0:5353
0.8043
0.6641
0.9086
0.6468
Trajeglish" (Ptilion al. 2023)
0,4504
0.3929
0,5382
0.92226
0.8309
0.6396
0.3364
0.645]
InteractionFFeemer***
0,4294
0.2394
0.529]
0.4780
0.3776
0.9391
0
0.6466
0.9347
0.6587
Table A8: Per-component metrics, as defined and computed within WOSAC using the WOMD validation split. *-scores publicly reported on WOMD test set at https: / /wwwwwww........../.. wwt.coo///////oo wwwm//pe//////////////////eeeeee http///ppe/aaaalalleeaeaaaa hhtpcchaaaleneeeeeeeeeeeaaaaaael weeleeegg/2222 222.3/ 10022///imm/ sim-agentss and from Montali et al. (2023). All scores are derived using the 2023 metric definitions. Shaded rows correspond to approaches that do not use training data. We see that GIGAFLOW, in zero-shot evaluation, is capable of producing effective driving performance that approaches expert policies specifically developed for imitative traffic modeling using the provided training data, even without having been shown WOMD data or maps previously.
Collisions
Overall
Ego
Vehicles
Pedestrians
Bicycles
Method
Total
-C
Total
-C
C
Total
-C
C
Total
-C
Total
-C
C
Expert Demo.
0,0337
0.0000
1.0000
0.0048
0.0000
1.0000
0.0076
0.0000
1.0000
0.2986
0.0000
1,0000
0.0547
0.0000
1.0000
GIGAFLOW (ours)
0.0322
0.0085
0.7082
0.0021
0.0006
0,3223
0.0080
0.0051
0.3700
0.2391
0.0029
0.7938
0.0291
0.0003
0.5268
Stationary
0.0220
0.0017
0.6015
0.0021
0.0007
0.2832
0.0035
0.0012
0.3513
0,2012
0.0003
0.6731
0.0242
0.0001
0.3556
Off-road
Overall
Ego
Vehicles
Pedestrians
Bicycles
Total
-0
Total
-0
0
Total
-0
Total
-0
Total
-0
Expert Demo
0.1263
0.0000
1.0000
0.0127
0.0000
1.0000
0.0599
0.0000
1.0000
0,8513
1.0000
0.3611
0.0000
1,0000
GIGAFLOW (ours)
0.1 105
0.0072
0.8252
0.0071
0.0003
0.5367
0.0409
0.0051
0.6033
0.8385
0.0014
0.9847
0.2397
0.0013
0.6635
Stationary
0.0959
0.0060
0.7053
0.0041
0.0001
0.3220
0.0298
0.0052
0.5117
0.6999
0.0002
0.8201
0.2346
0,0002
0.6493
Table A9: Calculated rates by which vehicles controlled by either the GIGAFLOW policy or with the baseline stationary policy are assessed to be in collision (condition C) or drive off-road (condition O), grouped by actor type and whether these outcomes were assessed to occur in the ground truth. We anchor this analysis by the provided expert demonstrations recorded in WOMD. GIGAFLOW produces the expected vehiele behavior when the ground truth trajectories do not collide or drive off-road, achieving incidence rates well below one percent. While not directly rewarded in WOSAC scoring, the aversion of GIGAFLOW to colliding and off-road driving is evident among circumstances where the ground truth trajectories are assessed to do so. Aside from pedestrians, all other actors controlled by GIGAFLOW have incidence rates far lower than anticipated (where generated rollouts are expected to collide or drive off-road if the underlying ground truth trajectory does so). Notably, there is a proportion of trajectories that are initialized to be in collision or off-road (unsurprising given the presence of pedestrians) since the incidence rates of the stationary policy are not zero for these events in this conditional analysis. From this, we conclude that GIGAFLOW avoids an excess amount of collisions or off-road behavior as the incidence rates do not significantly increase when applying the eelf policy.
25

---

Robust Autonomy Emerges from Self-Plas
D022
4002
Oicenter-bias
O'goal
O1-center
Occomfort
O1-align
Orvel-align
Figure Al: Conditional analysis.
26

---

Robust Autonomy Emerges from Self-Play
Advantage filtering ablation: nuPlan score
Advantage filtering ablation: Carla LAV score
86
nuPann resccivv chalengg score
80
86
80
Caraa
70
nuPlon reactive, advarlage fibering
Carls LAV, advarlage fitering
nuPian reactive, advantage fhhoring
LAV, advantage fiterinng
10
20
90
60
90
40
60
70
Advantage fitternng ablation: crash rate self-play
Advantage filtering ablation: self-play episode timeouts
Robusiness, advantage litering
Timeouls, advantage fitering
advantage fitering
advantage fitering
Timed apisadiss
0.6
40
0.2
80
Training lime, h
Training lime, h
Figure A2: Training with and without advantage filtering (see Algorithm 1). Top row: nuPlan reactive challenge and Carla LAV Benchmark results. Bottom row: accident rate and task completion rate in self-play.
reward condttining
No dynamics
no rindomzed dynamics paramsters
nuFlan reactive
No randomized dynam naniiss paramoters
Carla LAV
dynamics
All festures
15
20
25
Combined enror rale, %
Figure A3: Aggregate impact of algorithmic features used GIGAFLOW. The length each bar indicates the cumulative percentage points required attain perfect score across three benchmarks.
27

---

Robust Autonomy Emerges from Self-Play
Combined colliaion and off-rodd rate, knitrash
10a
109
10a
10
Sim, Irequency during training. (At) 3.33 Hz
8
10
16
Decision making frequency, Hz
Figure A4: Accident rate in GIGAFLOW self-play different values of simulationnddecisoon--aaal frequency.
28


Reviews:
[{"review_id": "MFx5lD1vnd", "reviewer": "Reviewer_SjGz", "summary": "This paper presents GIGAFLOW, a batched simulator that supports large-scale simulation to train robust driving policies via self-play. Through learning from a massive scale of self-play, the learned policy demonstrates superior performance compared to the prior state-of-the-art on multiple challenging benchmarks, without seeing any human demonstration in training.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.\n\n> The intuition and motivation to use one policy for all agents is under-discussed in this work. Furthermore, the ablation study of comparing shared policy and separate policies for different types of agents is not reported in this work.\n\nN separate policies are technically hard to implement efficiently at scale for two reasons:\nComputational cost grows, as there is a significant amount of routing required from agent states to potentially non-uniformly batched inference and training steps / sequences\nSeparate policies likely train slower: Each separate policy sees 1/N amount of experience. Prior evidence suggests that embedding a population in a single policy can accelerate learning [1].\n\nThe amount of policies required to cover our continuous reward parametrization can potentially be quite large.\n\n[1] Liu, Siqi, et al. \"Neural population learning beyond symmetric zero-sum games.\" arXiv preprint arXiv:2401.05133 (2024).\n\n\n> Although this work shows promising results of learning robust policy via self-play, it is notable that all policies presented in this paper operate on highly abstracted vector space without sensor inputs, which is different from real-world driving agents that reason over their online sensor data.\n\nThe reviewer is right that scaling this work to sensor simulation or real-world deployment will require additional work, and presents an exciting avenue of future work. One thing to note is that we are the first work to show that a single policy can generalize well across several different vector maps (Carla, nuPlan, Waymo). The main reason for this is that our policy does not directly see the vector representation, but rather lane boundaries etc. These inputs can likely be inferred by a sense stack, although getting a good noise profile will be part of future work.", "claims_and_evidence": "Yes, the claims made in this submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Strengths of methods and evaluation:\n- **Large-scale reinforcement learning for driving**. The key component of the proposed method is to apply large-scale reinforcement learning for learning robust driving policy. Although reinforcement learning has been adopted for driving in prior works, it is the first time attempt to conduct reinforcement learning on such a large scale with over 42 years of experience. \n- **Strong generalization of learned policy on multiple benchmarks**. Due to the learning on massive experience with a wide span of scenarios, the learned policy demonstrates strong generalization to unseen environments in a zero-shot manner. Additionally, it sets new records for several challenging benchmarks like CARLA and nuplan without special adaptations or additional learning. These results provide encouraging signals for future studies on applying reinforcement learning to broader areas including driving and robotics. \n\nIssue of the method:\n- **Lack of study on the model design**.  This work mainly focuses on the simulation and experiment aspects with a lack of detailed investigation for model designs. For instance, according to L145-150 (right), this method uses a single policy for different types of agents with conditioning $C_{conditioning}$ as an indicator for the agent type. However, the intuition and motivation to use one policy for all agents is under-discussed in this work. Furthermore, the ablation study of comparing shared policy and separate policies for different types of agents is not reported in this work. Given the distinction between different agents, such investigations could further improve the soundness of this work.", "theoretical_claims": "No issues with the theoretical aspect of this work.", "experimental_designs_or_analyses": "Strengths of experiments:\n\n- **Extensive experimental results on multiple benchmarks**. This work presents extensive experimental results to demonstrate the superiority of the proposed method, such as Fig1 and Table (A5, A6, A7) in supp. Providing cross-environment results would empirically require a lot of effort, and thus should be encouraged and valued.\n\n- **Abundant qualitative analyses**. Qualitative visualizations in Figures 3 and 4 enhance the readability of this work and clearly illustrate the policy behavior during and after training.", "relation_to_broader_scientific_literature": "The key contributions of the paper relate to the areas of autonomous driving, traffic simulation, large-scale training, and applicable reinforcement learning.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Other Strengths:\n- This paper is well-written, well-organized, and easy to follow.\n- This paper focuses on a long-standing yet challenging problem, which is to establish a robust driving agent without extensive human demonstrations. This paper shows encouraging results in that research direction.\n\nLimitation yet NOT weaknesses of this work:\n- Although this work shows promising results of learning robust policy via self-play, it is notable that all policies presented in this paper operate on highly abstracted vector space without sensor inputs, which is different from real-world driving agents that reason over their online sensor data. Therefore, there is still a significant gap in applying this method to establish real-world driving policy. The gap lies in the difficulty of high-dimensional sensor simulation which requires much more computation compared to low-dimensional vector-space simulation in this paper. However, we should be aware that sensor simulation refers to another branch of work and is not the focus of this paper, thus it should not be viewed as a weakness of this work.\n\n\nAlso see strengths and weaknesses in the above sections.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lNM4SA9hdQ", "reviewer": "Reviewer_eomV", "summary": "This paper presents a batched driving simulator called GIGAFLOW that enables large-scale self-play, i.e., randomly initializing scenarios and learning robust driving behaviors with pre-defined RL rewards. The GIGAFLOW world creates worlds based on eight maps, spawns agents on random locations with randomly perturbed accelerations, sizes, goals, and other necessary parameters. It defines rewards w.r.t. safety, progress, traffic rules, and comforts. It trains a unified policy for all agents with conditions of various driving behaviors as input. The policy is trained with PPO and a variant of prioritized experience replay to filter unimportant samples. Through learning on GIGAFLOW only, the policy achieves state-of-the-art performance on multiple driving benchmarks, including CARLA, nuPlan, Waymax, and Waymo Sim Agent.\n\n## update after rebuttal\nThe authors clarified most of my concerns and confusion on technical details during the rebuttal. This paper presents a promising direction of self-play to realize robust motion planning. The novelty, contribution, and final results all demonstrate its high quality. I did not give a higher score mainly because self-play has been adopted in other domains, which limits the broader impact on the ML community. Its reliance on accurate maps and perception results may also limit its real-world application, but I believe the industry can acquire insights from the paper and address the bottlenecks for large-scale deployment soon.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.\n\n> GPUDrive\n\nWe are aware of the work, and have been in close communication with the authors of GPUDrive from the start of their project. We will cross-cite each other's work for the camera-ready paper.\n\n> ... requires accurate high-definition (HD) maps to achieve robust planning, which are not available for most areas\n\nTrue, training requires HD-maps. Unclear how big the reliance on HD maps is during deployment. Our policy generalizes across several different vector map formats (Carla, nuPlan, Waymo). It does not directly see the vector representation, but rather an intermediate representation. A performant sense stack should be able to construct this representation. \n\n> The paper adopts the Frenet coordinates, which potentially encourage the vehicles driving on the center of lanes\n\nThis is a misunderstanding. The Frenet frame provides a policy with a rotation invariant representation of its local world. However it does not limit its driving ability. The policy choses to go into the opposing lane if required (see Figure 4). After all, PPO optimized the policy on such a huge amount of experience that it seems inconceivable that biases like lane-centering would survive only because of the input representation.\n\n> ... Sec. F.3? ... Figure A4?\n\nOur training environment is augmented with erratic drivers and severe dynamics noise. Sec F.3 evaluates robustness under ordinary conditions. In this regime, our agents achieve over 3 million kms of driving between incidents of any kind. This corresponds to approximately 99.99999997% of transitions in the simulator being collision-free. To our best knowledge, this is one of the first examples in the literature where such level of robustness is demonstrated for a stochastic policy trained with RL.\n\nThe policy drives better with a higher decision making frequency (Fig A4) as it has to commit to \"shorter\" actions (lower reaction time).\n\nWill clarify in final paper.\n\n> ... Could the authors clarify which other rewards can potentially be used for more delicate designs?\n\nOur reward encodes road rules, but not behavior or expected interactions between agents. We do not use:\n\n- Penalties shaping expected proximity to other agents\n- Expected speeds around other agents\n- Forward progress\n\nThese terms trade short term performance with generalizability.\n\n> ... Lines 258-268 ... how the policy autonomously adapts to various scoring criteria if they are contradictory.\n\nOur eval is zero-shot. We use a single reward condition for all benchmarks, there is no autonomous adaptation.\n\n> Would using a single policy ... result in unstable training, ...?\n\nIt does not. Ultimately, the major reward the agent receives is to reach a goal. The randomized components of the reward are secondary. This is also reflected in the behaviors learned. The agent first learns to move towards the goal and collect close-by goals, only after that will it pay attention to red lights, driving direction etc.\nThe training curves are stable. We will add them to the appendix.\n\n> Routing information / potential label leakage?\n\nWe do not use the expert trajectory in any way (no leakage). For nuPlan, we use a point in the final lane segment as the goal (no routing information, our policy learned to route locally during training). For waymax, we use the final location of the log ego vehicle projected forward 20m along the lane to ensure that the agent continues driving if it arrived early. In neither setup, our agent sees more information that baselines.\n\n> 2024 WOSAC leaderboard?\n\nWorking on it, unfortunately wont finish in time for this rebuttal.\n\n> Clarifications on Figure A3 and randomization\n\nThe metrics for nuPlan and Waymax are very compressed. Both benchmarks have a certain number of scenarios that are unsolvable and thus a non-zero optimal policy. Carla has fewer scenarios that are completely unsolvable, and its metric degrades less gracefully.\nReward randomization does help for all methods (nuPlan 93.08 -> 93.17; Waymax 98.79 -> 98.92).\n\n> Detailed number of data used.\n\nOver 99% of trajectories are used during training and need to be simulated to completion to calculate rewards and advantages for each timestep. Figure 1 is an accurate comparison of sizes. Gigaflow is about five orders of magnitude larger than prior public attempts.\n\nWe filter 80% of time-steps (see section 2.2). This filtering happens after simulating rollouts. Even accounting for this filtering, RL training sees at least four orders of magnitude more data than a single pass over any publicly available data would provide.\n\n> Typos\n\nThank you. Will be fixed.\n\n> Supplementary videos\n\nSorry, we forgot to upload them: https://drive.google.com/drive/folders/1hlDrPDQIu56-iq039fwOa18g8aUGSKYl?usp=sharing\n\n> Open-sourcing\n\nWe would very much like to open-source this work, but cannot commit due to approval requirements beyond our control.", "claims_and_evidence": "The paper's claims are solid overall. The method does not involve human data and uses self-play with RL only, in alignment with the claims. The experimental results show that it realizes state-of-the-art performance on benchmarks and outperforms specific algorithms.", "methods_and_evaluation_criteria": "The method, self-play, is simple and sound. To make it effective, the authors propose several improvements, including hash-based map encoding for acceleration, the simple importance sampling strategy, careful randomization and filtering designs, etc. These methods make sense for the problem and are validated in the ablation study in the supplementary material.", "theoretical_claims": "Not applicable. This paper does not include theoretical claims.", "experimental_designs_or_analyses": "The authors incorporate multiple benchmarks, including open-loop and closed-loop evaluations. I think the experimental results and analysis are valid and sound, with concrete descriptions on how experimental settings are adapted in the supplementary material.", "relation_to_broader_scientific_literature": "The methodology in this paper is mainly self-play, which has been explored in some existing applications such as StarCraft gaming and Go Chess.", "essential_references_not_discussed": "I think one closely related prior work is GPUDrive [1]. It also supports large-scale simulation or RL training with 1 million FPS, which is comparable to the speed of GIGAFLOW (1.2M as shown in Table A1). Though GPUDrive mainly targets fast training agents on WOMD, it can probably support self-play as well. Additionally, GPUDrive supports sensor simulation. I suggest the authors include a discussion of the comparisons with GPUDrive in the revision, which may be important to evaluate the contribution of GIGAFLOW in the aspect of batched simulation.\n\n[1] Kazemkhani, S., Pandya, A., Cornelisse, D., Shacklett, B. and Vinitsky, E., 2024. GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS. arXiv preprint arXiv:2408.01584.", "other_strengths_and_weaknesses": "Strength:\n1. The map hashing part involves great engineering effort, which is crucial to designing an accelerated simulator for batched RL training.\n2. Conditioning various agent types to the policy and using a unified policy for training is interesting.\n3. Great visualizations. Very neat and informative.\n4. Results are strong, with a single policy adapted to various benchmarks.\n\nWeakness:\n1. One potential weakness of this method is the reliance on maps. This work put great efforts into encoding maps to accelerate information acquisition during training and calculating geometric constraints. However, the vectorized representation requires accurate high-definition (HD) maps to achieve robust planning, which are not available for most areas. To enlarge the functional regions, the current driving industry is focusing on map-free or map-light methods, using navigation maps or (inaccurate) crowdsourced maps only. Therefore, this issue may limit the broader application for the real-world driving industry.\n2. The paper adopts the Frenet coordinates, which potentially encourage the vehicles driving on the center of lanes. This is reasonable in most cases and is prevailing in current motion benchmarks. However, corner cases in real life sometimes do not satisfy this assumption. For example, it requires the vehicle to deviate to the opposite lane to bypass an obstacle in the front. The method adopts a discrete set of actions (12 in total) as well to simplify the problem. I understand that the authors have fully analyzed the failure cases in current benchmarks, yet I am wondering if it is possible to come up with more challenging scenarios to discuss the drawbacks of the proposed method. Besides, another potential benchmark in academia could be Bench2Drive [2]. This weakness also relates to W1 in terms of maps.\n3. I hope the authors could provide further clarifications on Figure A3. The results seem to indicate that randomization mainly works for CARLA experiments, while the impact for nuPlan and Waymax is relatively minor. I think the ablations in the appendix, Sec. F, are more important to get more insights from the proposed method, compared to the visualizations in Figure 4.\n4. The RL algorithm is trained in GIGAFLOW with billions of miles, but a large portion of data is filtered. Could the authors provide a more detailed number for the ultimate data used and compare it with specific data in various benchmarks? As those dataset-specific algorithms assume all the public dataset is useful, this comparison would be more direct and intuitive to see how much valuable data is needed.\n\n[2] Jia, X., Yang, Z., Li, Q., Zhang, Z. and Yan, J., 2024. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877.", "other_comments_or_suggestions": "1. Typos: Line 29/401 (e.g.,), Line 85 (Dashed Lines -> The trophy and the thumbtack?), Line761 (intersect), Line979 (GPUs), additional blank lines before Eq. 1 and Eq. 5.\n2. There are multiple sentences stating that videos are in the supplementary material in benchmark evaluations (Sec. E, Lines 1088/1051), but they are not provided in fact.\n3. It would be helpful if the work could be open-sourced to benefit the community and the industry.", "questions_for_authors": "1. What is the meaning of \"long-form evaluation\" in Sec. F.3? Besides, what is the relationship of long-form evaluation with the decision-making frequency in Figure A4? I hope the authors can clarify this section in the revision.\n2. In multiple places, the authors state that they use only a minimalistic reward function (Line 43) or do not design delicate reward terms (Line 149). Meanwhile, I think the rewards listed in Sec. B.3 and Table A2 have already included the most adaptable rewards in RL for driving. Could the authors clarify which other rewards can potentially be used for more delicate designs?\n3. The description in Lines 258-268 is very interesting. But I am curious how the policy autonomously adapts to various scoring criteria if they are contradictory.\n4. Would using a single policy to train all agents, with various conditions and randomization, result in unstable training, especially during the beginning stages?\n5. How are the routing information and goals applied for short-term planning benchmarks like nuPlan? Will there be label leakage or additional information used when comparing with other methods?\n6. Sec. E.4 adopts WOSAC 2023 results. How about the public 2024 WOSAC leaderboard?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kaElV4eaRJ", "reviewer": "Reviewer_EHkB", "summary": "The authors introduce a simulator which is capable of efficiently simulating joint traffic scenarios at scale. Using this simulator, they train a population of driving policies using self-play reinforcement learning. When evaluated on CARLA, NuPlan and Waymax, the authors report state of the art performance despite never training on these datasets.\n\n## Update After Rebuttal\n\nDuring the rebuttal, the authors addressed my concerns regarding fine tuning of the reward and committed to including additional references. I believe that this work warrants acceptance. However, I believe that the manner in which the authors report their performance on Waymax is misleading. I strongly urge them to improve the clarity as discussed in the my review to improve their work further.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and list a detailed reply for all questions raised.\n\nTo contextualize this rebuttal we would like to clarify one of the main differences between Agent and Environment Simulation (WOSAC) and learning a driving policy (this work). The goal of WOSAC (and motion forecasting more generally) is to learn how human traffic generally behaves. If a vehicle collides or goes off-road (in the real world, or in a log due to sensor noise) a good agent and environment simulator will simulate this event. The WOSAC benchmark contains 3% collision events and 12% off-road events. A competent driving policy should never collide, and rarely go off-road.\n\nFor example, the best Agent and Environment Simulations (SMART etc) should reproduce the logged trajectory almost perfectly to maximize their score. A good policy will not because certain noisy trajectories lead to collisions and off-road events.\n\nIt was never our aim to claim to be state-of-the-art on WOSAC, nor should a competent driving policy be expected to. We show that in interactive driving simulations (CARLA, nuPlan, Waymax) our policy is a competent driver which gets to its goals safely, robustly, and efficiently. However, it does not necessarily achieve this in a human-like manner, nor does it imitate the sense noise of the waymo perception stack. This leads to a lower WOSAC score.\nThe only mention of WOSAC in the main paper is on line 429. We primarily mention it as it is interesting, but not critical to the claims of the paper, that we achieve a non-trivial WOSAC score.\n\n> Additionally references\n\nThank you. We will add them in the final version.\n\n> For the Waymax benchmark, the authors propose a Score metric which is not present in the original Waymax paper as their summary metric...\n\nIt is unfortunately quite easy to trade individual metrics off in the Waymax benchmark. Yes BC and Wayformer perform better in Log ADE (following the demonstrated trajectory). They clearly should as this was their training objective. However, they fall short across all other simulation objectives with a collision rate of 4+%.\n\n> As noted in the Claims and Evidence section, WOSAC should be evaluated on the newer 2024 leaderboard.\n\nWe are working on this, and hopefully will have numbers for the final version.\n\n> It is unclear how much hyperparameter tuning of C_reward was needed to achieve state of the art performance on each dataset. / How much tuning (if any) was done to select the reward hyper parameters for each benchmark?\n\nAll benchmarks use the same reward hyperparameters (the mean of the randomized range). We did not tune reward parameters as we considered this overfitting to the target benchmark (and thus would invalidate our zero-shot generalization claim). There is a single difference between all benchmarks, which is the radius of the target region. Carla requires more precise navigation (the agent needs to follow the given route points closely.) than Waymax and nuPlan and uses a tighter target radius. We will add these values to the appendix in the final version.\n\n> How does the performance of SoTA WOSAC policies compare against your policy on the Waymax benchmark? I would specifically like to see the performance of SMART which is available open source here.\n\nSMART is certainly performing better at Agent and Environment Simulation (WOSAC). The aim of the WOSAC results was to show the policy learns human-like driving, and not to claim state-of-the-art on this benchmark. As such, we sharply outperform SMART on collision scores, our primary metric of interest, but underperform on distance to logged trajectories, which our paper did not target and is not always relevant for a self-driving car.\n\n> Have any other prior art used the baseline implementations presented in the Waymax paper as a benchmark?\n\nAs developed, the tools developed in the waymax paper are primarily intended for the development of human-like simulation agents. However, as the data is interesting and can be used to measure robustness, other works have started to use it in similar manners to those used here [1, 2, 3].\n\n[1] Cornelisse, D., Pandya, A., Joseph, K., Suárez, J., & Vinitsky, E. (2025). Building reliable sim driving agents by scaling self-play. arXiv preprint arXiv:2502.14706.\n\n[2] Xiao, L., Liu, J. J., Ye, X., Yang, W., & Wang, J. (2024). EasyChauffeur: A Baseline Advancing Simplicity and Efficiency on Waymax. arXiv preprint arXiv:2408.16375.\n\n[3] Charraut, V., Tournaire, T., Doulazmi, W., & Buhet, T. (2025). V-Max: Making RL practical for Autonomous Driving. arXiv preprint arXiv:2503.08388.\n\n\n> Will you open-source this simulator to allow others to reproduce your results?\n\nWe would very much like to open-source this work, but cannot commit due to approval requirements beyond our control.", "claims_and_evidence": "Although technically correct, I have some concerns regarding the framing of the author's claims of emergent state of the art behaviour. The authors use three driving benchmarks to support their claim. Of these, NuPlan is a strong result, as it is widely used in this field. CARLA is less commonly used. I have no issues with either of these evaluations. My major concern is with the author's Waymo based evaluations. In the literature, the Waymo Open Motion Dataset is an extremely popular and competitive benchmark. The authors compare against the baseline methods reported in the original 2023 Waymax simulator paper [1] and prominently report their improvements (ie Figure 1 right). To my knowledge, I have never seen this paper's metrics used as a benchmark for evaluating policy performance. As I understand it, the official and substantially more widely used evaluation for the Waymo Open Motion Dataset is the Waymo Sim Agents Challenge. On this domain, the authors do not achieve state of the art performance, despite only comparing their methods to 2023 leaderboard and not the newer 2024 leaderboard where state of the art has further improved.\n\n[1] \"Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research\" Gulino et al 2023", "methods_and_evaluation_criteria": "In addition to my concerns about the SoTA claim, I have a few concerns about the evaluation criteria\n\n1. For the Waymax benchmark, the authors propose a Score metric which is not present in the original Waymax paper as their summary metric. Although this metric captures realism regarding collisions, off-road driving, and distance driven it does not incorporate the displacement error with respect to the logged policy. Notably, Gigaflow performs worse than all policies except DQN on this metric.\n\n2. The self-play policy is conditioned on $C_{reward}$ which the authors argue has a significant impact on the behaviour of the agent. However, $C_{reward}$ is not reported for any of their benchmarks. It is unclear how much hyperparameter tuning of $C_{reward}$ was needed to achieve state of the art performance on each dataset.\n\n3. As noted in the Claims and Evidence section, WOSAC should be evaluated on the newer 2024 leaderboard.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria", "relation_to_broader_scientific_literature": "This paper provides a counterpoint to the large body of literature related to learning traffic behavioral models. Specifically, the majority of prior art is based off various imitation learning techniques, leveraging large scale datasets of recorded traffic behaviour. The quality and realism of the policy reported in this paper opens up new possibilities for developing traffic policies through RL.", "essential_references_not_discussed": "The authors fail to include several more recent SoTA policy models as they do not compare to the 2024 WOSAC leaderboard. For example\n\n- SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction (Wu et al. 2024)*\n- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (Zhou et al. 2024)\n\nThis paper references two papers on fine-tuning behavioural cloning polcies with RL. I believe this subject area is highly related and warrants further discussion. For example:\n\n- Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving (Peng et al. 2024)\n- Learning to Drive via Asymmetric Self-Play (Zhang et al 2024)\n- Scaling Is All You Need: Autonomous Driving with JAX-Accelerated Reinforcement Learning (Harmel et al 2023)\n\n* Smart 7M results from Table 1 of Wu et al. corresponds to the InteractionFormer results from table A8 of your paper", "other_strengths_and_weaknesses": "In my opinion, this paper is very well done in almost every aspect. It shows evidence that large scale reinforcement learning is a promising alternative to the dominant paradigm of supervised learning for traffic modelling. The analysis of the policy performance is well done and the details provided in the appendix are thorough. Frankly, had the authors simply reported their reasonable WOSAC performance instead of claiming Waymo SoTA through their \"Waymax benchmark\" I would wholeheartedly recommend acceptance. Their SoTA claim in this area undermines an otherwise excellent paper.\n\nStrengths\n- The performance reported by this paper is remarkable given the lack of supervised training on human data.\n- The qualitative analysis of the resulting policies is robust\n- The paper is clear and well written.\n- The supplementary provides many of the details required to reproduce the paper\n\nWeaknesses\n- While obviously a strong behavioural policy, the state of the art claims on Waymo data are unconvincing.\n- The authors report training time compute but do not report hyperparameter tuning procedures or compute budget\n- Although the authors spend a large portion of the paper describing the details of their simulator, they do not offer to open source it, making reproduction of their results challenging.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "My primary grievances with this paper surround reporting of evaluation hyperparameters and results on Waymo data. I will gladly raise my score if my questions are answered in these areas\n\n1. What reward conditional signal was used for each of the reported benchmarks. How much tuning (if any) was done to select the reward hyper parameters for each benchmark?\n2. How does the performance of SoTA WOSAC policies compare against your policy on the Waymax benchmark? I would specifically like to see the performance of SMART which is available open source [here](https://github.com/rainmaker22/smart).\n3. Have any other prior art used the baseline implementations presented in the Waymax paper as a benchmark? I was unable to find any other work which uses these results as a benchmark.\n4. The performance of Gigaflow seems highly dependent on the details of your simulator. Will you open-source this simulator to allow others to reproduce your results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a reinforcement learning approach to train driving policies, without human demonstration. The paper uses gigaflow, to run massive numbers of simulations from three benchmark driving environments. Although the paper used a metric that they define, the evaluation seems fair given it’s a millage measure in dense traffic. It is a good contribution to the icml community.
Decision: Accept (poster)