SPACeR: Self-Play Anchoring with Centralized Reference Models
Authors: Wei-Jer Chang
Akshay Rangesh
Kevin Joseph
Matthew Strong
Masayoshi Tomizuka
yihan hu
Wei Zhan
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-agent reinforcement learning
traffic simulation
autonomous vehicles
planning
Primary area: reinforcement learning
TL;DR: We anchor self-play reinforcement learning to a pretrained tokenized model to enable fast, reactive, and realistic multi-agent simulation.

Abstract: Developing autonomous vehicles (AVs) requires not only safety and efficiency, but also realistic, human-like behaviors that are socially aware and predictable. Achieving this requires sim agent policies that are human-like, fast, and scalable in multi-agent settings. Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data, producing realistic policies. However, these models are computationally expensive, slow during inference, and struggle to adapt in reactive, closed-loop scenarios. In contrast, self-play reinforcement learning (RL) scales efficiently and naturally captures multi-agent interactions, but it often relies on heuristics and reward shaping, and the resulting policies can diverge from human norms. We propose human-like self-play, a framework that leverages a pretrained tokenized autoregressive motion model as a centralized reference policy to guide decentralized self-play. The reference model provides likelihood rewards and KL divergence, anchoring policies to the human driving distribution while preserving RL scalability. Evaluated on the Waymo Sim Agents Challenge, our method achieves competitive performance with imitation-learned policies while being up to 10× faster at inference and 50× smaller in parameter size than large generative models. In addition, we demonstrate in closed-loop ego planning evaluation tasks that our sim agents can effectively measure planner quality with fast and scalable traffic simulation, establishing a new paradigm for testing autonomous driving policies.

Published as a conference paper ICLR 2026
SPACER:
SELF-PLAY ANCHORING WITH CENTRALIZED
REFERENCE MODELS
Wei-Jer Chang
Akshay Rangesh', Kevin Joseph 1.3
Matthew Strong 144 Masayoshi Tomizuka", Yihan Hu', Wei Zha
Applied Intuition 2 University of California, Berkeley Stanford University
3 New York University
ABSTRACT
Developing autonomous vehicles (AVs) requires not only safety and efficiency, but also realistic, human-like behaviors that are socially aware and predictable. Achieving this re- quires sim agent policies that are human-like, like, fast, and scalable in multi-agent settings, Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data, producing realis- tic policies. However, these models are computationally expensive, slow during inference, and struggle to adapt in reactive, closed-loop scenarios. In contrast, self-play reinforce- ment learning (RL) scales efficiently and naturally captures multi-agent interactions, but it often relies on heuristics and reward shaping, and the resulting policies can diverge from human norms. We propose human-like self-play, framework that leverages pretrained tokenized autoregressive motion model as a centralized reference policy to guide decen- tralized self-play. The reference model provides likelihood rewards and KL divergence, anchoring policies to the human driving distribution while preserving RL scalability. Eval- uated on the Waymo Sim Agents Challenge, our method achieves competitive performance with imitation-learmed policies while being up to 10x faster at inference and 50x smaller addition, we demonstrate closed-
in parameter size than large generative models. loop ego planning evaluation tasks that our sim agents effectively measure planner quality with fast and scalable traffic simulation, establishing a new paradigm for testing autonomous driving policies.
1 INTRODUCTION
Developing autonomous vehicles (AVs) that can safely and smoothly share the road with human drivers is fundamental challenge. The difficulty lies not only ensuring safety and efficiency, but also in producing human-like behavior: policies must be predictable, socially aware, and capable of interacting effectively in complex multi-agent environments. To achieve this, is essential to build realistic simulation policies that AVs can interact with in scalable, closed-loop testing before deployment real roads.
Simulation policies must satisfy two key requirements: realism and reactivity. Realism refers to human-litt like behavior, while reactivity captures meaningful responses to other agents. These properties enable AVs handle interactive, multi-agent scenarios. Broadly, there are two paradigms for constructing such policies: imitation learning and self-play reinforcement learning (RL).
Imitation learning starts from expert demonstrations, with recent advances including tokenized models Phil- ion al. (2024): Wu et al. (2024); Zhang al. (2025) and diffusion models Zhong al. (2023): Jiang al. (2024) to generate realistic multi-agen behaviors. The key advantage of imitation learning is that it learns
"Work done during internship Applied Intuition.
'Corespondence: wei. zhan@appliedd.co

---

Published as a conference paper at ICLR 2026
directly from the human data distribution, producing realistic behavior. However, such approaches struggle in reactive settings and often require computationally heavy models such as Transformer-bant architectures that limit scalability, as we demonstrate in section 4.3
On the other hand, self- eelyy reinforcement learning (RL) has become popular Cusumanoo Towner et al. (2025): Kazemkhani et al.] (2024) for building driving policies. In self-play, agents learn policies by re- peatedly playing against one another in closed-loop interaction. However, successful training often requires extensive heuristics and careful reward shaping, and the resulting policies can still diverge from human norms, leading to unrealistic and non-human like behaviors.
To address both challenges, we introduce SPACER, which leverages a pretrained tokenized model Wu et al. (2024) to provide reference policy distribution and likelihood signal over the outputs of self-play policies (fig. D. This serves as an on- on-policy reward provider, shaping learning toward human-like behavior while preserving the scalability of self-play RL. Specifically, we align the self-play policy's 's action space with that of the tokenized model, enabling tractable likelihood estimation and KL-based distributional alignment. This design eliminates the need for ground-truth logged trajectories and instead provides probabilistic estimates that directly guide learning.
We validate our approach on the Waymo Sim Agents Challenge (WOSAC) Montali al. (2023), where SPACER significantly improves realism and human-likeness compared to prior self-play RL methods. In addition benchmark performance, we perform closed-loop policy evaluation across diverse planners, using the reference tokenized model as a baseline. Our experiments show that Hlman--Like Self-Play policies are more reactive and avoid the dalse-positive collisions often seen in imitation- based approaches, yielding more realistic and reliable estimates for planner evaluation.
Beyond performance, the resulting policies are remarkably lightweight: decentralized MLPs with only 665k parameters that run over 10x faster and are up to 50x smaller than most tokenized models. This efficiency enables scalable, real-time multi-agent simulation at unprecedented scale, while maintaining the realism necessary for AV testing and deployment.
2 RELATED WORKS
Self-Play RL for autonomous driving
Self-play reinforcement learning has been studied extensively in multi-agent reinforcement learning (MARL) as way for agents learn by interacting with copies of themselves [arroll et al. (011)); Yu ct al. (2022); Chen et al. (2024) and has recently been applied to autonomous driving Zhang et al. (2024): et al. (202)). One representative work, GIGAFlow Cusumano- Towner et al. 2025 demonstrates that large-scale self-play can produce robust autonomous driving policies, showcasing the scalability and effectiveness of self self-play for autonomy. Nevertheless, one of the main chal- lenges for self-play methods in real-world deployment is the divergence from human behaviors. Policies that maximize rewards may behave unpredictably, and be unable to coordinate with humans safely. While Human Regular-ee PPO ProCCorelissee & Vinitsky (2024) introduces small amount of demonstration data a regularizer to bias policies toward human- human-likeness, our approach leverages a large-scale tokenized model to provide human-like reward signal during multi-agen learning, making it the first demonstrate competi- tive performance with state-of -fre--fftheeeeee the-art imitation learning policies while preserving the scalability of self- play.
Kazemkhani al (2024): Cusumano-Towwe
Imitation-Learning Based Traffic simulation
Recently, data-driven methods like imitation learning have gained popularity due to their capacity to learn from expert data-distribution with minimal human effort. Early works have explored generative imitation- learning approaches for modeling human traffic behaviors Kuefler et al. (2017): Bhattacharyya et al. (2020). Recent works can mainly be categorized diffusion models and tokenized models. Diffusion Models such
2

---

Published as conference paper ICLR 2026
BATCHED SIMULATIONS
Full Scene
Agents
Context
B1
ag
Br
AGENT ROLLOUTS
Tokenized Reference Model (Huer
KL
humarikk
p(a, a2ze)
((0)
Dzz
Decentralized Model (m₂)
Figure 1: Overview of SPACER Framework. Self-play reinforcement learning is anchored to a pretrained tokenized reference model Tet which provides likenes distributional signal. The self-play policy Ty is decentralized and conditioned on local observations, whereas Tee is centralized and conditioned on the full scene context.
as Zhong et al. 2023) Chang et al. (2024) 2025); Jiang et al. (024) offer flexibility and controllability to potentially generate long-tai tail driving scenarios. On the other hand, tokenized models such as SMART Wu et al. (2224) or CAT-K Zhang et al. (2025) have become more popular due to their high realism and capac- ity to simulate realistic multi-agent behavior. However, both approaches are computationally expensive inference time: diffusion models require multiple denoising steps, and tokenized models generate actions sequentially. This limits scalability large-scale closed-loop simulation. In this work, we instead leverage pretrained initation-learning models to guide self-play RL toward human-like behaviors. There are several works that focus on RL fine-tume fine-tuning pretrained umitation-learninn models Peng et al. (0024); Chen et al. (202);; Cao al. (2024): ) for example, using Group Relative Proximal Optimization improve realism and controllability [Chnn et al. (2025), leveraging human feedback for post- training alignment Cao et al. (2024), exploiting implicit preferences from pre-training demonstrations to avoid costly human annotations Tian & Goel| (2025). In contrast to this weerrinnthen-fineture paradigm, we take an RL-firrt approach, where self-play serves the foundation and inntaation-eeaningg models are incorporated as reward provider.
3 HUMAN-LIKE SELF-PLAY
We aim to learn human-like driving policy (o(a))) whose behaviors match the real human driving dis- tribution closed-loop. The desired policy should reach goals, avoid collisions, and stay on-road, while producing predictable and realistic behaviors.
3.1 PROBLEM FORMULATION
We formulate this partially observable multi-sequential decision-making problem. At each timestep f, the global simulator state E S encodes the static road graph and the dynamic states of all agents. Each agent i receives only partial observation of (s,,, i) within its local feld-of-view. The agent then
3

---

Published as a conference paper ICLR 2026
chooses an action of E A (e.g., acceleration, steering, or maneuver) according to its policy To lo). and the environment transitions according to $1+1 ~ T(s .!....,,.""""
The key challenge in training human-like policy xe is that task-based rewards (e.g., reaching goals, avoid- ing collisions, or staying on the road) encourage efficiency and safety but do not guarantee realism. Such rewards often lead to behaviors that maximize task success while diverging from human driving norms, for example, accelerating unnaturally to reach goals or performing overly sharp maneuvers to avoid collisions. To address this, we introduce pretrained reference policy Tet that captures the human driving distribution and provides realism signal during self- play, anchoring learning to behaviors observed in real data.
task
+
humanlike (St: ac):
[()) = Crro (0: A(r)) - B DKL(TO( ox) Tref(- (t)),
(2)
Eq. 2c combines three components:
Task performance:
(r)o(0) is the standard PPO objective Schulman al. (0117) encouraging agents to reach goals, avoid collisions, and stay on the road.
Human-likeness reward: Inspired by prior work |Eccntreel al. (0023) we define a log-
likelihood based reward that provides dense, per-timestep feedback by scoring each executed action under the reference distribution:
Thumanlike (Sr, ae) log Trer (a; (t).
This formulation corresponds to maximizing the likelihood of human-like actions, thereby encour- aging realistic driving behaviors beyond task success.
Distributional alignment: The KL divergence (m o.) Tyer(- | 8t)) acts as a dense signal at every timestep, directly encouraging the policy align its action distribution with the human driving distribution captured by the reference model. For notational simplicity, we omit the agent index i; the KL divergence is computed independently for each agent.
3.2 PRETRAINED REFERENCE TOKENIZED MODEL
To incorporate human-likeness into self-play, we introduce pretrained reference policy Ther, trained on real-world human driving trajectories, to serve as a proxy for the human driving distribution. Unlike task- based rewards, which only encourage efficiency and safety, Tet provides a distributional signal that anchors policies toward realistic behavior. In principle, Tet can be any data-driven generative motion model, such diffusion model or a tokenized model. In this work, focus tokenized models (e.g., SMART jet al. (2024), CAT-K ZZhang et al. (2025)), since they provide tractable likelihoods estimates suitable for distributional alignment.
2024) operate in autoregressive fashion: given the past joint actions derr and the scene context e (e.g.. road graph and initial states), the model predicts a distribution over the next joint action a; = (a)..... a; where denotes the discrete timestep and N is the number of agents the scene. Under conditional independence assumption across agents, this factorizes as
Centralized Reference Tokenized Model. Tokenized models such as SMART Wu et al.
N p(a, ace, c) II pla ace, c).
(4)
E
In our framework, we treat p(-) as the pretrained reference policy Traf, which provides a distinct distribution for each agent's is action of at every timestep, conditioned on the shared scene context and action history.
4

---

Published as a conference paper ICLR 2026
Importantly, the ability of Tref assign a distinct distribution to each agent's action at every timestep directly addresses central challenge in multi-agent reinforcement learning: the credit assignment problem. It is often unclear which agent, and at which timestep, is responsible for a positive or negative outcome. This fine-grained feedback, grounded in the distributional signal of Tref, enables shaping learning signals on a per-agent, per-timestep basis rather than relying solely on sparse, trajectory-level rewards.
(1) Tractable training without autoregressive sampling. Unlike autoregressive generation, which requires sequential token sampling, our approach only requires single forward pass of the reference model per rollout batch. This provides the full per-agent, per-timestep action distribution, making the training pipeline efficient and scalable.
(2) Aligned action space. By ensuring that Te and Teef operate over the same discrete action space, we avoid online tokenization during training. This alignment not only reduces computational overhead but also enables the KL divergence be computed directly in closed form:
(TI(To(- ox) Trar(- (-1 [())) = Σ
Ta(a| ox) log To ot) Tre (a 8t)
(S)
aEA
(3) Privileged information and comparison to log-replay data. Note that in Eq 2 the reference tokenized model centralized, observing the full state of all agents rather than local views. This setup resembles privileged information in teacher-student framework in Robotics and Computer Vision He et al. (2024); Caron et al. (202I). Unlike log replay. the reference model generalizes beyond recorded trajectories and provides guidance in unseen self-play states.
Reference Models vs. VOSACMetrices WOSAC metrics Wu et al. (2024) estimate per-feature likelihoods (kinematic, interactive, and map-based) based ground-truth future trajectories. In contrast, tokenized reference models directly define a distribution over entire action sequences, offering two advantages: (i) evaluation does require logged future trajectories, and (ii) the model provides principled sequence- level distribution rather than feature-wise likelihoods tied to manual design metrics.
3.3 PRACTICAL. CONSIDERATION OF BUILDING HUMAN-LIKE SELF-PLAY
Here, we outline common problems in standard RL settings and the corresponding considerations for build- ing human-like self-play.
Goal-reaching reward and post-goal behavior In previous works Kazemkhani et al. (2024); Cornelisse et al. (0025), agents typically rewarded only upon reaching their goal, and are removed from the sce- nario once the goal is reached. This formulation has two main drawbacks: (i) agents are incentivized to accelerate unnaturally in order reach the goal as quickly as possible, and (ii) the number of active agents decreases after goals are reached, making the scenario progressively casier. Qualitatively, with this original formulation, if the agents are not disappeared, agents would mostly stop once reached the goal, making the scenarios unrealistic. To address this issue, we adopt goal-dropout: agents are trained with and without goal conditioning while receiving a terminal goal reward. This reduces reliance on explicit goal inputs, We further show that, when anchored to a reference model, the explicit goal reward can be removed without performance loss.
Tokenized Action Space and policy frequency: Contrary to previous works that utilize unicycle dynamics, we align the action space with the reference model by adopting a tokenized trajectory action space with K-disk from the data [Pilioon et al.] (2024); Wu al. (2024). This ensures compatibility for computing likelihoods and KL divergence without online tokenization. Therefore, the simulation frequency and policy frequency may differ; for example, the policy operate at 5 Hz while the simulator runs 10 Hz.
5

---

Published as a conference paper ICLR 2026
4 EXPERIMENTAL RESULTS
We validate whether self play policies trained with reference model produce behaviors that are both human- the Waymo Open Motion Dataset (WOMD) [WOMD]]Ettngerr et al. (2021), measuring how closely the learned policies match the human driving distribution and how effectively they adapt to interactions in closed-loop simulation.
like and reactive. Experiments are conducted on large-scale traffic scenarios from
4.1 IMPLEMENTATION DETAILS
We conduct all experiments in GPUDrive Kazemkhani al. (2024), GPU-accelerated, data-driven simu- lator built on WOMD Ettinger et al. (021). Each WOMD scenario spans 9 seconds; following the setup of Montali et al. (023), , we initialize at second and simulate the remaining 8 seconds. We train 10k scenarios. In the main vehicle experiments (table I. we control only vehicles, while pedestrians and cyclists follow their logged trajectories. For the VRU experiments (table 2). we train SPACeR by controlling all agent types, but report metrics only on pedestrian/cyclist targets.
Partially Observed Stochastic Game Hansen et al. (2004), where agents act simultaneously under partial observability. Each controlled vehicle receives a local observation of an ego-centric coordinate frame, including nearby vehicles, lane geometry, goal points (optionally), and relevant road features within 50 m radius. Agents do not receive temporal history, and all features are normalized the range [-1,1] 1].
Observation Space. We model the RL problem
Action Space. We adopt tokenized trajectory action space following Philion et al. (2024): ; Wu et al. (2024). Using the K-disk algorithm, we cluster short-horizon trajectories in Cartesian space into 200 discrete tokens. Each token represents 0.1-second step with horizon length of 2, corresponding to a 5 Hz action frequency. This setup balances two considerations: providing sufficiently fine-grained distributional signals from the SMART reference model while keeping memory usage manageable. Since actions are defined directly in Cartesian space, no explicit dynamics model is assumed; the simulator advances the scenario according to the selected token.
Reward Formulation. For each agent, the task reward is
rusk (st, (s) = Wgooal I[Goal achieved]- -Welllddddddlllldddd I [Collided]
Wottred [[Offrood]] + Whumanlike humanlike (st, az),
where I[-] E 13. By default, we set Weollided = Woffroad = 0.73. The weights Wgoal and Whamanlike are varied in the following ablation experiments study the trade-off between task completion, safety, and human-likeness. Note that other than the reward, we also directly compare the KL divergence between trained policy and referenced policy, where we use CAT-K/Zhan al. (2025) as the reference model.
Model Architecture and Training Details: We use a late-fusion feedforward model Kazemkhani et al. (2024); Cornelisse et al. (2025), where ego, partner, and road-graph features are each embedded with a two-layer MLP and then concatenated. The fused representation is fed into an actor head and critic head. During training, we control up to 64 agents with shared decentralized policy πT. where each agent samples actions based on its local observation. We optimize with Proximal Policy Optimization (PPO) Schulman jt al. (2017). with hyperparameters provided in section A.3 Training is performed on single NVIDIA A100 GPU for 1 billion environment steps. During training, all VRUs (pedestrians and cyclists) follow the logged trajectories except for the VRU experiments in Table where we train SPACeR for all agent types, and calculate metrics on VRUs.
4.2 HUMANLIKE SELF-PLAY EVALUATION ON WOSAC
To measure humanlike behaviors, we adopt the evaluation protocol from the Waymo Open Sim Agent Chal- lenge (WOSAC) Montali et al. (2023). WOSAC evaluates the distributional realism of simulated agents by
6

---

Published as conference paper ICLR 2026
Method
Composite ?
Kinematic
Interactive f
Map f
minADE 4
Collision +
Off-road
Throughput 1
PPO
0.710
0.327 1.011
0.751
0.875
12.725 2.53
0.038 10005
0.053
211.8
HF-PPO
0.0
0.341 +.
0.756 0.1
0.0
12.254 1.0
0.006
0
211.8
SPACeR
0.741 + 0.00
0.411 o.u
0.779 + out
0,880 0:00
4.101 +0.00
0.036 oom
0.056 0.00
211.8
SMART
0.720
0.450
0.725 0.792
0.870
1.840
0.17
0.13
22.5 0:00
CAT-K
0.766
0.490
0.890
1.470
0.06
0.09
22.5
Table I: Results on the WOSAC Validation Set. Our proposed method outperforms other self-play approaches across all realism metrics, while achieving ~00 higher throughput than imitation-learning (shaded) methods, with competitive performance and lower collision/off-road rates. Throughput is mea- sured in scenarios/sec 5 Hz on a single A100 GPU.
comparing their rollouts against held-out human driving data, details provied in section A.5 Metrics assess kinematics (e.g,, speed, acceleration). inter-agent interactions, and adherence to map constraints, with results aggregated into composite realism score.
PPO (Self-Play). Decentralized PPO trained only with the task reward Itask Kazemkhani al. (2024); no realism signals are used.
HR-PPO Cornelisse & Vinitsky 2024 PPO regularized toward behavioral cloning (BC) reference using KL divergence only (no likelihood term). For fairness, the BC model shares the same backbone as is decentralized, and conditions only on local observations; it is trained on WOMD training split.
Imitation learning baselines. We evaluate two tokenized closed-loop
Wu et al. (024)
and CAT-K iZhang et al.] (2025). Both use the same backbone and action vocabulary (K = 200) and run
5 Hz. SMART is first trained by behavior cloning WOMD and then fine-tuned following the CAT-K K closed-loop fine-tuning protocol; Implementation details and heerparaaeetess are provided section A.3
Discussion of Main Results. In table 0 anchor- ing self-play with a pretrained reference model sub- stantially improves realism across all Sim Agent metrics. By contrast, HR-PPO reduces minADE but yields only modest realism gain, whereas our method achieves clear improvements in kine- matics, underscoring the value of alignment with a strong reference model. Relative to imitation- learning methods, our approach attains lower colli- sion and off-road rates and even surpasses SMART in composite realism.
0.78
SPACeR Policy Realism
0.76
0.3M
1M
3M
0.74
0.72
D.3M
3M
3M
0.68
SMART
CATK
0.66
Vocab 100
Vocab
0.64
Vocab 200
For efficiency and scale, our decentralized MLP has ~65k parameters versus ~3.2M for CAT-K (~500 smaller). Briefly, we observe ~ ~00 higher closed-loop throughput (single A100, Hz); see the rightmost column of table [1 and section 4.3 for de- tails. CAT-K K remains the strongest IL baseline, but method preserves human likeness at substan- tially higher speed and greater reactivity, which we further demonstrate in the closed-loop policy evaluation (section 4.3).
0.64
0.66 0.68 0.70 0.72 O. 0.74 0.76
Reference Model Realism
0.78
Figure 2: Reference Model Quality Effect on SPACeR Realism. SPACeR maintains high compos- ite realism 0.73) even when anchored to a reference model of only 0.3 parameters with realism 0.64.
Reference Model Quality, We further study how reference model quality affects SPACeR performance by training SMART models of three sizes: 0.3M, 1M, and 3M parameters. Each model is also fine-tuned
7

---

Published as a conference paper ICLR 2026
Method
Composite
Kinematic 1 Interactive
Map t minADE
PPO
0.648
0.242
0.683
0,835
7.712
HR-PPO
0.668
0.285
0.700
0.847
7.014
SPACeR (Ours)
0.729
0.413
0.762
0.866
2.066
Table 2: VRU realism metrics on WOSAC (pedestrians and cyclists). SPACeR outperforms PPO and HR-PPO by large margin, achieving substantial gains across all realism metrics and minADE.
with CAT-K, producing total of six reference models (fig. 2 SPACeR policy still reaches 0.732 realism even with the smallest underperforming reference model (0.636). We observe similarly stable performance when varying the token vocabulary size with 3M params ( (100, 200, 400; different markers in fig. 2 larger vocabularies increase reference model's realism, but the resulting SPACeR policies all remain clustered around 0.73. This indicates that the reference model serves mainly as a soft prior that guides humanlike behavior, rather than target for direct imitation.
Because SPACeR agents learn through closed-loop interaction, they can refine their behavior beyond the limitations of the reference model. In addition, using reference model that has itself undergone closed- loop fine-tuning (CAT-K) provides stronger behavioral signals, which further improves the resulting SPACeR agents performance.
Evaluation on Pedestrian and Cyclist Behavior. We evaluate SPACeR on the WOSAC validation set restricted pedestrians and cyclists isolate VRU behavior (table Although absolute composite re- alism scores are 0.1 lower than for vehicles, SPACeR still improves performance by large margin over PPO and HR-PPO across all metries, including Composite, Kinematic, Interactive, and Map realism, as well as minADE. This demonstrates that the anchoring mechanism remains effective even under the higher stochasticity and variability of VRU motion. Additional design choices and component ablations for VRU simulation are provided in Appendix section A.1
Effect of Anchoring Parameters on SPACeR Performance. We investigate the effect the anchoring parameters. namely the likelihood weight a and the KL alignment weight 3. by activating each term inde- pendently. Top-1 refers to evaluating agents by always taking the single most probable action, whereas top-5 evaluates realism under stochastic sampling from the top five most probable actions. Since SimAgent mea- sures distributional realism over 32 joint rollouts, maintaining diversity through top-k sampling generally yields higher composite scores Montali et al. 2023)
The likelihood and KL alignment terms play differ- ent roles in shaping policy behavior. Optimizing the likelihood term alone increases log likelihood but reduces diversity, as reflected by drop in entropy in the training curves (Appendix fig. AT This pro- duces small improvements top-1 composite real- ism but harms top-5 performance. In contrast, the
Composite Score
Top-
0.725
Top-5
0.700
0.675-
0.650
KL alignment term raises both likelihood and KI.
0.625
consistency while preserving entropy, which im- proves both top-1 and top-5 realism. The training curves in Appendix fig. Al illustrate how log like- lihood, KL divergence, and entropy evolve under each parameter setting.
0.0
0,001
0.01
Likelihood Weight (a) DxL Weight (B)
Anchoring parameter ablation. Likelihood-only improves top-1, while KI. alignment improves both top-1 and top-5 realism.
Figure 3:
the ablation study (table A2
KL alignment con-
tributes most realism. While log-likelihood wards are popular Escontrela et al. (2023), their impact on realism is modest, likely because the real-world driving distribution is highly multi-modal, maximizing likelihood alone does not yield stable suffi-
8

---

Published as conference paper ICLR 2026
CAT-K
SPACER
SPADeR GT
vs
Score:
PDM Score: O.
1.0
Collision Score: 0.0
Collision
631
Ego Policy Agent
A.L
8.42
0.2
PDM Score: 38
PDM Score: 0.
0.0
Collision
Comfort
Off-Road
FDM
Collision
Collision
Figure 4: Correlation Coefficients of PDM Scores Across Policy Evaluation Strategies. Using our ap- proach leads to consistently lower correlations with ground-truth log replays across all metrics, suggesting more realistic penalization unsafe policies-especially for collisions.
ciently discriminative signal. A second insight is that once the policy is anchored to the reference distribu- tion, the explicit goal-reaching reward can be removed, further improving realism.
4.3 SPACER AGENTS FOR FAST & ACCURATE CLOSED-LOOD LOOP PLANNER EVALUATION
key application of humanlike agents is the closed-loop evaluation of ego-vehicle motion planners. In this setup, the planner under test controls the ego vehicle, while surrounding agents are driven by learned poli- cies. This enables realistic modeling of inter-agent interactions, reactive behaviors, and humanlike decision- making without risking safety.
We evaluate our SPACER agents along two dimensions: closed-loop evaluation realism and throughput. Unlike distributional realism in section 4.2 which measures similarity to human data, realism here refers how sim agents change planner rankings under closed-loop interaction.
To assess realism, compare against two baselines: (i) open-loop log-replay evaluation, and (ii) closed- loop simulation using CAT-K Zhang et (2025), state-of-the-art traffic simulator on WOSAC. We evaluate a diverse set of planners: 18 self-play-trained policies, 10 sampling-base based planners Treiber et al. (2000) (more details can be found in A.7p. For each planner, compute PDM scores Dauner et al. (2024) across multiple scenes under three traffic-simulation strategies: ground- truth logs (GT), CAT-K rollouts, and SPACER agent policies. We then measure the correlation of PDM scores between strategies. As shown in fig. 4 our approach yields consistently lower correlations with GT replays when compared to CAT-K. We interpret this as evidence that our simulation is more reactive, and suppresses unrealistic planner behavior more effectively-particularr examples (see fig. and supplementary videos) further highlight that SPACER agents respond naturally in diverse scenarios, avoiding collisions when reasonable and minimizing unrealistic off-road behaviors.
Frenet planners, and 10 IDM-
in collision scenarios. Qualitative
For throughput, benchmark against SMART Wu al. (2024), leading multi-agent motion generator on the WOSAC leaderboard MMnttll et al. (2023). To ensure fairness, we run both methods at 5 Hz for full 8-second episodes single NVIDIA A100 GPU. SMART achieves 22.5 + 0.01 scenarios/sec, while method reaches 211.8 5.64 scenarios/sec a ~ 10x speedup. Moreover, GPUDrive can be optimized for another
efficiency gain Cusumano-Tooue
al. (2025) Suárez (024). All experiments

---

Published as a conference paper ICLR 2026
were run on dual Intel Xeon Platinum 8358 (64 cores / 128 threads, 2.6 GHz) server with single A100 GPU, and results are averaged over 5 seeds.
summary, SPACER agents deliver both more realistic closed-loop evaluation and significantly higher throughput, enabling reliable and scalable benchmarking of ego-motion planning policies.
5 DISCUSSION
Limitation WOSAC Metrics. During experiments, we found that WOSAC metries can produce mis- leading scores. In fig- [ the logged agent turns into parking lot, while SPACER continues straight without crossing the curb. Although this behavior has an off-road rate of 0.0, WOSAC penalizes it with a low map score because the metric rewards reproducing the logged trajectory rather than recognizing alternative valid behaviors. Likewise, when logs contain sensor noise leading to collisions or off-road trajectories, WOSAC assigns higher likelihood agents that repeat these errors Wang al. (2025). We provide more qualitative results in section A.6 This reveals a key limitation: WOSAC evaluates similarity to logged distributions, necessarily safety or numan-likeness, and can misalign with the goals of self-play RL.
Simulating VRUs (Pedestrians and Cyclists). SPACeR improves pedestrian and cyclist realism compared prior baseline in large margin, showing that self-play anchoring transfers beyond vehicle agents sec- tion 4.2 Nevertheless, VRU self-play remains underexplored: current WOSAC metrics (e.g.. collision, off- road) are tailored to vehicles and do capture sidewalk adherence, crosswalk usage, or other pedestrian specific behaviors. Reward design must likewise reflect these VRU-specific considerations. In this work we primarily use likelihood and collision signals for VRU motion. Developing VRU-aware metrics, reward shaping, and scene-level infrastructure is an important direction for enabling more realistic pedestrian and cyclist simulation.
Training Efficiency Bottlenecks. In our current set- ting. each run requires roughly 24-48 hours, partly due GPUDrive's lack of multi-GPU support. Future ex- tensions could exploit multi-GPU training or alternative backends such as PufferLib Suárezz (2024), which has demonstrated order-of ofaor-f---ggntudee speedups. Memory us- age also limits scalability, especially for architectures like SMART that encode pairwise interactions explicitly. Recent advances in memory-effiie -effiie-efficient design (e.g.. Zhao ct al. )2025) may help reduce overhead and increase training throughput.
Map Score: 23 Othenad sata
Ground Truth
SPACER
Figure 5: Example scenarios where WOSAC metrics produce unrealistic estimates
CONCLUSION
We introduced SPACER, which anchors self-play RL to pretrained tokenized model via KL. alignment, providing lightweight realism signals without relying on logged trajectories. On the Waymo Sim Agent Challenge, it achieves higher realism than prior self-play while producing policies that are ~500 smaller and ~00 faster than state-of-the of-the-art art imitation models. Our policies remain reactive and humanlike, avoiding the unrealistic collisions often observed in imitation methods during closed- planner evaluation. Together, these results position SPACER as a step toward scalable, real-time closed-loop evaluation, and ultimately training, of planners under realistic large-scale traffic scenarios.
Reproducibility Statement. We provide detailed implementation and training settings in Appendix A.1- A.2. Our approach are included in Sections 3.2, 3.3, and 4.1 ensure clarity and reproducibility.
10

---

Published as a conference paper ICLR 2026
ACKNOWLEDGMENT
This work was part of W.J. Chang's summer internship at Applied Intuition, and he is also supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752 Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The authors would like to thank Eric and Andrew for infrastructure support.
REFERENCES
Raunak Bhattacharyya, Blake Wulfe, Derek Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel J. Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. CoRR, abs/2006.06412, 2020. URL /ttp:/p::///xx..orr/////////////
06412
Yulong Cao, Boris Ivanovie, Chaowei Xiao, and Marco Pavone. Reinforcement learning with human feed- back for realistic traffic simulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14428-144 14434. IEEE, 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630-9640, 2021.
Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Safe- sim: Safety-critical closed-loop traffic simulation with diffussion-controllbllee adversaries. In Computer Vision ECCV 2024 (Lecture Notes in Computer Science, vol. 15079), pp. 242-255 Cham, Switzer- land, 2024. Springer. doi: 10.1007998--3-031-2666--- 14. URL httpt::://// /oi. 110.10111000
378-3-031-72664-4
Wei-Jer Chang, Wei Zhan, Masayoshi Tomizuka, Manmohan Chandraker, and Francesco Pittaluga. Lang- traj: Diffusion model and dataset for language-condiiioned trajectory simulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 26622-26631, October 2025.
Keyu Chen, Wenchao Sun, Hao Cheng, and Sifa Zheng. Rift: Closed-loop rl fine-tuning for realistic and controllable traffic simulation. arXiv preprint arXiv:2505.03334 2025.
Yuxin Chen, Chen Tang, Thomas Tian, Chenran Li, Jinning Li, Masayoshi Tomizuka, and Wei Zhan. Quan- tifying interaction level between agents helps cost-efficient generalization in aueti-agent reinforcement learning. Reinforcement Learning Jourmal, 1-1950-1964, 2024. URL pnttps:/rrl cs umass.
edu/20244papess RLJ RRTC_004 253 pdd
Daphne Cornelisse and Eugene Vinitsky. Human-compatie driving partners through data-regularized self- play reinforcement learning. arXiv preprint arXiv:2403.19643 2024.
Daphne Cornelisse, Aarav Pandya, Kevin Joseph, Joseph Suárez, and Eugene Vinitsky. Building reliable sim driving agents by scaling self-play, arXiv preprint arXiv:2502.144700 2025.
Marco Cusumano-Towwe David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor Killian, Stuart Bowers, Ozan Sener, et al. Robust autonomy emerges from self- self-play. arXiv preprint 2rXiv:2502.033449 2025.
11

---

Published as a conference paper ICLR 2026
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706-
28719, 2024.
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems, 11668760-687788 2023.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9710-9719, 9719, 2021.
Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Max- imilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. imitation: Clean imitation learning implementations. arXiv preprint arXiv:2211.11972 2022. BRLhhtts:://ee org/ aas/
11972
Eric A. Hansen, Daniel S. Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observ- able stochastic games. Proceedings of the 19th National Conference Artificial Intelligence (AAAI), pp. 709-715, 2004.
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanodd whole-body teleoperation and learning, 2024.
Max Jiang, Yijing Bai, Andre Cormman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kul- shrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efficient and controllable driving simulation initialization and rollout. Advances in Neural Information Processing Systems, 37: 55729-55760, 2024.
Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinitsky, Gpudrive: Data-driven, multi-agent driving simulation at million fps. arXiv preprini arXiv:2408.01588 2024.
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel J. Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 204-2112211221122200 221, IEEE, 2017. doi:
Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang. Shimon Whiteson, Brandyn White, and Dragomir Anguelov. The waymo open sim agents challenge. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 36, pp. 59151-59171. Curran Associates, Inc., 2023. URL https: //arxiv.org//////e /2tp./2335552 12032 arXiv preprint wrXiv:2305.12032
Zhenghao Peng. Wenjie Luo, Yiren Lu, Tianyi Shen, Cole Gulino, Ari Seff, and Justin Fu. Improving agent behaviors with rl fine-tuning for autonomous driving. In European Conference on Computer Vision, pp. 165-181. Springer, 2024.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. In International Conference on Learning Representations (ICLR 2024), 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. In Proceedings of the 34th International Conference on Machine Learning (ICML), abs 2017. URL bsttps://arte wttp:///.ww/w/b///000000222
12

---

Published as conference paper ICLR 2026
Joseph Suárez. Pufferlib: Making reinforcement learning libraries and environments play nice, 2024. URL
wttp/rggg.bba...a//b/////// 12905
Ran Tian and Kratarth Goel. Direct post-training preference alignment for multi-agent motion genera- tion models using implicit feedback from pre-training demonstrations. arXiv preprint arXiv:2503220105, 2025.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
Mingyi Wang, Jingke Wang, Tengju Ye, Junbo Chen, and Kaicheng Yu. Do Ilm modules generalize? a study on motion generation for autonomous driving. arXiv preprint 2.Xiv:2509.02754 2025.
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion genera- tion via next-token prediction. Advances in Neural Information Processing Systems, 37:1 177114048-110777
2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 11-24611-22622 2022.
Chris Zhang, Sourav Biswas, Kelvin Wong, Kion Fallah, Lunjun Zhang, Dian Chen, Sergio Casas, and Raquel Urtasun. Learning drive via asymmetric self-play. In European Conference on Computer Vision, pp. 149-168. Springer, 2024.
Zhejun Zhang, Peter Karkus, Maximilian Igl. Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the Computer 2025.
Vision and Pattern Recognition Conference, pp.
Jianbo Zhao, Taiyu Ban, Zhihao Liu, Hangning Zhou, Xiyang Wang, Qibin Zhou, Hailong Qin. Mu Yang, Lei Liu, and Bin Li. Drope: Directional rotary position embedding for efficient agent interaction model- ing. arXiv preprint arXiv:2503.15022 2025.
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guidee guided traffic simulation via scene-level diffusion. In Conference on robot learning, pp. 144-177. PMLR, 2023.
13

---

Published as conference paper ICLR 2026
Table Al: Ablation study on key components the SPACeR adapted to VRU simulation settings. Each row removes one design choice. Metrics are reported over VRU target agents only.
Ablation
Composite
Kinematic f
Interactive t
Map-based f
minADE
Full Model
0.729
0.413
0.762
0.866
2.066
-goal-reaching weight
0.728
0.405
0.769
0.859
2.295
mmulti-action head
0.685
0.323
0.742
0.818
3.416
- reference KL. loss
0.607
0.222
0.626
0.804
12.844
Variant
Composite.1
minADE.
Trsk only
0.70
14,43
Goal LLH
0.69
21.05
Goal + KL
0.73
4.08
KL Vind
0.74
4.73
KL + Thr LLI
0.74
4.68
Table A2: Ablations on WOSAC (validation). Tint = infraction penalties (off-road, collision); LLH log- likelihood reward; KL alignment essential for realism; goals unnecessary.
A APPENDIX
A.1 DETAILS SPACER DESIGN CHOICES FOR SIMULATING VRU
To extend SPACeR to simulating VRUs (pedestrians and cyclists), we ablate several design choices, as shown in table A1 We use separate action heads for each agent type and include a one-hot agent-type indicator in the ego observation so the policy can specialize its behavior. We also find that the reference KL loss consistently improves VRU realism, helping reduce unstable or abrupt pedestrian behaviors. Finally, both the goal-reaching loss and the multi-action head contribute to stable and human-like VRU motion, with noticeable drops in kinematic and interaction realism when either component removed.
For evaluation, all agents in the scene (including vehicles) are simulated, but WOSAC metrics are computed exclusively for pedestrian and cyclist targets. For training, we follow the same approach as Wu et al. (2024). where different agent types use their token vocabularies.
A.2 ANCHORING PARAMETERS ABLATION STUDY
In the ablation study table 2 KL alignment contributes most to realism. While log-likelihood rewards are popular Escontrela et al. (2023), their impact on realism is modest-likely because the real-world driving distribution is highly multi-modal, so maximizing likelihood alone does not yield a stable sufficiently discriminative signal. A second insight is that, once the policy is anchored to the reference distribution, the explicit goal reward can be removed, which further improves realism.
we visualize the training dynamics of the anchoring terms. Likelihood-only optimization steadily increases the log-likelihood but also drives entropy downward, indicating reduced diversity training pro- gresses. In contrast, adding KL. alignment not only increases log- ligeeikl but also keeps the KL term stable and preserves higher entropy, preventing collapse and supporting more reliable top-k realism.
fig. Al
14

---

Published as conference paper ICLR 2026
Log Likeslthond
-1
Entropy
-2
2.0.0
0.001
2 0.1
Training Steps
Training Steps
Training Steps
(a)
3.0
2.5
Log Likeslhhod
2.0
Eatzor
8.0
1.5
0-0.03
1.0
A f=0.11
3.00
0.5
Tef
169
Tell
Training Steps
Training Steps
Training Steps
(b)
Anchoring parameters: effects of likelihood and KL-divergence weights. (a) Likelihood weight effect without KI. loss. (b) KI. divergence weight sweep without likelihood loss. Likelihood-only optimization increases the probability of the most likely action but collapses entropy, while KL alignment maintains diversity and improves likelihood.
Figure Al:
A.3 TRAINING HYPERPARAMTERS AND IMPLEMENTATION DETAILS OF SELF-PLAY
The core PPO hyperparameters are summarized in table A3 When training without a reference model, employ 600 parallel worlds single A100 (80 GB, PCIe) GPU. For human-regularized PPO and reference-model training, we reduce the number of worlds to 300, which leads to an approximately 2x slowdown due to the lack of multi-GPU support in GPUDRIVE with madrona backend. With multi-GPU support, we would expect comparable throughput to the reference-free setting. Our policy/value network is configured with an input embedding dimension of 64, hidden dimension of 128. and a dropout rate of 0.01. To reduce GPU memory consumption in the reference-mode cap the maximum number unique scenarios per batch at 200 to increase training speed, and (2) we limit the maximum number of map elements per agent from 200 Kazemkhani et al. (2024) to 120. table A3
setting, we highlight two adjustments: (1) we
Human-Ree Heinaan-Regulrizeed PPO Baseline. We follow the setup of human-regularized PPO Cornelisse & Vinit- sky (2024), but adapt it to our tokenized trajectory action space instead of low-level control actions. To train the behavior cloning (BC) reference policy, we use observation-actio action pairs from the full Waymo Open Motion Dataset (WOMD) rather than the 200-scenario subset originally used, ensuring a stronger expert baseline. Our BC model is parameterized with roughly 2x the capacity of the self-play policy network to increase expressiveness, achieving a validation accuracy of 92%. We implement BC training using the imitation package Gleave et al. (2022) and train for 60 epochs on the full dataset.
During HR-PPO training, we regularize the learned policy against the BC reference policy using a KL- divergence penalty with weight 3 = 0.01. Larger values of 3 destabilize training, while using tokenized model can generally increase to 3 = 1.0 w/ similar performance. In addition, when applying human- regularized PPO in the original action space, found it necessary to clamp the minimum reference log-
15

---

Published as conference paper ICLR 2026
Table A3: PPO Training Hyperparameters.
Parameter
Value
Description
seed
42
Random seed.
total timestes
1,000,000,000
Total number of environment timesteps.
batch_sizz
131,072
Timesteps collected per rollout.
minibatch size
8,192
Timesteps per optimization minibatch.
learning_ran rate
3e-4
Optimizer learning rate.
anneal.r
false
Learning rate annealing.
gamma
0.99
Discount factor.
gae lambda
0.95
GAE parameter X.
update.epochs epochs
Optimization epochs per rollout.
norm _dy
true
Normalize advantages.
clip.coef
0.2
PPO policy clip coefficient.
clos_vloss
false
Clip value loss.
vf.clip.coef
0.2
Value function clipping coefficient.
coef
0.0001
Entropy coefficient.
vf coef
0.3
Value loss coefficient.
max mexx_grad_ormm
0.5
Gradient clipping (max 1.2 norm).
probability 10-20 avoid unstable gradients; interestingly, this instability does occur when using tokenized trajectory reference model.
A.4 TRAINING DETAILS OF SMART AND CATK.
We pretrain the base SMART model on 16x A A100 (80 GB, PCIe) GPUs for 10 epochs and select the check- point with the best validation loss (5 10 10-4). We then apply closed-loop supervised finetuning as described in Zhang et al. 2025) for 6 additional epochs with an effective batch size of 64 and a learning rate of 1 10 The final model contains 3.2M parameters and is trained on the full training dataset of approxi- mately 500k scenarios. To make the self-play policy more responsive while maintaining balance between control frequency and memory usage, we use a sampling frequency of 5 Hz instead of the original Hz.
A.5 WAYMO SIM AGENT CHALLENGE METRICS
To evaluate whether the trained policies are humanlike, we follow the evaluation protocol of the Waymo Open Sim Agent Challenge (WOSAC Montali al. (2023), which measures how closely the distribution of simulated rollouts matches the ground-trui distribution across kinematics, agent interactions, and map adherence.
[Wottc|Montalii
The WOSAC metrics quantify how closely the distribution of simulated rollouts matches the ground-true truth distribution, across multiple aspects such as kinematics, agent interactions, and map adherence.
Concretely, for each scenario containing up to 128 agents simulated for 8 seconds, we generate 32 multi- agent rollout samples. For target agent in scenario i and statistic F,. the negative likeliood (NLL) the ground-truth outcome under the empirical distribution of simulated samples is defined as:
NLL(i, a, t.j) = log Pi.ja (F, (x " a, (i))
(6)
where Pi.j.a()) (:) denotes the empirical distribution constructed from the simulated samples, and x xi, a, the true trajectory at time Lower values indicate that the simulation better reflects observed behavior.
16

---

Published as conference paper ICLR 2026
To obtain a per-agent summary, we aggregate over valid timesteps:
m(a, i, exp
N(i, (-N(, oli t) (.
where a)) -, v(i, is the number of valid timesteps for agent The scenario-level score is then computed as the average across all evaluated agents:
(1,))
Auargt Em(a,
(8)
with Aurget denoting the number of target agents the scenario.
In our experiments, we restrict evaluation to vehicles only: agents corresponding to pedestrians or cyclists are excluded as targets and fixed to their ground-truth trajectories. All reported results computed on a
2% validation subset of the dataset /Zhang et al. (2025).
A.6 WOSAC FAILURE QUALITATIVE RESULTS
More qualitative results of WOSAC limitation provided in fig. A2
A.7
PLANNER VARIANTS OVERVIEW
Our evaluation framework covers two categories: learning- and rule-based planners.
Learning-based We trained 22 self-play reinforcement learning policies in GPUDrive Kazemkhani et al. (2024), all sharing the same decentralized late-fusion MLP architecture (controlling up agents) but differing in reward weights (see Sec. A.5.1).
Rule-based. We include two families of classical planners: Frenet-based trajectory samplers and Intelligent Driver Model (IDM) planners, each with 10 parameterized variants ranging from conservative to aggressive driving styles.
A.7.1 SELF-PLAY POLICIES
addition to handcrafted planners, we trained 22 self-play policies using the GPUDrive framework Kazemkhani et al. (2024). All policies share the same decentralized late-fusion MLP architecture (con- trolling up to 64 agents), but differ in reward weighting.
We varied the weights of three components: goal-reaching (Wgoal 0.5)), collision penalties (Weollided E {0, -0.37, -0.75, +0.13), and off-road penalties (Woftread E -0.375, -0.75, +0.1)). This grid covers both standard reward shaping (non-negative penalties) and unconventional settings with negative weights, where agents are encouraged to collide or go off-road.
The resulting 22 policies span wide spectrum of behaviors, from highly conservative (strong safety penal- ties) to adversarial (negative penalties), allowing us to stress-test planner evaluation under diverse multi- agent conditions. All policies trained for 1B environment steps 10k resampled WOMD scenarios, with 600 parallel worlds and up to 64 controlled agents per rollout.
A.7.2 FRENET-BASED PLANNERS
The Frenet planner uses quintic/quartic polynomials to generate trajectory candidates in the Frenet frame, evaluating them based weighted cost function that considers:
17

---

Published as conference paper ICLR 2026
Agent
Interectivn
Map Score: -626 offroad Rate:
Offrood Ulkelioood: 00000
Ground Truth
SPACER
Figure A2: Additional qualitative examples of WOSAC limitations. Top row: due sensor noise, the ground-truth agents collided in the data; consequently, safe simulated behaviors are assigned low collision likelihood. Bottom row: the ground-truth agent collided with the road edge, so remaining on-road given near-zero off-road likelihood and low map score. These cases illustrate how WOSAC penalize safe and realistic behaviors when they diverge from noisy or imperfect logged trajectories.
Lateral deviation from centerline (Wattral)
Velocity tracking (Wyelocity)
Acceleration smoothness (Wacceleration)
Progress along path (Wprogres))
Jerk minimization (Wjek)
Collision avoidance (collision penalty)
Table summarizes the 10 Frenet-based planner variants, showing their key characteristics including speed ranges, Jateral weights, safety focus levels, and sampling densities. These variants range from conservative focused configurations aggressive high-speed optimized settings.
18

---

Published as conference paper ICLR 2026
Table A4: Summary of Frenet-based Planner Variants
Variant
Description
Speed (m/s)
Lateral Weight
Safety Focus
Sampling" (d,v,t)
Key Features
Baseline
Balanced
0-30
10.0
Medium
10,5,3
Standard configuration
Aggressive
High progress
0-35
5.0
Low
10,5,3
Progress weight 2.0
Conservative
Safety-first
0-20
50,0
High
10,5,3
Collision penalty 5000
Smooth Rider
Comfort
0-30
20.0
Medium 10,5,3
Jerk weight 3.0
Lane Keeper
Centerline
0-30
100.0
Medium 15,5,3
Lateral span 1.5m
Wide Search
Comprehensive
0-30
10.0
Medium
20,10,7
Large search space Reduced horizon 40 horizon steps Collision disabled
Fast Planner
Quick
0-30
10.0
Medium
5,3,2
Long Horizon
Strategic
0-30
10.0
Medium
10,5,3
No Collision
Test baseline
0-30
10.0
None 10,5,3
High Speed
Highway
5-40
10.0
Medium
10,5,3
Velocity span 15
Sampling notation: (d.v.t) represents (lateral samples, velocity samples, time samples)
A.7.3 IDM-BASED PLANNERS
The IDM planner implements the Intelligent Driver Model for longitudinal control combined with PID controller for lateral tracking. Key parameters include:
Desired velocity (vo)
Minimum spacing (sg)
Safe time headway (T)
Maximum acceleration (a)
Comfortable deceleration (b)
Aggressiveness factor (0.0-1.0)
Table A5 presents the 10 IDM-hased planner variants, each configured to represent different driving styles from cautious urban driving aggressive highway scenarios. The aggressiveness factor plays key role in determining the overall behavior of each variant.
Table providds detailed comparison of the key configuration parameters for representative variants from both planner types, highlighting the differences in their weight distributions and fundamental parameters that lead their distinct behaviors.
19

---

Published as conference paper ICLR 2026
Table A5: Summary of IDM-based Planner Variants
Variant
Description
Desired Vel (m/s)
Min Gap
Headway
Aggress." Factor
Special
8g (m)
T (s)
Features
IDM Baseline
Standard
30
2.0
1.5
0.5
Balanced behavior
IDM Conservative
Cautious
25
3.0
2.0
0.2
Safety factor 1.5
IDM Aggressive
Dynamic
1.5
1.0
0.8
Safety factor 0.9
IDM Comfort
Smooth
28
2.5
1.8
0.3
Max jerk 2.0
IDM Highway
High-speed
3.0
1.2
0.6
Perception 100m
IDM City
Urban
15
2.0
1.5
0.4
Perception 30m
IDM Truck
Heavy
4.0
2.0
0.3
Length 8.0m
IDM Emergency
Urgent
40
1.5
0.8
0.9
Max accel 4.0
IDM Adaptive
Balanced
30
2.5
1.5
0.5
Reaction 0.2s
IDM Defensive
Safety
25
4.0
2.5
0.1
TTC== 3.0s
Aggressiveness factor:
Ranges from (very conservative) to 1.0 (very aggressive)
TTC: Time-to-collision threshold
Table A6: Key Configuration Parameters Comparison
Parameter
Baseline
Aggressive
Conservative
Smooth
Lane Keeper
Frenet Planner Weights
Lateral (wr)
10.0
5.0
50.0
20.0
100.0
Velocity (w,)
1.0
0.5
1.0
2.0
1.0
Acceleration (w.)
1.0
1.0
3.0
5.0
1.0
Progress (wp)
1.0
2.0
1.0
1.0
1.0
Jerk (wy)
0.5
0.5
1.5
3.0
0.5
IDM Parameters
Desired vel (ug)
30.0
35.0
25.0
28.0
Min spacing (sp)
2.0
1.5
3,0
2.5
Time headway (I')
1.5
1.0
2.0
1.8
Max accel (a)
2.0
3.0
1.5
1.5
Comfort decel
3.0
4.0
2.0
2.0
Note: All planners use dt=0. Is time step and wheelbasc=2.88m
20


Reviews:
[{"review_id": "uSodXqD6iN", "reviewer": "Reviewer_tztw", "summary": "This paper proposes SPACeR (Self-Play Anchoring with Centralized Reference), a novel framework that integrates the scalability of self-play reinforcement learning with the realism of imitation learning. The key idea is to anchor decentralized self-play policies to a pretrained, centralized tokenized reference model via a KL-divergence alignment and a log-likelihood reward, enabling human-like behavior without relying on logged trajectories. This design allows SPACeR to achieve realistic, reactive, and efficient driving agents that are up to 10× faster and 50× smaller than large generative imitation models.", "questions": "Q1. How sensitive is SPACeR to the quality and coverage of the reference model? For instance, would using a smaller or domain-shifted tokenized model significantly affect policy realism or stability?\n\nQ2. Could the reference signal (KL and likelihood reward) be updated dynamically or distilled into a lightweight surrogate during training to reduce reliance on a fixed pretrained model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. SPACeR introduces an elegant integration of self-play reinforcement learning with a pretrained tokenized reference model, using KL-divergence alignment and likelihood-based rewards to anchor decentralized policies toward human-like behaviors. The approach is conceptually clean, easy to implement, and avoids heavy reliance on heuristic reward shaping or large generative models.\n\nS2. The proposed lightweight decentralized policy (≈65k parameters) achieves over 10× faster inference and 50× smaller model size than state-of-the-art imitation-learning methods (e.g., SMART, CAT-K), while maintaining comparable realism. This demonstrates clear practical potential for large-scale, real-time autonomous driving simulation and planner evaluation.", "weaknesses": "W1. The effectiveness of SPACeR heavily relies on the pretrained tokenized reference model. If the reference distribution is biased or limited in coverage, the learned self-play policies may inherit those biases and fail to generalize to unseen or long-tail behaviors. The paper would benefit from a sensitivity or ablation analysis on different reference model qualities.\n\nW2. The current experiments are restricted to vehicle agents, without incorporating pedestrians, cyclists, or mixed-traffic interactions. Since realistic urban driving often involves diverse and heterogeneous agents, demonstrating SPACeR’s adaptability to such settings would significantly strengthen its generality and practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aYh1RfWxQU", "reviewer": "Reviewer_nA91", "summary": "This paper presents SPACER (Self-Play Anchoring with Centralized Reference Models), a framework for training large-scale, human-like driving agents in simulation. The key idea is to anchor reinforcement learning (RL) agents trained via self-play to a centralized, pretrained imitation-learning (IL) model. The centralized model serves as a human-likeness reference, providing both a log-likelihood reward and a KL-divergence regularization term that guide the self-play policy toward realistic behaviors while retaining scalability and reactivity. The approach achieves significantly improved realism on the Waymo Sim Agents Challenge compared to baseline self-play RL and imitation-only models, with 10× faster inference and 50× smaller model size.", "questions": "How sensitive is the performance to the choice or size of the reference imitation model?\n\nCould the anchoring approach be generalized to other domains (e.g., pedestrian or cyclist simulation)?\n\nDoes the KL regularization ever constrain the policy too strongly, reducing behavioral diversity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Clearly identifies and addresses the gap between imitation learning (realistic but non-reactive) and self-play RL (reactive but unrealistic).\n\n- Introduces a principled anchoring mechanism through likelihood and KL-based regularization.\n\n- Demonstrates clear ablation and comparative analysis against strong baselines.", "weaknesses": "- The paper could elaborate on how anchoring parameters affect the trade-off between realism and exploration.\n\n- The reference model’s dependence on large imitation datasets may limit generalization to low-data domains.\n\n- The related-work section would benefit from citing related studies:\n\nA. Kuefler et al., “Imitating Driver Behavior with Generative Adversarial Networks,” IEEE IV 2017.\n\nR. P. Bhattacharyya et al., “Modeling Human Driving Behavior through Generative Adversarial Imitation Learning,” CoRR 2020.\n\nH. Chen, T. Ji, S. Liu, and K. Driggs-Campbell, “Combining Model-Based Controllers and Generative Adversarial Imitation Learning for Traffic Simulation,” IEEE ITSC 2022.\n\nK. Brown, K. Driggs-Campbell, and M. Kochenderfer, “Modeling and Prediction of Human Driver Behavior: A Survey,” arXiv:2006.08832, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T4PGCowPJI", "reviewer": "Reviewer_Vct6", "summary": "* This paper introduces SPACER, a novel framework for training realistic and human-like sim agents that combines RL and IL.\n* The key idea is to have a lightweight, decentralized student RL policy that is anchored to a large pre-trained centralized teacher policy (reference model).\n* The reference model is pre-trained to capture human driving data and provides a realism signal as a reward as well as a KL divergence term to the student during self-play.\n* The KL divergence can be computed efficiently in closed-loop due to RL agent and reference model action space alignment.\nSPACER achieves competitive realism scores with 50x fewer parameters and 10x faster inference as well as lower collision rates and superior reactivity in closed-loop planner evaluation", "questions": "* The ablation shows that the realism reward signal provides hardly any benefit and only the KL term is critical. This result is surprising to me. Do you have an explanation for it? Did you study this more?\n* How sensitive is your method to the size of the discrete action space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper tackles an important practical problem: Creating sim agents that are realistic and human-like, yet, fast and cheap to run.\n* The SPACER results are compelling: The small model performs well on WOSAC on the composite metric and clearly outperforms the baselines in collisions and offroad events.\n* The ablation results clearly show the importance of the KL divergence term.\n* The authors provide good examples of WOSAC weaknesses.", "weaknesses": "* VRUs are not controlled and follow logs, which is a significant shortcoming.\n* Unclear benefits of the realism reward signal in the ablation. Also see my question below.\n* While the RL agent inference is fast, training is expensive (first need to train a reference policy, then run inference on it during RL).\n* The HR-PPO baseline is decentralized, which makes it weaker since SPACER can access a centralized reference model during training. A fairer comparison could be a centralized HR-PPO policy distilled into a decentralized one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)