Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning
Authors: Muleilan Pei
Shaoshuai Shi
Shaojie Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving
Reinforcement Fine-Tuning
Multi-agent Traffic Simulation
Primary area: applications to robotics, autonomy, planning
TL;DR: A novel R1-style Reinforcement Fine-Tuning (RFT) paradigm for multi-agent traffic simulation in autonomous driving.

Abstract: Scalable and realistic simulation of multi-agent traffic behavior is critical for advancing autonomous driving technologies. Although existing data-driven simulators have made significant strides in this domain, they predominantly rely on supervised learning to align simulated distributions with real-world driving scenarios. A persistent challenge, however, lies in the distributional shift that arises between training and testing, which often undermines model generalization in unseen environments. To address this limitation, we propose SMART-R1, a novel R1-style reinforcement fine-tuning paradigm tailored for next-token prediction models to better align agent behavior with human preferences and evaluation metrics. Our approach introduces a metric-oriented policy optimization algorithm to improve distribution alignment and an iterative "SFT-RFT-SFT" training strategy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) to maximize performance gains. Extensive experiments on the large-scale Waymo Open Motion Dataset (WOMD) validate the effectiveness of this simple yet powerful R1-style training framework in enhancing foundation models. The results on the Waymo Open Sim Agents Challenge (WOSAC) showcase that SMART-R1 achieves state-of-the-art performance with an overall realism meta score of 0.7858, ranking first on the leaderboard at the time of submission.

Published as a conference paper at ICLR 2026
ADVANCING MULTI- TRAFFIC SIMULATION VIA R1-STYLE REINFORCEMENT
Muleilan Pei', Shaoshuai Shi?; Shaojie Shen'
Hong Kong University of Science and Technology Voyager Research, Didi Chuxing {mpei, eeshao jie ie jieeeust..t.... shaoshuai asse cs com
ABSTRACT
Scalable and realistic simulation of multi-agent traffic behavior is critical for ad- vancing autonomous driving technologies. Although existing data-driven simu- lators have made significant strides in this domain, they predominantly rely on supervised learning align simulated distributions with real-world driving sce- narios. A persistent challenge, however, lies in the distributional shift that arises between training and testing, which often undermines model generalization in un- seen environments. To address this limitation, we propose SMART-RI, a novel RI-style reinforcement fine-tuning paradigm tailored for next-token prediction models to better align agent behavior with human preferences and evaluation met- rics. Our approach introduces metric-oriented policy optimization algorithm to improve distribution alignment and an iterative "SFT-RFT-SFT" training strat- egy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) maximize performance gains. Extensive experiments on the large-scale Waymo Open Motion Dataset (WOMD) validate the effectiveness of this simple yet powerful R RI-style training framework in enhancing foundation models. The results on the Waymo Open Sim Agents Challenge (WOSAC) show- case that SMART-RI achieves state-of- of- oftte--f--he-rrr performance with overall realism meta score of 0,7858, ranking first on the leaderboard at the time of submission.
INTRODUCTION
Simulating multi-agent traffic behaviors plays pivotal role in ensuring the safety and reliability of autonomous driving systems. However, modeling realistic and scalable traffic behaviors remains highly challenging due to the inherent uncertainty and multi-modality of human driving. Traditional simulators that simply replay logged data lack reactive capability, while rulc-based methods, such as the Intelligent Driver Model (IDM) (Treiber al., 2000), depend on handcrafted heuristics and fail to capture the diversity and realism of human behavior. To overcome these limitations, recent research (Montali et al. 2023) has reframed realistic traffic simulation as a distribution matching problem, aiming to align the behavior distributions of simulated agents with those observed in real- world data. The guiding principle is that, as simulation quality improves, the distributions of learned simulated agent behaviors should assign higher likelihoods to real-world logged samples.
Early learning-basse simulators largely adopted architectures designed for motion forecasting, such as encoder-decoder Transformers (Shi et al.) 2024, Zhou et al. 2023). Yet. simulation fundamen- tally differs from trajectory prediction: it requires (i) a closed-loop setup. (ii) consistent scene-level multi-agent futures, and (iii) recovery of the underlying behavior distribution rather than simple tra- jectory imitation. These distinctions often cause motion predictors to perform poorly when adapted to simulation. More recent advances in high-performing simulators have primarily followed two di- rections: diffusion-basee based models (Jiang et al. 2024 Huang et al.) 2024) and autoregressive models (Seff et al.) 20233] Philion al. (0024). Diffusion models can generate diverse multi-modal futures by modeling joint trajectory distributions, but they suffer from computational inefficiency and strug- gle capture multi-agen interactions. In contrast, autoregressive approaches treat agent behaviors as discrete tokens under the Tekt-Token Prediction (NTP) paradigm (Wu et al.) 2024 Zhou et al.) 2024), generating interactive and realistic multi-agent behaviors through minimizing cross-entropy
"Correponndigg Author.

---

Published as a conference paper at ICLR 2026
Reward
Open-Loop NTP
Closed-Loop
Closed-Loop RFT
Closed-Loop SFT
SFT
Figure 1: Training pipeline of SMART-RI. The solid square represents the agent state at the cur- rent timestep. The Open-Loop NTP predicts the next single-step state, while Closed-Loop SFT rolls out entire trajectories autoregressively to identify those closest to the ground truth. Both stages are optimized with per-token cross-entropy loss. contrast, Closed-Loop RFT performs full rollouts but aligns trajectories with evaluation preferences through reward feedback and policy optimization.
loss against the distribution of logged tokens. However, because NTP models generate behaviors in an autoregressive manner, they are prone to encountering the covariate shift issue: small prediction errors accumulate during closed-loop simulation rollouts. Inspired by post-training strategies for Large Language Models (LLMs) (Ouyang et al.) 2022). a Supervised Fine-Tuning (SFT) technique with Closest Among Top-K (CAT-K) rollouts (Zhang et al., (2025) has been proposed mitigate compounding errors, significantly improving simulation realism.
Despite the remarkable performance of NTP models in traffic simulation, fundamental limitation persists: the training objectives of current imitative or Behavior Cloning (BC) models are not explic- itly aligned with ultimate eehavior-characterinn off-road rates. These outcome metrics are simulation scalar, sparse, and non-differentiable, making them unsuitable as direct trainable loss functions for gradient- basde optimization. Consequently, relying solely on BC or SFT is insufficient for realistic simulation agents.
goals of simulators, such reducing collisions or
To bridge this gap. we draw inspiration from cutting-edge Large Reasoning Models (LRMs) such as OpenAl-ol (Jaech et al. (2024) and DeepSeek-RI R1 (Guo et al. 2025), which leverage Reinforcement Learning from Human Feedback (RLHF) or Reinforcement Fine- TuninTuning (RFT) to align model be- haviors with specific preferences. In this regard, we introduce an R R1-style RFT training paradigm for traffic simulation, as demonstrated in Figure [] Specifically, we build on SMART (Wu et al.) 2024), strong open-loop NTP foundation model, and employ CAT-K rollouts (Zhang et al. 2025 for closed-loop SFT. In the subsequent RFT stage, we refrain from directly applying the Group Rel- ative Policy Optimization (GRPO) (Shao et al. 2024) algorithm for preference alignment. Although GRPO eliminates the necessity for explicit value function approximations by utilizing normalized relative rewards within each group, its dependence
average rewards across multiple sampled roll- outs introduces inherent sampling bias and undermines the reliability of advantage estimation for policy optimization (Zheng al. 2025). Given the relatively predictable reward expectations in our task, we instead propose Metric-oriented Policy Optimization (MPO) strategy, which exploits this prior knowledge to guide policy updates in RFT more efficiently and effectively.
Moreover, the conventional practice in LRMs typically involves leveraging SFT as a warm-up up stage before RFT, which poses the issue of catastrophic forgetting (Chen et al. 2025). resulting in per- formance degradation. To address this concern, following the training pipeline of DeepSeek-R1, which alternates multiple rounds of SFT and RFT to refine reasoning performance, we introduce an additional closed-loop SFT phase after RFT. This yields iterative "SFT-RFT-SFT" (RI-style) post-training scheme that balances optimization for metric-specific objectives with preservation of the behavioral distribution learned during SFT, thereby further enhancing overall simulation realism.
In summary, the primary contributions of this paper are as follows:
(1) We introduce SMART-RI, to our knowledge the first RI l-sttyle post- post-training paradigm for multi- agent traffic simulation, which combines SFT and RFT to better align simulated behaviors with human preferences and evaluation metrics.
(2) We develop a simple yet effective Metric-oriented Policy Optimization (MPO) strategy for RFT to reinforce the model towards targeted evaluation metrics.
2

---

Published as a conference paper at ICLR 2026
(3) Our proposed post-fine-tue cest-fine-tuiing tuning pipeline, structured "SFT-RFT-SFT"" empowers SMART-RI to achieve state-of thr performance on the 2025 Waymo Open Sim Agents Challenge, rank- ing first on the leaderboard at the time of submission.
2 RELATED WORK
Traffic Behavior Modeling. Accurately modeling the behaviors of traffic agents is a core compo- nent of autonomous driving systems. Early approaches represent behavior in a discrete and sparse decision space (e.g.. lane-keeping or left-turning) but such abstractions are overly coarse to capture the diversity of real-world driving (Cunningham
al., 2015). More recent work instead models behavior as sequences of future position points (Pei et al. 2025bb a). and employs encoder-decoder architectures to generate continuous trajectories by framing the task as regression (Song al. 2022 Shi et al. 2025). In contrast, emerging approaches characterize traffic behavior as language mod- eling problem, where trajectories are discretized into motion tokens to form an action vocabulary. The resulting trajectories are then produced via next-token prediction in an autoregressive manner (Philion et al. 2024 Seff et al. 2023). This paradigm naturally captures multi-modal and interactive behaviors among agents, making well-suited for realistic traffic simulation.
Multi-agent Simulation. Multi-agent traffic simulation crucial for evaluating the robustness and safety of autonomous vehicles (Shi et al., 20266). Traditional simulators often depend rule-based policies (Treiber et al., 2000) for traffic participants, which fail to reproduce the complexity and realism of human driving. To address this, data-driven approaches have been proposed to enable scalable and realistic simulation. Early methods repurpose trajectory prediction models for simu- lation 4Shi al., 2024 Zhou al., 2023); however, they lack explicit interaction modeling and exhibit poor stability in closed-loop rollouts. Recent advances in generative modeling have mainly followed two directions: diffussionbbsed methods (Jiang al.) 2024 Hu et al. 2024) and autore- gressive methods (Wu et al. 2024, Zhou et al.] 2024] Peng et al. 2024). While diffusion models can capture multi-modality, they sulter from computational inefficiency and degrade in long-horizon scenarios. Consequently, research momentum has shifted toward autoregressive models. In line with these trends, we adopt the next-token prediction paradigm for the multi multi-agent traffic simulation task.
Reinforcement Fine-Tuning, Tuning. Post-training is widely utilized in LLMs to enhance the model align- ment with human preferences. Early efforts primarily concentrate on SFT, while more recent work explores RFT to improve reasoning capabilities. For example, OpenAl-ol leverages Proximal Policy Optimization (PPO) (Schulman et al. 2017) for RLHF Subsequently, Direct Preference Optimiza- tion (DPO) Rafailov et al. (223)) is proposed to simplify the RLHF process by eliminating the need for training a reward model. More recently, DeepSeek-Ri introduces GRPO (Shao et al. 2024) within multi-stage training pipeline that alternates between SFT and RFT to progressively refine reasoning performance. Building on this trend, a series of RI Rs-slyle approaches have been proposed to extend RFT-based alignment to domains such vision reasoning (Huang et al. 2025 Tan al.) 2023) and trajectory planning (Li al.] (2025] Tang et al.) 2025). However, most prior works employ RFT-empowered LLMs as auxiliary components for downstream tasks, rather than integrating RFT directly into the downstream domain itself. In contrast, to the best of our knowledge, our work is the first to apply RI Rtysl RFT directly the domain of multi multi-agent traffic simulation.
METHODOLOGY
3.1 PRELIMINARIES
Problem Formulation. For multi-agent traffic simulation, the problem is often framed as learning a policy (8.)SS22 C). where S. = N denotes the state of N agents in the scene at time step t, Sei = 1.N is the historical states, and C represents the contextual information such as features from High-Definition (HD) maps. The trainable model parameters are denoted by 0. The objective to learn joint distribution of agent behaviors within the of input information over a time horizon r:
P(S So. C) = II To Te(S,
(1)
3

---

Published as a conference paper at ICLR 2026
NTP
Ref.
Agent2Agent Self-Attn.
MPO
Tokenization
Map Tokens
Map2Agent Cross-Attn.
Reward Model
Temporal Self-Attn.
Driving Context
Agent Takens
SFT
RFT
Framework of SMART-RI. The model first tokenizes the driving scene context into agent motion tokens and map tokens. In the BC pretraining stage, the model is optimized under the standard NTP paradigm. During the SFT stage, CAT-K rollouts are used to refine the model a closed-loop setting. Finally, in the RFT stage, the proposed MPO algorithm aligns the policy with larget evaluation metrics, further improving simulation realism.
Figure 2:
NTP Models for Traffic Simulation. To model agent behaviors within the Next-Token Prediction (NTP) framework, we often preprocess the driving context by discretizing and clustering continuous agent trajectories into motion token vocabulary V {mi k==1 {m comprising \V] template motion tokens my- Accordingly, the policy =0(S,[5<e: C) can be reformulated as predicting categorical distribution over the indices of the motion token vocabulary at each time step i.e.
To (e. C) = II no (ki)S C*),
where X denotes the tokenized format of the inputs X.
Closed-Loop SFT. To further mitigate the covariate shift introduced by open-loop behavior cloning, the CAT-K rollout strategy (Zhang et al., 2025) can be leveraged for closed-loop SFT in the traffic simulation task. Specifically, after obtaining the pre-trained foundation model, we autoregressively unroll the policy to generate tokenized trajectories and construct recovery motion indices to stay close to the ground truth. This closed-loop SFT approach effectively reduces covariate shift in NTP models and leads to improved overall performance.
3.2 FRAMEWORK OVERVIEW
The overall architecture of our proposed SMART-RI is illustrated in Figure stage training paradigm consisting of Behavior Cloning (BC) pretraining, closed-loop SFT, and RFT. Given a real-world scenario, following prior work (Wu al. 2024 Zhang al., 2025 we first discretize continuous trajectories and map polylines into agent motion tokens and map tokens using K-disks clustering algorithm (Philion et al. 2024). These tokens are then processed by a stack of attention layers: temporal self-attention layer capture sequential dependencies in agent motions, a saapto-agenn
follows multi-
layer to incorporate map-relevant information, and an agent-to-agent self-attention layer to model interactions among traffic participants for generating joint multi-agent behaviors. The outputs from these modules produce the next-token logits. the BC pretraining stage, the model is optimized via token-level classification. The resulting pretrained base model subsequently refined through closed-loop SFT, where CAT-K rollouts are employed to mitigate the covariate shift inherent BC. Finally, in order align the model with metrie preferences, we introduce an RFT stage and develop simple yet effective Metric-oriented Policy Optimization (MPO) algorithm for refinement, further enhancing overall simulation realism.
3.3
POLICY OPTIMIZATION
Considering that both the open-loop pre-trained model and the closed-loop SET stage are trained using cross-entropy loss learn the distribution of logged data, they lack direct optimization for
4

---

Published as a conference paper at ICLR 2026
Algorithm Metric-oriented Policy Optimization (MPO)
Input: Initial policy T....
Output: Optimized RFT policy TT
1: Initialize the reference model =0a f- m...
2: Initialize the policy model Te < 78mm
3: for iteration ... do
4: Roll out multi-agent trajectories in the scene via Tg.
5: Compute the reward function for the simulation outcomes using the Realism Meta metric.
6: Compute the simplified generalized advantage estimation A via Equation B
7: Compute the unbiased estimation of per-token KL divergence [To| TT,a) via Equation
8: Update policy Te by minimizing LMPO via Equation 5
9: end for
task-specific evaluation metrics. To bridge this gap, we introduce RFT to explicitly align the pre- trained NTP model with designated target metric.
We begin by formulating the behavior simulation problem as a Markov Decision Process (MDP), where each tokenized motion corresponds to element in the action space. The policy my determin- istically transitions each agent to its next state by selecting the associated motion token. The reward model r is defined by the Realism Meta metric computed through the official evaluation protocol.
Building upon the MDP formulation, we propose the Metric-oriented Policy Optimization (MPO) maximize the cumulative Realism Meta score. As illustrated in Figure 3 for each traffic scenario, we autoregressively roll out complete trajec- tories for all agents and evaluate the resulting simulation using the official Realism Meta metric. This metric serves as a reward model, assigning a scalar score assess the quality of the generated behaviors. Note that we can also apply other preferences to reinforce the alignment. Unlike GRPO (Shao et al. 2024). which relies generating group of comple- tions for comparison, our approach simplifies the estimation of the advantage function, in our specific task, we have the expected re- ward value for the model to achieve. Specifi- cally, we compute the Generalized Advantage Estimation (GAE). denoted A, as follows:
Policy Rollouts
Reward
Policy Model
DKL
Cauo
A
Policy Rollouts
Reference Model
Policy Optimization
A= a
(3)
Figure 3: Pipeline of the Metric-oriented Policy Op- timization (MPO).
where a is an empirical threshold used to nor- malize the reward signal and facilitate relative evaluation for efficient policy optimization.
We also incorporate a per-token KL divergence penalty between the updated policy me and refer- ence model Tog using the following unbiased estimator:
log
1.
(4)
The final training objective for MPO then given by:
LMPO T
3DKL (To TT.d)
(5)
where To a no- no-gradient copy of the policy, and a tunable coefficient balancing the advantage term against the KL. penalty. The complete pipeline of MPO is summarized in Algorithm.
3.4 R1-STYLE TRAINING PARADIGM
Since models fine-tuned with RFT risk deviating from the original distribution learned during pre- training and SFT, leading to catastrophic forgetting (Chen al. 2025), we draw inspiration from
5

---

Published as a conference paper at ICLR 2026
the multi-stage post-training pipeline of DeepSeek-RI (Guo et al., 2025 and introduce iterative "SFT-RFT-SFT" SFFTSSFT" strategy. Starting from a pretrained base model, we first apply CAT- rollouts for closed-loop SFT to mitigate covariate shift and stabilize the policy. We then employ the proposed MPO technique to further refine the model by explicitly aligning it with evaluation metrics. Finally, conduct additional round of closed-loop SFT to restore fidelity the logged data distribution, thereby enhancing simulation performance.
Empirically, this RI-style paradigm consistently outperforms standalone SFT or RFT (see Table 3 We attribute its effectiveness to striking a balance between optimizing for metric-driven objectives and preserving generalization from real-world data, and thus mitigating forgetting and improving the realism of simulated traffic behaviors.
3.5 TRAINING OBJECTIVES
Given our proposed framework, which consists of three phases, the training process is decomposed into three stages with distinct training objectives. the open-loop pretraining stage, we optimize the NTP task by minimizing the cross-entropy loss between the ground-truth token distribution and the prediction. Given the corresponding ground-truth index, denoted as k the training objective for policy rollouts can be formally defined follows:
N
Lbuse
log (To(ki))S2,,)))))
(6)
In the closed-loop SFT stage, we perform CAT-K rollouts (Zhang et al. 2025) to obtain a sequence of motion tokens and identify the motion index that is closest to the ground-truie trajectories. The model is again optimized using cross-entropy loss, with Equation 6 applied by replacing k with k. to directly align the policy
Finally, in the RFT stage, we employ the LMPO, as defined in Section 3.3 with target evaluation metrics.
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
Real-World Dataset. We train our model on the large-scale Waymo Open Motion Dataset (WOMD) (Fitinger et al. 2021) and evaluate on the Waymo Open Sim Agents Challenge (WOSAC) beneh- mark (Montali et al. 2023). WOMD contains approximately 487k training scenarios, 44k validation scenarios, and 45k test scenarios. Each scenario spans 9.1 seconds at 10 Hz. Given 10-step historical tracks, the current state, and the corresponding HD map, the task is targeted to simulate 32 realistic joint 8-second futures (including 80-step centroid coordinates and heading of the objects' boxes) for all the agents (up 128) in the scene.
Evaluation Metrics. We adopt the official WOSAC evaluation metrics, which frame simulation distribution matching problem: assessing how well the density induced by simulated rollouts assigns likelihood to real-world trajectories. The primary evaluation criterion is the Realism Meta metric, weighted combination of three sub-metrics, including Kinematic, Interactive, and Map Adherence, each comprising several finer-grained measures (see Table 2b. In addition, we report the minimum Average Displacement Error (minADE), a standard open-loop prediction metric that serves as complementary reference for motion forecasting accuracy. More details can be found in the official WOSAC documentation (Montali al. 2023) and Appendix A.I
Implementation Details. We choose SMART (Wu et al. 2024) as our bascline NTP model. For both the open-loop NTP and closed-loop SFT training processes, we follow the official implemen- tations of Zhang et al. (2025) to ensure fair comparison. Specifically, we pretrain the foundation autoregressive model SMART-iny (7M parameters) for 64 epochs to obtain the base model. We then perform post-training, including SFT and RFT. Instead of conducting 32-epoch SFT as the original implementation of Zhang et al. (2025), we split the post- post-training into three phases: (1) an initial SFT stage of 16 epochs, (2) RFT with our proposed MPO algorithm. and (3) final SFT stage of 16 epochs. For MPO, we set the empirical reward threshold to a 0.77 and the KL penalty coefficient to 3 0.04. Note that model ensembling or post-processing tricks employed, as

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on the 2025 Waymo Open Sim Agents Challenge (WOSAC)
leaderboard. The Realism Meta metric serves as the official ranking metric, shaded in gray. The best and second- -bst results are highlighted in bold and underline, respectively. "*" indicates results reproduced using the official implementation and re-evaluated under the 2025 version of metries.
Realism Meta metric
Kinematic
Interactive Map-based
min
model
Method
metrics 1
metrics
metries
ADE
params
SMART Wu ct al.
2024
0.78 14
0.4854
0.8089
0.9153
1,3931
7M
SimFormer Wang ct al. 2025
0.7820
0.4920
0.8060
0.9167
1.3221
7M
UniMM (Lnn
2025
0.7829
0,4914
0.8089
0.9161
1.2949
4M
ComBOT Rossert
2025
0.7837
0.4899
0.8102
0,9175
1.3687
N.A.
RLFTSim (Ahmadi cl al. 2025
0.7844
0.4893
0,8128
0,9164
1.3470
7M
CLSFT Zhang ct al. 2025
0.7846
0.4931
0.8106
0.9177
1,3065
7M
TrajTok Zhang al.
2026
0.7852
0.4887
0.8116
0.9207
1.3179
10M
SMART-R1 (Ours)
0.7858
0.4944
0.8110
0.9201
1.2885
7M
Table 2: Performance comparison among the SMART series across all sub-metrics the test split,
Kinematic metrics
Interactive metries
Map-based metries
Model
Lin.
Lin. Ang. Ang.
Disttt Collision T.T.C.
Dist. to
Offroad Traf. Light
Speed f Acc. f Speed Acc.
Obj. f
t
Road Edge
Violation
SMART-base SMART-SFT
0.3719 0.3984 0.5156 0.6557 0.3877 0.9693
0.8290
0.6757
0.9501
0.9809
0.3868
0.4066 0.5203 0.6588 0.3922 0.9702
0.8302
0.6814
0.9524
0.9805
SMART-RI
0.3867 0.4078 0.5215 0.6614
0.3917
0.9709 0.8303
0.6820
0.9554
0,9817
trajectories must be sampled autoregressively for the simulation task. In addition, no external data beyond WOMD is used during training.
4.2 QUANTITATIVE RESULTS
We compare proposed SMART-RI against other state-of-the-art methods the challenging 2025 WOSAC benchmark. As shown in Table [ we report results from several top-ranking pub- lished approaches evaluated under the 2025 version of the metrics. Our SMART-RI achieves first place on the leaderboard, attaining a Realism Meta metric of 0.7858, underscoring its ability to sim- ulate multi-agent future behaviors that closely match real-world scenarios. Moreover, SMART-RI achieves leading performance in both kinematic metrics and minADE, highlighting its capacity to effectively capture agent motion profiles while delivering the best open-loop prediction aceuracy.
To further analyze its effectiveness, we provide a detailed comparison within the SMART series, including the base model SMART-base, the closed-loop fine-tuned SMART-SFT, and our proposed SMART-RI, across all sub-metrics, listed in Table The results show that RFT consistently im- proves performance across most evaluation aspects, with particularly notable gains on safety-critica metrics such collisions, off-road rate, and traffic light violations, whose objectives difficult to optimize with supervised learning, as these outcomes are often non-differentiable scalars. These findings demonstrate that incorporating RFT not only enhances behavioral interaction and safety but also improves overall simulation realism, all without introducing additional model parameters.
4.3 ABLATION STUDIES
We conduct thorough ablation studies to validate the effectiveness of key components SMART- R1. Due to the high computational cost of evaluating the full validation set, we follow Zhang ct al. (2025) that performs efficient ablations on a 2% validation split, which has been verified to yield consistent conclusions with the full set.
Effect of the R1-Style Training Paradigm. We first investigate the impact of the proposed post- training paradigm. As shown in T the post-fine-tue wene--ttiinggge strategy consistently improves the sim- ulation quality over the BC-pretrained model. Specificallyy, inserting an RFT phase after the initial SFT yields clear gains in the Realism Meta metric, as RFT directly optimizes toward the evaluation

---

Published as a conference paper at ICLR 2026
Table 3: Effect of post-fine-tuning strategies.
Table Effect of different RFT policies.
1 SET RFT 2nd SFT
Realism Meta metric
min
Method
Realism Meta metric
min
ADE
ADE +
0.7725
1.4335
SMART-SFT
0.7734
1.3982
0.7734
1.3982
w/ PPO
0.7611
1.5126
0,7730
1,4188
w/ DPO
0,7682
1.5465
0.7740
1,4216
w/ GRPO
0.7701
1,4553
0.7746
1,4194
w/ MPO
0.7740
1.4216
Table 5: Effect of empirical threshold.
Table 6: Effect of KL regularization.
Threshold a
Realism Meta metric
min
KL coefficient
Realism Meta metric
min
ADE
/
ADE +
0.76
0.7735
1.4348
0.004 0.04
0.7732
1.4387
0.77
0.7740
1.4216
0.7740
1,4216
0.78
0.7727
1.4492
0.4
0.7724
1,4215
objective. Notably, applying two consecutive SFT phases without RFT degrades performance, derscoring the importance of the intermediate RFT stage. In contrast, applying an additional SFT phase after RFT further boosts performance, confirming the effectiveness of our RI-style strategy. This finding aligns with Chen et al. (2025), which notes that models trained with SFT followed by RFT often suffer from catastrophic forgetting. In summary, SFT restores the logged data distribu- tion, while RFT shifts the policy toward the metric objective; the "SFT-RFT-SFT" iterative training pipeline balances these two objectives, yielding higher overall realism.
Effect of the Policy Optimization. We further compare several representative policy optimization methods, including PPO, DPO, GRPO, and our proposed MPO. Using SMART with one-stage SFT as the baseline, observe in Table that PPO, DPO, and GRPO all degrade performance. PPO suffers from unstable training due to its actor-critic structure, where the value model is often hard to optimize. DPO and GRPO avoid value approximation but rely sampled rollouts, which can bias policy updates toward suboptimal candidate behaviors. In contrast, our metric-oriented policy directly leverages the relatively predictable reward expectation prior knowledge, effectively guid- ing optimization. A key factor influencing the performance of MPO the choice of the empirical threshold As shown in T both higher and lower thresholds degrade results, since higher val- ues provide too few positive rewards, while lower values make the criterion too lenient. It illustrates that policies are encouraged when exceeding the threshold and penalized otherwise. Therefore, with the baseline reward averaging around 0.77, setting 0.77 yields the best simulation performance.
Effect of the KL Regularization. We finally study the impact of the KL regularization during RFT. Table 6 shows that when the KL penalty coefficient 3 is too small, the policy deviates excessively from the reference model, leading to degraded performance by discarding prior knowledge from BC pretraining and SFT. Conversely, when 3 is too large, the reference model dominates, limiting improvements from the reward signal and overlooking safety-critical aspects such as collision and off-road rates. A balanced choice of 3 preserves the reference distribution while allowing effective metric-driven optimization.
4.4 QUALITATIVE RESULTS
We qualitatively evaluate SMART-RI on the WOSAC validation split, focusing on challenging in- tersection scenario. As shown in Figure we visualize future trajectories across multiple timesteps. Two distinct rollouts (R1 and R2) for the same scene generated by our model are presented, which, while differing from the logged data, remain realistic and highlight the diversity of SMART-RI.
In Rollout RI (second row of Figure the model exhibits conservative behavior. The upper ellipse shows vehicle slowing down to yield pedestrian crossing the road before making right turn, while the lower ellipse highlights both the white car and the blue ego car (performing an unprotected left turn) decelerating negotiate passage through the intersection. By contrast, Rollout R2 (final
8

---

Published as a conference paper at ICLR 2026
Os
2s
4s
6s
8s
GT
B
R1
R2
Figure 4: Simulation results of SMART-RI. The top row shows the ground-trun truth (GT) scenario, while the rows below illustrate representative rollouts generated by model for the same scene. The blue box denotes the ego vehicle, and white boxes denote other traffic participants, including cars and pedestrians. Transparent boxes indicate ground-truth agent positions, whereas solid boxes represent simulated agents. Green lanes indicate traversable paths under a green light, while red lanes denote restricted paths.
row) demonstrates a more aggressive strategy. The upper ellipse shows the car executing fast right turn, while the lower ellipse depicts the straight-moving car quickly passing through the intersection, with the ego car also advancing more rapidly than in the logged data.
These results demonstrate that SMART-RI generate diverse, plausible, and realistic traffic behaviors across different rollouts, effectively cap- turing both conservative and aggressive driving styles. Additional qualitative results are presented in Appendix A.3 with corresponding simulation videos available in the supplementary material.
not only imitate demonstrated behaviors but also
CONCLUSION
In this paper, we introduce SMART-RI, to our knowledge, the first Ri-style reinforcement fine- tuning paradigm for the next-token prediction model in the domain of traffic simulation. SMART-RI incorporates a Metric-oriented Policy Optimization (MPO) algorithm to explicitly guide the model toward desired objectives, and employs an iterative "SFT-RFT-SFT training strategy to effectively leverage SFT in conjunction with RFT. Experimental results demonstrate that approach sub- stantially enhances the overall simulation realism and achieves state-of wate-of-thee-ad --th-artt performance on the 2025 Waymo Open Sim Agents Challenge with Realism Meta metric of 0.7858 and minADE of 1.2885, highlighting the promise of RFT for advancing multi- agent traffic simulation toward more realistic and safety-critical applications.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the Hong Kong Ph.D. Fellowship Scheme and in part by the HKUST-DJI Joint Innovation Laboratory. The authors thank the Program Chairs for their support and assistance, as well as Lijun Yu, Shizhe Diao, Jun Cen, Xiaogang Jia, Jieqi Shi, Tin-Hing Wong, Xuesong Chen, Chen Shi, Muxin Liu, Ruixiang Li, Can Wang, and Jianfei Yang.
REFERENCES
Ehsan Ahmadi et al. Riftsim: Multi-agent traffic simulation via reinforcement learning fine-tuning. In CVPR Workshop on Autonomous Driving (WAD), 2025.
Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Beyond two-stage training: Cooperative sft and for IIm reasoning. arXiv preprint arXiv:2509.06944, 2025.
Alexander G Cunningham, Enric Galceran, Ryan M Eustice, and Edwin Olson. Mpdm: Multipolicy decision-maken in dynamic, uncertain environments for autonomous driving. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp- 1670-1677. IERE, 2015.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9710-9719, 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-rl incentivizes reasoning in llms through reinforcement learning. Nature, pp. 633-638, 2025.
Yihan Hu, Siqi Chai, Zhening Yang. Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solving motion planning tasks with scalable generative model. In European Conference on Computer Vision, pp. 386-404. Springer, 2024.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint urXiv:2003.06749, 2025.
Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile behavior diffusion for generalized traffic agent simulation. arXiv preprint 0tl.:24044.05224. 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint 4rX1122412.16720, 2024.
Max Jiang, Yijing Bai, Andre Comman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efficient and con- trollable driving simulation initialization and rollout. Advances in Neural Information Processing Systems, 37:55729-55760, 2024.
Yue Li, Meng Tian, Dechang Zhu, Jiangtong Zhu, Zhenyu Lin, Zhiwei Xiong, and Xinhai Zhao. Drive-rl: Bridging reasoning and planning in vlms for autonomous driving with reinforcement learning. arXiv preprint arXiv:2506. 18234, 2025.
Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang. Rong Xiong, and Yue Wang. Revisit mixture models for multi-agent simulation: Experimental study within a unified frame- work. arXiv preprint arXiv:2501. 17015, 2025.
Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents chal- lenge. Advances in Neural Information Processing Systems, 55-5911, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
10

---

Published as a conference paper at ICLR 2026
Muleilan Pei, Shaoshuai Shi, Xuesong Chen, Xu Liu, and Shaojie Shen. Foresight in motion: Re- inforcing trajectory prediction with reward heuristics. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 28303-28312 2025a.
Muleilan Pei, Shaoshuai Shi, Lu Zhang, Peiliang Li, and Shaojie Shen. Goirl: Graph-oriented inverse reinforcement learning for multimodal trajectory prediction. In International Conference on Machine Learning, 2025b.
Zhenghao Peng, Wenjie Luo, Yiren Lu, Tianyi Shen, Cole Gulino, Ari Seff, and Justin Fu, Improving agent behaviors with fine-tuning for autonomous driving. In European Conference on Computer Vision, pp. 165-181. Springer, 2024.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token predic- tion. In The Twelfth International Conference on Learning Representations, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53777 2023.
Christian Rossert, Johannes Drever, and Lukas Brostek. combot: an ensemble combination model combining results from smart-tiny-clsft with a cognitive behavior mode. In CVPR Workshop on Autonomous Driving (WAD), 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1.XX::1777.06377, 2017.
Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Re- faat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8579-8590, 2023.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330. 2024.
Chen Shi, Shaoshuai Shi, Kehua Sheng, Bo Zhang, and Li Jiang. Drivex: Omni scene modeling for learning generalizable world knowledge in autonomous driving. Proceedings of the IEEF/CVF International Conference on Computer Vision 2025.
Chen Shi, Shaoshuai Shi, Xiaoyang Lyu, Chunyang Liu, Kehua Sheng, Bo Zhang, and Li Jiang. Unisplat: Unified spatio-temporal fusion via 3d latent scaffolds for dynamic driving scene recon- struction. The Fourleenth International Conference on Learning Reprexentations, 2026.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mir+:: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 46(5):3955-3971, 2024.
Haoran Song, Di Luan, Wenchao Ding, Michael Y Wang, and Qifeng Chen. Learning to predict vehicle trajectories with model-based planning. In Conference on Robot Learning, pp. 1035- 1045. PMLR, 2022.
Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Xiansheng Chen, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning of vision language models. In Advances in neural information processing systems, 2025.
Xiaolong Tang, Meina Kan, Shiguang Shan, and Xilin Chen. Plan-r1: Safe and feasible trajectory planning as language modeling. arXiv preprint arXiv:2505. 2025.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical review E, 62(2):1805, 2000.
Sen Wang. Jianrong Xu, Xiaoyong Zhang, Fangqiao Hu, Zhijun Huang. JiaChen Luo, Kechen Zhu, Jiaxiang Zhu, Yong Zhou, and Zhenwu Chen. Simformer: Improving tokenization of agents and maps with transformers for multi-agent simulation (Ist place in the waymo open scenario generation challenge 2025). CVPR Workshop on Autonomous Driving (WAD), 2025.
11

---

Published as a conference paper at ICLR 2026
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems, 37:
(14048-114071, 2024.
Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4422-5432. 2025.
Zhiyuan Zhang, Xiaosong Jia, Guanyu Chen, Qifeng Li, Zuxuan Wu, and Junchi Yan. Trajtok: What makes for good trajectory tokenizer in behavior generation. In The Fourteenth International Conference on Learning Representations, 2026.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807,, 2025.
Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Qenext:
A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint MrXi::2306110503 2023.
Zikang Zhou, Haibo Hu, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li. Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. Advances in Neural Information Processing Systems, 37:79597-79617, 2024.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 2025 VERSION OF WOSAC METRICS
In this section, provide detailed overview of the evaluation metrics used in the 2025 Waymo Open Sim Agents Challenge (WOSAC) (Montali et al., 2023). The WOSAC metrics are updated annually, with adjustments made the computation methods, the included sub-metries, and their respective weights. Herein, we summarize the 2025 version of the metrics. Compared to the previ- ous year, the most significant changes are: (i) the introduction of new traffic light violation metric, and (ii) new filtering mechanism for vehicles in the time-to-collision metric calculation. Below, we detail the definitions and the corresponding weights w of the sub-metrics within the three main categories: kinematic, interactive, and map-based metrics.
(1) Kinematic metrics evaluate agent motion features and include four sub-metrics:
Linear speed (w 0.05): magnitude of the displacement in (2, y, )) each step.
Linear acceleration (wc 0.05): signed change in speed magnitude between consecutive steps.
Angular speed (w 0.05): signed change in yaw between consecutive steps.
Angular acceleration (w 0.05): signed change in angular speed between consecutive steps.
(2) Interactive metrics assess how agents interact with each other and include three sub-metrics:
Collision indication (w 0.25): boolean indicating whether an object collides with another agent.
Distance to nearest object (w = 0.10): minimum distance from the agent any other agent. Negative values indicate overlap (collision).
Time to collision (w 0.10): predicted time (in seconds) before the agent collides with another agent, assuming constant velocity. This metric is filtered for vehicles only.
(3) Map-based metrics capture how agents behave relative to the environment and include three sub-metrics in the 2025 version:
Off-road indication (w 0.25): boolean indicating whether the agent is outside the drivable region of the map.
Distance to road edge (w 0.05): minimum distance to the boundary of the drivable region. Negative values indicate the agent is inside the road, positive values indicate otherwise.
Traffic light violation (w 0.05): boolean indicating whether vehicle runs red light. This metric is filtered for vehicles only.
Finally, the Realism Meta metric is computed as weighted combination of the above sub-metries, providing comprehensive measure of overall simulation realism and performance.
A.2 ADDITIONAL QUANTITATIVE RESULTS
In this section, we present detailed quantitative results from the top-ranking published approaches listed in Table D evaluated across sub-metrics of the WOSAC leaderboard to enable a com- prehensive comparison. All metrics follow higher-is-bettee better criterion. As shown in Table 7 our proposed SMART-RI not only achieves the best performance the overall Realism Meta metric but also ranks first or second across most individual sub-metrics.
Notably, TrajTok (Zhang et al. 2026) performs relatively well on map-based metries. This can be attributed to its trajectory tokenizer, which integrates rule-based priors. However, such design choices make its generated trajectories less human-like which explains its weaker performance on kinematic metrics. Meanwhile, comBot (Rossert al.) 2025), ensemble-based method, faces scalability concerns, particularly for large-scale real-time simulation.
In summary, SMART-RI delivers consistently competitive results across nearly all sub-metrics and achieves the top score the overall Realism Meta evaluation, underscoring its effectiveness in producing realistic and high-fidelity traffic simulations.
13

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison across all sub-metries on the 2025 WOSAC leaderboard. The Realism Meta metric serves as the official ranking metric. The best and second-best results are highlighted in bold and underline, respectively. "** indicates results reproduced using the official implementation and re-evaluated under the 2025 version of metrics.
Realism Meta
Kinematic metrics
Interactive metrics
Map-based metries
Model
Lin.
Lin. Ang. Ang. Distto Collision T.T.C.
Dist. to
Offroad Traf. Light
metric
Speed
f Acc.
Speed f Acc. Obj. t
Road Edge
f
Violation
SMART*
0.7814
0.3719 0.3984 0.5156 0.3877 0.9693
0.8290
0.6757
0.9501
0,9809
SimFormer
0.7820
0.3857 0.4066 0.5191 0.6565 0.3922
0.9619 0.8301
0.6814
0.9514
0,9785
UniMM
0.7829
0,3836 0,4160 0.5168 0.399
0.9680 0.8293
0.6791
0.9505
0.9811
COmBOT
0.7837
0,3802 0.4033 0.5181 0.390
0.9703 0.8297
0.6790
0,9525
0,9630
RLFTSim
0.7844
0.3793
0.4026 0,5184 0.3898
0.9755
0.8293
0.6779
0,9512
0,9809
CLSFT
0.7846
0.3868
0.4066 0.5203 0.6588 0.3922
0.9702 0.8302
0.6814
0.9524
0,9805
TrajTok
0,7852
0,3823
0.3988 0.5194 0.3921
0.9742
0.8246
0.6845
0.9555
0.9829
SMART-RI
0.7858
0.3867 0.4078 0,5215 0.3911
0.9709
0.8303
0.6820
0.9554
0.9817
A.3
ADDITIONAL QUALITATIVE RESULTS
We present additional qualitative results on the WOSAC validation split. Figure iillustrates chal- lenging U-turn scenario, where the simulated ego vehicle (blue box the ellipse) successfully com- pletes the U-turn in exact agreement with the logged trajectory. Figure 2 highlights an intersection with multi-agent negotiation, focusing on two white vehicles (ellipse) engaged in a strong interac- tion. We provide three rollouts demonstrating distinct outcomes: (R1) conservative waiting with mutual yielding at the intersection, (R2) the left- efft-turning vehicle yiclding to the through-goin -ooigg vehi- cle, and (R3) the through-going vehicle yielding the left-turning vehicle. Figure presents another intersection. featuring negotiation between the ego vehicle (blue box) and a traffic participant (white box). SMART-R1 generates three realistic behaviors: (R1) replication of the logged trajectory, (R2) a conservative maneuver, and (R3) an aggressive maneuver. The corresponding simulation videos provided in the supplementary material.
These visualizations further highlight the ability of SMART-RI to simulate diverse, plausible, and realistic multi-agent traffic behaviors, underscoring its strong capability in capturing complex inter- actions in real-world scenarios.
A.4 DECLARATION OF LLM USAGE
Large Language Models (LLMs) were employed solely to assist with language refinement and manuscript polishing, including grammar correction and improvements in clarity and readability. All scientific concepts, contributions, methodological developments, experimental designs, data analy- ses, and conclusions are the original work of the authors. The authors have carefully reviewed and validated all content and take full responsibility for the accuracy and integrity this manuscript.
14

---

Published as a conference paper at ICLR 2026
50
25
45
6$
85
GT
Figure 5: Simulation results of SMART-RI in U-turn scenario. The top shows the ground- truth (GT) scenario. The blue box denotes the ego vehicle, and white boxes denote other traffic participants. including cars and pedestrians. Transparent boxes indicate ground-truth agent posi- tions, whereas solid boxes represent simulated agents. Green lanes indicate traversable paths under green light, while red lanes denote restricted paths. The visualization depicts the simulated ego vehicle completing the U-turn in exact agreement with the logged trajectory.
15

---

Published as a conference paper at ICLR 2026
0s
2s
4s
66
-8s
GT
R1
R2
R3
Figure 6: Simulation results of SMART-RI at an intersection with multi-agent negotiation, focusing on two vehicles engaged in a strong interaction. The top row shows the ground-truth (GT) scenario, while the three rows below illustrate three representative rollouts generated by our model for the same scene. The blue box denotes the ego vehicle, and white boxes denote other traffic participants, including cars and pedestrians. Transparent boxes indicate ground-truth agent positions, whereas solid boxes represent simulated agents. The visualizations depict three distinct behaviors: (R1) conservative waiting with mutual yielding the intersection, (R2) the left-turning vehicle yielding to the through-going vehicle, and (R3) the through-going vehicle yielding to the left-turning vehicle.
16

---

Published as a conference paper at ICLR 2026
Os
2s M
4s
6s
8s
M
M
2
1::00
1:300
GT
MM
5
5
R1
150
1:300 JCM
1C
R2
SW
90
5
C3
R3
Figure 7: Simulation results SMART-RI at an intersection, featuring negotiation between the ego vehicle and traffic participant. The top row shows the ground-truth (GT) scenario. while the three rows below illustrate three representative rollouts generated by our model for same scene. The blue box denotes the ego vehicle, and white boxes denote other traffic participants, including cars and pedestrians. Transparent boxes indicate ground-truth agent positions, whereas solid boxes represent simulated agents. The visualizations depict three distinct behaviors: (R1) replication of the logged trajectory, (R2) a conservative maneuver, and (R3) an aggressive maneuver.
17


Reviews:
[{"review_id": "2RWjpHbpSn", "reviewer": "Reviewer_YBue", "summary": "The paper introduces a trajectory forecasting system that is based on SMART and adds DeepSeek-R1-style training, which ultimately leads to SotA performance on a difficult dataset.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **S.1:** The idea is simple and well-justified. It's not methodologically revolutionary but the results are very impressive.\n- **S.2:** The writing is clear, the illustrations are simple but good.\n- **S.3:** I really like your ablations, especially table 3-6. I think this is very good scientific practice.", "weaknesses": "- **W.1:** Reproducibility. I really dislike when papers don't submit any code with their submission for a system that's quite complex. Especially since your method is built on top of an existing codebase, I think it would've been great if you could've included an anonymous github link or a zip file with the submission.\n- **W.2:** Performance/Train time? Along the same lines of the previous comment: how long was this trained, on which hardware? How fast is inference?\n- **W.3:** The writing is moooostly clear but there are some sharp edges that I think need to be improved. In the abstract, you already mentioned \"R1-style...\" without mentioning what this is referring to (e.g.\"the SFT-RL-SFT training regimen of DeepSeek-R1\") and you often refer to CAT-K rollouts, which I'm not familiar with and since it seems important to your method, it'd be nice to have a quick explanation in the main body of your paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5099haZglH", "reviewer": "Reviewer_uZAq", "summary": "- This paper tries to address the problem of distribution shift in the learned (supervised learning) simulators affecting their generalization capabilities.\n- The authors propose SMART-R1 to align agent behavior with human preferences and relevant evaluation metrics for agent traffic simulation tasks.\n- The authors introduce an iterative (interleaving) SFT-RFT-SFT post-training pipeline to maximize the performance and minimize the skills learned in previous training runs (catastrophic forgetting).\n  - They introduce Metric-oriented Policy Optimization (MPO) to exploit the knowledge from priors and can try to optimize for collision, off-route cases, etc. that might not be there in the SFT data. \n- With the above recipe the proposed method was able achieve best performance on Waymo Open Sim Agents Challenge (WOSAC)\n\nGiven the comments related to weaknesses and limitations, I can have the rating as 2. Flexible to move this during or after rebuttal.", "questions": "1. Was keeping a KL term in the SFT tried?\n\n2. Was there any KL term in the RL loss to minimize the divergence from the base policy while maximizing returns?\n\n3. Was any other regularizer tried to prevent forgetting (for example, Elastic Weight Consolidation)?\n\n4. Was there any KL term in the second SFT training (refer to [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://arxiv.org/abs/2509.04259))?\n\n5. How many agents are considered in the scenario and how are selected or filtered, if any?\n\n6. Line 397: “Notably, applying two consecutive SFT phases” - isn't this just one big SFT stage with 2x epochs?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors presented the text, figures and tables nicely.\n- Converting the trajectories into motion tokens makes it easier for the sequence to be fed to the transformer and formulate it as a next-token-prediction (NTP) task.\n- The closed-loop SFT seems to reduce the covariate shift in NTP models leading to improved performance.\n- RFT stage aligns the model with metric preferences and enhances realism. \n- The paper does a good job in performing experiments and ablations to study the effect KL regularization, effect of different post-fine-tuning methods, effects of using different RL algorithms for RFT.", "weaknesses": "The approach proposed has off-late been a pretty common fine-tuning strategy for many robotics tasks. It is appreciated that the authors have done research in incorporating the multi-step post-training pipeline serving a dual purpose of maximizing performance and minimizing forgetting. However, there has been some work done on dealing with forgetting as well in LLMs ([Towards a Unified View of Large Language Model Post-Training](https://arxiv.org/pdf/2509.04419)) .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1CCQLyk0i", "reviewer": "Reviewer_tSgk", "summary": "This paper proposes to use reinforcement learning to post-train the Next-token-prediction-based motion planning model. It uses a three-stage post-training method, SFT-RLFT-SFT, and finally achieves SOTA on Waymo Open Sim Agent Challenge (WOSAC).", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Agent simulation is an important problem for self-driving, especially building agents with realistic behaviors.\n2. The paper proposes to use RL post-training to explore better behaviors to increase the realism and achieve SOTA on the commonly used benchmark WOSAC. Comprehensive experiment results, including ablations, demonstrate the effectiveness of the proposed method.\n3. The paper writing is good and easy to understand.", "weaknesses": "1. The method is mainly designed for an NTP-based model, while new adaptations are needed for continuous prediction models like diffusion models, which may limit its future application.\n2. It would be better to evaluate the method on more end-to-end planning benchmarks like nuPlan to show its general capabilities, which can further increase its impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SjjZT4Q5B4", "reviewer": "Reviewer_PHPM", "summary": "The paper apply the R1 training style to the traffic model learning task to solve the distribution shift problem. It directly uses the waymo evaluation metrics to assign the reward. The method achives good result in the Waymo Sim Agent benchmark.", "questions": "1. How do you assign the rewad among the 32 rollout? Do all 32 rollouts use the same reward?\n2. What is the performance of using the original KL penalty?\n3. What is the performance of using more iterations of SFT and RFT or making them optimize together?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The performance of the method is good on the Waymo Sim Agent benchmark.\n2. The paper is good written and easy to follow.\n3. The ablation experiments are extensive.", "weaknesses": "1. The contribution of the paper is limited. It looks like an engineering work to apply the existing R1 training style to finetune an existing model based on CATK. \n2. The metric-oriented policy optimization method directly uses the evaluation metric as the reward, which makes its strong performance less meaningful. It may simply overfit to the metric itself rather than improving genuine behavioral realism across other metrics.\n3. Taking the waymo sim agent score as reward is time consuming because the computation is slow which requires 32 rollouts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)