Adv-BMT: Bidirectional Motion Transformer for Safety-Critical Traffic Scenario Generation
Authors: Yuxin Liu
Zhenghao Peng
Xuanhao Cui
Bolei Zhou
Venue: NeurIPS 2025 poster ()
Keywords: autonomous driving
reinforcement learning
robotics
Primary area: deep_learning
TL;DR: We propose the Adv-BMT framework to augment real-world scenarios with diverse and realistic adversarial interactions.

Abstract: Scenario-based testing is essential for validating the performance of autonomous driving (AD) systems. However, such testing is limited by the scarcity of long-tailed, safety-critical scenarios in existing datasets collected in the real world. To tackle the data issue, we propose the Adv-BMT framework, which augments real-world scenarios with diverse and realistic adversarial interactions. The core component of Adv-BMT is a bidirectional motion transformer (BMT) model to perform inverse traffic motion predictions, which takes the last frame of the scenario as input and reconstruct the traffic in the inverse of chronological order, till the initial time step. The Adv-BMT framework is a two-stage pipeline: it first conducts adversarial initializations and then inverse motion predictions. Different from previous work, we do not need any collision data for pretraining and are still able to generate realistic and diverse collision interactions. Our experimental results validate the quality of generated collision scenarios by Adv-BMT: training in our augmented dataset would reduce episode collision rates by 20\% compared to previous work. The code will be made available.

Adv-BMT: Bidirectional Motion Transformer for Safety-Critical Traffic Scenario Generation
Yuxin Liu" Zhenghao Peng" Xuanhao Cui
Bolei Zhou
University of California, Los Angeles
Abstract
Scenario-basee testing is essential for validating the performance of autonomous driving (AD) systems. However, such testing is limited by the scarcity of long- tailed, safety-critical scenarios in existing datasets collected in the real world. To tackle the data issue, we propose the Adv-BMT framework, which augments real- world scenarios with diverse and realistic adversarial traffic interactions. The core component of Adv-BMT is bidirectional motion transformer (BMT) model perform inverse traffic motion predictions, which takes agent information in the last time step of the scenario as input, and reconstruct the traffic in the inverse chronological order until the initial time step. The Adv-BMT framework is two- staged pipeline: it first conducts adversarial initializations and then inverse motion predictions. Different from previous work, do not need any collision data for pretraining, and are able to generate realistic and diverse collision interactions. Our experimental results validate the quality of generated collision scenarios by Ady-BMT.: training in our augmented dataset would reduce episode collision rates by 20%. Demo and code are available http:://metadrriverss github.ion io/ adv bmt/
Introduction
In recent years, autonomous driving (AD) agents have achieved unprecedented performance in simulations 3 9 23 However, handling corner traffic situations, especially collision scenarios, remains major challenge. A major cause that safety-critical scenarios are missing from real-world driving datasets due to high costs and risks of data collections. Without enough collision training data, is hard for the autonomous driving (AD) planners and prediction models learn safe driving in challenging and risky scenarios. This motivates the need for simulating different real-world accidents. To generate realistic collision trajectories, previous works [15 27 18 leveraged learned real-world traffic priors, and optimized predictions on Dollision-encouraging objectives. However, our evaluations these baselines reveal that the generated behaviors are insufficiently diverse and yield low collision generation rate.
We tackle this challenge by rethinking the motion prediction architecture itself. We introduce the Bidirectional Motion Transformer (BMT), a new model that learns to predict both future and history trajectories for all agents, conditioned their current states. Similar to recent autoregressive traffic forecasting models [24] 16 300, BMT tokenizes continuous trajectories into discrete control actions. Distinct from prior work, BMT employs a temporally reversible tokenization scheme that enables unified forward (future) and reverse (history) motion prediction within the same framework.
We utilize BMT model into Adv-BMT framework for realistic and diverse collision traffic gener- ations from real-world driving data. While existing works follow a standard paradigm, which selects convenient neighbor agent and modifies the behavior attack the ego agent, Adv-BMT inserts new
Equaal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Adv-BMT
Realswordd Driving Log
Reverse Prediction
Collsion Scenario
Adversaria Inibalication
Reject Sampling
Adaissy Ape
Sudtin speed changer Dact
Higett
Figure 1: Overview of Adv-BMT. The framework mainly consists of three steps: (1) it adds an adversarial agent (ADV) with a sampled collision state into the current scenario; (2) it predicts reversely for the adversarial trajectory; (3) it performs rule-based checks and rejects physically implausible ones.
agents (ADV) with diverse collision interactions, maintaining realistic interaction with other traffic agents. In short, Adv-BMT is a three-staged pipeline: first, it initializes diverse collision frames be- tween adversary agent (ADV) and ego vehicle; then, it reconstructs the adversarial trajectories via BMT's reverse predictions; finally, it conducts rule-based checks and rejects trajectories with physically implausible collision initializations. An overview of framework is illustrated in Fig. It is worth noting that that there is no collision data included in our model training.
Adv-BMT is designed with multiple generation modes to support varying levels of agent interactions and realism. By default, traffic agents follow their recorded trajectories to preserve consistency with the real-world driving log. while the adversarial agent is generated to interact within this fixed context. This enables targeted scenario editing while maintaining overall scene plausibility. To address limitations in interactivity, Adv-BMT also supports closed-loop reverse prediction mode, in which all agents are jointly predicted to generate fully reactive traffic scenario, without teacher-forcin any agent behaviors during predictions. Additionally, optional forward refinement step allows traffic agents to respond to the newly introduced adversary, enabling a more interactive and dynamically consistent outcome. Together, these modes allow Adv-BMT to balance realism, controllability, and diversities for different use cases.
We summarize contributions follows: (1) We introduce the bidirectional motion transformer with temporally reversible motion tokenizations: (2) We develop Ady-BMT for realistic and diverse safety-critical traffic simulations; (3) We leverage Adv-BMT in closed-log loop setting to dynamically create challenging environments for reinforcement learning agents.
2
Related Work
Driving Motion Prediction. The task of motion prediction focuses forecasting the future trajec- tories of traffic participants conditioned on their initial map context and agent states. Recent work uses transformer models to autoregressive sequence modeling. A set of work utilizes discretized motion tokenizations to perform next token predictions: Trajeglish [12] discretizes motion repre- sentations using k-disk- based tokenization scheme represent position and angle differences for relative movements. MTR++ [17] directly represents motion on continuous space in Gaussian mixture distributions. MotionL.M [16] models trajectory deltas. SMART builds a k-disk-clustered vocabulary of motion tokens, containing coordinates, heading, and shape. BehaviorGPT 30 per- forms next-patch predictions with future motion chunks, instead of single-step predictions. The BMT model constructs two sets of motion tokens based inter- frer-ffram accelerations, enabling both

---

forward prediction of future motions and inverse reconstruction of past motions. Another set of works
29
28
13
25 22 leverage diffusion models for motion predictions.
Safety-Critien Traflic Scenario Generation. To generate collision traffic, classic line of previous works use two-staged method: first uses traffic prior to generate realistic agent trajectories, then use collision-sensitii objectives for trajectory optimization or candidate selections. STRIVE [T models traffic by variational autoencoders but requires computationally expensive per-scenario optimizations. CAT [27] leverages transformer motion decoder [5] and selects the collision trajectories and simulates MetaDrive [9] for -lop adversarial environments. SEAL [88 uses skill- based adversarial policy with collision-related objectives. SafeSim []] proposes diffusion model with test-time collision-sensitvvv guidance loss control the collision type and adversarial agent selections. Another set of work such as 14 use a reinforcement learning (RI.) based approach, which parametrizes trajectories and goal constraints to generate safety-critical interactions. AdvSim [21] directly perturbs actor trajectories using kinematic model and optimizes via black-box adversarial loss. Another line recent work such as 2 uses a conditional normalizing flow to model the distribution of real-world cafety-critical trajectories. CrashAgent (7] and LCTGen [19] leverage free-form texts as inputs and extract embeddings to parameterize scene initializations and agent driving directions. Different from previous approaches, Adv-BMT generates collision scenarios in three steps: first samples a collision state, then conduct reverse predictions, finally forcast traffic agents accordingly.
3
Method
Classical motion prediction models forecast future trajectories based on the current states traffic agents. Building this foundation, we propose the reverse motion prediction problem. To address both tasks within a unified framework, we introduce the Bidirectional Motion Transformer (BMT) model, which is able to perform both tasks. Finally, present the Adv-BMT framework and describe how it leverages BMT model as the core to generate realistic and diverse safety-critica interactions.
3.1 Bidirectional Motion Prediction
We first introduce the bidirectional motion pre- diction task, shown in Fig. 2 Consider a traffic scenario with at most N agents and prediction horizon of steps. The trajectory agent represented r' {rg, ri...... 7 where each state -े E Rd encodes its position, velocity, and heading at time t. We introduce a prediction direction indicator D (Forward, Reverse), ), to specify whether the model predicts future or past motion over the horizon.
Forward Prediction
Reverse Prediction
For each predicted traffic agent, we construct sequence of motion tokens Z {zf,..., ) by applying the motion tokenization function <(.) between consecutive states. In the forward setting, the tokens are derived in chronological order, from to Tÿ. In the reverse setting, the temporal order of the trajectory is inverted, and the tokens generated by applying 6(-) back- ward from the current state toward the initial state. This formulation yields bidirectional token sequence that enables BMT model both forward and reverse motion dynamics in a unified token space. The tokenization function Φ(-) is further
Figure 2: Bidirectional predictions the ego agent (red). BMT supports predictions for future motions (left) and historical motions (right) jointly for all prediction agents.
detailed in Section 3.2
3.2 Bidirectional Motion Transformer (BMT)
Token Space. BMT's bidirectional motion tokens are derived from simplified bicycle dynamics model. Both forward and reverse tokens are defined over the same shared token space-a set

---

of discrete bins of acceleration and yaw rate pairs. We uniformly quantize the control space of accelerations [=a amax and yaw rates E mmax maxl into bins each, yielding a total of K2 discrete motion tokens, where 6max 10m/ 8, mmax = 2. and K = 33.
Motion Token Reconstruction. A key step in BMT is to reconstruct continuous trajectories from discrete motion tokens, bridging between physical motion and the token vocabulary. Each token encodes a low-level control command defined by an acceleration and yaw rate Starting from the current agent state r = (z. y. 8, v), the token is applied over a time interval At to update the agent's s position, heading, and velocity. BMT adapts midpoint integration to propagate the state, which provides a stable and realistic approximation of vehicle dynamics. Intuitively, the speed and heading updated using their midpoint values, ensuring that the reconstructed trajectory remains smooth and physically consistent. This reconstruction procedure allows BMT to decode token sequences into continuous motion trajectories in both forward and reverse prediction settings.
Model Architecture. The BMT architecture overview is shown in Fig. BMT has scene
encoding component used to obtain embeddings for scenario contexts with separate embeddings for map polylines, traffic lights, and agent initial states. Then, use Fourier-encoded edge features [20] to represent the spatial and directional information between these encoded entities, which are then passed to the transformer encoder with self-attention layers for the relational embeddings.
The prediction decoder predicts subsequent mo- tion tokens in autoregressive manner, with only initial agent information for the first frame, along with the scene embedding obtained from the Scene Encoder. The motion decoder incor- porates self-attention over the initial agent token embeddings, and three relation computations separately: agent-to-agent (a2a), agent-to- toe (a2t), and agent-to-scene (a2s), with each rela- tion embedding then passed to cross-attention layers. The output agent embeddings are con- catenated and repeated number of times. The output agent motion embeddings are mapped to the vocabulary of discretized motion tokens through MLPs. More details can be found in the Appendix.
Tolema baciourd prodicionn
T-11
43
prediction
Map Encoder
Motion Predictor
<S>
CRDN
Figure 3: BMT architecture. BMT consists of a scene encoder and GPT-style motion decoder. It employs two sets of motion tokens for forward and reverse predictions to generate the next-step token for each agent. All predictions are conditioned only the map information and the one-step cur- rent state of all predicted agents.
Training. BMT is trained to learn policy that reproduces the distribution of real-world driv- ing behaviors. At each step, the model predicts discrete motion token for every agent based on its past tokens, current state, and the scene context. To align these predictions with the ground-truth behaviors in the dataset, we minimize cross-entropy loss between the predicted and observed token distributions:
Luran
Ep
Σ
M log no (o) xi
where Ty is the token prediction policy parameterized by Intuitively, this objective encourages the model to assign high probability to motion tokens that correspond to real trajectories, thereby capturing the joint distribution of multi-agent actions in traffic.
During inference, BMT generates motion tokens autoregressively, sampling one token at time conditioned on its previous predictions. We apply nucleus (top-p) sampling to promote behavioral diversity while remaining faithful to the learned distribution. To mitigate exposure bias, the model rolls out on its own sampled tokens rather than the ground-truth sequence, ensuring consistency between training and inference.

---

Waymo
Adv-BMT
Ego Agent
Adheesary Agen:
Traflic Agent
Figure Diverse collision headings generated by Adv-BMT from the same driving log input, visualized from a third-person perspective.
3.3 Ady-BMT for Safety-critical Interaction
The BMT model serves as the core that captures the distribution of realistic multi-agent motions in Ady-BMT framework. Building upon this foundation, we develop Adv-BMT,, an adversarial scenario generation framework designed to generate realistic and safety-critical traffic interactions. The overview Adv-BMT framework is illustrated in Fig From the input scenario, Adv-BMT first samples diverse collision states between ego agent and new adversarial agent (ADV), then reconstructs complete trajectories via reverse motion predictions. A rule-based reject sampling mechanism is used for selecting collision initializations.
Diverse Adversarial Initialization. While ex- isting works select convenient neighbor agent and utilize its initial information, Ady-BMT inserts new agents (ADV) with diverse colli- sion initializations for different opponent inter- actions. We visualize an example result in 5 Formally, we define the collision state as to in- clude the position, time, velocity, and heading at the collision step. The collision time can be varied, sampled from the first second to the last time step of the ego trajectory length. Similarly, ADV's collision headings are randomly sampled. ADV's collision position can be calculated from
T
50
T 50
collision heading and the ego vehicle's collision position. Last but not least, collision velocity calculated from sampling a offset from ego vehicle's s speed at collision step.
Ego Agemt
Adversary Agent
Trafte Agent
Figure S: Diverse collision timings generated by Ady-BMTT from the same driving log input, visual- ized in bird's s-yye view.
Multi-agent Adversarial Interaction. key advantage of Adv-BMT lies in its ability to gen-
5

---

erate realistic and flexible multi- aggett interactions surrounding the adversarial agent (ADV), Starting from sampled collision initializations, the ADV is added into the scene together with existing traffic participants. BMT then performs reverse-time prediction to reconstruct plausible interaction histories that lead to the designated collision state. To balance realism, controllability, and scene reactivity, Adv-BMT provides several generation modes with different levels of multi-agent coupling:
1. Ady-BMT with replayed traffic agents: In this mode, traffic agents are teacher-forced follow their original recorded trajectories, while only the adversarial agent's history is predicted in reverse time. This preserves the original traffic flow and ensures high scene fidelity. feacher-foreing prevents unintended deviations (e.g., traffic agents switching lanes or diverging from the original trajectory) that may occur if the model predicts all agents in the closed-loop manner. As a result, BMT reconstructs the adversarial agent's 's trajectory to fit into the original scenario.
2. Ady-BMT with closed-loop reverse prediction: This mode removes teacher-forcing and jointly predicts the trajectories of all agents, including both traffic participants and the adversarial agent, in a single reverse prediction pass. By allowing all agents to evolve backward in time simultancously, the scene develops coherently as fully interactive scenario. This setting enables rich multi-agent dynamics and can produce fully synthesized safety-critical scenarios.
3. Ady-BMT with forward refinement: This hybrid mode first applies teacher-forced reverse prediction to generate the adversarial agent, then runs an additional forward prediction pass enable traffic agents to react to its presence. This introduces interaction and reactivity without discarding the structure of the original recorded scene, offering a middle ground between strict replay and fully closed-loop generation.
These complementary modes provide a controllable trade-off' between scene realism, interactivity, and controllability, enabling users to select the most suitable configuration for different evaluation objectives.
Rule-based Rejection Sampling. We design Adv-BMT to have diverse initializations, which do not guarantee the realism of the collision outcomes. To address this issue, we implement a rule-based filtering rejection mechanism for ADV candidates. Specifically, we first measure the driving distance and average speed of ADV candidates; if it moves too short and mostly wanders at the designated position waiting for the ego vehicle, then it is considered invalid. Mcanwhile, we check the max curvature (the rate of change for heading): given candidate ADV trajectory, we compute the curvature constraint using: R2 = 20,/ Ass, where Ad, is the absolute heading change between time steps, and As; is the displacement between consecutive positions. Ady-BMT rejects a prediction if max, (K+) > Kehreshold- where Ridresbold = 0.8 is a predefined curvature limit that we found useful. We enforce a curvature constraint to ensure that the trajectory remains within the predefined threshold, rejecting ADV candidates that exhibit unrealistically sharp turning behavior. With our straightforward rule-based rejection sampling mechanism, we are able to maintain realistic collision events between the ADV and the ego vehicle.
4
Experiments
We first assess the BMT model on its ability to generate realistic and diverse traffic behaviors in Section 4.1 We then evaluate quality of Ady-BMT scenarios compared to three baseline methods in Section 4.2 Furthermore, in Section 4.3 we evaluate the utility for the downstream learning task. and assess whether training a reinforcement learning (RL) planner in Adv-BMT scenarios lead to improved performance and robustness compared to log-replay traffic flows.
Dataset. All experiments use driving data from the Waymo Open Motion Dataset (WOMD)
with formats managed by ScenarioNet [8] WOMD contains 10Hz scenarios, each with second of history and 8 seconds of future trajectories. Each scenario includes up to 128 traffic agents including vehicles, cyclists, pedestrians along with high-definition maps. To reduce computational cost, we downsample each scenario to 2Hz, yielding 19 prediction steps. We randomly select 500 scenes for both open-loop evaluation and RL training. We use prediction modes for each scene during open-loop evaluations in Section and Section 4.2

---

Table We evaluate BMT under three types of prediction tasks in a closed-loop setting for all traffic agents, without replaying any agents. In the bidirectional mode, forward prediction is performed based on the reverse prediction results. The number of prediction modes set to six.
(a) Realism of BMT predictions.
Method
SFDE.ng
SFDEmin
SADEng
SADEmin
VchCollmn
VchCollag
JSDutlciit
JSDrre
Reverse
3.92
2.70
5.53
5.11
0.03
0.05
0.17
0.02
Forward
3.52
2.35
2.39
1.98
0.03
0.05
0.23
0.30
Reverse Forward
3.29
2.01
2.38
1.74
0.03
0.06
0.22
0.35
Diversity of BMT predictions.
Method
SDD
FDD
ADD
Reverse
8.35
3.13
Forward
10.78
4.40
Reverse Forward
8.49
12.78
7.00
Metrics. To evaluate the realism BMT predictions, we adopt standard open-loop prediction accuracy metrics: Scenario Final Displacement Error (SFDE) and Scenario Average Displacement Error (SADE). For each, we report results on both the best prediction mode and the average over all six prediction modes. We also measure the average number of agent collisions to capture interaction realism. To assess the diversity of generated trajectories, we report Final Displacement Diversity (FDD), Starting Displacement Diversity (SDD), and Average Displacement Diversity (ADD), which quantify the spread of predicted positions across all prediction modes. We also compute the -haannon Divergence (JSD) values over velocity, acceleration, and time-to- collision (TTC) distributions between generated and ground-truth trajectories. JSD measures the similarity between two probability distributions, with lower values indicating closer alignment between generated and real-world behaviors.
4.1 Evaluation of BMT Predictions
Results from Table a indicates that BMT performs realistic scenario generations in forward, back- ward, and bidirectional predictions. Reverse predictions perform slightly worse than forward predic- tions in realism metrics. Compared to single-pass prediction, bidirectional prediction exhibit higher prediction accuracy. Despite these differences. the overall collision rate remains reasonable and comparable for all prediction tasks, which indicates that BMT effectively generate realistic traffic interactions. The results in Table show that bidirectional prediction enhances the diversity of generated agent behaviors across all displacement diversity metrics. Forward prediction exhibits higher FDD and ADD scores than reverse prediction. This indicates that predicting future motions encourages more explorations for variant directions.
4.2 Evaluation of Adv-BMT Scenarios
In Table additionally use the adversarial attack success rate (i.e., the collision rate between adversary and the ego), ADV-Traffic Collision Rate (i.e..collisoon rates between adversary agents and traffic agents), and the average Agent Collision (i.e., average traffic agent collision rate) to indicate interaction realism. Two settings of Ady-BMT outperform baselines in both realism and diversity. This validates Ady-BMT's s design of collision initialization + reverse predictions, which couldn't be achieved by other methods. While Adv-BMT generates highly adversarial behaviors, it also preserves diversity of traffic interactions compared to baselines. FDD and SDD metries suggest that baselines generate nearly the same adversarial trajectories on the given scene. The JSD metrics suggest Adv-BMT outperforms in realism metrics compared to baselines. BMT model able to achieve realistic motion predictions indicted by Waymo Open Sim Agent Challenge metrics [II], which we add Table 6 in our appendix. Furthermore, the results for Adv-BMT with filtering indicate that our rule-based filter does not harm diversity and at the same time enhances the realism the generation speed comparisons among all methods. Our evaluations validate Adv-BMT as efficient framework for realistic, diverse, and safety-critice generation.
metrics. Tablee]]eemonstralll
7

---

Table 2: Diversity and realism of generated adversarial behaviors. We compare Adv-BMT scenarios with three baseline methods for safety-critical interactions, namely STRIVE [IS], CAT 277 and SEAL [8] consistent with baseline methods in this evaluation only, we force Ady-BMT to choose nearest neighboring agent and modify its behavior as the predicted adversary.
Method
FDD
ADD
JSDvel
JSD
JSDrcc
Attack Suce.
Agent Collmn
ADV-Traffic Collmin
CAT SEAL
0.00
0.00
0.13
0.16
0.12
0.48
0.10
0.3
0.00
0.00
0.18
0.25
0.17
0.55
0.12
0.32
STRIVE BMT_TH TF
0.01
0.00
0.11
0.22
0.12
0.38
0.09
0.32
1.22
0.49
0.10
0.17
0.13
1.00
0.13
0.22
BMT_AII
0.63
0.08
0.13
0.19
BMT TF + Filter
2.32
1.00
0.08
0.12
0.09
1.00
0.13
0.12
BMT_AII+ + Filler
1.97
0.83
0.13
0.08
0.12
0.09
Table 3: Generation speeds across methods. Among the four methods, CAT achieves the fastest generation speed, followed by Adv-BMT with a slightly lower speed. SEAL and STRIVE are comparatively slower.
Method
CAT
SEAL
STRIVE
Adv-BMT
500-avg (seconds)
0.80
2.36
1.02
Visualization. We simulate Adv-BMT scenarios in MetaDrive 9. rendering in both bird's e-ye view and third-person perspective. In Fig. 5 Fig. 4 and Fig. we show diverse Ady-BMT adversarial behaviors in the collision directions on the same ego agent in several real-world traffic flows from WOMD. Generated adversarial agents follow traffic rules and maintain realistic driving patterns. Adv-BMT supports diverse adversarial agent types including vehicles, pedestrians, and cyclists, as shown in Fig. 6 Adv-BMT makes full use of each driving log, which makes Adv-BMT suitable for AD testing and adversarial training. visual comparison between Adv-BMT and bascline methods is shown in Fig. 7 When baseline fails to select existing traffic agent that is convenient for safety-critical attack, Ady-BMT is able to imagine a new agent at an appropriate position to perform adversarial attacks the ego vehicle.
4.3 Adversarial Learning
To validate the value of Adv-BMT scenarios downstream autonomous driving (AD) tasks, we train reinforcement learning (RL) agent within augmented scenarios containing collision interactions generated by Adv-BMT and baseline methods. To determine the quality of the augmented training scenarios, we measure both the driving performance and the safety performance of the learned AD agent compared to learning on the original training set. In our experiment, we conduct two sets of training: (1) open-loop RL, where the agent is trained
fixed training with adversarial scenarios generated based on ground-trute -tuthh ego trajectories; and (2) closed-loop RL, where an adaptive adversarial agent attacks the current ego agent based its recent rollout trajectory records. The adaptive adversarial agent's motion is generated by Adv-BMT or a baseline method. Results are shown in Table 4 and Table 5
Setting. The training set contains 500 real-world scenarios randomly selected from the WOMD training We train Twin Delayed DDPG (TD3) agent for million steps using 8 random seeds (hyperparameters listed in the appendix). We measure the average reward, average step cost, average route completion rate (Compl.), and average episode cost (cost sum) for driving performance measurement. To evaluate the impact of adversarial training using Adv-BMT-gee -.M--eenrrttedd scenarios, we assess policy performance across two distinct validation environments: (1) 100 Waymo validation environments, which consist of unmodified real-world driving scenarios from WOMD validation set, and (2) 100 Adv-BMT environments, which the augmented collision scenarios from the 100 validation scenes.
to ensure robustness in MetaDrive 9
Analysis. Table reports open-loop evaluations comparing RL policies trained on Adv- BMT-generated scenarios against policies trained on baseline scenario sets. On the WOMD original validation environments, Adv-BMT-trained agents outperform baselines across metrics: episode cost decreases by 10% and collision rate by 8%, while reward and route-completion remain com-

---

Table Open-loop RL agent evaluation. Each WOMD scenario is augmented with one collision scenario, so that the ratio between real-world and anfety-critical scenes is 1:1. For each method, we augment the original dataset into a new training set.
(a) Evaluation in the Waymo validation environments
Training Scenarios
Reward f
Cost +
Completion
Collision
Cost Sum
Waymo 4
32.03 = 4.27
0.39 n.o
0.72 000
0.14 008
1.41 #035
CAT 227
30.37 1.99
0.39 nos
0.71 1 nos
0.14 non
1.73 1
STRIVE 15
31.30
0.40 noe
0,73
0.13 oza
1.51
SEAL [8
29.94
0.39
0.71
0.12
1.63
Ady-BMT
31.47 121
0.38 nos
0.73
0.11
1.35
Ady-BMT (Refined)
33.22
0.36 088
0.74
0.12 o08
1.39 +022
(b) Evaluation in the Adv-BMT validation environments
Training Scenario
Reward f
Cost
Completion
Collision
Cost Sum
Waymo 4
37.01 + 6.26
0.64 0.e
0.60 a.0
0.30 022
2.96 = 0.63
CAT 27
36.77 =: 4.55
0.62 nos
0.62 nos
0.29 6.0
3.09 0.5
STRIVE 15
37.72 :: s.s
0.63 uce
0.63 : 0.06
0,29 no4
2.92 a.8
SEAL [8
35.74 = 6.36
0.67 0.0
0.60 0.06
0:01
2.97 =03
Adv-BMTT
37.33 1.77
0.62 non
0.63 n0:
0.25 sos
2.41
Adv-BMT (Refined)
39.55 1.9
0.59 0.04
0.65 =:
0.27 no4
2.74
Table 5: Closed-loop RL agent evaluation. We use the same augmented training set as in the open-loop experiment. For the adaptive adversarial learning experiment, implement adaptive generator for Ady-BMT and CAT (27). We discard the other baseline methods due to their low generation speeds.
(a) Waymo Validation Environments
(b) Adv-BMiB BMT Environments
Generator
Reward
Cost
Completion
Collision..
Cost Sum
Generator
Reward
Cos
Completio
Collision
Cost Sum
CAT
$2.15 x.
0.38
0.74
0.10 =xw
202
CAT
30:47
0.62
0.53
0.22
2.51 asa
Ad--MI
33.13
0.39
0.74
6
1.25
Ad-
40.40
0.57
0.63
0.2
2.48
parable to the best baseline. Adding the optional forward refinement (Sec. 3.3 yields additional gains-reward increases by 6%, episode cost decreases by 5%, and route completion improves slightly. On safety-critical test environments, Adv-BMT further reduces episode cost by 17% and collision rate by 14% relative to the best baseline. With forward refinement, improvements in driving capability, reflected in higher reward and completion rates. These results indicate that training on Adv-BM' scenarios exposes agents to a broader distribution of high-risk interactions, enabling more robust, safety- policies. The extra gains from forward refinement underscore the importance of modeling reactive traffic around emerging incidents, which better prepares agents for cascading hazards. Motivated by these findings, we next evaluate all methods in closed-loop RL. setting.
observe further
Table 5) reports closed-loop RI. results under reactive adversarial traffic generators. Relative to the corresponding open-loop policy, the closed-loop policy improves across all metrics in both real-world and safitc-critical collision environments, indicating substantially safer and more robust AD policy, Compared with the strongest closed-loop baseline (CAT), our method achieves higher reward, lower collision rate, and a 38% reduction in episode cost. These gains suggest that training with Adv-BMT scenarios featuring diverse adversarial behaviors and realistic, traffic-consistent reactions better train the agent to anticipate and mitigate high-risk interactions. Learning in closed-loop further improves performance; by training the agent respond online reactive opponents, it becomes better at anticipating dangerous situations, and staying safe in unfamiliar or rapidly changing traffic conditions. The results also highlight the importance of the Ady-BMT design, which enables the generation of realistic and flexible multi-agent interactions with the adversaria agent.

---

Ego Agent
Adversary Agent
Agent
Figure 6: Ady-BMT can also generate scenarios of pedestrian and cyclist adversarial agents.
Input
CAT
STRIVE
SEAL
Adv-BMT
Ego Agent
Baseline Agent
Our Agent
Traffic Agent
Figure 7: Baseline method comparisons. We present several examples where Ady-BMT successfully generates collision interactions scenarios where all baseline methods fail. Unlike the baselines, Adv-BMT does not rely the proximity of neighboring agents to the ego vehicle, enabling more flexible and diverse adversarial attack strategies.
Conclusion
We introduced Ady-BMT, novel framework for generating diverse safety-critical driving scenarios with realistic traffic interactions. Built Bidirectional Motion Transformer with the ability to perform bidirectional motion prediction tasks, Adv-BMT is able to design new adversarial agent for each real-world scenario. A key advantage of Ady-BMT lies in its ability to generate realistic and flexible multi-agent interactions surrounding the adversarial agent. Unlike previous frameworks that select existing neighboring vehicle and modify the corresponding trajectory for adversarial attack, Adv-BMT initializes candidate collisions for new adversarial agent, reconstructs multi-agent trajectories via inverse prediction, and optionally applies forward refinement for reactive, traffic- consistent interactions. Adv-BMT is able to balance realism, controllability, and scene reactivity, Our evaluations validate the quality Adv-BMT scenarios in of interaction realism and diversity; furthermore, we demonstrate that learning within the Adv-BMT traffic flow improves AD agents' safety performance by clear margin.
Limitations. Ady-BMT relies on long token sequences to represent multi-agent traffic scenes, which results in high memory and computational demands. Our empirical study focuses on single downstream resk-reiinfnnemmnnt learning for an autonomous driving agent-So so broader applicability remains to be validated.
10

---

Acknowledgment: This work was supported by the NSF Grants CNS-2235012. IIS- IIS-2339769, and and CCF-2344955. ZP was supported by the Amazon Fellowship via UCLA Science Hub.
References
[1] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Safe- sim: Safety-critical closed-loop traffic simulation with cuffusionn-ontrollaie adversaries, 2024.
[2] Wenhao Ding, Baiming Chen, Bo Li, Kim Eun, and Ding Zhao. Multimodal safety-critical scenarios generation for decision-making algorithms evaluation, 2020.
[3] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. 2017.
[4] Scott Eutiger,, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Saheek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset, 2021.
[5] Junru Gu, Chen Sun, Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets, 2021.
Chiyu "Max" Jiang, Andre Comman, Cheolho Park, Benjamin Sapp. Yin Zhou, and Dragomir Anguelov. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 0644-9653, June 2023.
[7] Miao Li, Wenhao Ding, Haohong Lin, Yiqi Lyu, Yihang Yao, Yuyou Zhang, and Ding Zhao. Crashagent: Crash scenario generation via multi-modal reasoning. 2025.
[8] Quanyi Li, Zhenghao Peng, Lan Feng. Zhizheng Liu. Chenda Duan, Wenjie Mo. and Bolei Zhou. Scenari- onet: Open-source platform for large-scale traffic scenario simulation and modeling, 2023.
Quanyi Li, Zhenghao Peng, Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning, 2022.
[10] Jack Lu, Kelvin Wong, Chris Zhang, Simon Suo, and Raquel Urtasun. Scenecontrol: Diffusion for controllable traffic scene generation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16908-16914 16914, 2024.
[11] Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nick Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich. Zoey Yang. Shimon Whiteson, Brandyn White, Dragomir Anguelov. The waymo open sim agents challenge, 2023.
[12] Jonah Philion, Xue Bin Peng. and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. 2024.
[13] Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Kai Wang, and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 68873 68894.. Curran Associates, Inc., 2023.
[14] Joshua Ransiek, Johannes Plaum, Jacob Langner, and Eric Sax. Goose: Goal-conditioned reinforcement learning for safety-critical scenario generation, 2024.
[15] Davis Rempe, Jonah Philion, Leonidas J. Guibas, Sanja Fidler, and Or Litany. Generating useful accident- prone driving scenarios via leamedd traffic prior, 2022.
[16] Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling, 2023.
[17] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with
symmetric scene modeling and guided intention querying, 2024.
[18] Benjamin Stoler, Ingrid Navarro, Jonathan Francis, and Jean Oh. Seal: Towards safe autonomous driving via skill-enabled adversary learning for closed-loop scenario generation, 2025.
[19] Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuchl. Language conditioned traffic generation, 2023.

---

[20] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains, 2020.
[21] Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles, 2023.
[22] Sheng Wang. Ge Sun, Fulong Ma, Tianshuai Hu, Qiang Qin, Yongkang Song, Lei Zhu, and Junwei Liang. Dragtraffie: Interactive and controllable traffic scene generation for autonomous driving. In 2024 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 14241 142447, 2024.
[23] Wayne Wu, Honglin He, Jack He, Yiran Wang, Chenda Duan, Zhizheng Liu, Quanyi Li, and Bolei Zhou. Metaurban: An embodied simulation platform for urban micromobility. 2024.
[24] Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time motion generation via next-token prediction, 2024.
[25] Chejian Xu, Aleksandr Petiushko, Ding Zhao, and Li. Diffscene: Diffussion-asse safety-critical scenario generation for autonomous vehicles. Proceedings of the AAAI Conference on Artificial Intelligence, 39(8):8797-8888 8805, Apr. 2025.
[26] Chen Yang. Yangfan He, Aaron Xuxiang Tian, Dong Chen, Jianhui Wang. Tianyu Shi, Arsalan Heydarian, and Pei Liu. Wedt: World-centric diffusion transformer for traffic scene generation. In 2025 IEEE International Conference Robotics and Automation (ICRA), pages 6566-6566 2025.
[27] Linrui Zhang, Zhenghao Peng, Quanyi Li, and Bolei Zhou. Cat: Closed- loop adversarial training for safe end-to-end driving. 2023.
[28] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Languago-guider traffic simulation via scene-level diffusion, 2023.
[29] Ziyuan Zhong, Davis Rempe. Danfei Xu. Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation, 2022.
[30] Zikang Zhou, Haibo Hu, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li, Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next patch prediction, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our main claims supported by our experiments and arguments in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitation section the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
We don't have proofs and theories in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss all details and information needed to reproduce the experimental results in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: Yes, we publish our code and the dataset public already.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee 7/nips.com htc////bbb///uddddd//eeeeeeeisss ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The training and testing details are specified in our main paper and also appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss information of computer resources needed to reproduce in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
This paper comply with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are addressed in our supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no high risk for misuse of released data and model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our work is unrelated in using existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced in our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not engage in crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not engage in research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLM usage is not important, original, or non-standard component of our core methodology in this paper.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

We include additional technical details and results supplementing the main paper due to page limits. Appendix gives details on the model architecture and trainings, Appendix B gives details for experiments, and Appendix C and D provide additional results to supplement those in the main paper. In Appendix E, we discuss the broader impacts of our methodology.
A
Model Details
A.1 BMT Architecture
Preprocessing. The model takes real-world scenarios in the ScenarioNet format [ It performs a series of preprocessing steps on both static map features and dynamic agent trajectories. We compute the global bounds to extract a consistent map center and heading. Each map feature is decomposed into a sequence of vectorized segments, where each vector is represented by start and end points in 3D coordinates. These vectors are augmented with relative directional positions, segment headings, and lengths. Semantic attributes are binary indicators that encode feature types. Map feature types (e.g., lanes, crosswalks, lane markings, stop signs, etc.) are encoded as semantic indicators. Features are then centered the map center. Traffic-light states are also encoded and aligned with timesteps. Similarly, agent trajectories are centered for agent- agent-feature encoding, and position, heading, velocity, shape, and type are extracted over time to form temporal sequences. We extract 16-dimensional state vector at each timestep. This preprocessing ensures that all trajectories are represented in consistent spatial frame and expressed in an egocentric coordinate system.
BMT Motion Tokenization. We formulate motion prediction as an autoregressive next-token prediction problem, where each motion token corresponds to a discretized control input over fixed time interval. The tokenization process maps continuous agent yaw rate-into discrete 2D bins. During tokenization, we consider candidate tokens sampled from fixed grid; for each candidate, we simulate the resulting motion over short duration and select the best-matching action by minimizing the contour-aligmmett error between the predicted agent footprint and the ground-truth pose (position and heading). We adopt simplified bicycle model to parameterize agent motion using longitudinal acceleration and yaw rate within predefined bounds: acceleration [-10, 10) m/s² and yaw rate E [-x/2, T/ rad/s. Given predicted motion-token sequences, the trajectory is reconstructed by mapping tokens back to acceleration and yaw-rate changes.
acceleration and
In both forward and reverse directions, the same tokenizer used to decode motion tokens into continuous trajectories. Given initial state T; (It,, Vr. 0r, vx),, forward decoding simulates the next state T--1 by applying a tokenized control action 2, (a,, 5,) E A, where A is the discrete token space; repeating this autoregressively over the predicted tokens reconstructs the full trajectory Tec+T. For reverse decoding, given a known future state T1+1, the model evaluates candidate tokens 2, E A. simulates their inverse dynamics with At + At, and selects the token that best reconstructs the preceding state тg. Operating in both directions is a key distinction of our approach: forward prediction enables open-loop rollouts of future behaviors, whereas reverse prediction traces from desired outcome (e.g., a collision state) back to plausible initiating actions.
BMT Motion Decoder. The decoder follows GPT-like structure composed of stacked cross- attention blocks. Each block integrates three structured attention modules: agent-to-agent (A2A), agent-lo-temporal (A2T), and agent-to-scene (A2S). Relational information across modalities is encoded via relation embeddings: agent-to-agent relation embedding, agent-to-time relation em- bedding, and agent-to-scene relation embedding, each capturing context-specific spatial/temporal cues. For token construction, we use several embeddings, including the agent-type embedding, agent-shape embedding, agent-ID embedding, and motion-token embedding (which embeds discrete control tokens). A continuous-motiin feature embedding encodes acceleration and yaw-rate attributes. Auxiliary embeddings include the special-tokee embedding (to indicate sequence boundaries) and the reverse- preddicionn indicator embedding (to distinguish forward from reverse prediction modes). For each attention edge, a relative-relation embedding is computed using a Fourier encoder [20] and added to the key and/or value vectors. The input agent tokens are progressively updated across layers by aggregating contextual features from other agents, their temporal histories, and relevant map elements.
20

---

A.2 BMT Motion Decoding
Input. Specifically, each imput agent token to the BMT motion decoder is constructed by summing of embeddings of:
a. the motion token from the previous step (representing discretized acceleration and yaw rate),
b. agent shape (length, width, height),
c. agent type (e.g., vehicle, pedestrian),
d. agent identifier (embedded optionally),
special token type (c.g.. <start>, <end>, padding),
as well as continuous motion delta feature embedded via Fourier encoder. These components are projected into the same hidden dimension and summed to form the input motion token embedding.
Prediction Head and Output. After processing through all decoding layers, the final hidden state for each valid token is passed through two-layer ML. head produce logits the motion token space:
MLP(h) W2 <(W h) RKK
where (b-) denotes the GELU activation and K2 is the number of discrete motion tokens (from K K acceleration -aceleration-yaw bin grid). The resulting logits are used to predict the next motion token at each time step. During inference, we generate motion tokens using nucleus (top-p) sampling.
Trajectory Reconstruction Our model makes a prediction in the interval of 0.5 seconds. To simulate the effect of motion token over fixed time step At 0.5 adopt midpoint integration based on a simplified bicycle model. In forward predictions, given current state s = (Iz, yr, 8,, v2), the model computes the next speed and heading U++ U2 + Al and 0.+1 0,
Al.
The average speed and heading are then defined
++tr++1 and 0 [++00+1] In the reverse direction, the process inverted. Given a future state 11+1, the model enumerates all possible token candidates and inverts the dynamics: v, U441 quantities and are computed similarly and used to derive the previous position:
At and 0, 0s+1 At. The average
Ft = Z++1
cos(O) AI, Vt yt+1 - 7 sin( Al.
A.3 BMT Training
Training Loss. The decoder produces a logit tensor à E IR [XTTxXXIA where |A]] is the number of motion tokens (i.e., discretized acceleration-yaat yyw pairs). The supervision target the ground- truth token sequence z" E NBxTxN, derived by tokenizing agent trajectories. A binary mask m (0, 1) BxTxN specifies which tokens are valid and should contribute to the training loss. The training objective is computed over all valid entries using the cross-entropy loss:
Emnin
Σ mb.i.n
CE( k,,,
Ebt,m Mbrc b.t.n
where CE denotes the standard cross-entropy loss between the predicted logits and the ground-truth discrete token.
Reverse Prediction. During reverse prediction, the model measures metries separately for forward and reverse token predictions. Let e BxTXX be a binary indicator for whether each token comes from reverse prediction. Then compute separate metrics:
Σ Emb.a by 1[2..em 25.1
tovere
Mmn.. by,e.
Entropy reverse
Σ m Σ mb,4,n bb, H(Sb.a.)
with analogous expressions for forward prediction (i.c., for 1 bb.t.)).
21

---

Table 6: BMT Model Parameters.
Component
Parameters
Size (MB)
Scene Encoder
902,080
3.44
Map Polyline Encoder
22,656
0.09
Traffic Light Embedding MLP
1,024
0,00
Scene Relation Embedding
117,184
0.45
Scene Transformer Encoder
744,448
2.84
Scene Encoder Output Projection
16,512
0.06
Scene Output Pre-Normalization
256
0.00
Motion Decoder
4,385,025
16.73
Multi-Cross Attention Decoder
2,881,536
10.99
Motion Prediction Head
157,121
0,60
Motion Prediction Pre- Normalization
256
0.00
Agent-to-Agent Relation Embedding
418,432
1.60
Agent-to- Relation Embedding
418,432
1.60
Agent-to-Scene Relation Emhedding
117,184
0.45
Agent Type Embedding
640
0.00
Motion Token Embedding
139,520
0.53
Agent Shape Embedding
17,152
0,07
Agent ID Embedding
16,384
0.06
Continuous Motion Feature Embedding
217,600
0.83
Special Token Embedding
512
0.00
Reverse Prediction Indicator Embedding
256
0.00
Total
5,287,105
20.17
Metrics. To measure the quality and diversity of the model's 's predictions during training, we track the perplexity:
M Σ 1 = a),
Perplexity = exp
Σ aEAA
Pa log(Pa + +c)) where
Pa
M 1=1
and M is the number of valid tokens. We also track the number of distinct tokens used by both predictions and ground truth:
Cluster
Σ1p > 0).
REA
Total Loss. The total loss is the sum of all enabled components:
Liotal Lmain + Amap AngCCe-totll
with default weights Amap Azg 1.
A.4 Training Details
Our model has 5.2 million trainable parameters, with details indicated in Table We trained BMT model on the training set of the Waymo Open Motion Dataset (4). WOMD contains 480K real-world traffic with each scenario of length 9 seconds; traffic are composed by agents of vehicle, pedestrian, and bicycle; Each scenario comes with high-fidelity road map. During training, we use NVIDIA RTX A6000 GPUs for our model training and fine-tunings. We trained BMT two stages, each with hyper-parametert indicated in Table 7 In the first stage, we pre-trained BMT for forward prediction only with million steps. Then, we fine-tuned BMT with reverse motion prediction in fine-tuning with totally 1.5 million steps. We use AdamW optimizer for learning rate scheduling.
22

---

Table BMT Training settings.
Forward Prediction
Reverse Prediction
Hyper-parameter
Value
Hypereparameter
Value
Training steps
10F6
Training steps
15E6
Batch sizes
Batch size
2
Training Time (h)
185
Training Time (h)
310
Sampling Topp
0.95
Sampling Topp
0.95
Sampling temperature
1.0
Sampling temperature
1.0
Learning Rates
3E-4
Learning Rates
3E-4
Ego Agent
Adversary
Traffic Agent
Opposite Direction
Lane-Change
JUR
Turn
Sudden brake
Cross
REAR-END
Figure 8: Diverse adversarial behaviors generated by Adv-BMT.
B Additional Experiment Results
Training Environment. We conduct reinforcement learning experiments using the MetaDrive ScenarioEnv [9], which provides standardized driving environments for training and evaluating au- tonomous agents. Each environment encodes sensor observations including LiDAR-based surround- ings and physical dynamics. Specifically, the observation space consists of three key components: (i) Ego state, which contains the ego vehicle's 's current physical state such speed, heading, and steering; and (ii) surroundings, which encodes nearby traffic objects.
Actions are continuous and correspond to low-level vehicle control commands. The agent outputs 2D normalized vector, which then mapped to steering angle, throttle (acceleration), and brake signal. The environment includes compositional reward structure combining driving progress, collision
23

---

Table 8: RL training settings.
Adv-BMT
TD3
Hyper-parameter
Value
Value
Scenario Horizon
9s
Discounted Factor
0.99
History Horizon
Os
Train Batch Size
1024
Collision Step
1s-9s
Learning Rate
IE-4
Prediction Mode
8
Policy Delay
200
Policy Training Steps
10E6
Target Network
0.005
Table 9: Realism results for BMT's reverse prediction.
Method
SFDE
SFDE
SADE
SADE
Collagentmin
Collsgentare
Reverse
3.92
2.69
5.53
5.11
0.03
0.05
Reverse + Adv-init
7.17
5.56
7.68
0.13
0.16
Reverse + Adv-init + Filter
6.93
5.30
7.58
7.02
0.12
0.15
Table 10: Diversity results for BMT's reverse prediction.
Method
FDD
ADD
JSD
JSD
JSDrrc
Reverse
8.35
3.13
0.17
0.71
0.02
Reverse Adv-init
9.89
3.82
0.22
0.64
Reverse + Adv-init Filter
10.27
3.99
0.22
0.73
0.09
penalties, and road boundary violations. Driving reward is measured by forward lane progress, while penalties are applied for collisions with other vehicles or drifting off-roodd
Hyperparameter. The settings of our open-loop and closed-loop adversarial RI experiments shown in table Note that in our closed-loop learning, Adv-BMT takes one frame of agent information as input for adversarial generations, whereas all baseline methods take one second agent history.
C Quantitative Results
C.1
Ablation Study
In Tad Table TO reverse predictions with adversarial initializations exhibit greater deviation from the ground-truth data and yield improved diversity. This behavior is expected, since adversarial initializations modify the terminal positions and headings selected agents, which propagates backward into more varied histories, Importantly, our rulc-based filtering does not reduce diversity; rather, it preserves multimodality while improving realism metrics by removing physically implausible trajectories.
C.2 Waymo Open Sim Agents Challenge
We evaluate BMT results on 400 WOMD validation scenarios using the Waymo Open Sim Agents Challenge (WOSAC) 2025 (TT]. Evaluation results are summarized in Table IT] For the metrics, smaller values of minADE indicate more accurate predictions, whereas larger values for the remaining metrics indicate better performance. From the results, we observe that forward prediction achieves much better performance than reverse prediction across all metrics, except for similar performance in angular speed, angular acceleration, distance to the nearest object, and TTC. Note that the training times for forward prediction and reverse prediction are similar (10E6 and 15E6 steps, respectively). The WOSAC results indicate that our BMT model is better at predicting future motion than historical motion.
24

---

Table 11: WOSAC Evaluation results of BMT.
Metrics
Reverse
Forward
Linear speed
0.375
0.393
Linear acceleration
0.394
0,405
Angular speed
0.441
0.428
Angular acceleration
0.594
0.593
Distance to nearest object
0.405
0.388
Collision
0.521
0.951
Time to Collide
0.840
0.840
Distance to road edge
0.675
0.683
Offroad
0.564
0.934
Realism
0.554
0.753
Kinematic
0.451
0.455
Interactive
0.566
0.801
Map
0.596
0.862
minADE
2.148
1.344
Metametric
0.554
0.753
Upon reviewing the collisions detected in reverse predictions with real initializations, found that most occurred in crowded parking-lot areas, where clusters of parked vehicles or pedestrians are close together. These were flagged collisions by WOSAC's evaluation API, even though they may not represent meaningful agent-agent interactions. This explains why metrics such as ADE and FDE remain comparable across methods despite differences in collision scores.
The performance gap is primarily due to our training strategy: the model was pre-trained for forward prediction (around 800k steps) and then fine-tuned for bidirectional prediction (around 1.5M steps). We have included these details in Section A.4 (Training Details).
D Qualitative Results
Visualizations. Figure and Figure present of example scenarios results generated by Adv-BMT. Across different scenarios, the adversarial agent exhibits diverse range of safety-critical driving behaviors, demonstrating their ability interact plausibly with realistic traffic participants. The visualizations illustrate that Adv-BMTT can generate multiple distinct collision outcomes from single driving log. This highlights key advantage of Ady-BMT over baseline methods, which tend to produce identical or highly similar adversarial behaviors for the same input scenario.
Demo Video. We submit video within our supplementary materials. Here we provide vi- sualizations with case studies through animated simulations of Ady-BMT scenarios, which in- clude different types vehicles, pedestrians, and bicycle agents. More demos are available at https::/netadrivee wereetaadiiieree..mm thub. 10/ botdv/bbtt/
E Broader Impacts
Our work introduces a novel model for generating safety-critical traffic scenarios, aiming to improve the safety reliability and driving robustness of AD systems. By modeling both forward and reverse motion trajectories, our framework enables controllable and diverse simulation rare and high-risk traffic events. Our framework, Ady-BMT, benefits the development and testing of safer autonomous agents by exposing failure cases under challenging interactions. However, generating adversarial scenarios may potentially raise concerns about potential misuse, such as crafting unrealistic or malicious simulations. To address this, our approach is designed for research and evaluation within closed simulation environments. We encourage responsible usages of model and encourage integrating them into safety validation pipelines with appropriate regulations.
25

---

Figure Diverse adversarial behaviors generated by Adv-BMT.
26


Reviews:
[{"review_id": "i8vB6yEIV6", "reviewer": "Reviewer_KrZM", "summary": "This paper introduces Adv-BMT, a framework for generating diverse and realistic safety-critical traffic scenarios by augmenting real-world driving data. The core of the framework is a novel Bidirectional Motion Transformer (BMT) that can predict agent trajectories both forwards and backwards in time. The generation process is a two-stage pipeline: it first defines a future collision state between the ego vehicle and a new adversarial agent (adversarial initialization) and then uses the BMT's reverse-prediction capability to reconstruct the adversarial trajectory leading up to that collision. This approach allows the framework to generate a high rate of successful collisions without needing any prior collision data for training.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- **Novel and highly effective generation method**: The core concept of defining a collision and predicting backwards is a creative and powerful way to ensure a high rate of adversarial events. This approach effectively sidesteps the common challenge of data scarcity for safety-critical events. The method is highly effective, achieving a high adversarial collision rate and being significantly faster than baselines.\n- **Value for downstream AD training**: The paper provides strong evidence that the generated scenarios are not just adversarial but also useful for improving agent performance. An RL agent trained on the dataset augmented by Adv-BMT showed superior safety, reducing collision rates by 20% compared to agents trained with data from previous methods. This demonstrates the practical value of the framework for creating more robust AD systems.\n\n**Weaknesses:**\n- **Realism of \"inserted\" adversaries**: The framework's primary method is to insert a new adversarial agent into an existing scene. This raises questions about the semantic realism of the scenario, as the adversary does not originate naturally from the existing traffic flow. While rule-based filters ensure kinematic plausibility, the agent might still behave in ways that are inconsistent with a typical driver's intent in that specific context. Also, I am wondering if the generated scenarios are all solvable.\n- **Limited reactivity of background traffic**: In the primary reverse-prediction pipeline, the trajectories of existing background vehicles are \"teacher-forced\" to remain unchanged. This means the surrounding traffic does not react to the new, aggressive adversarial agent, which is unrealistic. While an optional \"forward refinement\" step is proposed to address this, the core method generates scenarios with limited multi-agent interactivity.\n- **Limited scope of adversarial scenarios**: As acknowledged by the authors, the framework currently only considers adversarial behaviors among vehicle agents. It does not generate scenarios involving other critical road users like pedestrians or cyclists, which are essential for comprehensive safety validation of any AD system.", "was_revised": true, "final_justification": "My concerns are mostly addressed. I encourage the authors to incoporate additional experiments in the final version.This work represents a meaningful step forward in the field of safety-critical scenario generation.", "rebuttal": "We appreciate the reviewer's valuable concerns and questions. \n\n>1. The framework's primary method is to insert a new adversarial agent into an existing scene. This raises questions about the semantic realism of the scenario, as the adversary does not originate naturally from the existing traffic flow. While rule-based filters ensure kinematic plausibility, the agent might still behave in ways that are inconsistent with a typical driver's intent in that specific context. \n\nAdv-BMT is designed with multiple generation modes to support varying levels of agent interactions and realism. By default, traffic agents follow their recorded trajectories to preserve consistency with the real-world driving log, while the adversarial agent is generated to interact within this fixed context. This enables targeted scenario editing while maintaining overall scene plausibility. To address limitations in interactivity, Adv-BMT also supports a closed-loop reverse prediction mode, in which all agents are jointly predicted to generate a fully reactive traffic scenario, without teacher-forcing any agent behaviors during predictions. Additionally, an optional forward refinement step allows traffic agents to respond to the newly introduced adversary, enabling a more interactive and dynamically consistent outcome. Together, these modes allow Adv-BMT to balance realism, controllability, and diversities for different use cases.\n\n\n>2. Also, I am wondering if the generated scenarios are all solvable.\n\nWe interpret \"solvable\" to mean that the autonomous agent is able to complete the scenario without violating safety constraints, such as collisions or failures to reach the destination. As shown in Table 3 (open-loop RL) and Table 4 (closed-loop RL), agents trained on Adv-BMT scenarios consistently achieved lower collision rates and reduced trajectory costs compared to baseline methods. Specifically, in the open-loop setting, our agent outperformed the best baseline by 5% in average cost, 14% in collision rate, and 17% in average trajectory cost. In the closed-loop setting, we also observed improvements of 5% in average cost and 1% in average trajectory cost over the best baseline. These improvements indicate that the scenarios generated by Adv-BMT are learnable and solvable by an RL agent. Furthermore, the completion rates reported in Table 3(b) and Table 4(b) show that at least 65% of the Adv-BMT scenarios are successfully completed by the agent. These results together suggest that Adv-BMT produces challenging yet solvable scenarios that are valuable for developing safer driving policies.\n\n\n>3. Limited reactivity of background traffic: In the primary reverse-prediction pipeline, the trajectories of existing background vehicles are \"teacher-forced\" to remain unchanged. This means the surrounding traffic does not react to the new, aggressive adversarial agent, which is unrealistic. While an optional \"forward refinement\" step is proposed to address this, the core method generates scenarios with limited multi-agent interactivity.\n\nAdv-BMT is designed with flexible generation modes that support varying levels of multi-agent interactions:\n\n**Adv-BMT with traffic agent teacher-forced**: In this generation mode, traffic agents are teacher-forced with their recorded trajectories. The new adversarial agent interacts with replayed traffic agents in the driving log to keep the high scene fidelity by preserving the original traffic flow.\n\n**Adv-BMT with closed-loop reverse prediction**: all agents including traffic participants and the adversarial agent are predicted jointly. It allows the entire scene to evolve coherently, enabling multi-agent interactions and producing fully synthesized scenarios with ego collisions.\n\n**Adv-BMT with forward refinement**: To introduce reactivity without sacrificing the structure of the recorded scene, this mode performs an additional forward-prediction step after generating the adversarial agent in teacher-forced mode. This allows background agents to adjust their behavior in response to the adversarial agent’s presence.\n\nThese complementary modes offer a trade-off between scene realism, controllability, and interactivity, enabling users to select the appropriate setting based on their evaluation needs.\n\n**Ablation Study for Adv-BMT with baselines**: \n\na. Realism of generated Adversarial behavior\n| Method              | Adversarial Attack Success | avg SFDE | min SFDE | avg SADE | min SADE | min Agent Collision | avg Agent Collision | min ADV-Traffic Collision | avg ADV-Traffic Collision |\n|---------------------|----------------------------|----------|----------|----------|----------|---------------------|---------------------|---------------------------|---------------------------|\n| CAT                 | 0.48                       | 1.01     | 1.01     | 0.31     | 0.31     | 0.1                 | 0.1                 | 0.3                       | 0.3                       |\n| SEAL                | 0.55                       | 1.26     | 1.26     | 0.38     | 0.38     | 0.12                | 0.12                | 0.32                      | 0.32                      |\n| STRIVE              | 0.38                       | 0.63     | 0.63     | 0.17     | 0.17     | 0.09                | 0.09                | 0.32                      | 0.32                      |\n| BMT_TF              | 1                          | 2.27     | 1.67     | 1.51     | 1.3      | 0.13                | 0.15                | 0.22                      | 0.39                      |\n| BMT_All             | 1                          | 2.36     | 1.63     | 1.56     | 1.29     | 0.13                | 0.15                | 0.19                      | 0.37                      |\n| BMT_TF_with_Filter  | 1                          | 2.28     | 1.35     | 1.53     | 1.15     | 0.13                | 0.14                | 0.12                      | 0.33                      |\n| BMT_All_with_Filter | 1                          | 1.97     | 1.23     | 1.4      | 1.09     | 0.12                | 0.14                | 0.09                      | 0.33                      |\n|                     |                            |          |          |          |          |                     |                     |                           |                           |\n|                     |                            |          |          |          |          |                     |                     |                           |                           |\n|                     |                            |          |          |          |          |                     |                     |                           |                           |\n\n\nb. Diversity of generated Adversarial behavior\n| Method              | Final Displacement Diversity | Average Displacement Diversity | Velocity JSD | Acceleration JSD | Time-To-Collide JSD |\n|---------------------|------------------------------|--------------------------------|--------------|------------------|---------------------|\n| CAT                 | 0                            | 0                              | 0.13         | 0.16             | 0.12                |\n| SEAL                | 0                            | 0                              | 0.18         | 0.25             | 0.17                |\n| STRIVE              | 0.01                         | 0                              | 0.11         | 0.22             | 0.12                |\n| BMT_TF              | 1.22                         | 0.49                           | 0.1          | 0.17             | 0.13                |\n| BMT_All             | 1.51                         | 0.63                           | 0.1          | 0.17             | 0.08                |\n| BMT_TF_with_Filter  | 2.32                         | 1                              | 0.08         | 0.12             | 0.09                |\n| BMT_All_with_Filter | 1.97                         | 0.83                           | 0.13         | 0.14             | 0.08                |\n\n(Note: here the min/avg refers to the mode selection method for each scene)\n\n**Analysis**: Comparing adversarial agents generated under different modes (BMT_TF for teacher-forced traffic agents, and BMT_All for fully interactive generation), we observe a trade-off between realism and diversity. Specifically, BMT_TF generated ADV trajectories have slightly better realism-related metrics (lower ADE/FDE), while BMT_All exhibits better behavioral diversity and interaction, as indicated by greater displacement diversity metrics and lower ADV-traffic collision rates. \n\nWe appreciate the reviewer’s questions and concerns, and we will emphasize in revision that Adv-BMT is designed to flexibly support multiple interaction modes, enabling a controllable balance between realism and diversity based on application needs.\n\n\n>4. Limited scope of adversarial scenarios: As acknowledged by the authors, the framework currently only considers adversarial behaviors among vehicle agents. It does not generate scenarios involving other critical road users like pedestrians or cyclists, which are essential for comprehensive safety validation of any AD system.\n\nThank you for raising this important point. We would like to clarify that Adv-BMT does in fact support the generation of adversarial pedestrian and bicycle agents, enabled by the fact that our prediction model has been trained on scenarios involving these agent types. Specifically, adversarial pedestrians can be initialized in scenes where the ego vehicle’s trajectory is close to existing crosswalks on the map, limiting their applicability to a subset of maps. In contrast, adversarial bicycle agents can be initialized more flexibly across a wider range of scenes. We appreciate the reviewer’s feedback and will revise the manuscript accordingly by updating the limitation section and adding visualizations to better reflect this capability."}, {"review_id": "HlOoT9Eoln", "reviewer": "Reviewer_qsAw", "summary": "Summary\n\nThis paper addresses the challenge of limited coverage of long-tailed, safety-critical scenarios in existing real-world autonomous driving (AD) datasets, which hampers effective scenario-based validation. To overcome this limitation, the authors propose Adv-BMT, a framework designed to augment real-world scenarios with realistic and diverse adversarial interactions.\nAt the core of the method is a Bidirectional Motion Transformer (BMT) model that performs inverse traffic motion prediction. Given the final frame of a scenario, BMT reconstructs the preceding traffic interactions in reverse chronological order, back to the initial state. The framework follows a two-stage pipeline: it first performs adversarial initialization, then applies the inverse motion prediction to synthesize plausible collision scenarios. Unlike prior work, Adv-BMT does not require access to real-world collision data for pretraining, yet is still capable of generating realistic and diverse collision events. Experiments demonstrate that training with the augmented scenarios produced by Adv-BMT reduces the episode-level collision rate by 20% compared to previous methods. The authors also state their intention to release the code.", "questions": "1. The bidirectional prediction process in the paper is not clearly explained. Is it correct that the model first predicts from the last frame to the first (reverse), and then from the first frame to the last (forward)? During the reverse process in Adv-BMT, it is stated that the ego and all non-ADV agents are teacher-forced — in the forward refinement stage, are the trajectories of the ADV agent and ego vehicle (from the reverse process) also teacher-forced? In Table 1, for the \"Bidirection\" setting, is it correct that no teacher-forcing is applied at all?\n\n2. There are some concerns regarding completeness of the experimental comparison. Why is the generation speed not compared against CAT? Could you also include the Adv-BMT results in Table 2 for completeness?\n\n3. The related work section is missing several relevant baselines. For example, there is no mention of SEAL, and diffusion-based trajectory generation methods are not discussed. In particular, works that apply diffusion with a fixed goal for trajectory denoising are highly relevant and deserve comparison or discussion.\n\n4. Could the authors provide more detail about the rule-based filtering process? Specifically, how many trajectories were generated initially, and how many were filtered out? What is the performance impact if the rule-based filtering is removed (e.g., on Table 2 results)? Since Adv-BMT applies rule-based filtering while many competing methods do not, this raises fairness concerns in the evaluation.\n\n5. You mention that you will release the code. Could the authors clarify the timeline and the scope of the release?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. The paper introduces the Bidirectional Motion Transformer (BMT), which employs temporally reversible motion tokenization to enable inverse traffic motion prediction.\n\n2. The proposed method can generate diverse safety-critical scenarios efficiently, demonstrating both high scenario diversity and fast generation speed.\n\n---\n\n### **Weaknesses**\n\n1. The writing could be improved for clarity. For example:\n   - Table 2 does not explain what \"SDD\" refers to.\n   - Tables 3 and 4 could benefit from highlighting the second-best results for better comparison.\n   - In Table 1, the number of frames corresponding to each context length should be clearly specified. Does \"context 0\" represent a single-frame input? How many frames are included in 1 second?\n\n2. The overall novelty appears to be limited", "was_revised": true, "final_justification": "The additional experiments and clarifications have addressed part of my concerns. However, since some of the rebuttal content was submitted beyond the valid rebuttal period, I will maintain the current score in accordance with NeurIPS rules.", "rebuttal": "We appreciate the reviewer's suggestions and questions. We will address point by point. \n\n\n>1. The writing could be improved for clarity. For example: Table 2 does not explain what \"SDD\" refers to. Tables 3 and 4 could benefit from highlighting the second-best results for better comparison.\n\nThank you for the helpful suggestions regarding clarity. We will revise the manuscript to include a clear definition of Start Displacement Diversity (SDD), which quantifies the variability in predicted starting positions across different modes. Additionally, we will update Tables 3 and 4 to highlight the second-best results, improving visual comparison across methods.\n\n\n>2. In Table 1, the number of frames corresponding to each context length should be clearly specified. Does \"context 0\" represent a single-frame input? How many frames are included in 1 second?\n\nYes, thank you for the question. A 0-second context refers to the setting where BMT takes only a single-frame of agent motion as input. In contrast, a 1-second context corresponds to 10 input frames, based on a frame rate of 10 Hz. We will revise the manuscript to clarify this definition in Table 1 and the corresponding text.\n\n>3. The overall novelty appears to be limited.\n\nWe want to emphasize the novelty of Adv-BMT compared to other baseline methods. Adv-BMT is designed with flexible generation modes that support varying levels of multi-agent interaction:\n\n**Adv-BMT with traffic agent teacher-forced:** In this generation mode, traffic agents are teacher-forced with their recorded trajectories. The new adversarial agent interacts with replayed traffic agents in the driving log to keep the high scene fidelity by preserving the original traffic flow.\n\n**Adv-BMT with closed-loop reverse prediction:** all agents including traffic participants and the adversarial agent are predicted jointly. It allows the entire scene to evolve coherently, enabling multi-agent interactions and producing fully synthesized scenarios with ego collisions.\n\n**Adv-BMT with forward refinement:** To introduce reactivity without sacrificing the structure of the recorded scene, this mode performs an additional forward-prediction step after generating the adversarial agent in teacher-forced mode. This allows background agents to adjust their behavior in response to the adversarial agent’s presence.\nThese complementary modes offer a trade-off between scene realism, controllability, and interactivity, enabling users to select the appropriate setting based on their evaluation needs.\n\nWe thank the reviewer for the insightful feedback and will clarify in the paper that Adv-BMT is designed to support a range of interaction modes, allowing users to balance realism and diversity according to their specific use cases.\n\n>4. The bidirectional prediction process in the paper is not clearly explained. Is it correct that the model first predicts from the last frame to the first (reverse), and then from the first frame to the last (forward)? \n\nYes, that is correct. In the bidirectional prediction process, BMT first performs reverse prediction, generating the initial frame from the given final frame. Then, using the predicted initial frame for all agents, BMT performs forward prediction to reconstruct the full trajectory up to the final frame. We will revise the manuscript to clarify this process.\n\n>5. During the reverse process in Adv-BMT, it is stated that the ego and all non-ADV agents are teacher-forced — in the forward refinement stage, are the trajectories of the ADV agent and ego vehicle (from the reverse process) also teacher-forced?\n \nIn the reverse pass, the ego and traffic agents are teacher-forced while the adversarial agent is predicted. In the subsequent forward pass, the ego and adversarial agents (i.e., the collision participants) are teacher-forced, and the traffic agents are predicted to allow for interaction with the adversarial behavior. We will clarify this bidirectional teacher-forcing setup in the manuscript.\n\n>5. In Table 1, for the \"Bidirection\" setting, is it correct that no teacher-forcing is applied at all?\n\nYes, that is correct. In the \"Bidirection\" setting in Table 1, the forward pass uses the predicted initial states from the reverse pass without applying teacher-forcing to any agents. This setup allows us to evaluate the realism and consistency of fully generated trajectories under our bidirectional prediction pipeline. We will clarify this in the revised manuscript.\n\nAblation Studies Evaluation Results:\nRealism \n| Method            | avg SFDE | min SFDE | avg SADE | min SADE | veh_coll_scenario_count | veh_coll_min | veh_coll_avg |\n|-------------------|----------|----------|----------|----------|-------------------------|--------------|--------------|\n| Reverse           | 3.92     | 2.7      | 5.53     | 5.11     | 0.7                     | 0.03         | 0.05         |\n| Forward           | 3.52     | 2.35     | 2.39     | 1.98     | 0.6                     | 0.03         | 0.05         |\n| Reverse + Forward | 3.29     | 2.01     | 2.38     | 1.74     | 0.73                    | 0.03         | 0.06         |\n\n\nDiversity\n| Method            | Start Displacement Diversity | Final Displacement Diversity | Average Displacement Diversity | Velocity JSD | Acceleration JSD | Time-To-Collide JSD |\n|-------------------|------------------------------|------------------------------|--------------------------------|--------------|------------------|---------------------|\n| Reverse           | 8.35                         | 0                            | 3.13                           | 0.17         | 0.71             | 0.02                |\n| Forward           | 0                            | 10.78                        | 4.4                            | 0.23         | 0.69             | 0.3                 |\n| Reverse + Forward | 8.49                         | 12.78                        | 7                              | 0.22         | 0.77             | 0.35                |\n\n**Analysis:** The results show that bidirectional prediction (Reverse + Forward) enhances the diversity of generated agent behaviors across all displacement diversity metrics. In addition, it improves the realism of predictions, as evidenced by lower SFDE and SADE values, with a slight regression in average agent collision rate compared to the reverse-only and forward-only settings. This supports the effectiveness of the forward-refinement design in Adv-BMT to improve the realism of surrounding traffic participants.\n\n>6. There are some concerns regarding completeness of the experimental comparison. Why is the generation speed not compared against CAT? \n\nThank you for pointing this out. The omission of CAT’s generation speed in Table 2 was an oversight on our part. We will correct this in the revised version and include CAT’s generation speed for a complete and fair comparison. \n| Method  | CAT  | SEAL | STRIVE | Adv-BMT |\n|---------|------|------|--------|---------|\n| 500-avg | 0.8  | 2.36 | 9.53   | 1.02    |\n| 500-std | 0.09 | 0.41 | 0.28   | 0.02    |"}, {"review_id": "UxFPOL6Mgi", "reviewer": "Reviewer_v8s5", "summary": "This paper introduces a new method for generating realistic and adversarial scenarios by reverse prediction from a collision scenario. They demonstrate that scenarios generated with Adv-BMT improve robustness and safety of downstream policies, surpassing several prior adversarial scenario generation approaches.", "questions": "1. Address the weakness.\n2. Why is the improvement from training the agent using RL with the scenarios generated with the BMT method so limited?\n3. Why does predicting history motion achive much worse realism than predicting future motion?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Generating the collision scenario resersely is novel. \n2. The paper is well-writen and figure is clear.\n3. The experiment is extensive with real world data.\n\nWeakness:\n1. The realism of the reversely generated scenario is questionable. The result in table 2 demonstrates the baseline can achive better realism. Table 8 also demonstrates the reservely generated scenarios have a poor WOSAC realism meta metric.\n2. The method uses rule-based filter but it is not applied to the baseline method. So the comparison is unfair.\n3. In figure 4, different method uses different adversarial agents. Consider using the same adversarial agent and explain why the trajectory of the baseline method is so weird.", "was_revised": true, "final_justification": "The additional experiments and clarifications have addressed my concerns.", "rebuttal": "We thank the reviewer for all valuable questions and suggestions. \n\n> 1. The realism of the reversely generated scenario is questionable. The result in table 2 demonstrates the baseline can achive better realism. Table 8 also demonstrates the reservely generated scenarios have a poor WOSAC realism meta metric.\n\nWe appreciate the reviewer’s concern regarding the realism of the reversely generated scenarios. As shown in Table 8, reverse prediction results exhibit lower scores in collision likelihood and offroad likelihood, which contributes to a lower WOSAC realism meta-metric compared to forward predictions. The reverse model is designed to predict the behavior of an adversarial agent, which allows predicted collisions and deviates from typical driving patterns.\nMore importantly, our ablation study table attached below shows that the reverse model produces adversarial behaviors with both higher realism and diversity than baseline generation strategies that do not incorporate learned backward agent dynamics. Rule-based filters improve generation quality for the adversarial agents and remove implausible or extreme behaviors, enhancing the realism of the final scenarios.\nWe leverage the high performance of forward predictions in Adv-BMT’s forward refinement step to allow realistic interactions among traffic agents. As supported by both our quantitative metrics and qualitative visualizations, the resulting scenarios are realistic, diverse, and safety-critical, serving as effective test cases for evaluating prediction models.\n\n>2. The method uses rule-based filter but it is not applied to the baseline method. So the comparison is unfair.\n\nWhen attempting to apply these filters to baseline methods such as CAT, SEAL and STRIVE, we realize that the adversarial behaviors produced by the implementations lack the variability needed to generate alternative trajectories once the initial ones were rejected by the filters. This is also indicated by the low FDD and SDD scores in our evaluation. So we update the ablation study and remove filtering from Adv-BMT in Table 2.\n\na. Realism of generated Adversarial behavior\n\n| Method              | Adversarial Attack Success | avg SFDE | min SFDE | avg SADE | min SADE | min Agent Collision | avg Agent Collision | min ADV-Traffic Collision | avg ADV-Traffic Collision |\n|---------------------|----------------------------|----------|----------|----------|----------|---------------------|---------------------|---------------------------|---------------------------|\n| CAT                 | 0.48                       | 1.01     | 1.01     | 0.31     | 0.31     | 0.1                 | 0.1                 | 0.3                       | 0.3                       |\n| SEAL                | 0.55                       | 1.26     | 1.26     | 0.38     | 0.38     | 0.12                | 0.12                | 0.32                      | 0.32                      |\n| STRIVE              | 0.38                       | 0.63     | 0.63     | 0.17     | 0.17     | 0.09                | 0.09                | 0.32                      | 0.32                      |\n| BMT_TF              | 1                          | 2.27     | 1.67     | 1.51     | 1.3      | 0.13                | 0.15                | 0.22                      | 0.39                      |\n| BMT_All             | 1                          | 2.36     | 1.63     | 1.56     | 1.29     | 0.13                | 0.15                | 0.19                      | 0.37                      |\n|                     |                            |          |          |          |          |                     |                     |                           |                           |\n              \n(Note: here the min/avg meaning the mode selection for each predicted scenario)\n\nb. Diversity of generated Adversarial behavior\n| Method              | Final Displacement Diversity | Average Displacement Diversity | Velocity JSD | Acceleration JSD | Time-To-Collide JSD |\n|---------------------|------------------------------|--------------------------------|--------------|------------------|---------------------|\n| CAT                 | 0                            | 0                              | 0.13         | 0.16             | 0.12                |\n| SEAL                | 0                            | 0                              | 0.18         | 0.25             | 0.17                |\n| STRIVE              | 0.01                         | 0                              | 0.11         | 0.22             | 0.12                |\n| BMT_TF              | 1.22                         | 0.49                           | 0.1          | 0.17             | 0.13                |\n| BMT_All             | 1.51                         | 0.63                           | 0.1          | 0.17             | 0.08                |\n\n\n>3. In figure 4, different method uses different adversarial agents. Consider using the same adversarial agent and explain why the trajectory of the baseline method is so weird.\n\nWe will update the figure to use the same initial adversarial agent across all methods for consistency, and revise the caption to explain any trajectory artifacts observed in the baseline methods.\n\n\n>4. Why is the improvement from training the agent using RL with the scenarios generated with the BMT method so limited?\n\nWe appreciate the reviewer’s question regarding the impact of using Adv-BMT-generated scenarios for reinforcement learning. As shown in Table 3 (open-loop RL) and Table 4 (closed-loop RL), agents trained on Adv-BMT scenarios consistently achieved lower collision rates and reduced trajectory costs compared to baseline methods. Specifically, in the open-loop setting, our agent outperformed the best baseline by 5% in average cost, 8% in collision rate, and 10% in average trajectory cost. In the closed-loop setting, we observed improvements of 3% in average cost, 10% in collision rate, and 38% in average trajectory cost. This highlights the value of the diverse and targeted adversarial scenarios generated by Adv-BMT in strengthening policy robustness through reinforcement learning.\n\n\n>5. Why does predicting history motion achive much worse realism than predicting future motion?\nWe attach our WOSAC results below:\n\na. Forward prediction:\n\n| Metric name                             | Value       |\n|----------------------------------------|-------------|\n| Realism meta-metric                    | 0.7694586   |\n| Linear Speed Likelihood                | 0.37768704  |\n| Linear Acceleration Likelihood         | 0.39826843  |\n| Angular Speed Likelihood               | 0.42033565  |\n| Angular Acceleration Likelihood        | 0.59226924  |\n| Distance To Nearest Object Likelihood  | 0.38673738  |\n| Collision Likelihood                   | 0.9600281   |\n| Time To Collision Likelihood           | 0.82912844  |\n| Distance To Road Edge Likelihood       | 0.6734454   |\n| Offroad Likelihood                     | 0.9424464   |\n| minADE                                 | 1.4130014   |\n\n\nb. Reverse prediction:\n\n| Metric Name                          | Value      |\n|-------------------------------------|------------|\n| Realism meta-metric                 | 0.5640852  |\n| Linear Speed Likelihood             | 0.36751205 |\n| Linear Acceleration Likelihood      | 0.38674814 |\n| Angular Speed Likelihood            | 0.43716192 |\n| Angular Acceleration Likelihood     | 0.59631693 |\n| Distance To Nearest Object Likelihood | 0.40195426 |\n| Collision Likelihood                | 0.5159998  |\n| Time To Collision Likelihood        | 0.8315656  |\n| Distance To Road Edge Likelihood    | 0.6667353  |\n| Offroad Likelihood                  | 0.5591505  |\n| minADE                              | 2.160784   |\n\nWe appreciate the reviewer’s question regarding the lower realism of the reversely generated scenarios. As shown in Table 8, the reverse model yields a lower overall WOSAC realism meta-metric (0.56) compared to the forward model (0.77). This gap is primarily due to reduced scores in collision likelihood (0.52 vs. 0.96) and offroad likelihood (0.56 vs. 0.94). These two metrics heavily influence the realism meta-metric and suggest that reverse-predicted trajectories more frequently result in unsafe or abnormal behaviors, such as agent collisions or off-road motions. Despite this, the reverse model performs comparably to the forward model in some dynamics-related metrics, such as linear and angular acceleration likelihood, indicating that the generated motions are still kinematically plausible."}]

Author rebuttal:

Meta-review: The paper introduces Adv-BMT, a framework for generating safety-critical traffic scenarios using a bidirectional motion transformer. It reconstructs trajectories in reverse from adversarial collision states, enabling diverse and realistic scenario generation without collision data pretraining. Results show improved diversity, realism, and 20% lower collision rates for RL agents trained with augmented data.

Strengths: creative use of reverse prediction; strong empirical results across realism, diversity, and downstream safety; flexible modes (teacher-forced, closed-loop, forward refinement) balancing realism and diversity; thorough ablations; code release planned.

Weaknesses: realism of adversarial agents questioned; limited background traffic reactivity; scope mainly focused on vehicles, with limited validation on pedestrians/cyclists; some fairness concerns due to rule-based filtering; clarity of writing and comparisons could be improved.

In the rebuttal, authors clarified bidirectional prediction, detailed teacher-forcing strategies, added ablations on filtering and initialization, and emphasized support for pedestrian/cyclist adversaries. Reviewers found concerns on realism and fairness partly resolved. While novelty and clarity were noted as moderate, the demonstrated utility and comprehensive evaluation led reviewers to lean positive. 

Despite limitations, the work addresses a key challenge in AD safety testing, proposes a novel and effective solution, and demonstrates strong practical impact through improved robustness of trained agents.
Decision: Accept (poster)