Title: Adv-BMT: Bidirectional Motion Transformer for Safety-Critical Traffic Scenario Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: autonomous driving

reinforcement learning

robotics
Abstract: Scenario-based testing is essential for validating the performance of autonomous driving (AD) systems. However, such testing is limited by the scarcity of long-tailed, safety-critical scenarios in existing datasets collected in the real world. To tackle the data issue, we propose the Adv-BMT framework, which augments real-world scenarios with diverse and realistic adversarial interactions. The core component of Adv-BMT is a bidirectional motion transformer (BMT) model to perform inverse traffic motion predictions, which takes the last frame of the scenario as input and reconstruct the traffic in the inverse of chronological order, till the initial time step. The Adv-BMT framework is a two-stage pipeline: it first conducts adversarial initializations and then inverse motion predictions. Different from previous work, we do not need any collision data for pretraining and are still able to generate realistic and diverse collision interactions. Our experimental results validate the quality of generated collision scenarios by Adv-BMT: training in our augmented dataset would reduce episode collision rates by 20\% compared to previous work. The code will be made available.


Peer reviews:
comments: SUMMARY:
This paper introduces a new method for generating realistic and adversarial scenarios by reverse prediction from a collision scenario. They demonstrate that scenarios generated with Adv-BMT improve robustness and safety of downstream policies, surpassing several prior adversarial scenario generation approaches.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Generating the collision scenario resersely is novel. 
2. The paper is well-writen and figure is clear.
3. The experiment is extensive with real world data.

Weakness:
1. The realism of the reversely generated scenario is questionable. The result in table 2 demonstrates the baseline can achive better realism. Table 8 also demonstrates the reservely generated scenarios have a poor WOSAC realism meta metric.
2. The method uses rule-based filter but it is not applied to the baseline method. So the comparison is unfair.
3. In figure 4, different method uses different adversarial agents. Consider using the same adversarial agent and explain why the trajectory of the baseline method is so weird.

QUESTIONS:
1. Address the weakness.
2. Why is the improvement from training the agent using RL with the scenarios generated with the BMT method so limited?
3. Why does predicting history motion achive much worse realism than predicting future motion?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary\n\nThis paper addresses the challenge of limited coverage of long-tailed, safety-critical scenarios in existing real-world autonomous driving (AD) datasets, which hampers effective scenario-based validation. To overcome this limitation, the authors propose Adv-BMT, a framework designed to augment real-world scenarios with realistic and diverse adversarial interactions.\nAt the core of the method is a Bidirectional Motion Transformer (BMT) model that performs inverse traffic motion prediction. Given the final frame of a scenario, BMT reconstructs the preceding traffic interactions in reverse chronological order, back to the initial state. The framework follows a two-stage pipeline: it first performs adversarial initialization, then applies the inverse motion prediction to synthesize plausible collision scenarios. Unlike prior work, Adv-BMT does not require access to real-world collision data for pretraining, yet is still capable of generating realistic and diverse collision events. Experiments demonstrate that training with the augmented scenarios produced by Adv-BMT reduces the episode-level collision rate by 20% compared to previous methods. The authors also state their intention to release the code.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. The paper introduces the Bidirectional Motion Transformer (BMT), which employs temporally reversible motion tokenization to enable inverse traffic motion prediction.\n\n2. The proposed method can generate diverse safety-critical scenarios efficiently, demonstrating both high scenario diversity and fast generation speed.\n\n---\n\n### **Weaknesses**\n\n1. The writing could be improved for clarity. For example:\n   - Table 2 does not explain what \"SDD\" refers to.\n   - Tables 3 and 4 could benefit from highlighting the second-best results for better comparison.\n   - In Table 1, the number of frames corresponding to each context length should be clearly specified. Does \"context 0\" represent a single-frame input? How many frames are included in 1 second?\n\n2. The overall novelty appears to be limited\n\nQUESTIONS:\n1. The bidirectional prediction process in the paper is not clearly explained. Is it correct that the model first predicts from the last frame to the first (reverse), and then from the first frame to the last (forward)? During the reverse process in Adv-BMT, it is stated that the ego and all non-ADV agents are teacher-forced — in the forward refinement stage, are the trajectories of the ADV agent and ego vehicle (from the reverse process) also teacher-forced? In Table 1, for the \"Bidirection\" setting, is it correct that no teacher-forcing is applied at all?\n\n2. There are some concerns regarding completeness of the experimental comparison. Why is the generation speed not compared against CAT? Could you also include the Adv-BMT results in Table 2 for completeness?\n\n3. The related work section is missing several relevant baselines. For example, there is no mention of SEAL, and diffusion-based trajectory generation methods are not discussed. In particular, works that apply diffusion with a fixed goal for trajectory denoising are highly relevant and deserve comparison or discussion.\n\n4. Could the authors provide more detail about the rule-based filtering process? Specifically, how many trajectories were generated initially, and how many were filtered out? What is the performance impact if the rule-based filtering is removed (e.g., on Table 2 results)? Since Adv-BMT applies rule-based filtering while many competing methods do not, this raises fairness concerns in the evaluation.\n\n5. You mention that you will release the code. Could the authors clarify the timeline and the scope of the release?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Adv-BMT, a framework for generating diverse and realistic safety-critical traffic scenarios by augmenting real-world driving data. The core of the framework is a novel Bidirectional Motion Transformer (BMT) that can predict agent trajectories both forwards and backwards in time. The generation process is a two-stage pipeline: it first defines a future collision state between the ego vehicle and a new adversarial agent (adversarial initialization) and then uses the BMT's reverse-prediction capability to reconstruct the adversarial trajectory leading up to that collision. This approach allows the framework to generate a high rate of successful collisions without needing any prior collision data for training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- **Novel and highly effective generation method**: The core concept of defining a collision and predicting backwards is a creative and powerful way to ensure a high rate of adversarial events. This approach effectively sidesteps the common challenge of data scarcity for safety-critical events. The method is highly effective, achieving a high adversarial collision rate and being significantly faster than baselines.\n- **Value for downstream AD training**: The paper provides strong evidence that the generated scenarios are not just adversarial but also useful for improving agent performance. An RL agent trained on the dataset augmented by Adv-BMT showed superior safety, reducing collision rates by 20% compared to agents trained with data from previous methods. This demonstrates the practical value of the framework for creating more robust AD systems.\n\n**Weaknesses:**\n- **Realism of \"inserted\" adversaries**: The framework's primary method is to insert a new adversarial agent into an existing scene. This raises questions about the semantic realism of the scenario, as the adversary does not originate naturally from the existing traffic flow. While rule-based filters ensure kinematic plausibility, the agent might still behave in ways that are inconsistent with a typical driver's intent in that specific context. Also, I am wondering if the generated scenarios are all solvable.\n- **Limited reactivity of background traffic**: In the primary reverse-prediction pipeline, the trajectories of existing background vehicles are \"teacher-forced\" to remain unchanged. This means the surrounding traffic does not react to the new, aggressive adversarial agent, which is unrealistic. While an optional \"forward refinement\" step is proposed to address this, the core method generates scenarios with limited multi-agent interactivity.\n- **Limited scope of adversarial scenarios**: As acknowledged by the authors, the framework currently only considers adversarial behaviors among vehicle agents. It does not generate scenarios involving other critical road users like pedestrians or cyclists, which are essential for comprehensive safety validation of any AD system.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}