Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous Driving
Authors: Haohong Lin
Yunzhi Zhang
Wenhao Ding
Jiajun Wu
Ding Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Autonomous Driving
World Model
Policy Adaptation
Primary area: reinforcement_learning

Abstract: End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap, we propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains a diffusion-based policy adapter to refine the base policy’s predictions and a multi-step Q value model to evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain, and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.

Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving
Haohong Lin', Yunzhi Zhang", Wenhao Ding2, .Jajun Wu², Ding Zhao
CCU, Stanford, "NVIDIA hachongl@andree cmu. edu
Abstract
End-to-end (E2E) autonomous driving models have demonstrated strong perfor- mance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap. we propose Model- based Policy Adaptation (MPA), general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first gen- erates diverse counterfactual trajectories using a geometry-consistee engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains diffusion-basee policy adapter to refine the base policy's predictions and a multi-step Q value model evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.
simulation
Introduction
End-to-End (E2E) autonomous driving models have made impressive strides by integrating perception, prediction, and planning into unified learning framework [12 3 4. Leveraging large-scale offline driving datasets, E2E models perform well under open-loop evaluation protocols, where the agent passively predicts future behaviors from offline recorded observation sequences. However, these models degrade closed-loop environments, where minor deviations accumulate over time, leading compounding errors, distribution shifts, and poor generalization to long-horizon scenarios. This performance gap reveals a core challenge: offline training based on empirical risk minimization does align with the online objective of maximizing cumulative reward, is illustrated in Figure [
To bridge this gap, recent efforts have turned to evaluating the closed-loop performance E2E agents. Some open-loop such NavSim proposes the approximate closed-loop evaluation with Predictive Driver Model Score an open-loop evaluation fashion. Other works introduced sensor simulation for the closed-loop evaluation, generating camera views based diffusion models (5 6
Neural Radiance Field (NeRF) 77 g 9 [0] or 3D Gaussian Splatting (3DGS) [2 [ 14
that enable photorealistic rendering of novel viewpoints. These tools provide fine-graines control over agent interventions and visual realism, making them promising testbeds for studying failure modes and recovery strategies. Existing works such as VAD [4) VAD-v2 (16), and Hydra-MDP [I7] design different scoring mechanisms to select the predicted motions for closed-loop control, yet these works either lack closed-loop evaluation results or evaluated in non-photorealistic simulator like CARLA [18] Most recently, RAD [19 incorporates reinforcement learning and uses 3DGS for online rollouts and evaluation, while the training of PPO agents be costly, and the value
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Open-Loop: Lowest MSE
MPA Training
MPA Inference
Multi-stop Q Valuee Model
Q- Q-Value
ServingAggnn
Policy Adapter
Adapter
Open-Loop: Cameea Sensor
E2E Driving Agent
Closed-Loop: 3DG8 Rendered
Counterfactual Data Generation
Closed-Loop Highest Reward
Ground Truth Trajectory
Driving
Figure 1: Left: Causes of closed-loop performance degradation in End-to-End driving, including observation and objective mismatches. Right: We propose counterfactual data generation to address the observation mismatch, and model-based policy adaptation framework tackling the objective mismatch.
critic is left unused at inference time. Among all the prior attempts, none of the works have curated counterfactual data into consideration during the training phase.
Our goal in this paper is to adapt the pretrained open-loop E2E driving agents from the real domain safe and generalizable closed-loop agents, with 4DGG-basee driving simulation data engine. We identify that the performance drop between the closed-loop and open-loop evaluations stems from two fundamental sources: (1) Observation mismatch the shift between training-time sensor inputs and deployment-timg timeentttemeeeeee closed-loop observations under perturbed behaviors from different data engines; (2) Objective mismatch the absence of meaningful reward feedback during offline imitation learning. which limits long-term reasoning.
We conducted preliminary experiments to demonstrate that the first mismatch is actually minor in the open-loop evaluation. Then we propose unified solution called Model-Basee Policy Adaptation (MPA), general framework that directly addresses both mismatches by separating and targeting their root causes. We first use the pretrained policy as an initialization of behavior policy to generate a coun- terfactual dataset using a high-fidelity 3DGS simulation engine. To mitigate observation mismatch, we design diffusion-baase residual policy adapter that conditions on diverse, counterfactual trajec- tories. This exposes the policy broad range of behaviors and visual scenes beyond those seen in the offline dataset. To address objective mismatch, then learn a Q-value model from the same counterfactual data that captures long-horizon outcomes and enables value-based assessment beyond rule-based metrics. MPA uses both components at inference time: the policy adapter generates residual action proposals conditioned on the current observation, and the value model performs inference-time scaling to select the action with the highest expected utility.
Our contributions in this work are threefold:
We analyze the root causes of closed-loop performance degradation in E2E agents and assess the fidelity of IDGS-basee simulation for modeling observation and behavior shifts.
We develop systematic counterfactual data curation pipeline using 3DGS rollouts and train the MPA with diffusion-bal based policy adapter and reward model to address observation and reward mismatches, respectively.
We demonstrate that inference-time scaling using the learned reward model significantly improves closed-loop performance on the nuScenes benchmark, particularly in safety-critical and out-of- domain scenarios.
2 Preliminary
2.1 Problem Formulation of End-to-End Autonomous Driving
We formulate closed-loop end-to-end (E2E) driving Partially Observed Markov Decision Process (POMDP) M = A, O,,, T), where
latent state space, A the action space, the transition dynamics, R the reward function, O the observation space, the discount factor, and T the planning horizon. each timestep, the agent receives an observation o, O and outputs a trajectory action a; A. The environment evolves according to P( ax) and emits observations via

---

(obs() In practice, the state includes the ego vehicle's IMU status and surrounding road entities' past poses and motion intents. These are often only partially observable. The action at represents sequence of future waypoints, which is translated into low-level throttle and steering
control sequences via an LQR controller [20], following prior benchmarks [21] 22 13 The observation o, is captured by real sensors or rendered by simulation engine during closed-loop evaluation. Notably, current open-loop E2E agents are trained using expert trajectories from behavior policy Wref inducing a state distribution (se) and yielding supervised model ee.. In contrast, closed-loop agents aim to maximize cumulative reward over time:
Open-loop
#* arg min
(s 10e (
Closed-loop T² arg max
E P ao..a P (r(se, ac)l
Simulation n arg max
E P(r 1 1). 10x(0r 1se 1), PaPat 1) [r((s,,))) at)]
This leads to an inherent objective mismatch: open-loop training minimizes imitation error under expert supervision, while closed-loop deployment optimizes long-horizon reward under evolving dynamics and partial observability, as is illustrated in Figure | Bridging this gap requires careful alignment of three components in equation ( the transition model P($1411/8r ar), which can be consistently approximated using vehicle dynamics; the observation model Poss (or (s), which may deviate from simulated sensors Possi and the reward function r(3r, az), which must be inferred from partial observations o, using learned value models.
To address these mismatches, we employ a counterfactual data generation supported by 3DGS- based observation model Pats. then design policy adapter that transforms the pretrained सिर into reward-aligned policy л under the guidance of a learned Q-value model, as outlined in Figure !
2.2
E2E driving Agents in the Closed-loop Simulation
A recent line of approach uses visual generative reconstruc-
L2 Error to Reference Trajectory (m)
Closed-loop
tion models for rendering photorealistic driving scenes from state parameterizations, utilizing the recent advancement in im-
6-
UnIAD (GT)
UniAD
age diffusion modeling (5), neural field rendering [10), and 3D
5
VAD (GT)
Gaussian Splatting [13]. This line of methods essentially learn
LTF (GT)
4
estimate Poos( (-st))):) in our POMDP formulation in Section 2.1
However, it is critical to verify that the visual quality of these
3
model outputs remains close to real-world scenes for them to serve as valid proxies for our closed-loop evaluation.
We conducted the following preliminary experiments study the fidelity of the closed-loop simulator and demonstrate the performance gap between open-loop and closed-loop evaluation. In Figure 2 we systematically study the difference in L2 error under open and closed-loop settings. We use 3DGS [3] to reconstruct the scenes from the nuScenes dataset and compare the performance difference using ground truth data in Figure 2 Among all three E2E policies, we see very close open-loop performance in motion prediction. This confirms the fidelity of the 3DGS-bars based simulation in its reconstruction quality.
0.5 Prediction Horizons (8)
1.0
1.5
2.0
2.5
3.0
Figure Comparison of average L2 error in the motion prediction under different prediction horizons.
Meanwhile, we also illustrate the L.2 error based on UniAD's closed-loop rollout trajectory. We can see that the prediction eтror becomes quite significant compared to the open-loop prediction as the prediction horizon grows. A non-ignorable 1.2 error in the short prediction horizon leads to out-of- distribution issues that result in such compounding errors. Increasing the planning frequency and shortening the planning horizon cannot necessarily close the gap between open-loop and closed-loop performance unless effective feedback guidance is provided to the E2E agents.

---

Counterfactual Generation
MPA Training and Inference
Denoising Adapter fo +
Residiall
Guidence
Low Raward
High Poward
ex: P Re.e T
Max-value Sampling
Forward noising
Multi-
Fittrn
áa
times
staans Consttaint
3DGS World Model
Weighted Sum
Counterfacoudd
Tenohe Fording true
Action
Scone Representation
History
Driving Agent
Trojectory
Driving Agent
Collision
Speed
Route
Multi-step Value Model (
Original Datasot
Counteriactual
Figure 3: Overview of Model-Based Policy Adaptation (MPA). Left: We propose a counterfactual data generation pipeline, where we first generate initial data of T Ttee trajectories rolled out with pretrained E2E policy and 3DGS-based world model, and then filter the generated data with reward and distance constraints to improve data realism, resulting in counterfactual (action, state, observation, reward) sequences. Right: We utilize the generated data train two MPA modules: (i) a diffusion policy adapter predicting residual actions on top of base E2E agent, and (ii) value model Q estimating multi-step cumulative rewards under different principles, such as collision and speed.
Proposed Method: Model-Based Policy Adaptation (MPA)
To bridge the observation and objective mismatches outlined in Section 2.1 we introduce Model- Based Policy Adaptation (MPA)-a unified framework for open-loop to closed-loop adaptation in end-to-end (E2E) autonomous driving. Figure 3 shows overview. This section is organized from left to right along the pipeline: Section 3.1 describes our model-based counterfactual data synthesis to address distribution mismatch. Section 3.2 details the training of a diffusion-basse policy adapter on the curated dataset. Section 3.3 presents the value model used to guide policy adaptation.
3.1 World Model-Based Counterfactual Data Generation
We generate counterfactual samples using a geometry-consister 3D Gaussian Splatting (3DGS) simulator 13), which renders photorealistic observations conditioned on poses of ego vehicle and surrounding agents, modeling the observation distribution (- As shown in Section 2.2 this rendering remains reference distribution.
as long as the rollout policy induces a state distribution close to the
To ensure reliable observations while introducing behavioral diversity, we randomly augment the predicted actions from a pretrained E2E policy *et by rotating, warping, and random noising operations, resulting in a noisy policy Φe o Ree.. The resulting distribution of an augmented behavior shown in Figure 3 Specifically, the augmentation of the trajectory consists of rotation angle ranging from [-10, 10), and warping ratio from [0.1, 1.0). as well as random Gaussian noise with standard deviation 0.05. These augmented trajectories will then be categorized into different modes, given their rotation angles and warping ratios for the multi-modal policy adapter. Under a teacher-forcing setup, we initialize the rollout of every driving scene from original reference state, and conduct counterfactual rollouts. To mitigate the exponentially growing search space as the rollout time horizon expands, we refer to the beam search algorithm 233 maintaining only the most rewarding candidates based on the cumulative reward calculated by the simulator using rule-based heuristics. The specific definition of these rewards be found in Appendix A A similar proposal aopposal-scoring.... ccoring--ellction selection pipeline is used in the prior literature 24 to mitigate the ceddiction-planiin misalignment of the motion planner.
To prevent rendering artifacts, we discard trajectories that deviate beyond a distance threshold or fall below minimum reward. The rollout horizon T determines how far into the future these

---

counterfactual behaviors extend; as we show in our experiments, longer horizons expose richer supervision signals for downstream learning, b ut iecrasee the risk of divergence from reference data.
The full generation procedure is summarized Algorithm [ With the generated counterfactual dataset at hand, we then conduct policy learning and value learning in the following sub-sections.
Rotated Trajectories
Algorithm 1: 3DGS-Based Counterfactual
12
Generation
Input: Ref. dataset Duef policy Trefs horizon T, thresholds re
10
Output: Counterfactual dataset D
foreach ref. traj (s) .1 ....... Dref do
So sp for = to T-1
ae 0g f
if dist( 8 > d or ((0, < <o then
break
Original Trjjedory
81+1, T2 4- Sim.step( sc, ae)
or + 3DGS Rendere
Y Postioo
Append (st, Qz, or, r1) to De
Figure 4: Ego's augmented trajectories after ro- tating, warping, and random noising operations.
3.2 Diffusion-Based Policy Adaptation
As we can see in the Figure the generated counterfactual actions fall into a multi-modal distribution. To capture these counterfactual actions, we propose a celfuusion-based policy adapter that refines the output of a frozen end-to-end (E2E) driving model by predicting residual trajectories Aa = a" a buse where a base Hx2 is the trajectory from a pretrained policy (e.g.. UniAD) and a" is the most rewarding trajectory sample from the counterfactual rollout dataset.
Training. To model the distribution over residuals, apply latent diffusion process. The forward step adds Gaussian noise over K steps:
(Aa (k) (a =N (VN(AA Vānda Vax (1 (1-6x)) -11-a)]I.
where (o) = Aa and ae is the cumulative noise schedule. The denoising network So is 1D U-Net that outputs multi-mode (o) from (k conditioned on the scene encoding z = Penc (o, Segu). ego history Sego. and base predicted trajectory a base The output of fo contains N modes, we denote the i-th mode as fo(...) ji.. It trained with the loss:
Lan = E J.c min (A) k, Bego: a hase [i]
An
where (k) = (a) + -- with e ~ N(O,I).
Inference. For inference, we use DDIM 25 to sample () and recover the adapted trajectory:
adapt
a basc
Aa (0).
This design adapts the knowledge from the pretrained base policy by conditioning their encoded scene context and base actions, enabling generalization beyond the counterfactual data domain.
3.3 Principled Q- Value Guided Inference-time Sampling
Training. While step-wise rewards can be computed given full access to 8t, estimating long-horizon returns is challenging under partial observability. Using the generated counterfactual dataset, we train multi-step action value model
5

---

Nominal Scenes
VAD
MPA (Ours)
Ground Trush
Reconetructed
Colliiin
Yieding
Ground Truth
Reconstructed
Collision
Bmiking
Figure 5: Qualitative Results in the in-domain and safety-critical scene. The silver car cuts in from the right side, forcing the ego vehicle to yield. Compared to the pretraind VAD, MPA-adapted policy can successfully brake and yield to the cut-in vehicles under the guidance of the Q-value model.
T Σ rr)
Q St, Ori T)
based on four interpretable principles: route following Qroute. lane distance Qaist. collision avoidance Qcollision: and speed compliance Spesdt. Each Q-function is trained independently to predict cumulative returns using (Ot, Segur at). Q = E (collision, dist.... + W/ x Q, We follow the prior literature (26 to
design our reward principles and their corresponding weights, with more details in Appendix A We provide thorough ablations of the Q Q-function components in Section
Inference. At inference, residual actions aa are sampled following the policy adapter inference procedure from Section 3.2 and the best proposal is selected via
Ad arg max (o,, Sego: buse
Do; T),
â"
buse
Ad"
AuEa
Compared to classifier- reward guidance 27, 28 our Q-value guidance offers feedback from longer planning horizons and avoids the gradient instability of the reward model.
4 Experiment Results
In our experiments, we aim to answer the following research questions. RQ1. Compared to the baselines, can MPA bring benefits to the E2E driving agents in a closed-loop evaluation in a generalizable way? RQ2. How does MPA benefit the safety-critical performance in the closed- loop evaluation? RQ3. How do different adapters and value guidance modules contribute to the performance of MPA? RQ4. How does MPA scale with the number of counterfactual planning steps in the data generation phase?
4.1 Experiment Settings
Dataset and Simulation Engine. We utilize the nuScenes dataset (29 that consists of 5.5 hours of driving data in Boston and Singapore. Every scene has a reference trajectory of 20 seconds. We use HUGSIM [K3] the simulation engine and evaluation benchmark. We train on split of 290 scenes in the nuScenes train-val split, and evaluate on three settings. (1) In-domain evaluation: the model will be tested on sub-split of 70 scenes, the surrounding dynamic entities (vehicles. pedestrians) will be replayed by a fixed ratio of their reference trajectory in the offline dataset. (2) Unseen nominal scene evaluation: the model will be tested on a sub-split of 70 scenes that are unseen yet during training, the surrounding dynamic entities (vehicles, pedestrians) are nominal and will be replayed by a fixed ratio of their reference trajectory in the offline dataset. (3) Safety-critical evaluation: the model will be tested on 10 scenes, where there exists one (or few) non- non-native agents to challenge the ego agents in an adversarial way. The simulation frequency 4 Hz. all the scenes, the termination occurs under one of the following five conditions: (i) full route completion, (ii) off-road events, (iii) collision events, (iv) too far from the reference trajectory, (v) maximum rollout time limits (50 seconds, 2.5x x of the reference trajectory) reached.

---

Table 1: In-Domain Closed-Loop Evaluation Results. All the evaluation metrics are higher the better. Bold means the best, and underlined is the best rumner-up for each metrics.
Model
Ego Status
Camera
Curation
RC
NC
DAC
TTC
COM
HDScore
UniAD
39,4
56.9
75.1
52.1
98.7
19,4
VAD
50.1
68.4
87.2
66.1
90.2
31.9
LTF
65.2
71.3
92.1
67.6
98.4
46.7
AD-MLP
X
13.4
80.2
86.2
79.4
90.1
6.5
BC-Safe
V
57.0
59.8
87.9
55.2
89.4
33.6
Diffusion
V
71.8
67.4 4
88.1
64.5
91.5
45.1
MPA (UmiAD)
93.6
76.4
92.8
72.8
91.8
66.4
MPA (VAD)
94.9
75.4
93.6
72.5
92.8
67.0
MPA (LTF)
93.1
70.8
90.9
67.9
94.9
60.0
Baselines. We compare the MPA with diverse baselines in E2E driving algorithms that fall in the two following categories. (1) Pretrained base policy with open-loop training manner: we compare with the performance of UniAD [3) VAD 4 and LTF [2] on the HUGSIM dataset. We further build on our MPA with these policies. (2) E2E agents trained with curated counterfactual dataset: We further train several baseline policies with the curated dataset. AD-MLP 30 utilizes the ego's velocity, acceleration, past trajectories, and high-level command as the input, which is recognized as a naive baseline for the closed-loop Driving tasks. BC-Safe uses the safe segments in the counterfactual datasets to train and End-to-End policies. Diffusion adopts the implementation of [32 in the scene encoding and utilizes a DDIM-based sampler (25) instead of truncated denoising during the inference time. To ensure fair comparison between the MPA and the second category of the baselines, all the approach uses pretrained ResNet [33] as the perception backbone to encode the RGB inputs from perspective cameras. The ego status includes velocity, acceleration, and navigation landmarks from the history frames.
Metrics. We follow the evaluation protocol in HUGSIM 13 which is inspired by the NAVSIM- based metrics [22]. The metrics include Route Completion (RC), Non-Collision (NC), Driveable Area Compliance (DAC), Time-To-Colliss
(TTC), Comfort (COM), HUGSIM Driving Score (HDScore). Specifically, HDScore is computed with the above metrics along with Route Completion (RC), instead of the Ego Progress (EP) in PDMS (22). HDScore is the weighted sum as follows:
Eme (TTC. COM) weight,,, score
HDScore RC
E{
II
scorem
Eme(TTc..om) weight,,
t= (NC, DAC)
We list all the metrics with (x 100) in the tables. All the metrics fall in [0.0, 100). We attach more details of these metrics and other information settings in the Appendix(B.]
4.2 Main Results and Analysis (RQ1, RQ2)
To answer RQI, we first evaluate the closed-loop performance for in- and out-of-domain scenes, as shown in Figure All the reported MPA approaches are evaluated with 20 action samples at inference time. In-domain scenes refer to the scenes that are used to generate counterfactual training data in Singapore. We evaluate the quantitative closed-loop results these training scenes in Table []] MPA-based E2E driving agents achieved better results compared to their pretrained counterparts, as well as three baseline methods trained on counterfactual curated dataset, especially in the most important metrics, RC and HDScore. The baseline AD-MLPP moves very conservatively, so the NC and TTC are low, yet the RC also quite low as it barely completes the assigned routes for the challenging E2E scenes. Besides, the NC score in HUGSIM is bit underestimated compared to NAVSIM, as HUGSIM crodes the vehicle boxes compared to the actual size by the point clouds. This leads to few false 'collision' signals during the evaluation. Yet. most of them will not cause collision that terminates the entire episode in the closed-loop simulation. This is why we observe some high RC with mediocre NC metrics. This still means the ego agents are capable of navigating around different collisions and oof-road maneuvers to reach the goal in reasonable way, and will result in good HDScore.
We further evaluate the closed-loop performance under the unseen scenes. We select 70 scenes in Boston that not accessible in the curated counterfactual dataset. The qualitative results shown
7

---

Table Out-of-Domain Closed-Loop Evaluation Results in unseen nominal and safety-critical scenes. All the evaluation metrics are higher the better. Bold means the best, and underlined is the best runner-up for each metrics.
Unseen Nominal Scenes
Safety- Critical Scenes
Model
RC
NC
DAC
TTC
COM
HDScore
RC
NC
DAC
TTC
COM
HDScore
UniAD
39.3
56.6
74.0
52.6
98.2
22.2
11.4
76,2
82.1
57.8
95.9
4.5
VAD
45.4
64.8
86.2
62.0
95.9
29.3
25.4
77.0
88.3
73.2
88.4
16.0
LTF
63.3
64.8
86.5
62.8
98.2
41.9
35.1
80.9
96.8
78.1
100.0
24.2
AD-MLP
7.6
71.6
82.2
69.8
92.3
3.3
4.9
93.5
96.2
93.4
85,9
4.3
BC-Safe
59.2
59.8
81.2
56.3
95.9
34,6
20.2
80.1
91.7
67.3
86.7
13.5
Diffusion
57.9
62.1
83.5
58.3
96.2
35.1
20.9
84.3
92.3
72.4
86.3
13.1
MPA (UmiAD)
93.7
69.5
92.9
66.6
97.6
60.9
95.1
76.8
98.9
74.2
97.7
70.4
(VAD)
90.9
71.0
94,4
68.8
97.7
61.2
96.6
79.8
99.0
77.3
97.7
74.7
MPA LLF))
91.8
68.3
91.0
66.5
96.9
57.0
87.3
72.0
94.0
66.9
97.8
56.3
(PA.U
(PA
MPA.
UniAD
WAD
LTF
NG
DAC
TTC
COM
HDScore
Mentes Value
75 15
72-
90-
72-
68
8
64
85-
64-
92
60-
80
75
acoumtoractual Stops
#Connterfccuu Stops
#Coumeerfacua Steps
#Counter Stops
#coutterfccuu Steps
@Coutterfactual Stops
Figure 6: Impact of Rollout Steps (T from Algorithm 1 during the counterfactual data generation. We fix the sample size to six during the closed-loop evaluation for all the MPA variants. MPA trained with more counterfactual steps results in better test-time performance, the Q value model takes more future steps as the supervision signals.
in Table (left). Compared to the in-domain results in Table 1 we can see significantly degraded performance in AD-MLP and Diffusion, while the pretrained E2E policies still perform similarly as they do in the in-domain scenes. We observe that the MPA agents built upon the pretrained E2E policies are still optimal and quite robust under the unseen scenes. All three variants have comparable HDScore with their in-domain evaluation. This demonstrates the generalizability of the proposed adapter and value model under unseen scene contexts.
4.3 Ablation Studies (RQ3, RQ4)
We conduct ablation studies to analyze the contribution of the three main modules of MPA: (i) counterfactual dataset generation, (ii) policy adapter, and (iii) Q-value guidance.
Counterfactual Dataset. We further analyze the impact of the curated counterfactual dataset by ablating the step size during the rollout of counterfactual data. Then we train the MPA over the curated dataset and evaluate its performance in the unseen scenes in Boston, similar to the setting in Table 7 (left). We illustrate the trends of evaluation metrics with respect to the step sizes in the counterfactual dataset in Figure [ MPA benefits from longer counterfactual steps, as there would be more informative supervision for the value function training in the future steps.
Policy Adapter. In Table we evaluate few variants of MPA (UniAD). Due to the space limits. the results of ablation studies for MPA (VAD) and MPA (LTF) will be postponed to the Appendix B We evaluate the unseen scene's results across different variants. The comparison between ID-5 and ID-6 (ours) demonstrates the effectiveness of the adapter in better route completion. Under the safety-critical scenes, the adapter brings ~00 increase to the out route completion, leading to significantly higher HDScore compared to the policies without the adapter. We also investigate the impact of mode capacity of policy adapter to the driving performances in safety-critical testing splits in Figure 7 The results show that a smaller capacity will influence the DAC and RC metrics, leading to lower HDScore.
Principles of Q-Value Guidance. In Table we remove different principles in the state-action value function Q. Compared to the ID-5 variants, ID-1 removes the route information used in all the
8

---

MPA (UrVAC))
MPAAWW
MPA (TT))
UnIAD
VAD
LTF
DAC
COM
HDScore
DU
59
75
60"
58
93
70
92
45-
72
90
56
57
24
36 22
16
22
36 22
16 22
16
2
16
#ideepter
W'Adepter
#Adapter
#dddeperr
#Adepper
Figure 7: Impact of Adapter Mode Scales on the Performance under Safety-Critical Scenarios. The results show that larger sizes of modalities consistently bring benefits to the driving performance RC, DAC, and HDScore, and the benefit especially large under the smaller mode sizes.
Table Ablation Study on MPA's variants on the UniAD base policy, Top:
Unseen scenes that are nominal but not appearing in the training dataset. Bottom: Safety- scenes with adversarial surrounding agents. Bold means the best, and underlined is the best runner- up for each metrics.
ID
Qroute
Q
Q collissin
Opeeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
r
6.9
81.2
95.1
81.0
100
5.1
V
83.9
57.0
81.0
53.6
99.4
43.2
V
89.2
70.8
95.6
68.6
99.4
60.8
90.4
68.9
91.8
56.6
91.1
71.5
94.1
69.4
99.4
60.9
/
93.7
69.5
92.9
66.6
97.6
60.9
Qroute
Qaist
Qcolissoon
Qspeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
4.6
86.0
98.3
79.3
90.1
3.6
65.1
65.6
85.7
53.8
39.5
V
d
57.7
82.4
99.0
69.6
84.6
39.2
4
V
V
79.3
82.9
98.5
68.0
93.9
50.1
V
V
75.6
81.2
98.8
78.6
99.7
55.3
95.1
76.8
98.9
74.2
97.7
70.4
baselines, which leads drastically degraded performance. ID-2 removes the distance function the reference route, significantly degrading driveable area compliance and non-at-fault collision metrics. ID-3 removes the collision values, and ID-4 removes the speeding value function. Both of them have an impact on HDScore, especially in anfety-critica situations. The reason that NC still seemed to be high for ID-3 in safety-critical scenes is that the available frame length before collision short, which makes the denominator in NC smaller compared to the other group. However, when we look the RC metrics, the drop when removing Qcollision significant, as the agents will encounter collision and end to episode earlier than the nominal cases.
Related Works
End-to-End Autonomous Driving. End-to-End (E2E) autonomous driving has achieved significant progress by jointly training the detection, tracking, prediction, and planning modules avoid information loss throughout the cascading system. ST-P3 [1] and UniAD 3 propose unified E2E frameworks that achieve state-of-the-art open-loop performance the nuScenes dataset [29]. VAD encodes the driving scene with a vectorized representation and incorporates query-based planning modules, and VAD-v2 [4] further designs probabilistic planning approach and improves the closed-loop performance over the CARLA [18) benchmark. LAW (34) enhances VAD's performance with auxiliary supervision signal from latent world model. Hydra-MDP [17] follows VAD-v2's query-based framework and conducts multi-target hydra distillation with set of scoring rules. SparseDrive [35] uses a cearse-centric pipeline to get rid of BEV features in the E2E driving [32 designs truncated denoising scheme for the diffusion model to generate multi-mode motion trajectories. With the prosperity of foundation models, a series incorporate Large Language Models (LLMs) and Vision-Language Models (VLMs) into the E2E planning pipeline. Despite the benefits of commonsense reasoning with foundation models, most of the existing models still focus the open-loop evaluation or
pipeline, and DiffusionDrive
of works 33
37
38
39
40

---

the approximate closed-loop metrics from the open-loop evaluation 22 which lack counterfactual reasoning for the safety-critice scenarios. A recent work RAD 19 pays attention to the photorealistic closed-loop evaluation and utilizes imitation learning and online reinforcement learning to fine-tune the E2E driving agents. Yet, the value functions trained for the Proximal Policy Optimization (PPO) agents are not effectively integrated during the inference. MPA aims to use the value model as an effective inference-time guidance to make the E2E agents more robust in closed-loop evaluation.
Counterfactual Data Generation. Counterfactual data generation has been explored within the context of offline reinforcement learning. Wang al. 41 utilize a learned model to autonomously generate additional offline data, thereby enhancing the training of sequence models. OASIS 22 introduces method to produce counterfactual data by modulating guidance signals during diffusion model inference. In high-stakes decision -aaiing domains such as autonomous driving, generating counterfactual data essential due to the limited presence of safety-critical scenarios in existing datasets. Previous research has addressed the trade-off between realism and controllability in safety- critical scenario generation by integrating various constraints. These include inference-time sampling techniques [43] retrieval- -rnmmeeel- generation 44.. low-rank fine-tuning approaches [45]. and language-condee conditioned generation methods [46]. However, these efforts primarily focus on behavioral scenario generation without incorporating visual information. With advancements in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), recent studies such as DriveArena (5) and MagicDrive [47] have begun developing E2E simulators for closed-loop evaluation. Similarly, RAD 19 employs a DDDGS-basee simulator for RL fine-tuning. Notably, to date, no existing work has focused on generating E2E counterfactual data within E2E simulators.
Reward Model for Inference-Time Scaling. Recent LLM research has shown the power of the reward model in LLM's inference-time scaling 48 49 In sequential decision-making problems, the reward model was explicitly used for inference-ting time supervision signals, such as the guidance for the diffusion- policy models [50] [51, (52. For the closed-loop autonomous driving and decision- making tasks, several prior works incorporate reward models as the classifier- based guidance steer the diffusion model's sampling process [53 54 28 Other format of the guidance signals include signal temporal logic (STL.) guidance, language-basse
guidance (55), adversarial guidance [56]
57 58). and game-theoretic guidance [59]. In the autonomous driving domain, prior works like DiffusionDrivee (32) utilize truncated denoising for the diffusion models without additional classifier guidance. DiffAD utilizes action conditional guidance for E2E driving [60]. Diffusion-ES (27). Gen-Drive [61, and Diffusion-Planmee
(28) utilize customized reward models as test-time guidance for the non-E2E driving tasks. To the best of our knowledge, MPA is the first work incorporating the driving reward model for the inference-ting time scaling of E2E driving agents.
Conclusion
In this work, we introduce MPA, general framework for improving the closed-loop trustworthiness of E2E autonomous driving agents. MPA begins by generating high-quality counterfactual trajectories through ceoametry-coniistrrr rollouts in 3DGS-based simulation environment. This results in better data coverage while preserving visual fidelity. With the counterfactual dataset, MPA further trains diffusion- based policy adapter to refine base policy predictions and leverages a multi- principle value model to guide inference-time decision making. These components allow pretrained agents to generate and evaluate multiple trajectory proposals, selecting actions that optimize long-term driving outcomes. Experimental results on nuScenes data and HUGSIM benchmark demonstrate MPA's effectiveness in boosting safety and generalizability.
Limitations. Despite these promising results, our approach assumes reliable rendering from 3DGS under constrained trajectory deviations and currently decouples value modeling from policy opti- mization. Future work includes extending our current results to diverse driving datasets, exploring the online RL training over the 3DGS simulator, and deploying MPA to the multi-modal foundation models to enhance reasoning capability for more severe distribution shifts in autonomous driving.
Acknowledgments and Disclosure of Funding
This work is in part supported by ONR MURI N00014-24-1-2748 NSF CNS-2047454, and CMU Safety 21 of the University Transportation Program. YZ in part supported by the Stanford Interdisciplinary Graduate Fellowship.
10

---

References
[1] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-basee autonomous driving via pratial-empoort feature learning. In European Conference on Computer Vision, pages 333-549. Springer, 2022.
[2] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-basee sensor fusion for autonomous driving. IEEE 2022.
Iransactions on pattern analysis and machine intelligence,
[3] Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang. et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853-17862, 2023.
[4] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8355 2023.
[5] Xuemeng Yang. Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A glosed-loop generative simulation platform for autonomous driving. arXiv preprint arXiv:2408. 00415, 2024.
[6] Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405. /7398, 2024.
[7] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[8] Ze Yang, Yun Chen, Jingkang Wang. Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1389-1399, June 2023.
[9] Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. arXiv preprint arXiv:2311.15260, 2023.
[10] William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Åström, Michael Felsberg, and Christoffer Petersson. Neuroncap: Photorealistic closed-loop safety testing for autonomous driving. In European Conference on Computer Vision, pages 161-177. Springer, 2024.
[11] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., (2(4):139-1) 2023.
[12] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In ECCV, 2024.
[13] Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugsim: A real-time, p and closed-loop simulator for autonomous driving. arXiv preprint arXi::24/2.01718 2024.
[14] Ziyang Xie, Zhizheng Liu, Zhenghao Peng, Wayne Wu, and Bolei Zhou. Vid2sim: Realistic and interactive simulation from video for urban navigation. arXiv preprint NKKi:2500.06699 2025.
[15] Junhao Ge, Zuhong Liu, Longteng Fan, Yifan Jiang, Jiaqi Su, Yiming Li, Zhejun Zhang, and Siheng Chen. Unraveling the effects of synthetic data end-to-end autonomous driving. arXiv preprint arXiv:2503. 18108, 2025.
[16] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning, arXiv preprint a324: 2024.

---

[17] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406. 06978, 2024.
[18] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[19] Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, et al. Rad: Training an end-to-end driving policy via large-scale 3dgs-based reinforcement learning. arXiv preprint arXiv: 13144, 2025.
[20] Norman Lehtomaki, NJAM Sandell, and Michael Athans. Robustness results in linear-quadratic gaussian based multivariable control designs. IEEE Transactions on Automatic Control, 26(1):75-93, 1981.
[21] Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards learning-basee planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 629-636. IEEE, 2024.
[22] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang. Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706-2877 2024.
[23] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances neural information processing systems, 27, 2014.
[24] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with mis- conceptions about learning-based vehicle motion planning. In Conference on Robot Learning, pages 1268-1281. PMLR, 2023.
[25] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. [26] Jianyu Chen, Bodi Yuan, and Masayoshi Tomizuka. Model-free deep reinforcement learning for urban autonomous driving. In 2019 IEEE intelligent transportation systems conference (ITSC), pages 2765-2771. IEEE, 2019.
[27] Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung- Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki. Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following, arXiv preprint arXiv:2402.06555, 2024.
[28] Yinan Zheng, Ruiming Liang. Kexin Zheng. Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501. 15564, 2025.
[29] H Caesar, V Bankiti, AH Lang, S Vora, VE Liong, Xu, A Krishnan, Y Pan, G Baldan, and O Beijbom. nuscenes: A multimodal dataset for autonomous driving. arxiv. 2019.
[30] Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoging Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 10430, 2023.
[31] Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning, 2019.
[32] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang. et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139 2024.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[34] Yingyan Li, Lue Fan, Jiawei He. Yugi Wang. Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.0 08481, 2024.
[35] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation, arXiv preprint arXiv:2405.19620, 2024.
12

---

[36] Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. In NeurlPS 2023 Foundation Models for Decision Making Workshop.
[37] Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, XianPeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. In 8th Anmual Conference on Robot Learning.
[38] Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14769, 2024.
[39] Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv preprint arXiv:2407.00955, 2024.
[40] Vahid Behzadan and Arslan Munir. Vulnerabilityy of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recognition, pages 262-275. Springer, 2017.
[41] Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Boot- strapped transformer for offline reinforeement learning. Advances in Neural Information Processing Systems, 35:34748-34777 2022.
[42] Yihang Yao, Zhepeng Cen, Wenhao Ding, Hachong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 11:78451-78477 2024.
[43] Shuhan Tan, Kelvin Wong, Shenlong Wang. Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 892-901, 2021.
[44] Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone, Realgen: Retrieval augmented generation for controllable traffic scenarios. arXiv preprint arXiv:2312.13303 2023.
[45] Robert Dyro, Matthew Foutter, Ruolin Li, Luigi Di Lillo, Edward Schmerling. Xilin Zhou, and Marco Pavone. Realistic extreme behavior generation for improved av testing. arXiv preprint arXiv:22009 10669, 2024.
[46] Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krachenbuehl. Lan- guage conditioned traffic generation. In Conference on Robot Learning, pages 27 2714-2752. PMLR, 2023.
[47] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Ya Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023.
[48] Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033: 2024.
[49] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504. 02495, 2025.
[50] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
[51] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
[52] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations.
[53] Chiyu Jiang, Andre Cormman, Cheolho Park, Benjamin Sapp. Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9644-9653, 2023.
13

---

[54] Chiyu Max Jiang, Yijing Bai, Andre Cormman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Wheatley Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efffccinnt and controllable driving simulation initialization and rollout. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[55] Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guide traffic simulation via scene-level diffusion. In Conference on Robot Learning, pages 144-177. PMLR, 2023.
[56] Chejian Xu, Ding Zhao, Alberto Sangiovanni- Vincentelli, and Bo Li. Diffscene: Diffusion- based safety-critical scenario generation for autonomous vehicles. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023.
[57] Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chan- draker. Controllable safety-critical closed-loop traffic simulation via guided diffusion. arXiv preprint arXiv:2401.00331, 2023.
[58] Yuting Xie, Xianda Guo, Cong Wang, Kunhua Liu, and Long Chen. Advdiffuser: Generating ad- versarial safety- driving scenarios via guided diffusion. arXiv preprint arXiv:2410.08533, 2024.
[59] Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile scene-consistent traffic scenario generation optimization with diffusion. arXiv preprint arXiv:2404.02524, 2024.
[60] Tao Wang, Cong Zhang, Xingguang Qu, Kun Li, Weiwei Liu, and Chang Huang. Diffad: A unified diffusion modeling approach for autonomous driving. arXiv preprint ar 12110 2025.
[61] Zhiyu Huang, Xinshuo Weng. Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint arXiv:2410.05582. 2024.
[62] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Bric Cousineau, Benjamin Burchfiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137. 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are provided in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
They are discussed in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide necessary implementation details in section section and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We provide an open access to the code and data collection scripts in the anonymous link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiinyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the experiment settings and details Appendix A and B.T
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation, as well as the significance level (p value) in Appendix B.3
Guidelines: We
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://aaaaaps.cppppp.l//ttttat
Answer: [Yes]
Justification: Provided in the OpenReview form.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Provided in Appendix C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: This paper adapts from the pre-tre pre-trained end-to-ent autonomous driving models. We describe safeguards in the appendix and will also attach when releasing our model checkpoints and dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the baselines are properly cited and introduced in Appendix A.3
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

A Auxiliary Details of MPA Implementation
A.1 Data Collection and Preprocessing
The environments we use to rollout counterfactual data and policy evaluation stay the same with HUGSIM benchmark 13
State, Action, and Observation Space of Driving Simulator. consists of different entities, which will be input to the MPA.
The driving simulator we use 13
Observation (RGB): The rendered 6 camera views have resolution of 450 800 each. This results (6, 3, 450, 800) tensor.
The pose of every agent can be represented as a tuple (x, y, yaw). We also include the velocity, acceleration, and route information (represented as longitude progress and lateral distance) in the past frames. For the state in HUGSIM, the ego's 's history trajectory
State and Action (Trajectory):
will be (Thia: 8) array. For the action output from the agents, it will be (Tim: 2) trajectory that consists of the 2D position at each frame. This trajectory will be solved by an LQR controller [20) generate the corresponding throttle and steering sequence.
Intermediate Representation: all three methods will have intermediate representation as BEV, where UniAD and LTF have a rasterized map, and VAD has a vectorized map.
Mauristics-Basede Reward Labeling. The step-wing wise reward heuristics include the longitude reward (route completion), lateral distance reward (distance), collision penalty, and far from trajectory penalty. The Q-value is the truncated cumulative sum of the T step reward, where I the planning horizon as was set up to 5 steps (1.25 s) in our experiments.
Q
Ti+k
Based on these principles, we use the following reward heuristics to label the state-action transition pairs in the simulation. The first term is the longitude reward, denoted as route completion Proute- At timestep I,
Troute de d
where de be the projected longitudinal distance along the route from the starting point to the closest point route the current ego position determined pos by st-
The second term is the lateral distance penalty between the ego position and the route, denoted as distance reward rdist:
Tdist - |||pos ego
poS point 2|2-
Following the simulation setup, the off-road and collision penalty will be - -100 respectively, once collision between the ego vehicle and other traffic agents or between the ego vehicle and background static objects occurs. This penalty term can be described as:
1, min |pos gch < Sevtlisions
Feullision
-1, 1, posego $ Driveable Area,
0, otherwise.
To improve the smoothness of driving, we use the speed penalty when the agents go overspeeding:
{- [
&velo), Uego > 8yelos
sspeed
0.
otherwise.
Constrained Rollout with Pretrained E2E Agents. We use the pretrained UniAD 3. VAD and LTF [2] base behavior policies to generate counterfactual datasets to further train the MPA. We use the planned trajectory for every timestep and augment with different rotation angles (-5.09 to +5.0%) and scales (0.1 times 2.0 times), leading 21 counterfactual behaviors at single observation
21

---

frame. The key parameter during the counterfactual generation is to filter out the undesired samples based on the reward heuristics section discussed in the previous subsection.
For the policy adapter, we exclude all collision samples with a reward of Pcollision -1, although these samples are retained in the training dataset of the Q-value network. To ensure reliable learning for both the value network and the policy adapter. it is essential that the 3DGS-rendered datasets exhibit high visual realism. We assess the realism of these renderings using the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) metrics, as shown in Figure g The plot provides insight into acceptable lateral distance thresholds for maintaining the figure, both FID and KID degrade with increasing lateral displacement. Empirically, lateral deviation of less than 0.6 from the closest reference trajectory yields sufficiently realistic renderings. with KID remaining below 0.3 and FID generally under 50.
simulation. As illustrated in
FID and KID vs Lateral Displacement
0.07
FID
80
KID
0.06
0.05
60
0.04
40
0.03
0.02
20
0.01
0.00
0.0
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
>0.8
Distance Bin (meters)
Figure 8: FID (4) and KID (4) with respect to the lateral distance to the ground truth trajectory.
A.2 Implementation Details in Training and Inference
Network Architecture. The architectures of the diffusion adapter and the value model are illustrated in Figure 9 Both models take the ego history information and action as input. The key difference that the adapter model takes the base action from the pretrained model, noisy residual action for the diffusion block, and intermediate representations from the base model, while the value model takes raw camera RGB inputs and final actions to be evaluated and selected at inference time.
Computing Resources. The experiments are run on a server with AMD EPYC 7542 32-Core Processor CPU with 256 threads, 4NVIDIA A5000 graphics, and 252 GB memory. For one experiment, it takes around 6 hours to train around 20 epochs for the policy adapter and value model. At inference time, the inference of the 3DGS-based world model and pretrained base driving model can be run on single A5000. The inference time of pretrained baselines (UniAD, VAD) is around 0.5s, and the 3DGS world model takes around 0.3s to render all six camera observations with static and dynamic objects in the traffic scenes, and they are running two separate worker nodes following the HUGSIM [13]. For one single scenario, the maximum step is 200. Most scenarios will be finished less than one-third of the maximum time. In total, it will take an average of minute for the evaluation per scene.
Hyperparameter Table. We list the hyperparameters used in MPA as follows. They include the hyperparameters for data curation, diffusion adapter, and the value model.
The link to our anonymous codebase is attached at: http://mnooyynooossooooocccccccr
A.3 Additional Description of the Baselines
UniAD K presents a unified autonomous driving framework that jointly learns perception, predic- tion, and planning within single Transformer-base architecture. The method explicitly formulates
22

---

Table 4: Hyperparameters for Our Methods.
Component
Value
Description
Policy Adapter Architecture
BEV encoder
ResNet-18
CNN Encoder for BEV input.
Ego encoder
128-dim
Encodes ego history features.
Action encoder
128-dim
Encodes the base actions and noisy target future trajectories.
Fused input dimension
960
Concatenated input vector.
Latent fusion module
ID U-Net
Applies down/up-sampling over 960-dim vector.
Latent dimension
256
Feature dimension used throughout latent layers.
Residual prediction heads
1-20
Each outputs 12-dimensional trajectory residual.
Mixture weight head
1
Outputs logits over modes.
DDIM steps (training)
25
Number of noisy timesteps during training.
Value Network Architecture
Image encoder
ResNet-18 Mean pooling
Shared pretrained CNN across 6 stitched views. Average of per-view features.
Image fusion
History steps Tast
5
History length of the ego agents
Hidden dimension
512 MLP
Used throughout encoders and MLPs.
Final decoder
Two-layer predictor for value scalar output.
Training Settings (Policy Adapter)
Batch size
256
Samples per mini-batch.
Learning rate
1 10
Initial LR with cosine decay.
Optimizer
AdamW
With weight decay of 10-4 1000
Epochs
Total training iterations.
Gradient clipping
1.0
Global gradient norm threshold.
Training Settings (Value Network)
Batch size
128
Samples per mini-batch Optimized via Adam. Total training epochs.
Learning rate
1 10
Epochs
100
Validation ratio
0.10 Weighted
90% train, 10% validation.
Sampler
Inverse-frequency sampling for value balance (collision value only). Stitched from 2x3 grid of 800x450 crops.
Image input format
6x(3x224x400)
Inference-Time
DDIM steps (trainig/sammplnng)
5
Number of diffusion time steps.
Noise schedule
Linear
3: in 10-4, 0.02].
DDIM 7)
0.0
Deterministic sampling. Penalty weight of route progression
Route Completion Weight Whoute
1000.0
Lateral Distance Weight Wdist
50.0
Penalty weight of lateral distance away from reference
Collision Value Weight Weollision
100.0
Collision penalty weight
Speed limit Weight Weollision
50.0
Speed penalty weight beyond the limit threshold with 36kph
23

---

with Apache Predicted Denoised
HUGSIM Rollout Reward Heuristics
Action Decoder
True Ego Future
Denoised Representation
(9I2)
Q-Value Supervision
U-Net
MSE Losol MLEE
Cumulative Principle (Colliaion, Over-Speed...))
Fused Representation
Fusion Projection (MLP)
Value Decoder (MLP)
Noisy Emb.
Action
History Embedding
Visuat Embedding
Action Embedding
History Embadding
Visual
Emb.
Embedding
(512.)
(128.)
(512.)
(512.
(S12)
(512)
Action Encoder (MLP)
History Encodar (MLP)
RGB Encoder (ResNet)
Action Enooder (MLP)
History
Mean Pooling
Encoder (MLP)
RGB Encoder
(ResNet)
Noiey Action
Base Action
Ego History
Intermediate
Action
History
6-Camera Views
apwed.
Hettry pos, VINA, speed, wnoel. Innami disance...
wocel., intwal (Ta).8)
BEV 22,224, 224)
Image
Orac 2)
(Ta
(haas
224, 204)
The policy adapter consists of the visual encoder, ego history encoder, and action encoder. The embeddings of all three encoders are further fused and passed through 1D U-Net to get the denoised actions. Right: the Q-value model consists of a pretrained visual encoder ResNet--18 with freeze weights. The remaining parts include similar action encoder and history encoder, and each value principle is supervised by the cumulative step-wise heuristic rewards defined during the counterfactual data rollouts in the HUGSIM.
Figure 9: The Architecture of Policy Adapter and Q-Value Model. Left:
the driving task as an autoregressive trajectory generation problem, leveraging dense Bird's Eye View (BEV) representations to enable strong coordination between scene understanding and mo- tion planning. Through large-scale offline training. UniAD achieves state-of-the-ar performance under open-loop evaluation protocols, demonstrating the effectiveness of end-to-end learning for tonomous driving tasks. The codebase in open-sourced at httgss//gtthubccoooo/nDDiiieeii with Apache License 2.0.
VAD extends the end-to-end paradigm with a focus on visual grounding and scalable deploy- ment. The framework employs multi-camera inputs to extract spatial-temporal features and utilizes Transformer blocks to capture interactions across different road agents. By removing the reliance practical real-world deployment while maintaining high-quality planning capabilities through direct alignment of vision features with future motion targets. The codebase in open-sourced httpss//gihubb.oo//uu/////// License 2.0.
on LiDAR sensors high-definition maps, VAD enables
Latent TransFuser represents camera-only variant of the TransFuser architecture that operates without LiDAR inputs, relying exclusively on multi-view camera imagery. The method fuses features from both perspective and Bird's Eye View representations using cross-attention mechanisms, enabling robust reasoning about scene layout and agent behaviors. Despite the absence of LiDAR data, this approach achieves competitive performance by leveraging latent representations and hierarchical fusion strategies that preserve the spatial structure and temporal dynamics essential for autonomous driving. The codebase in open-soureed at https://gthhbb..ommmmyyuuuuu//// License 2.0.
with Apache
BC-Safe is implemented by filtering unsafe trajectories and conducting behavior cloning on the E2E driving problem [3T]. We adapt the implementation from the safe imitation learning repositories in wttp:/ehwwwooooeeenneeeettteeeee
with the MIT License.

---

Diffusion baseline is implemented by conditioning the action generation on the history trajectory and RGB images directly [32.
We adopt DDIM implementation from the diffusion policy [62 at http:://hhhh..cmmmmaaaaaaatttttt pollcc with the MIT License.
B Additional Experiments Details
In this section, we provide additional experiment results and algorithm implementation details.
B.1 Additional Environment Description
Evaluation Metries. metrics below:
Following NAVSIM 22 and HUGSIM
13 we describe all evaluation
RC: Route Completion score. It ranges from representing the progress of the ego vehicle before encountering severe failure such collision, off-road driving, deviating far from the preset trajectory, or exceeding the time limit.
NC: Non-Collision score. This evaluates the absence of at-fault collisions between the ego agent and surrounding static or dynamic traffic entities. Since the 3DGS model may erode some objects with inconsistent pointclouds, the NC tends be lower-estimated as some near miss will be counted collision samples due to the reconstruction error of 3DGS.
DAC: Driveable Area Compliance score. Slightly different from NAVSIM, which has ground truth map information, HUGSIM considers violation of the driveable area the ratio of the area within the current ego pose that stays inside the ground point clouds is smaller than 0.3, If this ratio is between 0.3 and 0.5, then the DAC score 0.5. If the ratio is greater than 0.5, the DAC score 1.0.
TTC: Time- To-e-To-Collision score. If the ego agent fails pass collision check within the next 0.5 with surrounding objects (different from 0.9 in NAVSIM), the TTC score is O; otherwise, it is 1.
COM: Comfort score. is if the longitudinal and lateral acceleration, yaw rate, yaw acceleration, and Iongitudinal jerk all fall below their respective thresholds; otherwise, it is O.
HDScore: HUGSIM Driving Score. As introduced in the main text, it is computed based on the above metrics, replacing the Ego Progress in NAVSIM with the Route Completion score:
Scorem X Em (TTC. COM) weight,,, X scorem
HDScore = RC
II
T
Eme(Tic, COM) weight,,.
1=
(NC, DAC)
Dataset Overview. Our environment includes a training split 290 scenes. The distribution of dynamic objects among the training scenes is illustrated below. Most of the scenes have no more than 40 dynamic traffic entities. Yet this already sufficient craft diverse and complex traffic scenes.
Distribution of Controllable Dynamic Objects of the Training Scenes
Number of Scenes
40
30
20
10
0-2
3.5
6-8
9-11
12-14
15-17
18-20
21-23
24-26
27-29
30-32
33-35
36-58
240
Number of Controllable Dynamic Objects
Figure 10: A histogram of the number of dynamic objects in the scenes.
25

---

B.2 Additional Quantitative Results
Additional Ablation Studies As shown in Table 5 and Table we extend the ablation study of MPA to the VAD and LTF base policies. In both settings, the full model with all Q-value guidance and the adapter (ID-6) consistently achieves the highest route completion and HDScore across nominal and safety-critical scenarios. Regarding the adapter, its removal leads to similar performance drops for both VAD and LTT,, suggesting that while helpful, the value model plays more critical role in these cases. Ablating individual Q-components reveals distinct impacts: removing Qroate (ID-1) severely degrades nearly all metrics, especially RC and TTC; removing Qdist (ID-2) significantly harms DAC and HDScore; excluding Qcollision (ID-3) causes agents terminate earlier in safety-critic scenes due to collisions, lowering RC and HDScore; and omitting Qpeeed (ID-4) reduces COM and slightly affects TTC, leading to modest decline in HDScore. These findings about value guidance modules align with our results on the UniAD policy and further validate the generalizability of our MPA design.
Table 5: Ablation Study on MPA's variants on the VAD base policy. Top: Unseen scenes that are nominal but not appearing in the training dataset. Bottom: Safety-critics scenes with adversarial surrounding agents. Bold means the best, and underlined is the best runner-up for each metrics.
Qroute
Qdist
Qcollision
Qspeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
10.2
77.2
94.5
77.1
100.0
7.6
82.2
59.5
86.1
57.5
97.2
47.7
90.6
70.5
94.5
69.0
97.7
60.7
92.2
71.4
94.0
68.9
97.6
61.9
88.9
73.6
94.8
71.2
97.8
62.6
/
90.9
71.0
94.4
68.8
97.7
61.2
ID
Qroute
Qaist
Qualision
Qspeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
14.4
89.2
99.3
84.8
96.4
13.1
V
76.9
72.3
90.6
60.5
99.3
52.8
V
76.1
78.4
99.8
68.6
97.7
55.0
V
98.2
80.9
99.8
70.9
99.3
71.8
98.3
81.6
99.4
71.3
72.2
96.6
79.8
99.0
77.3
97.7
74.7
Table 6: Ablation Study MPA's variants the LTF base policy. Top: Unseen scenes that nominal but not appearing in the training dataset. Bottom: Safety-critical scenes with adversarial
surrounding agents. Bold means the best, and underlined is the best runner-up for each metries.
ID
Qreate
Quist
Qcollision
Qxpeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
11.0
80.0
95.5
79.8
94.9
7.9
85.3
63.4
83.0
58.9
97.2
48.3
89.6
67.6
90.9
65.3
96.9
55.4
4
91.1
66.4
84.7
63.1
96.6
49.8
89.8
69.6
91.7
64.7
94.7
57.9
91.8
68.3
91.0
66.5
96.9
57.0
ID
Qreate
Qata
Qcollision
Qxpeed
Adapter
RC
NC
DAC
TTC
COM
HDScore
7.2
94.1
100.0
92.8
67.5
6.2
V
63.4
74.2
91.4
63.5
99.2
36.5
V
v
59,4
76.1
98.6
61.4
98.9
39.3
r
88.7
74.9
93.9
54.7
89.9
76.9
97.8
70.1
61.7
V
87.3
72.0
94.0
66.9
97.8
56.3
Impact of the Size of Adapter Modalities We evaluate the performance of MPA models over the safety-critical testing environments. The results show that larger sizes of modalities consistently bring benefits to the driving performance in RC, DAC, and HDScore, especially under the smaller mode sizes from to 8. Properly adding the number of modes benefits the closed-loop driving performance.
Statistical Signifieance Analysis. Table demonstratee 3-seed over the safety-critic scenes. The results show reasonably consistent performance of all the methods with low standard deviation
26

---

across different random seeds. Our MPA is still significantly better than the baselines among all the safety-critical scenes.
Table 7: Statistical Significance Results unseen safety-critical scenes. We illustrate the mean and standard deviation of all the metrics tested with three random seeds for all the baseline methods and our approach. All the evaluation metrics are higher the better. Bold means the best, and underlined the best runner-up for each metrics. Based the and standard deviation results among three random seeds, MPA-empowered methods can significantly outperform baselines in RC and
HDScore under the safety-critical scenes.
Model
RC
NC
DAC
TTC
COM
HDScore
UniAD
11.410.05
76.91n022
81.820.38
58.310.39
83.0=9.15
4.3
VAD
23.77
76.59
88.3
69.7+
96. 15.47
+2.9
LTF
34.0+:0.8
81.0++0.9
96.1
72.3+4421
99,4-0.46
22.0+1.69
AD-MLP
4.33
93.0++0.68
96.9=00.7
91.3
70.4+
4. 4.1.00.9
BC-Safe
20.8:1076
19.3
91.7
65.4
95. 65.6-16.27
13.1
Diffusion
26.54:5004
87.1
95.7=12.6
72.6+0.78
92.6=14.88
16.7+3.59
MPA (UniAD)
89. :9.19
79.6::2.66
99.17
70.6:t2.64
97.2:10.62
63.9
MPA (VAD)
97.8
81.8:t:..6
99.2=t0.17
73.8++2.54
98.8=00.55
13.6
MPA (LTF)
19.2
73.4412.46
96.7+1.89
67.3+2.14
97.5+1.17
58.5+230
B.3
Additional Qualitative Studies
We illustrate more qualitative results in both counterfactual data generation and closed-loop evaluation of different E2E driving agents.
More Counterfactual Data Generation Examples. As illustrated in Figure we illustrate some additional counterfactual data generation examples on scenarios of the training split. The results show that the augmented data improves the data coverage, leading to more robust policy and value learning in MPA.
Comparison of Driving Performance of In-Domain and Safety-Critical Scenes. We further visualize the performance MPA and baselines nominal and safety-critical scenarios in Figure [2] and Figure [3 We can observe that the baselines without the value model usually fail to meet the safe constraints, leading catastrophic consequences such collision or getting oof-road.
27

---

Scene-0001
(h) Scene-0004
(c) Scene- 0007
(d) Scene-0008
(e) Scene-0011
(f) Scene-0013
Figure 11: Examples of Counterfactual Data Generation. Red lines represent the ground truth trajectory. Orange lines represent the augmented counterfactual trajectories that are used to train Q-value model and policy adapter. Blue lines represent the other agents' behavior at the scene.
28

---

(UniAD)
MPA (NA)
MPA (LFF)
biffusion
Un
WAD
MPX (UnIAD)
MPA (VAD)
MPX (TT)
Figure 12: Qualitative Results of Scene-0001. Top: In-distribution scenes where ego agents turn left and yield to pedestrians. Bottom: Safety-critical scenes, ego agents turn left and encounter fast-approaching oncoming vehicle. MPA agents take high value actions rather than low value ones.
29

---

Difuucion
(nmiD
(LF)
AU-MUP
BC Bo-Safe
Diffusion
WAD
MPA LOAA
MPA (VAD)
MPA (
Figure 13:
Qualitative Results of Scene-0004. Top: In-distribution scenes where ego agents move forward and cross the intersection. Bottom: Safety-critical scenes with an agent cutting in from the left lane, ego agents need to brake and yield to the cut-in agent. MPA agents take high value actions instead of low value ones.
30

---

C
Broader Impact
The proposed Model-Based Policy Adaptation (MPA) framework enhances end-to-end autonomous driving by improving safety and robustness in complex scenarios through high-fidelity simulation and reward-guides inference. This can accelerate deployment and reduce accidents. However, risks in- clude biased reward models, gaps between simulated and real-world performance, potential misuse for surveillance or adversarial purposes, and privacy concerns from real-world data. Mitigation strategies include gated model release, reward interpretability, simulation validation, and data anonymization.
31


Reviews:
[{"review_id": "tBLW09s4sk", "reviewer": "Reviewer_s9Gr", "summary": "The paper presents Model4 based Policy Adaptation (MPA), a general framework that attempts to fortify pretrained end-to-end driving policies against distributional shift. MPA first uses a simulator to create many counterfactual/ “what-if” driving situations the original data never saw. Using this synthetic corpus, it jointly learns a diffusion-based policy adapter that refines the agent’s raw trajectory proposals and a multi-step Q-value estimator that forecasts the long-horizon utility of each candidate. At inference, the adapter produces multiple feasible trajectories and the Q-value network selects the one with the greatest expected return, thereby delivering safer, more robust driving behavior without retraining the underlying policy.", "questions": "Please see weaknesses section", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper tackles model drift in closed-loop driving, where small deviations compound over time and cause severe distribution shift. The topic is both important and under active investigation.\n\n2. The writing is generally clear, the pipeline is easy to follow, and figures are helpful.\n\n**Weaknesses and questions:**\n\n1. Combining synthetic counterfactual data with a learned value head is well-established in model-based RL. The contribution appears to be an incremental recombination of known pieces rather than a fundamentally new idea. \n\n2. The motivation for all the submodules (simulator, diffusion adapter, Q-network) is not clear.\n\n\n3. Why not have the Q-value network directly on the counterfactual trajectories and select among them, bypassing the diffusion adapter? What advantage does the diffusion adapter offer?\n\n\n4. I missed mention and comparison to key model-based RL literature (e.g., PlaNet, Dreamer, MBRL for autonomous driving).\n\n\n5. Algorithms addressing compounding error by online data aggregation (e.g., DAgger) or by combining imitation with value optimization are absent.\n\n\n6. I missed ablation to some of the components (aligned with comment 2 and 3).\n\n\n7. There are quite some similar recent works aligning with the model-based counterfactual data generation for the same problem setting. These are not cited. \n\n**References:**\n\n1.\tHafner, Danijar, et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint arXiv:1912.01603 (2019).\n\n2.\tHafner, Danijar, et al. \"Mastering atari with discrete world models.\" arXiv preprint arXiv:2010.02193 (2020).\n\n3.\tRoss, Stéphane, Geoffrey Gordon, and Drew Bagnell. \"A reduction of imitation learning and structured prediction to no-regret online learning.\" Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.\n\n4.\tDey, S., Paassen, B., Nair, S. R., Boughorbel, S., & Schilling, A. F. (2024). Continual Learning from Simulated Interactions via Multitask Prospective Rehearsal for Bionic Limb Behavior Modeling. arXiv preprint arXiv:2405.01114.", "was_revised": true, "final_justification": "Based on the discussion with the authors and the constructive exchange with AC, I have decided to maintain my initial rating of 3, as certain components remain insufficiently clear. However, I completely understand and have no objection if the paper is accepted.", "rebuttal": "We express our gratitude to reviewer s9GR for their feedback and for acknowledging our contribution in closed-loop autonomous driving and clear presentation in the methodology pipeline. We provide our response to the questions as below.\n\n**Q1. Regarding the 'incremental recombination of known pieces'.**\n\nWe thank the reviewer for raising this clarification point regarding our major contribution. While pairing synthetic data with a value head is well-explored in MBRL, our proposed MPA addresses an open challenge in the real world problem: photorealistic end-to-end closed-loop driving. MPA boosts HDScore over UniAD/VAD/LTF in a scalable evaluation pipeline using nuScenes dataset. To our knowledge, no prior MBRL or E2E driving work unifies sensor-level counterfactual generation, diffusion-based residual control, and online multi-principle value guidance in a single real-time system. We believe that MPA could offer a meaningful advance in E2E driving problem beyond an incremental recombination.\n\n**Q2 & Q6. The motivation for all the submodules (simulator, diffusion adapter, Q-network) is not clear. The corresponding ablation results (on these submodules) are unclear.**\n\n**Motivation of each module:** We thank the authors for raising this important aspect of our key methodology. \n\n* The adopted simulator HUGSIM provides a systematic benchmark for closed-loop end-to-end autonomous driving, which is missing in the preexisting open-loop evaluation in end-to-end autonomous driving research. The 3DGS model for the training scenes will be used to synthesize our counterfactual observations with action and reward annotations, composing the training dataset of MPA. \n* The diffusion adapter aims to provide diverse residual action proposals to effectively correct the output actions from pretrained base policy, such as UniAD, VAD, etc. \n* The Q-network aims to evaluate the quality of the corrected trajectory proposals from diffusion adapter by different aspects: driveable area compliance, collision avoidance, route following, and speeding constraints. \n\n**Ablation results of each module:** The effectiveness of counterfactual data generation is visualized in Figure 5, where more counterfactual rollout steps yield to better final performance. The ablation studies in Table 3 of our original manuscript show the effectiveness of Q value function guidance and the policy adapter. \n\n**Q3. What advantage does the diffusion adapter offer?**\n\nAs introduced in Q2, the diffusion adapter offers more diverse and high-quality proposals of action correction, which are learned based on the behavior from counterfactual data.  This effect of the policy adapter is demonstrated in the ablation studies of Table 3 in our original manuscript, where method without policy adapter (ID-5) suffered performance degradation compared to the full model, especially under safety-critical scenarios.\n\n**Q4. Discussion and comparison with Model-Based RL approaches.**\n\nWe thank the reviewer for raising the line of model-based RL research. World model indeed plays an important role in end-to-end autonomous driving. We will add more discussion and references in our revised manuscripts with existing MBRL and works like PlaNet, Dreamer as the reviewer mentioned. \n\nBesides the current baselines, we compare our method with a latent world model-based end-to-end driving framework LAW [1]. LAW follows VAD and incorporates additional latent world model loss as an auxiliary regularization when predicting actions during pretraining. We use the output trajectory from LAW as the base action, and further incorporate imagination rollout in MBRL using the same value functions as MPA for LAW, denoted as **LAW-Imagine**. We use LAW's checkpoint from their public codebase, and report the results of LAW-Imagine on HUGSIM in **Table R1.1-R1.3**. \nThe results show that with world model augmentation and added imagination rollout, **LAW-Imagine** can outperform three pretrained base policies UniAD, VAD, and LTF in route completion, and it prevails in non-collision and time-to-collision. However, the comfort of LAW-Imagine is the far worse than the other approaches, which means the agents possibly driving in a zig-zag manner to guarantee safety. As a result, the Route completion and HDScore of LAW-Imagine is not as good as MPA-enhanced base policies. \n\n\n**Q5. Comparison with imitation learning with online data aggretation.**\n\nWe extend our BC-Safe baseline [2] with **online data aggregation** [3], i.e. rolling out new counterfactual dataset based on the trained policy and aggregate them to the next round of policy learning. We conduct one round of such data aggregation on the training dataset, and report the performance of **BC-Safe-DAgger** for both training (in-domain), testing (unseen scenes), and safety-critical scenes in **Table R1.1-R1.3**. The results show that data aggregation helps improve BC-Safe's route completion performance for in-domain setting and safety-critical setting, but has limited improvements in the unseen scenes. \n\n**Table R1.1: Additional in-domain results and comparison with existing base policies with MPA. Bold means the best for individual metrics.**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|57.0|59.8|87.9|55.2|89.4|33.6|\n|**BC-Safe-DAgger (New)**|63.4|64.6|83.3|53.9|94.9|33.0|\n|**LAW-Imagine (New)**|77.3|**84.7**|94.6|**82.6**|41.8|50.9|\n|UniAD|39.4|56.9|75.1|52.1|**98.7**|19.4|\n|VAD|50.1|68.4|87.2|66.1|90.2|31.9|\n|LTF|65.2|71.3|92.1|67.6|98.4|46.7|\n|MPA (UniAD)|93.6|76.4|92.8|72.8|91.8|66.4|\n|MPA (VAD)|**94.9**|75.4|**93.6**|72.5|92.8|**67.0**|\n|MPA (LTF)|93.1|70.8|90.9|67.9|94.9|60.0|\n\n**Table R1.2: Additional results under unseen scenes and comparison with existing base policies with MPA. Bold means the best for individual metrics.**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|59.2|59.8|81.2|56.3|95.9|34.6|\n|**BC-Safe-DAgger (New)**|57.6|61.9|79.0|54.1|94.5|30.5|\n|**LAW-Imagine (New)**|79.5|**79.6**|96.6|**78.3**|38.6|49.5|\n|UniAD|39.3|56.6|74.0|52.6|**98.2**|22.2|\n|VAD|45.4|64.8|86.2|62.0|95.9|29.3|\n|LTF|63.3|64.8|86.5|62.8|**98.2**|41.9|\n|MPA (UniAD)|**93.7**|69.5|92.9|66.6|97.6|60.9|\n|MPA (VAD)|90.9|71.0|**94.4**|68.8|97.7|**61.2**|\n|MPA (LTF)|91.8|68.3|91.0|66.5|96.9|57.0|\n\n**Table R1.3: Additional results under safety-critical scenes and comparison with existing base policies with MPA. Bold means the best for individual metrics.**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|20.2|80.1|91.7|67.3|86.7|13.5|\n|**BC-Safe-DAgger (New)**|35.9|78.4|93.3|66.8|99.7|17.6|\n|**LAW-Imagine (New)**|71.7|**85.1**|98.4|**81.7**|44.8|50.5|\n|UniAD|11.4|76.2|82.1|57.8|95.9|4.5|\n|VAD|25.4|77.0|88.3|73.2|88.4|16.0|\n|LTF|35.1|80.9|96.8|78.1|**100.0**|24.2|\n|MPA (UniAD)|95.1|76.8|98.9|74.2|97.7|70.4|\n|MPA (VAD)|**96.6**|79.8|**99.0**|77.3|97.7|**74.7**|\n|MPA (LTF)|87.3|72.0|94.0|66.9|97.8|56.3|\n\n**Q7. Missing literature review on counterfactual data augmentation in Model-Based RL.**\n\nWe thank the authors for raising these related works, indeed some of the counterfactual data augmentation works contribute to the generalizability of MBRL agents, e.g. [4, 5]. We will add the discussion on them in the related work in our revised manuscript. \n\n\n> [1] Li, Yingyan, et al. \"Enhancing End-to-End Autonomous Driving with Latent World Model.\" ICLR 2025.\n> \n> [2] Pan, Yunpeng, et al. \"Agile autonomous driving using end-to-end deep imitation learning.\" RSS 2018.\n> \n> [3] Ross, Stéphane, et al. \"A reduction of imitation learning and structured prediction to no-regret online learning.\" AISTATS 2011.\n> \n> [4] Pitis, Silviu, Elliot Creager, and Animesh Garg. \"Counterfactual data augmentation using locally factored dynamics.\" NeurIPS 2020.\n> \n> [5] Pitis, Silviu, et al. \"Mocoda: Model-based counterfactual data augmentation.\" NeurIPS 2022."}, {"review_id": "VLGLHMOggr", "reviewer": "Reviewer_hzLc", "summary": "The authors present a system designed to adapt open-loop end-to-end driving agents for closed-loop deployment. The proposed framework comprises three key components: (1) counterfactual data generation using 3DGS, (2) a diffusion-based residual policy adaptor to address observation mismatches, and (3) a Q-value model to capture outcomes. The authors identify objective mismatch as the primary cause of closed-loop performance degradation and validate their approach through experiments on the nuScenes dataset.", "questions": "In addition to the issues mentioned above:\nSince the HDScore is influenced by multiple factors such as RC, NC, and TTC, why is the HDScore particularly high for MPA? Which specific factors contribute most significantly to the reported HDScore?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Quality\nThe paper presents a reasonable idea, and the system’s performance is supported by the released experimental results. However, there are several major concerns.\n(1) The writing lacks clarity, with insufficient explanation of the motivation and component design details.\n(2) The claims in the introduction \"the performance drop in close-loop ealuation stems from observation mismatch and objective mismatch \" \"first mismatch is actually minor in the open-loop evaluation\" are not substantiated by evidence or results in the current version.\n\nClarity\nThe problem definition is clearly stated. Several concerns remain regarding the motivation and component design details. \n(1) the explanation in lines 103–111 is unclear, particularly regarding how the prediction horizon is controlled and how experiments with 3DGS are conducted. \n(2) We cannot find evidence Figure/Table to support the claim \"the fidelity of ... qualicy.\"\n(3) The r(s_t, a_t) and Sim.step is ambiguous, though it can be inferred that HDSim is used. \n(4) the icons used in the MPA column of Figure 4 and in Figure 1 are not well explained, leading to confusion. \n\nSignificance\nThe work is of interest to the industry, and the results demonstrate the effectiveness of the proposed method. Validation on the HDSim environment based on the nuScenes dataset is appropriate. However, the results could be more comprehensive, for example by including experiments on an additional dataset and providing more visualization cases.\n\nOriginality\n\nThe paper proposes a system that integrates previously explored techniques, such as the use of 3DGS for simulating actions and outcomes (as in RAD) and inference-time scaling with customized rewards. The novelty lies in combining these components into a unified framework and applying them to end-to-end driving agents.", "was_revised": true, "final_justification": "Thanks for your detailed response. I have raised my score on clarity due to the additional response.", "rebuttal": "We thank Reviewer hzLc for recognizing the well definition and industry-relevance of our work, as well as the effectiveness of our unified MPA framework on the nuScenes closed-loop benchmarks. We answer each question in turn as follows. \n\n**W1. Regarding insufficient motivation and details on individual components.**\n\nOur MPA composes of three major components: (1) counterfactual data generation, (2) diffusion adapter, and (3) Q value network for inference-time scaling. \n1. The counterfactual data generation aims to provide better data coverage by conducting random sampling based on the behavior policy. The 3DGS model for the training scenes will be used to synthesize our counterfactual observations with action and reward annotations, composing the training data for diffusion adapter and Q value networks. \n2. The diffusion adapter aims to provide diverse residual action proposals to effectively correct the output actions from pretrained base policy, such as UniAD, VAD, etc. \n3. The Q-network aims to evaluate the quality of the corrected trajectory proposals from diffusion adapter by different aspects: driveable area compliance, collision avoidance, route following, and speeding constraints. \n\n**W2 & W4. Some claims are not substantiated by evidence or results in the current version.**\n\nWe clarify the links between our claims and the accompanying results below:\n\n1. **\"The performance drop in closed-loop evaluation stems from *observation mismatch* and *objective mismatch*.\"** \n\n   *Evidence:* This statement follows directly from the decomposition in **Eq. (1)**, which analytically separates the two sources of error. The objective-mismatch term captures the gap between the training loss and the online reward, while the observation-mismatch term accounts for perceptual discrepancies introduced by the simulator.\n\n2. **\"…the first mismatch is actually minor in the open-loop evaluation.\"** \n\n   *Evidence:* Figure 2 supports this point: prediction errors for policies using **ground-truth** visual inputs and those using **3DGS-rendered** inputs are nearly identical, indicating that observation mismatch has negligible impact under open-loop evaluation.\n\n3. **\"the fidelity of the 3DGS-based simulation in its reconstruction quality.\"** \n\n    *Evidence:* Figure 2 validates that the reconstructed data has minor impact on the prediction error of end-to-end autonomous driving policies, such as UniAD, VAD, and LTF. We further evaluate the rendering quality of novel view synthesis. We can see that the FID scores of 3DGS rendered images are around 20 when the view pose has small lateral displacements with the ground truth trajectories. \n\nWe will revise the manuscript to make the presentation more coherent between the analytical derivation and the experimental validation.\n\n**W3. In the preliminary experiment, how is the prediction horizon controlled and how are experiments with 3DGS conducted?**\n\nWe appreciate the reviewer’s request for clarification.\n\n**Prediction horizon.** Each base policy produces a sequence of future way-points spanning 3 s. To create the horizons shown in Fig. 2, we truncate that sequence after 1, 2, … H steps and compute the L2 error between the retained way-points and the ground-truth trajectory. Thus the horizon is controlled entirely by how many of the already-predicted way-points we evaluate—no additional forward passes are required.\n\n**3DGS experiments.** Following the HUGSIM protocol, we train a separate 3D Gaussian-Splatting (3DGS) model for every scene. Ground, static background, and dynamic objects are reconstructed as three disjoint Gaussian sets and composited at render time. In the preliminary evaluation, we run each policy twice: (1) on the original nuScenes camera images and (2) on the images rendered by the corresponding 3DGS model. Then we measure prediction errors with the same horizon-truncation procedure described above.\n\n**W5. What is the meaning of $r(s_t, a_t)$ and `env.step()` in the pseudo code?** \n\nWe apologize for the insufficient explanation in the current manuscript. Based on our MDP's definition in section 2.1, $r(s_t, a_t)$ is the reward feedback that HUGSIM returns to evaluate the performance based on the current driving states and actions. `env.step()` is called to update the next vehicle states as well as the corresponding visual observations with HUGSIM's 3DGS rendering engine. The collected counterfactual dataset is composed of (s, a, o, r), including the state action transition pairs, visual observations and the reward labels. They will later be used to train the policy adapter and value model. \n\n**W6. Icon explanation in Figure 1 and 3.**\n\nWe thank the reviewer for pointing out this concern. The icons in Figure 1 and 3 include the driving actions (throttle and steering), synthetic counterfactual dataset, frozen base E2E policies, as well as the trainable diffusion adapter and value networks. We will add a legend in our revised manuscript to help the audience better understand our pipeline details. \n\n\n**W7. Additional visualization and more comprehensive results on different datasets.** \n\n**Additional visualization.** Beyond Figure 4, we offer additional visualizations for all nine evaluated methods across four distinct scenarios in Appendix Figures 11 and 12. Each scenario is illustrated with four representative key frames, covering both safety-critical and non-safety-critical traffic contexts. We will include more qualitative examples in the appendix for our final revised version.\n\n**Additional baseline comparison.** We add additional baseline results on imitation learning with data aggregation (BC-Safe-DAgger), and model-based RL agent (LAW-Imagine). \n* DAgger [1] is an online imitation learning approach that extends our BC-Safe baseline in aggregating new online rollout data. We observe that BC-Safe-DAgger performs better compared to BC-Safe, particularly in the safety-critical scenes. But it still cannot outperform our proposed MPA (reported in Table 1 in the original manuscript). \n* LAW [2] follows VAD and incorporates additional latent world model loss as an auxiliary regularization when predicting actions during pretraining. We use the output trajectory from LAW as the base action, and further incorporate imagination rollout in MBRL using the same value functions as MPA for LAW, denoted as **LAW-Imagine**. We use LAW's checkpoint from their public codebase, and report the results of LAW-Imagine on HUGSIM in Table R1. The results show that with world model augmentation and added imagination rollout, **LAW-Imagine** can outperform three pretrained base policies -- UniAD, VAD, and LTF, but is still not as good as MPA (reported in Table 1 in the original manuscript) due to the significant drop in the comfort of the driving behaviors. \n\n**Table R1.1: Additional in-domain results**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|57.0|59.8|87.9|55.2|89.4|33.6|\n|**BC-Safe-DAgger**|63.4|64.6|83.3|53.9|94.9|33.0|\n|**LAW-Imagine**|77.3|84.7|94.6|82.6|41.8|50.9|\n\n**Table R1.2: Additional results under unseen scenes**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|59.2|59.8|81.2|56.3|95.9|34.6|\n|**BC-Safe-DAgger**|57.6|61.9|79.0|54.1|94.5|30.5|\n|**LAW-Imagine**|79.5|79.6|96.6|78.3|38.6|49.5|\n\n**Table R1.3: Additional results under safety-critical scenes**\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|BC-Safe|20.2|80.1|91.7|67.3|86.7|13.5|\n|**BC-Safe-DAgger**|35.9|78.4|93.3|66.8|99.7|17.6|\n|**LAW-Imagine**|71.7|85.1|98.4|81.7|44.8|50.5|\n\n**Additional dataset evaluation.** Besides additional baselines, we also evaluate MPA’s effectiveness on the Waymo dataset as well as on nuScenes. The Waymo dataset contains only three front-facing camera views, whereas nuScenes provides six. Consequently, we report the zero-shot performance of the nuScenes-trained LTF and MPA (LTF) models, which accept three-camera inputs. We evaluate on the first 40 scenes of the “NeRF on the Road” (NOTR) dataset used in EmerNeRF [3]. The results below demonstrate that MPA generalizes well and delivers consistent performance gains on datasets beyond nuScenes.\n\n\n|Method|RC|NC|DAC|TTC|COM|HDScore|\n|-|-|-|-|-|-|-|\n|LTF|0.48|0.67|0.54|0.62|0.61|0.24|\n|MPA (LTF)|0.71|0.76|0.65|0.71|0.71|0.37|\n\n\n\n**Q1. Which specific factors contribute most significantly to the reported HDScore?**\n\nAs we stated in the paper, the HDScore is computed as: \n\n$$\n\\\\text{HDScore}=\\\\text{RC} \\\\times \\\\frac{1}{T}\\\\sum_{t=0}^T \\\\Big\\\\{ \\\\prod_{m\\\\in \\\\{\\\\text{NC, DAC}\\\\}} \\\\text{score}_m \\\\times  \\frac{\\sum\\_{m \\\\in \\\\{\\\\text{TTC, COM}\\\\}}  \\\\text{weight}_m   \\\\times \\\\text{score}_m}{\\sum\\_{m\\\\in \\\\{\\\\text{TTC,COM} \\\\}} \\\\text{weight}_m} \\\\Big\\\\}_t.\n$$\n\nOur MPA method prevails baselines particularly in the **Route Completion (RC)** and **Driveable Area Compliance (DAC)** metrics, which leads to a higher HDScore. Please kindly check Tables 1 and 2 in our original manuscript for more details. \n\n> [1] Ross, Stéphane, et al. \"A reduction of imitation learning and structured prediction to no-regret online learning.\" AISTATS 2011\n> \n> [2] Li, Yingyan, et al. \"Enhancing End-to-End Autonomous Driving with Latent World Model.\" ICLR 2025.\n> \n> [3] Yang, Jiawei, et al. \"EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision.\" ICLR 2024"}, {"review_id": "7WCWGX6EvP", "reviewer": "Reviewer_2ZCm", "summary": "This paper addresses the performance degradation of end-to-end (E2E) autonomous driving models when transitioning from open-loop to closed-loop evaluation. The authors identify a fundamental mismatch is the objective mismatch due to the absence of reward feedback during offline imitation learning. To address that, they propose Model-Based Policy Adaptation (MPA), which consists of three components: counterfactual data generation using 3D Gaussian Splatting (3DGS) simulation, a diffusion-based policy adapter that predicts residual actions, and a multi-step Q-value model for inference-time action selection. The models are trained on nuScenes data and evaluated via HUGSIM simulation engine. Results show that the proposed MPA can outperform the baseline by a large margin, especially in the safety-critical scenes.", "questions": "1. In the qualitative results in Fig.4, it seems that MPA is not yielding/breaking compared to the VAD baseline according to the simulated surroundings (the position of road mark / pole, etc.) unless the images are not \"simulated on the same timestamp\". Could you be more specific about how you obtained the qualitative results?\n2. According to the original HUGSIM paper, UniSIM generally performs better than VAD on HDScore, which is contradictory to the authors results. Could you explain why it is the case?\n3. The weighted sum of the value model seems ad-hoc to me, could the authors explain how did these weight coefficient to be determined?\n4. What is the computation overhead for adding the value model and adapter during inference?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. While individual components are not novel, their combination for addressing the open-loop to closed-loop gap in E2E driving is original, particularly the use of counterfactual data generation with 3DGS simulation. \n2. The experimental evaluation is thorough, covering in-domain, out-of-domain, and safety-critical scenarios with consistent improvements across multiple base policies (UniAD, VAD, LTF). \n3. The overall methodology is clearly presented with good visual aids (Figures 1 and 3 effectively illustrate the approach). \n4. The work provides immediate practical value by showing how to improve existing E2E driving models' deployment safety, with HDScore improvements by a large margin.\n\nWeaknesses:\n1. The counterfactual data generation relies heavily on teacher-forcing, which may not explore sufficient behavioral diversity to handle truly out-of-distribution scenarios. \n2. The claim that observation mismatch is minor lacks convincing evidence—Figure 2 only addresses motion prediction, not visual fidelity. 3. The Q-value decomposition into four specific components appears arbitrary without ablation on alternative decompositions or theoretical justification. \n4. Computational costs are not discussed. \n5. For safety-critical scenes, only 10 scenes are used for evaluation, whether this is representative is questionable.", "was_revised": true, "final_justification": "I appreciate the authors' detailed response and decide to keep my score as accept.", "rebuttal": "We thank Reviewer 2ZCm for the thoughtful and constructive feedback, and for acknowledging the clarity, practical value, and strong empirical performance of our MPA framework in bridging open-loop and closed-loop end-to-end driving. We address each question as follows.\n\n**W1.The counterfactual data generation relies heavily on teacher-forcing, which may not explore sufficient behavioral diversity to handle truly out-of-distribution scenarios.**\n\nWe thank the reviewer for this insightful question. The key goal in our counterfactual generation is to **balance the diversity of the sampled trajectories and the fidelity of the rendered images.**\n\n* For the **diversity** of the data, we add random noise by scaling and rotating the predicted trajectories from the base policy. In this way, we can collect the state and action transition pairs under different future motions in the next few frames. We also conduct an ablation study on the counterfactual rollout steps in Figure 5 in our original manuscript. Longer counterfactual rollout steps bring in better diversity in the dataset. We can see that the performance of MPA increases significantly from step 0 to 3, and then converges around 4 and 5 steps. This suggests the current behavioral diversity in the counterfactual dataset is sufficient to train robust policy adapters and Q value models. \n* Meanwhile, for the sake of rendering **fidelity**, we need to constrain the divergence of the sampled counterfactual trajectories. As visualized in Figure 6 in our supplementary material, the quality of the 3DGS-rendered images degrades (FID gets larger than 50) as the lateral distance to the ground truth trajectories gets too large. To avoid this, we will truncate the samples that violates the distance or reward threshold during the counterfactual data generation phase, as included in Algorithm 1 in our main text. \n\n**W2. About the visual fidelity to support the observation mismatch claim.**\n\nHUGSIM allows us to script diverse, photorealistic behaviors for dynamic objects, enabling us to synthesize realistic safety-critical interactions and mitigate the scarcity of real-world interaction-rich data.\n\nAs shown in Fig. 6 of the supplementary material, HUGSIM renders high-fidelity images when the synthesized viewpoint remains close to the ground-truth path (FID ~20). Visual quality degrades only as the lateral deviation grows: once the offset exceeds ~0.8 m the FID can surpass 50. Such large deviations occasionally arise when the base policies (UniAD, VAD, LTF) drift during closed-loop rollouts. \n\nAccordingly, when producing counterfactual data, we will constrain the lateral displacement, ensuring every rendered observation remains photorealistic and avoids perception noise that could degrade MPA's final performance.\n\n\n\n**W3 & Q3. Further explanation on the Q-value decomposition into the specific components, as well as the weighted sum of different terms.**\n\nWe follow the prior reward recipes in [1] to decompose the essential value function terms in autonomous driving. Note that some of these terms are under different scales in our case. For instance, the route completion is 0 to 1 across all timesteps, and the difference between different timesteps will be very small. Distance to the driving lane has a unit of meters (m). The speed is m/s, and collision is a binary term of 0 or 1. \n\nIn order to select the value coefficients, we zoom into the empirical distribution of these individual reward term among the training data, then balancing them by timing them approximately by the reciprocol of their mean, i.e. $w_\\text{collision} \\approx 1/\\text{mean}(r_\\text{collision})$, forming the final weights coefficients as presented in the hyparameter table in appendix Table 4. \n\n**W4 & Q4. Computational costs are not discussed.**\n\nWe thank the reviewer for raising these valuable points. We ran our experiment on 2xNVIDIA RTX A5000 GPUs for 1000 rollout steps, and report the average time consumption among different modules in MPA: \n\n**Table R1. Inference overhead analysis on MPA.**\n|Component|Time(s)|#Params|\n|-|-|-|\n|Base Policy (UniAD)|0.67|131.81M|\n|Base Policy (VAD)|0.34|58.36M|\n|Base Policy (LTF)|0.33|56.04M|\n|Diffusion Adapter|0.09|12.26M|\n|Q Value Net|0.07|13.84M|\n\nCompared to the inference speed of base policy, the overhead of the additional parameters from MPA is small (a total of 0.16s, <25\\% inference time overhead for UniAD, <50\\% for VAD and LTF), while boosting up the final performance significantly given a considerably large amount of counterfactual samples and multi-step Q value feedback. We will explore optimizing more efficient (parallel) rollout for MPA in the future. \n\n\n**W5. Limited number of safety-critical scenes.**\n\nWe thank the reviewer for highlighting this concern. Generating diverse, safety-critical scenarios remains challenging because real-world crash data are scarce and reliable ego policies for automated stress-testing are still under development. Even without explicitly adversarial scenes, we find that state-of-the-art E2E policies (UniAD, VAD, LTF) frequently fail in **closed-loop rollouts**. Since the main focus of this paper is to first train a robust end-to-end policy, we manually design 10 safety-critical scenarios in the scenes in Singapore from nuScenes dataset.\n\nThe current ten safety-critical scenes we curated from HUGSIM also align with other recent benchmarks such as NeuroNCAP [2], where they evaluate the ego policy in five front scenes and five side scenes from the nuScenes dataset. \n\nIn the future, we intend to use MPA as a robust ego policy (i.e. \"victim policy\") that is ready for a more scalable safety-critical scenario generation pipeline in the end-to-end autonomous driving research.\n\n**Q1. Clarification on the qualitative results in Figure 4.**\n\nWe thank the reviewer for raising clarification on our important qualitative results. \nIn Figure 4, we presented two scenarios in two rows. The three columns show the non-safety-critical scenes (left), VAD rollout results under safety-critical scenes (middle), and MPA-VAD rollout results under safety-critical scenes (right). We only pick 2 key frames (pre-crash, and crash frame) for each scenario due to the space limit, highlighting the different outcomes of VAD (crashing) and MPA-VAD (safe) at the end of the evaluation episode. We further illustrate more comprehensive 4-frame qualitative results in Figure 11 and 12 in our supplementary materials. Please kindly check them if interested. \n\n**Q2. According to the original HUGSIM paper, UniAD generally performs better than VAD on HDScore, which is contradictory to the authors results. Could you explain why it is the case?**\n\n\nWe believe the difference stems primarily from the evaluation set.\n\n* **Scope of scenes**. The HUGSIM paper reports results on a small, high-quality subset of nuScenes (one scene in their initial release and eight scenes in their latest). To obtain broader coverage, we trained 3DGS reconstructions for more than 360 nuScenes scenes and evaluated on **290 training and 80 for testing** (including safety-critical) cases.\n* **Rendering fidelity and perception sensitivity.** While this larger set offers greater diversity, some of the newly reconstructed scenes could lead to lower perception quality than the hand-selected ones in the HUGSIM release. Compared to UniAD, VAD consumes a vectorized BEV scene representation and is more robust, thus ranking higher on our expanded evaluation set.\n\nDespite these differences, MPA boosts both UniAD and VAD on our extensive benchmark and the original HUGSIM scenes. We will clarify this point in the revision and explore a better balance between scene diversity and rendering fidelity in future work.\n\n\n> [1] Chen, Jianyu, Bodi Yuan, and Masayoshi Tomizuka. \"Model-free deep reinforcement learning for urban autonomous driving.\" IEEE ITSC 2019.\n> \n> [2] Ljungbergh, William, et al. \"Neuroncap: Photorealistic closed-loop safety testing for autonomous driving.\" ECCV 2024."}, {"review_id": "4328FVApFU", "reviewer": "Reviewer_7qBw", "summary": "The paper presents Model-based Policy Adaptation (MPA), a framework that improves the robustness of end-to-end driving policies in closed-loop settings. The submission first analyses issues of such driving models that occur under distribution shifts of closed-loop simulation. Moreover, a data generation pipeline is proposed that unrolls a reference policy on reference trajectories in a simulator based on 3D Gaussian Splatting (3DGS) to generate novel observations. This counterfactual data is utilized to (1) train a diffusion-based policy adapter for the trajectories and (2) an action-value model that guides the adapter during inference. Experiments using the HUGSIM simulation tools and the nuScenes dataset demonstrate the benefits of the proposed robustness strategy with several reference policies and ablate relevant design choices.", "questions": "- Could you comment on “…increasing the planning frequency and shortening the planning horizon cannot necessarily close the gap between open-loop and closed-loop performance, unless some effective feedback guidance is provided to the E2E agents.” in Section 2.2? I am not sure if that is a clear consequence from the analysis. \n- Are the reference trajectories simply the ground truth states of the ego vehicle, sampled at different time steps in the scene? Using the word “reference” for both the policy and the recorded dataset might not be ideal.\n- Could you provide some intuition or qualitative result on why LTF benefits less from the robustness strategy compared to UniAD and VAD (despite being more robust without MPA)? Is this related to the smaller intermediate representation? \n- Could you comment on the additional compute requirements of MPA (i.e. with the diffusion adapter and value prediction)? Do these modules substantially increase run-time?  \n\nI am quite positive about this work. I am open to increasing my overall score, if unclarities and questions can be sufficiently resolved.", "limitations": "The submission discusses limitations. The authors might want to consider commenting on the compute requirements, as such methods would need to be very efficient in a real-world setting. Moreover, I personally think that long simulations (i.e. in the synthetic CARLA simulator) might be required to support some claims regarding long-term planning benefits with the Q value model. Though, I respect that this is beyond the scope of this project.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The topic is highly relevant and well motivated in the submission. With end-to-end driving models being more and more used in industry and academia, addressing the closed-loop robustness is a promising field. The proposed framework does not pose compatibility constraints to the specific driving policy. The work appears highly valuable to the community due the quality of the work and tools that are likely open-sourced. \n- The paper is mostly well written. Specifically the experimental study is well structured and clear to follow. It is apparent from the paper and supplementary pdf that thorough work was required for the project.  \n- The experiments are conducted with closed-loop simulation based on state-of-the-art sensor simulation (i.e. 3DGS).  The claims are well supported. The authors discuss limitations of the approach. \n\nWeaknesses:\n- My main issue with the submission is that I do not find the counterfactual data generation procedure clearly described. This negatively impacts the understanding of the downstream diffusion adapter and value learning modules. I find it somewhat unclear how the reference trajectories are constructed, how the reference policy is applied (and on which time steps), and how the reward is calculated. A figure or more precise notation might be helpful. For example, Figure 3 provides a BEV overview where the colors are not defined, the trajectories are not clearly visible, and the procedure is not shown over the different timesteps. \n- I am uncertain how much value Section 2.2 (and partially Section 2.1) provides in the paper. It is well known that behavioral cloning suffers from distributional shifts and compounding errors, thus the section does not provide much value to the reader. I recommend shortening these parts or providing further insights or qualitative examples instead.", "was_revised": true, "final_justification": "I recommend to accept this paper. The topic is highly relevant, and the paper has a good evaluation setup and strong results.", "rebuttal": "We sincerely thank reviewer 7qBw for their insightful and inspiring feedback. We are glad to know the reviewer recognizes the motivation and community value of our work, the generality of the MPA framework, as well as its strong empirical performance. We provide our response to the questions below.\n\n**W1. Clarification on the counterfactual data generation procedure.**\n\nWe thank the reviewer for raising this important clarification question. We will make clarification on each question as follows. \n\n\n* **W1.1 (Q2) How are the reference trajectories constructed? Are they simply the ground truth states of the ego vehicle? How the reference policy is applied (and on which time steps)?** \n\nThe reference policy is not the ground truth states of ego vehicle. Instead, they are constructed by adding random noise to the base policy's prediction (UniAD, VAD, LTF). In practice, we conduct geometric transformation by rotating the trajectories with a random angles uniformly sampled from -5 degree to 5 degree, and also scaled the longitude motions with random scales from 0.1 to 2.0. \n\nThe the counterfactual data generated by this reference policy consists of over 20 counterfactual actions under the same states, and their reward will be annotated by the underlying states from HUGSIM during training time. Similar techniques are adopted by the PDM controller in [1]. \n\nNote that this 'reference policy' is only applied during training time to help collect counterfactual datasets. We will modify the term 'reference policy' to 'behavior policy' in our revised manuscript to avoid further misunderstanding. \n\n\n* **W1.2 How is the reward calculated?** \n\nThe reward is calculated by the underlying states from HUGSIM simulator. It is a weighted sum of route completion, lateral distance, speed, and the collision penalty. We attach a more detailed description of the reward definition in Appendix Section A.1. \n\n\n* **W1.3 A figure or more precise notation might be helpful. For example, Figure 3 provides a BEV overview where the colors are not defined, the trajectories are not clearly visible, and the procedure is not shown over the different timesteps.**\n\nWe thank the reviewer's suggestion in our visualization and will revise it to be more clear in our revised manuscripts. Currently, the BEV map we used is the identical as the direct BEV map prediction from our base policies, and the one in Figure 3 is the visualization from UniAD. For every step of rollouts, the BEV map will be updated, and the model will take in the most recent observations to correct the actions. We will add a multi-step demonstration to clearly demonstrate the inference-time pipeline of MPA. \n\n\n**W2. Regarding reorganizing on section 2.1 and 2.2.**\n\n\nWe thank the reviewer for giving this valuable suggestion on our paper writing. While compounding errors is widely acknowledged in RL community, prior end-to-end driving methods fail to resolve it due to the lack of high-fidelity counterfactual data generation and robust policy adaptation algorithm. We will shorten the section in our revision. \n\n\n\n**Q1. Clarification on the claim: \"…increasing the planning frequency and shortening the planning horizon cannot necessarily close the gap between open-loop and closed-loop performance, unless some effective feedback guidance is provided to the E2E agents.\"**\n\nWe thank the reviewer for the careful reading and thoughtful feedback on our presentation. We mainly draw this claim from Figure 2, where the closed-loop trajectory prediction yields to compounding error as the planning horizon enlarges. We also notice from Figure 2 that the L2 prediction error is non-neglegible **even under small prediction horizon** (0.5s\\~1s). This suggests that **behavior cloning loss is not a strong supervision signal** even for a short-horizon policy. Then we propose to use other guidance signal (such as reward in our counterfactual dataset) to help correcting the prediction online. We will add more elaboration between the analysis on Figure 2 and this claim to make the presentation more coherent. \n\n**Q3. Qualitative studies on the LTF's performance gain from MPA.**\n\nWe thank the reviewers for their thoughtful feedback. Since no images are allowed in the response, we cannot visualize the heatmap of predicted trajectories, but we'll add them in our revised version. **Alternatively**, in the tables below, we analyze the statistics in the driving behaviors of LTF, UniAD, VAD, and MPA correction effect on them. \n* We quantify the diversity in both **longitude** and **lateral** directions, for both in-distribution and unseen scenarios. We can see a clear difference in the longitude prediction's standard deviation between LTF and other methods without MPA. The metrics are reported with the future waypoints in 3 seconds with a unit of meter. In both tables, LTF base model is **more agile with higher standard deviation** compared to UniAD and VAD in the longitude direction, which demonstrates their capability to accelerate/decelerate to avoid collision or getting into other safety-critical conditions.\n* We also notice for all the methods, after MPA's correction, their standard deviation in longitude (and most lateral, except for VAD in OOD scenes) predictions will increase. This partly reveals some internal mechanism in how the MPA improves the behavior quality: it gives the base model **better agility** in both longitude and lateral movements.\n* Although MPA benefits LTF, the beneficial effect is not as large as the others. Since we use the same random trajectory transformation during the counterfactual data generation perspective, this may lead to worse coverage in LTF in the whole trajectory space, since the base action in LTF is already quite diversified.\n\n**Table R1.1 Standard deviation in in-distribution scenarios. Without MPA, LTF already has higher longitude behavior diversity compared to the other base policies.**\n|Method|Longitude (w/o MPA)|Longitude (w/ MPA)|Lateral (w/o MPA)|Lateral (w/ MPA)|\n|-|-|-|-|-|\n|UniAD|2.78|9.46|1.10|1.53|\n|VAD|1.53|9.16|0.82|1.28|\n|LTF|**8.77**|11.91|1.12|1.49|\n\n**Table R1.2 Standard deviation unseen scenarios, Without MPA, LTF already has higher longitude behavior diversity compared to the other base policies.**\n|Method|Longitude (w/o MPA)|Longitude (w/ MPA)|Lateral (w/o MPA)|Lateral (w/ MPA)|\n|-|-|-|-|-|\n|UniAD|2.64|10.74|0.99|1.17|\n|VAD|1.45|10.35|1.53|1.41|\n|LTF|**9.52**|12.86|1.06|1.32|\n\n\nWe also quantify the distribution of output action corrections $\\Delta \\hat{a} \\approx a^*-a_{base}$. In the following table, we quantify the standard deviation of lateral behavior correction from MPA in both in-distribution (ID) and unseen (OOD) scenes. MPA(LTF) has the highest deviation in lateral correction, which leads to more unstable steering actions. This potentially reduces the final performance in driveable area compliance (related to DAC) and distance keeping (related to NC and TTC) compared to MPA (UniAD) and MPA (VAD).\n\n**Table R2. MPA Correction Analysis. MPA (LTF) has higher lateral deviation compared to the other two methods.**\n|Method|Lateral Correction (ID)|Lateral Correction (OOD)|Steering (ID)|Steering (OOD)|\n|-|-|-|-|-|\n|MPA (UniAD)|1.00|1.02|0.12|0.12|\n|MPA (VAD)|0.71|1.12|0.14|0.18|\n|MPA (LTF)|**1.32**|**1.42**|**0.18**|**0.19**|\n\n**Regarding compact representation**: Given our current empirical results, there is no clear evidence that the intermediate representation of LTF has significant harm in the final performance of MPA. This intermediate representation will be fed to the policy adapter, while the Q-value models have similar capacity as they take in the raw camera observations. \n\n\n**Q4. Discussion on the computation efficiency.**\n\nWe ran our experiment on 2xNVIDIA RTX A5000 GPUs for 1000 rollout steps, and calculate the average time comsumption over different modules in MPA: \n\n**Table R3. Inference overhead analysis on MPA.**\n|Component | Time(s) | #Params |\n|-|-|-|\n|Base Policy (UniAD) | 0.67 | 131.81M | \n|Base Policy (VAD)| 0.34  | 58.36M |\n|Base Policy (LTF)| 0.33  | 56.04M |\n|Diffusion Adapter | 0.09 | 12.26M | \n|Q Value Net | 0.07 | 13.84M |\n\nCompared to the inference speed of base policy, the overhead of the additional parameters from MPA is small (a total of 0.16s, <25\\% inference time overhead for UniAD, <50\\% for VAD and LTF), while boosting up the final performance significantly given a considerably large amount of counterfactual samples and multi-step Q value feedback. We will explore optimizing more efficient (parallel) rollout for MPA in the future. \n\n> [1] Dauner, Daniel, et al. \"Parting with misconceptions about learning-based vehicle motion planning.\" Conference on Robot Learning. PMLR, 2023."}]

Author rebuttal:

Meta-review: This paper introduces Model-based Policy Adaptation (MPA), a framework to improve the robustness of end-to-end autonomous driving policies under distributional shift. A key challenge arises from objective mismatch, as offline imitation learning lacks reward feedback, leading to degraded closed-loop performance. To address this, MPA leverages counterfactual data generation using 3D Gaussian Splatting (3DGS) to create novel driving scenarios unseen during training. A diffusion-based policy adapter refines trajectory proposals by predicting residual corrections for the pretrained policy. In parallel, a multi-step Q-value model evaluates long-horizon outcomes to select the safest and most effective trajectory at inference. The combined system adapts open-loop policies for closed-loop deployment without retraining the original backbone. Experiments demonstrate substantial performance gains, particularly in safety-critical situations.  

The rewievers were all very interested in the problem and the proposed approach. They asked many questions, in particular about the validation of the method based on the various metrics proposed and the types of scenarios.
At the end of the rebuttal, where the authors provided many detailed answers and clarifications on their approach, further discussions with the AC cleared up some misunderstandings, and even if some reservations remain on the clarity of the gains that do not emerge for all the proposed metrics, the AC proposes to accept this submission, following the very positive opinions of 3 of the 4 Rs.
Decision: Accept (poster)