Title: Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous Driving

World Model

Policy Adaptation
Abstract: End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap, we propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains a diffusion-based policy adapter to refine the base policy’s predictions and a multi-step Q value model to evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain, and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.

Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving

Abstract

End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap, we propose Modelbased Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain, and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.

1 Introduction

End-to-End (E2E) autonomous driving models have made impressive strides by integrating perception, prediction, and planning into a unified learning framework [1, 2, 3, 4]. Leveraging large-scale offline driving datasets, E2E models perform well under open-loop evaluation protocols, where the agent passively predicts future behaviors from offline recorded observation sequences. However, these models degrade in closed-loop environments, where minor deviations accumulate over time, leading to compounding errors, distribution shifts, and poor generalization to long-horizon scenarios. This performance gap reveals a core challenge: offline training based on empirical risk minimization does not align with the online objective of maximizing cumulative reward, as is illustrated in Figure 1.

To bridge this gap, recent efforts have turned to evaluating the closed-loop performance of E2E agents. Some open-loop such as NavSim proposes the approximate closed-loop evaluation with a Predictive Driver Model Score in an open-loop evaluation fashion. Other works introduced sensor simulation for the closed-loop evaluation, generating camera views based on diffusion models [5, 6], Neural Radiance Field (NeRF) [7, 8, 9, 10] or 3D Gaussian Splatting (3DGS) [11, 12, 13, 14, 15] that enable photorealistic rendering of novel viewpoints. These tools provide fine-grained control over agent interventions and visual realism, making them promising testbeds for studying failure modes and recovery strategies. Existing works such as V AD [4], V AD-v2 [16], and Hydra-MDP [17] design different scoring mechanisms to select the predicted motions for closed-loop control, yet these works either lack closed-loop evaluation results or are evaluated in a non-photorealistic simulator like CARLA [18]. Most recently, RAD [19] incorporates reinforcement learning and uses 3DGS for online rollouts and evaluation, while the training of PPO agents can be costly, and the value

≈

≈

E2E Driving Agent

Multi-step Q-Value Model

!

!

❄

E2E Driving Agent

❄

Counterfactual Data

MPA Inference

Q-Value

E2E

Adapter

➕

Open-Loop:

Lowest MSE

E2E Driving Agent

!

Open-Loop:

Camera Sensor

Closed-Loop:

3DGS Rendered

Closed-Loop:

Highest Reward

critic is left unused at inference time. Among all the prior attempts, none of the works have curated counterfactual data into consideration during the training phase.

Our goal in this paper is to adapt the pretrained open-loop E2E driving agents from the real domain to safe and generalizable closed-loop agents, with a 3DGS-based driving simulation data engine. We identify that the performance drop between the closed-loop and open-loop evaluations stems from two fundamental sources: (1) Observation mismatch - the shift between training-time sensor inputs and deployment-time closed-loop observations under perturbed behaviors from different data engines; (2) Objective mismatch - the absence of meaningful reward feedback during offline imitation learning, which limits long-term reasoning.

We conducted preliminary experiments to demonstrate that the first mismatch is actually minor in the open-loop evaluation. Then we propose a unified solution called Model-Based Policy Adaptation (MPA) , a general framework that directly addresses both mismatches by separating and targeting their root causes. We first use the pretrained policy as an initialization of behavior policy to generate a counterfactual dataset using a high-fidelity 3DGS simulation engine. To mitigate observation mismatch, we design a diffusion-based residual policy adapter that conditions on diverse, counterfactual trajectories. This exposes the policy to a broad range of behaviors and visual scenes beyond those seen in the offline dataset. To address objective mismatch, then learn a Q-value model from the same counterfactual data that captures long-horizon outcomes and enables value-based assessment beyond rule-based metrics. MPA uses both components at inference time: the policy adapter generates residual action proposals conditioned on the current observation, and the value model performs inference-time scaling to select the action with the highest expected utility.

Our contributions in this work are threefold:

We analyze the root causes of closed-loop performance degradation in E2E agents and assess the fidelity of 3DGS-based simulation for modeling observation and behavior shifts.

We develop a systematic counterfactual data curation pipeline using 3DGS rollouts and train the MPA with a diffusion-based policy adapter and reward model to address observation and reward mismatches, respectively.

We demonstrate that inference-time scaling using the learned reward model significantly improves closed-loop performance on the nuScenes benchmark, particularly in safety-critical and out-ofdomain scenarios.

2 Preliminary

2.1 Problem Formulation of End-to-End Autonomous Driving

We formulate closed-loop end-to-end (E2E) driving as a Partially Observed Markov Decision Process (POMDP) M = ( S , A , P, R, O , γ, T ) , where S is the latent state space, A the action space, P the transition dynamics, R the reward function, O the observation space, γ the discount factor, and T the planning horizon. At each timestep, the agent receives an observation o t ∈ O and outputs a trajectory action a t ∈ A . The environment evolves according to P ( s t +1 | s t , a t ) and emits observations via

P obs ( o t | s t ) . In practice, the state s t includes the ego vehicle's IMU status and surrounding road entities' past poses and motion intents. These are often only partially observable. The action a t represents a sequence of future waypoints, which is translated into low-level throttle and steering control sequences via an LQR controller [20], following prior benchmarks [21, 22, 5, 13]. The observation o t is captured by real sensors or rendered by a simulation engine during closed-loop evaluation. Notably, current open-loop E2E agents are trained using expert trajectories from a behavior policy π ref, inducing a state distribution d ref ( s t ) and yielding a supervised model ˆ π ref. In contrast, closed-loop agents aim to maximize cumulative reward over time:

This leads to an inherent objective mismatch: open-loop training minimizes imitation error under expert supervision, while closed-loop deployment optimizes long-horizon reward under evolving dynamics and partial observability, as is illustrated in Figure 1. Bridging this gap requires careful alignment of three components in equation (1): the transition model P ( s t +1 | s t , a t ) , which can be consistently approximated using vehicle dynamics; the observation model P obs ( o t | s t ) , which may deviate from simulated sensors ̂ P obs; and the reward function r ( s t , a t ) , which must be inferred from partial observations o t using learned value models.

To address these mismatches, we employ a counterfactual data generation supported by 3DGSbased observation model ̂ P obs, then design a policy adapter that transforms the pretrained ˆ π ref into a reward-aligned policy π ∗ under the guidance of a learned Q-value model, as outlined in Figure 1.

2.2 E2E driving Agents in the Closed-loop Simulation

A recent line of approach uses visual generative or reconstruction models for rendering photorealistic driving scenes from state parameterizations, utilizing the recent advancement in image diffusion modeling [5], neural field rendering [10], and 3D Gaussian Splatting [13]. This line of methods essentially learn to estimate ˆ P obs ( ·| s t ) in our POMDP formulation in Section 2.1. However, it is critical to verify that the visual quality of these model outputs remains close to real-world scenes for them to serve as valid proxies for our closed-loop evaluation.

We conducted the following preliminary experiments to study the fidelity of the closed-loop simulator and demonstrate the performance gap between open-loop and closed-loop evaluation. In Figure 2, we systematically study the difference in L2 error under open and closed-loop settings. We use 3DGS [13] to reconstruct the scenes from the nuScenes dataset and compare the performance difference using ground truth data in Figure 2. Among all three E2E policies, we see a very close open-loop performance in motion prediction. This confirms the fidelity of the 3DGS-based simulation in its reconstruction quality.

0.5

1.0

1.5

2.0

2.5

3.0

Prediction Horizons (s)

0

1

2

3

4

5

6

L2 Error to Reference Trajectory (m)

Closed-loop (UniAD)

UniAD (GT)

UniAD (Recon)

VAD (Recon)

LTF (GT)

LTF (Recon)

Meanwhile, we also illustrate the L2 error based on UniAD's closed-loop rollout trajectory. We can see that the prediction error becomes quite significant compared to the open-loop prediction as the prediction horizon grows. A non-ignorable L2 error in the short prediction horizon leads to out-ofdistribution issues that result in such compounding errors. Increasing the planning frequency and shortening the planning horizon cannot necessarily close the gap between open-loop and closed-loop performance unless effective feedback guidance is provided to the E2E agents.

≈

≈

3DGS World Model

𝑓

!

🔥

E2E Driving Agent

❄

GT Trajectory

Teacher Forcing: true

𝑠

!

Filter

𝑟

!

Original Dataset

Distance

Constraint

Counterfactual

𝑎

!

(#)

Trajectories:

𝑎

!

#

, 𝑠

!%&

#

, 𝑜

!

#

, 𝑟

!

#

&

'

Multi-

step

E2E Driving Agent

❄

Counterfactual Data

Scene Representation

𝒛

Action

𝒂

History

𝒔

!"#

𝑎

∗

𝑎

%&'!

Δ𝑎 =

𝑎

∗

-𝑎

%&'!

Forward noising

Residual

𝑄

Collision

Speed

Route

Weighted Sum

Guidance

Dist.

H

times

Counterfactual Generation

MPA Training and Inference

LowReward

High Reward

3 Proposed Method: Model-Based Policy Adaptation (MPA)

To bridge the observation and objective mismatches outlined in Section 2.1, we introduce ModelBased Policy Adaptation (MPA)-a unified framework for open-loop to closed-loop adaptation in end-to-end (E2E) autonomous driving. Figure 3 shows an overview. This section is organized from left to right along the pipeline: Section 3.1 describes our model-based counterfactual data synthesis to address distribution mismatch. Section 3.2 details the training of a diffusion-based policy adapter on the curated dataset. Section 3.3 presents the value model used to guide policy adaptation.

3.1 World Model-Based Counterfactual Data Generation

We generate counterfactual samples using a geometry-consistent 3D Gaussian Splatting (3DGS) simulator [13], which renders photorealistic observations conditioned on poses of ego vehicle and surrounding agents, modeling the observation distribution ̂ P obs ( ·| s t ) . As shown in Section 2.2, this rendering remains high-fidelity as long as the rollout policy induces a state distribution close to the reference distribution.

To ensure reliable observations while introducing behavioral diversity, we randomly augment the predicted actions from a pretrained E2E policy ˆ π ref by rotating, warping, and random noising operations, resulting in a noisy policy ϕ β ◦ ˆ π ref. The resulting distribution of an augmented behavior is shown in Figure 4. Specifically, the augmentation of the trajectory consists of a rotation angle ranging from [ -10 , 10] , and a warping ratio from [0 . 1 , 1 . 0] , as well as random Gaussian noise with standard deviation 0.05. These augmented trajectories will then be categorized into different modes, given their rotation angles and warping ratios for the multi-modal policy adapter. Under a teacher-forcing setup, we initialize the rollout of every driving scene from an original reference state, and conduct counterfactual rollouts. To mitigate the exponentially growing search space as the rollout time horizon expands, we refer to the beam search algorithm [23], maintaining only the most rewarding candidates based on the cumulative reward calculated by the simulator using rule-based heuristics. The specific definition of these rewards can be found in Appendix A. A similar proposal-scoring-selection pipeline is used in the prior literature [24] to mitigate the prediction-planning misalignment of the motion planner.

To prevent rendering artifacts, we discard trajectories that deviate beyond a distance threshold or fall below a minimum reward. The rollout horizon T determines how far into the future these

counterfactual behaviors extend; as we show in our experiments, longer horizons expose richer supervision signals for downstream learning, b ut increase the risk of divergence from reference data.

The full generation procedure is summarized in Algorithm 1. With the generated counterfactual dataset at hand, we then conduct policy learning and value learning in the following sub-sections.

6

4

2

0

2

4

6

Y Position

0

2

4

6

8

10

12

X Position

Rotated Trajectories

Original Trajectory

Augmented Trajectories

3.2 Diffusion-Based Policy Adaptation

As we can see in the Figure 4, the generated counterfactual actions fall into a multi-modal distribution. To capture these counterfactual actions, we propose a diffusion-based policy adapter that refines the output of a frozen end-to-end (E2E) driving model by predicting residual trajectories ∆ a = a ∗ -a base , where a base ∈ R H × 2 is the trajectory from a pretrained policy (e.g., UniAD) and a ∗ is the most rewarding trajectory sample from the counterfactual rollout dataset.

Training. To model the distribution over residuals, we apply a latent diffusion process. The forward step adds Gaussian noise over K steps:

where ∆ a (0) = ∆ a and ¯ α k is the cumulative noise schedule. The denoising network f θ is a 1D U-Net that outputs multi-mode ∆ a (0) from ∆ a ( k ) , conditioned on the scene encoding z = ϕ enc ( o, s ego ) , ego history s ego, and base predicted trajectory a base . The output of f θ contains N modes, we denote the i -th mode as f θ ( ... )[ i ] . It is trained with the loss:

Inference. For inference, we use DDIM [25] to sample ∆ a (0) and recover the adapted trajectory:

This design adapts the knowledge from the pretrained base policy by conditioning on their encoded scene context and base actions, enabling generalization beyond the counterfactual data domain.

3.3 Principled Q-Value Guided Inference-time Sampling

Training. While step-wise rewards can be computed given full access to s t , estimating long-horizon returns is challenging under partial observability. Using the generated counterfactual dataset, we train a multi-step action value model

Algorithm 1: 3DGS-Based Counterfactual Generation

Input:

Ref. dataset

D

ref

, policy

ˆ

π

ref

,

horizon

T

, thresholds

δ

,

r

c

Output:

Counterfactual dataset

D

cf

foreach

ref. traj

(

s

ref

0

, s

ref

1

, . . .

)

in

D

ref

do

s

0

s

ref

0

for

t

= 0

to

T

-

1

do

a

t

∼

ϕ

β

◦

ˆ

π

ref

(

s

0:

t

)

if

dist

(

s

t

, s

ref

t

)

> δ

or

r

(

s

t

, a

t

)

< r

c

then

break

s

t

+1

, r

t

Sim.step

(

s

t

, a

t

)

o

t

3DGS.Render

(

s

t

)

Append

(

s

t

, a

t

, o

t

, r

t

)

to

D

cf

MPA (Ours)

VAD

Collision

Collision

Braking

Yielding

Reconstructed

Reconstructed

Ground Truth

Ground Truth

Low Reward

High Reward

based on four interpretable principles: route following Q route, lane distance Q dist, collision avoidance Q collision, and speed compliance Q speed. Each Q-function is trained independently to predict cumulative returns using ( o t , s ego , a t ) . Q = ∑ i ∈{ collision, dist, ... } w i × Q i . We follow the prior literature [26] to design our reward principles and their corresponding weights, with more details in Appendix A. We provide thorough ablations of the Q -function components in Section 4.3.

Inference. At inference, residual actions ∆ a are sampled following the policy adapter inference procedure from Section 3.2, and the best proposal is selected via

Compared to classifier-based reward guidance [27, 28], our Q-value guidance offers feedback from longer planning horizons and avoids the gradient instability of the reward model.

4 Experiment Results

In our experiments, we aim to answer the following research questions. RQ1 . Compared to the baselines, can MPA bring benefits to the E2E driving agents in a closed-loop evaluation in a generalizable way? RQ2 . How does MPA benefit the safety-critical performance in the closedloop evaluation? RQ3 . How do different adapters and value guidance modules contribute to the performance of MPA? RQ4 . How does MPA scale with the number of counterfactual planning steps in the data generation phase?

4.1 Experiment Settings

Dataset and Simulation Engine. We utilize the nuScenes dataset [29] that consists of 5.5 hours of driving data in Boston and Singapore. Every scene has a reference trajectory of 20 seconds. We use HUGSIM [13] as the simulation engine and evaluation benchmark. We train on a split of 290 scenes in the nuScenes train-val split, and evaluate on three settings. (1) In-domain evaluation : the model will be tested on a sub-split of 70 scenes, the surrounding dynamic entities (vehicles, pedestrians) will be replayed by a fixed ratio of their reference trajectory in the offline dataset. (2) Unseen nominal scene evaluation : the model will be tested on a sub-split of 70 scenes that are unseen yet during training, the surrounding dynamic entities (vehicles, pedestrians) are nominal and will be replayed by a fixed ratio of their reference trajectory in the offline dataset. (3) Safety-critical evaluation : the model will be tested on 10 scenes, where there exists one (or few) non-native agents to challenge the ego agents in an adversarial way. The simulation frequency is 4 Hz . In all the scenes, the termination occurs under one of the following five conditions: (i) full route completion, (ii) off-road events, (iii) collision events, (iv) too far from the reference trajectory, (v) maximum rollout time limits (50 seconds, 2.5 × of the reference trajectory) reached.

Baselines. We compare the MPA with diverse baselines in E2E driving algorithms that fall in the two following categories. (1) Pretrained base policy with open-loop training manner: we compare with the performance of UniAD [3], VAD [4] and LTF [2] on the HUGSIM dataset. We further build on our MPA with these policies. (2) E2E agents trained with curated counterfactual dataset: We further train several baseline policies with the curated dataset. AD-MLP [30] utilizes the ego's velocity, acceleration, past trajectories, and high-level command as the input, which is recognized as a naive baseline for the closed-loop Driving tasks. BC-Safe [31] uses the safe segments in the counterfactual datasets to train and End-to-End policies. Diffusion adopts the implementation of [32] in the scene encoding and utilizes a DDIM-based sampler [25] instead of truncated denoising during the inference time. To ensure a fair comparison between the MPA and the second category of the baselines, all the approach uses pretrained ResNet [33] as the perception backbone to encode the RGB inputs from 6 perspective cameras. The ego status includes velocity, acceleration, and navigation landmarks from the history frames.

Metrics. We follow the evaluation protocol in HUGSIM [13], which is inspired by the NAVSIMbased metrics [22]. The metrics include Route Completion (RC), Non-Collision (NC), Driveable Area Compliance (DAC), Time-To-Collision (TTC), Comfort (COM), HUGSIM Driving Score (HDScore). Specifically, HDScore is computed with the above metrics along with Route Completion (RC), instead of the Ego Progress (EP) in PDMS [22]. HDScore is the weighted sum as follows:

We list all the metrics with ( × 100 ) in the tables. All the metrics fall in [0 . 0 , 100] . We attach more details of these metrics and other information settings in the Appendix B.1.

4.2 Main Results and Analysis (RQ1, RQ2)

To answer RQ1, we first evaluate the closed-loop performance for in- and out-of-domain scenes, as shown in Figure 5. All the reported MPA approaches are evaluated with 20 action samples at inference time. In-domain scenes refer to the scenes that are used to generate counterfactual training data in Singapore. We evaluate the quantitative closed-loop results in these training scenes in Table 1. MPA-based E2E driving agents achieved better results compared to their pretrained counterparts, as well as three baseline methods trained on the counterfactual curated dataset, especially in the most important metrics, RC and HDScore. The baseline AD-MLP moves very conservatively, so the NC and TTC are low, yet the RC is also quite low as it barely completes the assigned routes for the challenging E2E scenes. Besides, the NC score in HUGSIM is a bit underestimated compared to NAVSIM, as HUGSIM erodes the vehicle boxes compared to the actual size by the point clouds. This leads to a few false 'collision' signals during the evaluation. Yet, most of them will not cause a collision that terminates the entire episode in the closed-loop simulation. This is why we observe some high RC with mediocre NC metrics. This still means the ego agents are capable of navigating around different collisions and off-road maneuvers to reach the goal in a reasonable way, and will result in a good HDScore.

We further evaluate the closed-loop performance under the unseen scenes. We select 70 scenes in Boston that are not accessible in the curated counterfactual dataset. The qualitative results are shown

MPA

(UniAD)

MPA

(VAD)

MPA

(LTF)

UniAD

VAD

LTF

in Table 7 (left). Compared to the in-domain results in Table 1, we can see a significantly degraded performance in AD-MLP and Diffusion, while the pretrained E2E policies still perform similarly as they do in the in-domain scenes. We observe that the MPA agents built upon the pretrained E2E policies are still optimal and quite robust under the unseen scenes. All three variants have comparable HDScore with their in-domain evaluation. This demonstrates the generalizability of the proposed adapter and value model under unseen scene contexts.

4.3 Ablation Studies (RQ3, RQ4)

We conduct ablation studies to analyze the contribution of the three main modules of MPA: (i) counterfactual dataset generation, (ii) policy adapter, and (iii) Q-value guidance.

Counterfactual Dataset. We further analyze the impact of the curated counterfactual dataset by ablating the step size during the rollout of counterfactual data. Then we train the MPA over the curated dataset and evaluate its performance in the unseen scenes in Boston, similar to the setting in Table 7 (left). We illustrate the trends of evaluation metrics with respect to the step sizes in the counterfactual dataset in Figure 6. MPA benefits from longer counterfactual steps, as there would be more informative supervision for the value function training in the future steps.

Policy Adapter. In Table 3, we evaluate a few variants of MPA (UniAD). Due to the space limits, the results of ablation studies for MPA (VAD) and MPA (LTF) will be postponed to the Appendix B. We evaluate the unseen scene's results across different variants. The comparison between ID-5 and ID-6 (ours) demonstrates the effectiveness of the adapter in a better route completion. Under the safety-critical scenes, the adapter brings ∼ 20% increase to the out route completion, leading to a significantly higher HDScore compared to the policies without the adapter. We also investigate the impact of mode capacity of policy adapter to the driving performances in safety-critical testing splits in Figure 7. The results show that a smaller capacity will influence the DAC and RC metrics, leading to a lower HDScore.

Principles of Q-Value Guidance. In Table 3, we remove different principles in the state-action value function Q . Compared to the ID-5 variants, ID-1 removes the route information used in all the

MPA

(UniAD)

MPA

(VAD)

MPA

(LTF)

UniAD

VAD

LTF

baselines, which leads to drastically degraded performance. ID-2 removes the distance function to the reference route, significantly degrading driveable area compliance and non-at-fault collision metrics. ID-3 removes the collision values, and ID-4 removes the speeding value function. Both of them have an impact on HDScore, especially in safety-critical situations. The reason that NC still seemed to be high for ID-3 in safety-critical scenes is that the available frame length before collision is short, which makes the denominator in NC smaller compared to the other group. However, when we look at the RC metrics, the drop when removing Q collision is significant, as the agents will encounter a collision and end to episode earlier than the nominal cases.

5 Related Works

End-to-End Autonomous Driving. End-to-End (E2E) autonomous driving has achieved significant progress by jointly training the detection, tracking, prediction, and planning modules to avoid information loss throughout the cascading system. ST-P3 [1] and UniAD [3] propose unified E2E frameworks that achieve state-of-the-art open-loop performance on the nuScenes dataset [29]. VAD [4] encodes the driving scene with a vectorized representation and incorporates query-based planning modules, and VAD-v2 [4] further designs a probabilistic planning approach and improves the closed-loop performance over the CARLA [18] benchmark. LAW [34] enhances VAD's performance with auxiliary supervision signal from latent world model. Hydra-MDP [17] follows VAD-v2's query-based framework and conducts multi-target hydra distillation with a set of scoring rules. SparseDrive [35] uses a sparse-centric pipeline to get rid of BEV features in the E2E driving pipeline, and DiffusionDrive [32] designs a truncated denoising scheme for the diffusion model to generate multi-mode motion trajectories. With the prosperity of foundation models, a series of works [36, 37, 38, 39, 40] incorporate Large Language Models (LLMs) and Vision-Language Models (VLMs) into the E2E planning pipeline. Despite the benefits of commonsense reasoning with foundation models, most of the existing models still focus on the open-loop evaluation or

the approximate closed-loop metrics from the open-loop evaluation [22], which lack counterfactual reasoning for the safety-critical scenarios. A recent work RAD [19] pays attention to the photorealistic closed-loop evaluation and utilizes imitation learning and online reinforcement learning to fine-tune the E2E driving agents. Yet, the value functions trained for the Proximal Policy Optimization (PPO) agents are not effectively integrated during the inference. MPA aims to use the value model as an effective inference-time guidance to make the E2E agents more robust in closed-loop evaluation.

Counterfactual Data Generation. Counterfactual data generation has been explored within the context of offline reinforcement learning. Wang et al. [41] utilize a learned model to autonomously generate additional offline data, thereby enhancing the training of sequence models. OASIS [42] introduces a method to produce counterfactual data by modulating guidance signals during diffusion model inference. In high-stakes decision-making domains such as autonomous driving, generating counterfactual data is essential due to the limited presence of safety-critical scenarios in existing datasets. Previous research has addressed the trade-off between realism and controllability in safetycritical scenario generation by integrating various constraints. These include inference-time sampling techniques [43], retrieval-augmented generation [44], low-rank fine-tuning approaches [45], and language-conditioned generation methods [46]. However, these efforts primarily focus on behavioral scenario generation without incorporating visual information. With advancements in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), recent studies such as DriveArena [5] and MagicDrive [47] have begun developing E2E simulators for closed-loop evaluation. Similarly, RAD [19] employs a 3DGS-based simulator for RL fine-tuning. Notably, to date, no existing work has focused on generating E2E counterfactual data within E2E simulators.

Reward Model for Inference-Time Scaling. Recent LLM research has shown the power of the reward model in LLM's inference-time scaling [48, 49]. In sequential decision-making problems, the reward model was explicitly used for inference-time supervision signals, such as the guidance for the diffusion-based policy models [50, 51, 52]. For the closed-loop autonomous driving and decisionmaking tasks, several prior works incorporate reward models as the classifier-based guidance to steer the diffusion model's sampling process [53, 54, 28]. Other format of the guidance signals include signal temporal logic (STL) guidance, language-based guidance [55], adversarial guidance [56, 57, 58], and game-theoretic guidance [59]. In the autonomous driving domain, prior works like DiffusionDrive [32] utilize truncated denoising for the diffusion models without additional classifier guidance. DiffAD utilizes action conditional guidance for E2E driving [60]. Diffusion-ES [27], Gen-Drive [61], and Diffusion-Planner [28] utilize customized reward models as test-time guidance for the non-E2E driving tasks. To the best of our knowledge, MPA is the first work incorporating the driving reward model for the inference-time scaling of E2E driving agents.

6 Conclusion

In this work, we introduce MPA, a general framework for improving the closed-loop trustworthiness of E2E autonomous driving agents. MPA begins by generating high-quality counterfactual trajectories through geometry-consistent rollouts in a 3DGS-based simulation environment. This results in a better data coverage while preserving visual fidelity. With the counterfactual dataset, MPA further trains a diffusion-based policy adapter to refine base policy predictions and leverages a multi-principle value model to guide inference-time decision making. These components allow pretrained agents to generate and evaluate multiple trajectory proposals, selecting actions that optimize long-term driving outcomes. Experimental results on nuScenes data and HUGSIM benchmark demonstrate MPA's effectiveness in boosting safety and generalizability.

Limitations. Despite these promising results, our approach assumes reliable rendering from 3DGS under constrained trajectory deviations and currently decouples value modeling from policy optimization. Future work includes extending our current results to diverse driving datasets, exploring the online RL training over the 3DGS simulator, and deploying MPA to the multi-modal foundation models to enhance reasoning capability for more severe distribution shifts in autonomous driving.

Acknowledgments and Disclosure of Funding

This work is in part supported by ONR MURI N00014-24-1-2748, NSF CNS-2047454, and CMU Safety 21 of the University Transportation Program. YZ is in part supported by the Stanford Interdisciplinary Graduate Fellowship.

References

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision , pages 533-549. Springer, 2022.

Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence , 45(11):12878-12895, 2022.

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 17853-17862, 2023.

Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8350, 2023.

Xuemeng Yang, Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving. arXiv preprint arXiv:2408.00415 , 2024.

Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398 , 2024.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1389-1399, June 2023.

Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. arXiv preprint arXiv:2311.15260 , 2023.

William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Åström, Michael Felsberg, and Christoffer Petersson. Neuroncap: Photorealistic closed-loop safety testing for autonomous driving. In European Conference on Computer Vision , pages 161-177. Springer, 2024.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. , 42(4):139-1, 2023.

Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In ECCV , 2024.

Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugsim: A real-time, photo-realistic and closed-loop simulator for autonomous driving. arXiv preprint arXiv:2412.01718 , 2024.

Ziyang Xie, Zhizheng Liu, Zhenghao Peng, Wayne Wu, and Bolei Zhou. Vid2sim: Realistic and interactive simulation from video for urban navigation. arXiv preprint arXiv:2501.06693 , 2025.

Junhao Ge, Zuhong Liu, Longteng Fan, Yifan Jiang, Jiaqi Su, Yiming Li, Zhejun Zhang, and Siheng Chen. Unraveling the effects of synthetic data on end-to-end autonomous driving. arXiv preprint arXiv:2503.18108 , 2025.

Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243 , 2024.

Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06978 , 2024.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning , pages 1-16. PMLR, 2017.

Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, et al. Rad: Training an end-to-end driving policy via large-scale 3dgs-based reinforcement learning. arXiv preprint arXiv:2502.13144 , 2025.

Norman Lehtomaki, NJAM Sandell, and Michael Athans. Robustness results in linear-quadratic gaussian based multivariable control designs. IEEE Transactions on Automatic Control , 26(1):75-93, 1981.

Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 629-636. IEEE, 2024.

Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven nonreactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems , 37:28706-28719, 2024.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014.

Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learning-based vehicle motion planning. In Conference on Robot Learning , pages 1268-1281. PMLR, 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.

Jianyu Chen, Bodi Yuan, and Masayoshi Tomizuka. Model-free deep reinforcement learning for urban autonomous driving. In 2019 IEEE intelligent transportation systems conference (ITSC) , pages 2765-2771. IEEE, 2019.

Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki. Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following. arXiv preprint arXiv:2402.06559 , 2024.

Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501.15564 , 2025.

H Caesar, V Bankiti, AH Lang, S Vora, VE Liong, Q Xu, A Krishnan, Y Pan, G Baldan, and O Beijbom. nuscenes: A multimodal dataset for autonomous driving. arxiv. 2019.

Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430 , 2023.

Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning, 2019.

Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139 , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.08481 , 2024.

Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint arXiv:2405.19620 , 2024.

Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. In NeurIPS 2023 Foundation Models for Decision Making Workshop .

Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, XianPeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. In 8th Annual Conference on Robot Learning .

Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14760-14769, 2024.

Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv preprint arXiv:2407.00959 , 2024.

Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recognition , pages 262-275. Springer, 2017.

Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:34748-34761, 2022.

Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, and Ding Zhao. Oasis: Conditional distribution shaping for offline safe reinforcement learning. Advances in Neural Information Processing Systems , 37:78451-78478, 2024.

Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 892-901, 2021.

Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. Realgen: Retrieval augmented generation for controllable traffic scenarios. arXiv preprint arXiv:2312.13303 , 2023.

Robert Dyro, Matthew Foutter, Ruolin Li, Luigi Di Lillo, Edward Schmerling, Xilin Zhou, and Marco Pavone. Realistic extreme behavior generation for improved av testing. arXiv preprint arXiv:2409.10669 , 2024.

Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuehl. Language conditioned traffic generation. In Conference on Robot Learning , pages 2714-2752. PMLR, 2023.

Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601 , 2023.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024.

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495 , 2025.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations .

Chiyu Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9644-9653, 2023.

Chiyu Max Jiang, Yijing Bai, Andre Cornman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Wheatley Lambert, Shuangyu Li, Xuanyu Zhou, et al. Scenediffuser: Efficient and controllable driving simulation initialization and rollout. In The Thirty-eighth Annual Conference on Neural Information Processing Systems .

Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guided traffic simulation via scene-level diffusion. In Conference on Robot Learning , pages 144-177. PMLR, 2023.

Chejian Xu, Ding Zhao, Alberto Sangiovanni-Vincentelli, and Bo Li. Diffscene: Diffusionbased safety-critical scenario generation for autonomous vehicles. In The Second Workshop on New Frontiers in Adversarial Machine Learning , 2023.

Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Controllable safety-critical closed-loop traffic simulation via guided diffusion. arXiv preprint arXiv:2401.00391 , 2023.

Yuting Xie, Xianda Guo, Cong Wang, Kunhua Liu, and Long Chen. Advdiffuser: Generating adversarial safety-critical driving scenarios via guided diffusion. arXiv preprint arXiv:2410.08453 , 2024.

Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile scene-consistent traffic scenario generation as optimization with diffusion. arXiv preprint arXiv:2404.02524 , 2024.

Tao Wang, Cong Zhang, Xingguang Qu, Kun Li, Weiwei Liu, and Chang Huang. Diffad: A unified diffusion modeling approach for autonomous driving. arXiv preprint arXiv:2503.12170 , 2025.

Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint arXiv:2410.05582 , 2024.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: They are provided in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: They are discussed in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide necessary implementation details in section 3, section 4, and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide an open access to the code and data collection scripts in the anonymous link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the experiment settings and details in Appendix A and B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean and standard deviation, as well as the significance level (p value) in Appendix B.3.

Guidelines: We

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Provided in the OpenReview form.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Provided in Appendix C

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: This paper adapts from the pre-trained end-to-end autonomous driving models. We describe safeguards in the appendix and will also attach it when releasing our model checkpoints and dataset.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the baselines are properly cited and introduced in Appendix A.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Auxiliary Details of MPA Implementation

A.1 Data Collection and Preprocessing

The environments we use to rollout counterfactual data and policy evaluation stay the same with HUGSIM benchmark [13].

State, Action, and Observation Space of Driving Simulator. The driving simulator we use [13] consists of different entities, which will be input to the MPA.

Observation (RGB): The rendered 6 camera views have a resolution of 450 × 800 each. This results in a (6 , 3 , 450 , 800) tensor.

State and Action (Trajectory): The pose of every agent can be represented as a tuple ( x, y, yaw ) . We also include the velocity, acceleration, and route information (represented as longitude progress and lateral distance) in the past frames. For the state in HUGSIM, the ego's history trajectory will be a ( T hist , 8) array. For the action output from the agents, it will be a ( T fut , 2) trajectory that consists of the 2D position at each frame. This trajectory will be solved by an LQR controller [20] to generate the corresponding throttle and steering sequence.

Intermediate Representation: all three methods will have an intermediate representation as BEV, where UniAD and LTF have a rasterized map, and VAD has a vectorized map.

Heuristics-Based Reward Labeling. The step-wise reward heuristics include the longitude reward (route completion), lateral distance reward (distance), collision penalty, and far from trajectory penalty. The Q-value is the truncated cumulative sum of the T -step reward, where T is the planning horizon as was set up to 5 steps (1.25 s ) in our experiments.

Based on these principles, we use the following reward heuristics to label the state-action transition pairs in the simulation. The first term is the longitude reward, denoted as route completion r route. At timestep t ,

where d t be the projected longitudinal distance along the route from the starting point to the closest point on route to the current ego position determined pos ego by s t .

The second term is the lateral distance penalty between the ego position and the route, denoted as distance reward r dist :

Following the simulation setup, the off-road and collision penalty will be -100 respectively, once a collision between the ego vehicle and other traffic agents or between the ego vehicle and background static objects occurs. This penalty term can be described as:

To improve the smoothness of driving, we use the speed penalty when the agents go overspeeding:

Constrained Rollout with Pretrained E2E Agents. We use the pretrained UniAD [3], VAD [4], and LTF [2] as base behavior policies to generate counterfactual datasets to further train the MPA. We use the planned trajectory for every timestep and augment it with different rotation angles (-5.0 ◦ to +5.0 ◦ ) and scales (0.1 times to 2.0 times), leading to 21 counterfactual behaviors at a single observation

frame. The key parameter during the counterfactual generation is to filter out the undesired samples based on the reward heuristics section discussed in the previous subsection.

For the policy adapter, we exclude all collision samples with a reward of r collision = -1 , although these samples are retained in the training dataset of the Q-value network. To ensure reliable learning for both the value network and the policy adapter, it is essential that the 3DGS-rendered datasets exhibit high visual realism. We assess the realism of these renderings using the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) metrics, as shown in Figure 8. The plot provides insight into acceptable lateral distance thresholds for maintaining high-fidelity simulation. As illustrated in the figure, both FID and KID degrade with increasing lateral displacement. Empirically, a lateral deviation of less than 0.6 m from the closest reference trajectory yields sufficiently realistic renderings, with KID remaining below 0.3 and FID generally under 50.

FID and KID vs Lateral Displacement

0.0

0.0~0.2

0.2~0.4

0.4~0.6

0.6~0.8

>0.8

0

20

40

60

80

FID

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

KID

FID

KID

A.2 Implementation Details in Training and Inference

Network Architecture. The architectures of the diffusion adapter and the value model are illustrated in Figure 9. Both models take the ego history information and action as input. The key difference is that the adapter model takes the base action from the pretrained model, a noisy residual action for the diffusion block, and intermediate representations from the base model, while the value model takes raw camera RGB inputs and final actions to be evaluated and selected at inference time.

Computing Resources. The experiments are run on a server with AMD EPYC 7542 32-Core Processor CPU with 256 threads, 4 × NVIDIA A5000 graphics, and 252 GB memory. For one experiment, it takes around 6 hours to train around 20 epochs for the policy adapter and value model. At inference time, the inference of the 3DGS-based world model and pretrained base driving model can be run on a single A5000. The inference time of pretrained baselines (UniAD, VAD) is around 0.5s, and the 3DGS world model takes around 0.3s to render all six camera observations with static and dynamic objects in the traffic scenes, and they are running two separate worker nodes following the HUGSIM [13]. For one single scenario, the maximum step is 200. Most scenarios will be finished less than one-third of the maximum time. In total, it will take an average of 1 minute for the evaluation per scene.

Hyperparameter Table. We list the hyperparameters used in MPA as follows. They include the hyperparameters for data curation, diffusion adapter, and the value model.

The link to our anonymous codebase is attached at: https://anonymous.4open.science/r/MPA-7432.

A.3 Additional Description of the Baselines

UniAD [3] presents a unified autonomous driving framework that jointly learns perception, prediction, and planning within a single Transformer-based architecture. The method explicitly formulates

History Encoder

(MLP)

RGB Encoder

(ResNet)

Action Encoder

(MLP)

t

1D U-Net

Predicted Denoised Actions

Base

Action

Intermediate

Noisy

Action

Noisy

Emb.

Action

Emb.

History pos., yaw, speed,

accel., lateral distance…

(𝑇

!"#$

, 8)

BEV

(3, 224, 224)

(𝑇

%&$

, 2)

(𝑇

%&$

, 2)

History

Embedding

Visual

Embedding

(512, )

(𝑇

%&$

, 2)

(128, )

(512, )

(512, )

Fusion Projection (MLP)

Fused Representation

(512, )

Denoised Representation

(512, )

Action Decoder

Action

6-Camera Views

History pos., yaw, speed,

accel., lateral distance…

(𝑇

!"#$

, 8)

Image

(3, 224, 224)

(𝑇

%&$

, 2)

RGB Encoder

(ResNet)

History

Encoder

(MLP)

Action

Encoder

(MLP)

Cumulative Reward of a Principle

(Collision, Over-Speed…)

Value Decoder (MLP)

Action

Embedding

History

Embedding

Visual

Embedding

(512, )

(512, )

(512, )

HUGSIM Rollout + Reward Heuristics

True Ego Future

Q--Value Supervision

MSE Loss/ MLE

MSE

Loss

the driving task as an autoregressive trajectory generation problem, leveraging dense Bird's Eye View (BEV) representations to enable strong coordination between scene understanding and motion planning. Through large-scale offline training, UniAD achieves state-of-the-art performance under open-loop evaluation protocols, demonstrating the effectiveness of end-to-end learning for autonomous driving tasks. The codebase in open-sourced at https://github.com/OpenDriveLab/UniAD/ with Apache License 2.0.

VAD [4] extends the end-to-end paradigm with a focus on visual grounding and scalable deployment. The framework employs multi-camera inputs to extract spatial-temporal features and utilizes Transformer blocks to capture interactions across different road agents. By removing the reliance on LiDAR sensors or high-definition maps, VAD enables more practical real-world deployment while maintaining high-quality planning capabilities through direct alignment of vision features with future motion targets. The codebase in open-sourced at https://github.com/hustvl/V AD/ with Apache License 2.0.

Latent TransFuser [2] represents a camera-only variant of the TransFuser architecture that operates without LiDAR inputs, relying exclusively on multi-view camera imagery. The method fuses features from both perspective and Bird's Eye View representations using cross-attention mechanisms, enabling robust reasoning about scene layout and agent behaviors. Despite the absence of LiDAR data, this approach achieves competitive performance by leveraging latent representations and hierarchical fusion strategies that preserve the spatial structure and temporal dynamics essential for autonomous driving. The codebase in open-sourced at https://github.com/hyzhou404/NAVSIM/ with Apache License 2.0.

BC-Safe is implemented by filtering unsafe trajectories and conducting behavior cloning on the E2E driving problem [31]. We adapt the implementation from the safe imitation learning repositories in https://github.com/HenryLHH/fusion/ with the MIT License.

Diffusion baseline is implemented by conditioning the action generation on the history trajectory and RGB images directly [32]. We adopt DDIM implementation from the diffusion policy [62] at https://github.com/real-stanford/diffusion_policy with the MIT License.

B Additional Experiments Details

In this section, we provide additional experiment results and algorithm implementation details.

B.1 Additional Environment Description

Evaluation Metrics. Following NAVSIM [22] and HUGSIM [13], we describe all evaluation metrics below:

RC : Route Completion score. It ranges from 0 to 1, representing the progress of the ego vehicle before encountering severe failure such as collision, off-road driving, deviating far from the preset trajectory, or exceeding the time limit.

NC : Non-Collision score. This evaluates the absence of at-fault collisions between the ego agent and surrounding static or dynamic traffic entities. Since the 3DGS model may erode some objects with inconsistent pointclouds, the NC tends to be lower-estimated as some near miss will be counted as collision samples due to the reconstruction error of 3DGS.

DAC : Driveable Area Compliance score. Slightly different from NAVSIM, which has ground truth map information, HUGSIM considers a violation of the driveable area if the ratio of the area within the current ego pose that stays inside the ground point clouds is smaller than 0.3. If this ratio is between 0.3 and 0.5, then the DAC score is 0.5. If the ratio is greater than 0.5, the DAC score is 1.0.

TTC : Time-To-Collision score. If the ego agent fails to pass a collision check within the next 0.5 s with surrounding objects (different from 0.9 s in NA VSIM), the TTC score is 0; otherwise, it is 1.

COM : Comfort score. It is 1 if the longitudinal and lateral acceleration, yaw rate, yaw acceleration, and longitudinal jerk all fall below their respective thresholds; otherwise, it is 0.

HDScore : HUGSIM Driving Score. As introduced in the main text, it is computed based on the above metrics, replacing the Ego Progress in NAVSIM with the Route Completion score:

Dataset Overview. Our environment includes a training split of 290 scenes. The distribution of dynamic objects among the training scenes is illustrated below. Most of the scenes have no more than 40 dynamic traffic entities. Yet this is already sufficient to craft diverse and complex traffic scenes.

Distribution of Controllable Dynamic Objects of the Training Scenes

0 2

3 5

6 8

9 11

12 14

15 17

18 20

21 23

24 26

27 29

30 32

33 35

36 38

>40

Number of Controllable Dynamic Objects

0

10

20

30

40

Number of Scenes

B.2 Additional Quantitative Results

Additional Ablation Studies As shown in Table 5 and Table 6, we extend the ablation study of MPAto the VAD and LTF base policies. In both settings, the full model with all Q-value guidance and the adapter (ID-6) consistently achieves the highest route completion and HDScore across nominal and safety-critical scenarios. Regarding the adapter, its removal leads to similar performance drops for both VAD and LTF, suggesting that while helpful, the value model plays a more critical role in these cases. Ablating individual Q-components reveals distinct impacts: removing Q route (ID-1) severely degrades nearly all metrics, especially RC and TTC; removing Q dist (ID-2) significantly harms DAC and HDScore; excluding Q collision (ID-3) causes agents to terminate earlier in safety-critical scenes due to collisions, lowering RC and HDScore; and omitting Q speed (ID-4) reduces COM and slightly affects TTC, leading to a modest decline in HDScore. These findings about value guidance modules align with our results on the UniAD policy and further validate the generalizability of our MPA design.

Impact of the Size of Adapter Modalities We evaluate the performance of MPA models over the safety-critical testing environments. The results show that larger sizes of modalities consistently bring benefits to the driving performance in RC, DAC, and HDScore, especially under the smaller mode sizes from 1 to 8. Properly adding the number of modes benefits the closed-loop driving performance.

Statistical Significance Analysis. Table 7 demonstrates 3-seed over the safety-critical scenes. The results show a reasonably consistent performance of all the methods with low standard deviation

across different random seeds. Our MPA is still significantly better than the baselines among all the safety-critical scenes.

B.3 Additional Qualitative Studies

Weillustrate more qualitative results in both counterfactual data generation and closed-loop evaluation of different E2E driving agents.

More Counterfactual Data Generation Examples. As illustrated in Figure 11, we illustrate some additional counterfactual data generation examples on six scenarios of the training split. The results show that the augmented data improves the data coverage, leading to more robust policy and value learning in MPA.

Comparison of Driving Performance of In-Domain and Safety-Critical Scenes. We further visualize the performance of MPA and baselines in nominal and safety-critical scenarios in Figure 12 and Figure 13. We can observe that the baselines without the value model usually fail to meet the safe constraints, leading to catastrophic consequences such as collision or getting off-road.

Y [m]

Y [m]

Y [m]

700

680

660

640

620

600

580

1680

1660

1640

1620

1600

1580

1560

680

660

640

620

600

580

900

920

940

960

980

1000

1020

1040

1060

X [m]

(a) Scene-0001

250

300

350

400

450

500

X [m]

1050

1075

1100

1125

1150

1175

1200

1225

Y [m]

(b) Scene-0004

800

825

850

875

900

925

950

975

X [m]

725

750

775

800

825

850

875

X [m]

1620

1640

1660

1680

1700

1720

1740

Y [m]

(d) Scene-0008

920

940

960

980

1000

1020

1040

1060

X [m]

(e) Scene-0011

1100

1150

1200

1250

1300

1350

X [m]

550

575

600

625

650

675

700

Y [m]

(f) Scene-0013

AD-MLP

BC-Safe

Diffusion

UniAD

VAD

MPA (UniAD)

MPA(VAD)

MPA(LTF)

AD-MLP

BC

Safe

Diffusion

UniAD

VAD

MPA (UniAD)

MPA(VAD)

MPA(LTF)

AD-MLP

BC-Safe

Diffusion

UniAD

VAD

MPA（UniAD)

MPA(VAD)

MPA(LTF)

AD-MLP

BC-Safe

Diffusion

UniAD

VAD

MPA（UniAD)

MPA(VAD)

MPA(LTF)

C Broader Impact

The proposed Model-Based Policy Adaptation (MPA) framework enhances end-to-end autonomous driving by improving safety and robustness in complex scenarios through high-fidelity simulation and reward-guided inference. This can accelerate deployment and reduce accidents. However, risks include biased reward models, gaps between simulated and real-world performance, potential misuse for surveillance or adversarial purposes, and privacy concerns from real-world data. Mitigation strategies include gated model release, reward interpretability, simulation validation, and data anonymization.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Model-based Policy Adaptation (MPA), a framework that improves the robustness of end-to-end driving policies in closed-loop settings. The submission first analyses issues of such driving models that occur under distribution shifts of closed-loop simulation. Moreover, a data generation pipeline is proposed that unrolls a reference policy on reference trajectories in a simulator based on 3D Gaussian Splatting (3DGS) to generate novel observations. This counterfactual data is utilized to (1) train a diffusion-based policy adapter for the trajectories and (2) an action-value model that guides the adapter during inference. Experiments using the HUGSIM simulation tools and the nuScenes dataset demonstrate the benefits of the proposed robustness strategy with several reference policies and ablate relevant design choices.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The topic is highly relevant and well motivated in the submission. With end-to-end driving models being more and more used in industry and academia, addressing the closed-loop robustness is a promising field. The proposed framework does not pose compatibility constraints to the specific driving policy. The work appears highly valuable to the community due the quality of the work and tools that are likely open-sourced. \n- The paper is mostly well written. Specifically the experimental study is well structured and clear to follow. It is apparent from the paper and supplementary pdf that thorough work was required for the project.  \n- The experiments are conducted with closed-loop simulation based on state-of-the-art sensor simulation (i.e. 3DGS).  The claims are well supported. The authors discuss limitations of the approach. \n\nWeaknesses:\n- My main issue with the submission is that I do not find the counterfactual data generation procedure clearly described. This negatively impacts the understanding of the downstream diffusion adapter and value learning modules. I find it somewhat unclear how the reference trajectories are constructed, how the reference policy is applied (and on which time steps), and how the reward is calculated. A figure or more precise notation might be helpful. For example, Figure 3 provides a BEV overview where the colors are not defined, the trajectories are not clearly visible, and the procedure is not shown over the different timesteps. \n- I am uncertain how much value Section 2.2 (and partially Section 2.1) provides in the paper. It is well known that behavioral cloning suffers from distributional shifts and compounding errors, thus the section does not provide much value to the reader. I recommend shortening these parts or providing further insights or qualitative examples instead.\n\nQUESTIONS:\n- Could you comment on “…increasing the planning frequency and shortening the planning horizon cannot necessarily close the gap between open-loop and closed-loop performance, unless some effective feedback guidance is provided to the E2E agents.” in Section 2.2? I am not sure if that is a clear consequence from the analysis. \n- Are the reference trajectories simply the ground truth states of the ego vehicle, sampled at different time steps in the scene? Using the word “reference” for both the policy and the recorded dataset might not be ideal.\n- Could you provide some intuition or qualitative result on why LTF benefits less from the robustness strategy compared to UniAD and VAD (despite being more robust without MPA)? Is this related to the smaller intermediate representation? \n- Could you comment on the additional compute requirements of MPA (i.e. with the diffusion adapter and value prediction)? Do these modules substantially increase run-time?  \n\nI am quite positive about this work. I am open to increasing my overall score, if unclarities and questions can be sufficiently resolved.\n\nLIMITATIONS:\nThe submission discusses limitations. The authors might want to consider commenting on the compute requirements, as such methods would need to be very efficient in a real-world setting. Moreover, I personally think that long simulations (i.e. in the synthetic CARLA simulator) might be required to support some claims regarding long-term planning benefits with the Q value model. Though, I respect that this is beyond the scope of this project.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the performance degradation of end-to-end (E2E) autonomous driving models when transitioning from open-loop to closed-loop evaluation. The authors identify a fundamental mismatch is the objective mismatch due to the absence of reward feedback during offline imitation learning. To address that, they propose Model-Based Policy Adaptation (MPA), which consists of three components: counterfactual data generation using 3D Gaussian Splatting (3DGS) simulation, a diffusion-based policy adapter that predicts residual actions, and a multi-step Q-value model for inference-time action selection. The models are trained on nuScenes data and evaluated via HUGSIM simulation engine. Results show that the proposed MPA can outperform the baseline by a large margin, especially in the safety-critical scenes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. While individual components are not novel, their combination for addressing the open-loop to closed-loop gap in E2E driving is original, particularly the use of counterfactual data generation with 3DGS simulation. \n2. The experimental evaluation is thorough, covering in-domain, out-of-domain, and safety-critical scenarios with consistent improvements across multiple base policies (UniAD, VAD, LTF). \n3. The overall methodology is clearly presented with good visual aids (Figures 1 and 3 effectively illustrate the approach). \n4. The work provides immediate practical value by showing how to improve existing E2E driving models' deployment safety, with HDScore improvements by a large margin.\n\nWeaknesses:\n1. The counterfactual data generation relies heavily on teacher-forcing, which may not explore sufficient behavioral diversity to handle truly out-of-distribution scenarios. \n2. The claim that observation mismatch is minor lacks convincing evidence—Figure 2 only addresses motion prediction, not visual fidelity. 3. The Q-value decomposition into four specific components appears arbitrary without ablation on alternative decompositions or theoretical justification. \n4. Computational costs are not discussed. \n5. For safety-critical scenes, only 10 scenes are used for evaluation, whether this is representative is questionable.\n\nQUESTIONS:\n1. In the qualitative results in Fig.4, it seems that MPA is not yielding/breaking compared to the VAD baseline according to the simulated surroundings (the position of road mark / pole, etc.) unless the images are not \"simulated on the same timestamp\". Could you be more specific about how you obtained the qualitative results?\n2. According to the original HUGSIM paper, UniSIM generally performs better than VAD on HDScore, which is contradictory to the authors results. Could you explain why it is the case?\n3. The weighted sum of the value model seems ad-hoc to me, could the authors explain how did these weight coefficient to be determined?\n4. What is the computation overhead for adding the value model and adapter during inference?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a system designed to adapt open-loop end-to-end driving agents for closed-loop deployment. The proposed framework comprises three key components: (1) counterfactual data generation using 3DGS, (2) a diffusion-based residual policy adaptor to address observation mismatches, and (3) a Q-value model to capture outcomes. The authors identify objective mismatch as the primary cause of closed-loop performance degradation and validate their approach through experiments on the nuScenes dataset.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\nThe paper presents a reasonable idea, and the system’s performance is supported by the released experimental results. However, there are several major concerns.\n(1) The writing lacks clarity, with insufficient explanation of the motivation and component design details.\n(2) The claims in the introduction \"the performance drop in close-loop ealuation stems from observation mismatch and objective mismatch \" \"first mismatch is actually minor in the open-loop evaluation\" are not substantiated by evidence or results in the current version.\n\nClarity\nThe problem definition is clearly stated. Several concerns remain regarding the motivation and component design details. \n(1) the explanation in lines 103–111 is unclear, particularly regarding how the prediction horizon is controlled and how experiments with 3DGS are conducted. \n(2) We cannot find evidence Figure/Table to support the claim \"the fidelity of ... qualicy.\"\n(3) The r(s_t, a_t) and Sim.step is ambiguous, though it can be inferred that HDSim is used. \n(4) the icons used in the MPA column of Figure 4 and in Figure 1 are not well explained, leading to confusion. \n\nSignificance\nThe work is of interest to the industry, and the results demonstrate the effectiveness of the proposed method. Validation on the HDSim environment based on the nuScenes dataset is appropriate. However, the results could be more comprehensive, for example by including experiments on an additional dataset and providing more visualization cases.\n\nOriginality\n\nThe paper proposes a system that integrates previously explored techniques, such as the use of 3DGS for simulating actions and outcomes (as in RAD) and inference-time scaling with customized rewards. The novelty lies in combining these components into a unified framework and applying them to end-to-end driving agents.\n\nQUESTIONS:\nIn addition to the issues mentioned above:\nSince the HDScore is influenced by multiple factors such as RC, NC, and TTC, why is the HDScore particularly high for MPA? Which specific factors contribute most significantly to the reported HDScore?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Model4 based Policy Adaptation (MPA), a general framework that attempts to fortify pretrained end-to-end driving policies against distributional shift. MPA first uses a simulator to create many counterfactual/ “what-if” driving situations the original data never saw. Using this synthetic corpus, it jointly learns a diffusion-based policy adapter that refines the agent’s raw trajectory proposals and a multi-step Q-value estimator that forecasts the long-horizon utility of each candidate. At inference, the adapter produces multiple feasible trajectories and the Q-value network selects the one with the greatest expected return, thereby delivering safer, more robust driving behavior without retraining the underlying policy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper tackles model drift in closed-loop driving, where small deviations compound over time and cause severe distribution shift. The topic is both important and under active investigation.\n\n2. The writing is generally clear, the pipeline is easy to follow, and figures are helpful.\n\n**Weaknesses and questions:**\n\n1. Combining synthetic counterfactual data with a learned value head is well-established in model-based RL. The contribution appears to be an incremental recombination of known pieces rather than a fundamentally new idea. \n\n2. The motivation for all the submodules (simulator, diffusion adapter, Q-network) is not clear.\n\n\n3. Why not have the Q-value network directly on the counterfactual trajectories and select among them, bypassing the diffusion adapter? What advantage does the diffusion adapter offer?\n\n\n4. I missed mention and comparison to key model-based RL literature (e.g., PlaNet, Dreamer, MBRL for autonomous driving).\n\n\n5. Algorithms addressing compounding error by online data aggregation (e.g., DAgger) or by combining imitation with value optimization are absent.\n\n\n6. I missed ablation to some of the components (aligned with comment 2 and 3).\n\n\n7. There are quite some similar recent works aligning with the model-based counterfactual data generation for the same problem setting. These are not cited. \n\n**References:**\n\n1.\tHafner, Danijar, et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint arXiv:1912.01603 (2019).\n\n2.\tHafner, Danijar, et al. \"Mastering atari with discrete world models.\" arXiv preprint arXiv:2010.02193 (2020).\n\n3.\tRoss, Stéphane, Geoffrey Gordon, and Drew Bagnell. \"A reduction of imitation learning and structured prediction to no-regret online learning.\" Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.\n\n4.\tDey, S., Paassen, B., Nair, S. R., Boughorbel, S., & Schilling, A. F. (2024). Continual Learning from Simulated Interactions via Multitask Prospective Rehearsal for Bionic Limb Behavior Modeling. arXiv preprint arXiv:2405.01114.\n\nQUESTIONS:\nPlease see weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}