Title: Learning to Drive with Two Minds: A Competitive Dual-Policy Approach in Latent World Models
Venue: NeurIPS 2025
Accepted: no
Keywords: Autonomous Driving

World Model

Reinforcement Learning
Abstract: Recent advances in generative video models such as SORA have renewed interest in using world models to simulate physical dynamics for embodied decision-making tasks like autonomous driving. In parallel, end-to-end driving frameworks have begun to incorporate latent world models that predict future latent states as an auxiliary objective, trained jointly with imitation learning to enhance the model’s planning capabilities. These models help encode environment dynamics and improve planning accuracy, but treat the world model as a passive auxiliary module. Separately, the Dreamer series has demonstrated the potential of using latent world models as simulators for reinforcement learning (RL), enabling agents to learn through imagined rollouts. However, combining imitation learning (IL) and RL in latent world models remains underexplored, and naive attempts to jointly optimize a shared policy often lead to instability and degraded performance. In this work, we propose a dual-policy framework that decouples IL and RL agents while sharing a common latent world model. The IL policy learns from expert driving data using supervised latent rollouts, while the RL policy explores the same latent environment via Dreamer-style training. Rather than fusing the two objectives, the agents are trained independently and compete during learning. Based on the outcome of their competition, knowledge—either expert behavior or exploratory experience—is selectively shared between agents. This architecture enables each policy to specialize while benefiting from the other’s strengths. Experiments in complex driving scenarios demonstrate that our approach outperforms imitation-only baselines, leading to more robust and generalizable autonomous driving policies. We will release our code on GitHub soon.

Abstract

Recent advances in generative video models such as SORA have renewed interest in using world models to simulate physical dynamics for embodied decisionmaking tasks like autonomous driving. In parallel, end-to-end driving frameworks have begun to incorporate latent world models that predict future latent states as an auxiliary objective, trained jointly with imitation learning to enhance the model's planning capabilities. These models help encode environment dynamics and improve planning accuracy, but treat the world model as a passive auxiliary module. Separately, the Dreamer series has demonstrated the potential of using latent world models as simulators for reinforcement learning (RL), enabling agents to learn through imagined rollouts. However, combining imitation learning (IL) and RL in latent world models remains underexplored, and naive attempts to jointly optimize a shared policy often lead to instability and degraded performance. In this work, we propose a dual-policy framework that decouples IL and RL agents while sharing a common latent world model. The IL policy learns from expert driving data using supervised latent rollouts, while the RL policy explores the same latent environment via Dreamer-style training. Rather than fusing the two objectives, the agents are trained independently and compete during learning. Based on the outcome of their competition, knowledge-either expert behavior or exploratory experience-is selectively shared between agents. This architecture enables each policy to specialize while benefiting from the other's strengths. Experiments in complex driving scenarios demonstrate that our approach outperforms imitationonly baselines, leading to more robust and generalizable autonomous driving policies. We will release our code on GitHub soon.

1 Introduction

End-to-end (E2E) learning has emerged as a dominant architecture in autonomous driving [19, 21, 40], enabling direct optimization of the entire driving pipeline-including perception, prediction, and planning-toward the final driving objective. By bypassing modular decompositions, this paradigm mitigates issues such as error accumulation and misaligned optimization objectives.

Before the rise of E2E approaches, autonomous driving models typically relied on manually defined 29 training objectives for each module. However, these objectives often introduced suboptimal biases 30 when integrated into a full system. In contrast, E2E learning allows models to learn directly from 31 expert demonstrations, focusing solely on observations and actions. Despite this advantage, E2E 32 methods still struggle with the long-tail problem: expert-collected datasets rarely cover rare or 33 safety-critical scenarios. 34

To address this, researchers have explored generative world models that conditionally synthesize 35 driving observations based on actions or user-provided modalities (e.g. BEV and text) [38, 39]. 36

These models can generate synthetic data to enrich training datasets, helping to alleviate long-tail 37 issues. More recently, the combination of generative and reconstruction models has enabled high38 fidelity scene generation [46, 33, 42], supporting closed-loop simulation and large-scale RL training 39 [41]. However, these methods are computationally intensive, as they often require training a new 40 reconstruction model (e.g., 3D Gaussian Splatting [23]) for each environment. 41

42

43

44

45

46

47

48

In this paper, we propose a more computationally efficient alternative for addressing the long-tail problem. Inspired by recent works integrating latent world models into end-to-end frameworks [28, 25], we adopt a latent world model rather than a pixel-level one. This design reduces computational overhead while preserving rich spatiotemporal representation. Our world model is trained jointly during the E2E imitation learning process, similar to model-based imitation learning (MBIL) [16]. The prediction objective enhances the model's ability to learn scene dynamics, aligning with predictive processing theories in human cognition [9].

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

While imitation learning allows agents to efficiently acquire expert behavior, it remains limited in generalization, especially in previously unseen scenarios-even with synthetic data augmentation. To overcome this, we draw inspiration from model-based reinforcement learning (MBRL), particularly the Dreamer series [12, 13, 14, 15]. We propose full utilize the latent world model with a learned reward model during E2E training rather than only set a new predictive task for the E2E model. This enables the agent to simulate imagined trajectories in latent space and optimize its policy using imagined rollouts.

By integrating model-based imitation and reinforcement learning into a unified framework, our agent benefits from both expert supervision and autonomous exploration. It can learn efficiently from demonstrations while improving through self-directed imagination-based learning in latent space.

Recent work such as RAD [6] also explores combining imitation and reinforcement learning. However, those methods rely on realistic 3D reconstructions for exploration, incurring significant computational cost. In contrast, our approach performs exploration entirely in latent space using a world model learned during imitation, eliminating the need for expensive 3D scene generation.

Our experimental results validate the effectiveness of our approach, achieving a substantial reduction in collision rate on the nuScenes benchmark, from 0.48% to 0.15%.

Our contributions are summarized as follows:

We introduce a competitive dual-policy architecture that decouples imitation learning and reinforcement learning agents, enabling them to co-train within a shared latent world model while specializing in expert imitation and exploratory behavior, respectively.

Wepropose a compute-efficient RL framework for autonomous driving , which eliminates the need for external simulators or costly scene reconstructions by leveraging a latent world model as an internal simulator for one step imagined.

We conduct extensive experiments on both the nuScenes and Navsim datasets , showing that our approach outperforms imitation-only and prior world-model-based baselines, achieving state-of-the-art results in terms of both planning accuracy and safety.

2 Preliminary

In end-to-end autonomous driving, the model receives a sequence of past and current observations 76 from multiple cameras mounted on the vehicle and predicts a future trajectory for ego car. At each 77 timestamp t , the observation o t consists of multi-view images. Most end-to-end models can be 78 viewed as encoder-decoder frameworks. They first encode the raw observations o t into intermediate 79 features, often in the form of a bird's-eye view (BEV) representation B t using architectures like 80 BEVFormer [43]. The BEV feature is then further processed to produce the latent scene state s t : 81

After extracting scene state, the planning head directly predicts the future action sequence based on 82 the latent state: 83

2

where T plan is the planning horizon, typically 3 to 4 seconds. Each action a t ∈ R represents a 2D 84 waypoint in the vehicle's local coordinate system, and the output sequence ˆ a t : t + Tplan -1 contains the 85 predicted waypoints from t to t + T plan -1 . 86

The planning head is typically trained using expert demonstrations. The predicted action sequence is 87 supervised using the corresponding ground-truth actions via an imitation loss: 88

Recently, latent world models have been introduced into end-to-end autonomous driving to help 89 the model better understand scene dynamics. Unlike pixel-level world models, latent world models 90 operate in the abstract feature space, which makes them more computationally efficient. Given the 91 current state s t and a proposed action sequence, the world model predicts a future state after T fut 92 steps: 93

where the world model predict the state after T fut frames in the future, given current state s t and the 94 proposed action sequence ˆ a t : t + T plan -1 . 95

To supervise the world model, we use the ground truth s t + T fut , which can be obtained by o t + T fut . 96 For interpretability, we first convert the latent space state to BEV space, then supervise the world 97 model's predictions in BEV space. This allow us to visualize the output of the latent world model 98 directly: 99

100

101

3 Method 102

Perception

Actor (IL)

Actor (RL)

Critic

reward

sample

advantage

Imitation

Learning

Reinforcement

Learning

Both

Perception

Input

@

t

Input

@

𝑡 + 𝑇

𝑓𝑢𝑡

Rule-based

reward

function

Reward Model

Latent

World Model

Supervision

s 𝑡

𝑎𝑡

𝑎 𝑡

expert

𝑠𝑡+𝑇𝑓𝑢𝑡

𝑠𝑡+𝑇𝑓𝑢𝑡

𝑠𝑡+𝑇𝑓𝑢𝑡

𝜋 ⋅ 𝑠 𝑡

)

𝑎 𝑡+𝑇 𝑓𝑢𝑡

(𝑖)

Ƹ 𝑠

𝑡+𝑇𝑓𝑢𝑡

(𝑖)

s 𝑡

s 𝑡

,

Ƹ 𝑠

𝑡+𝑇𝑓𝑢𝑡

(𝑖)

Exploration

Student Supervisor

B

Ƹ

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

3.1 Planning Head and Actor Modeling

Currently, most popular E2E autonomous driving model use a determined network to model the action sequence, showed in Eq.2. However, as the uncertainty and non-deterministic nature of planning [3] and the behaviors of the agents are highly multimodal [35], we follow this paradigm to build our planning head.

Firstly, since s t ∈ R N t × d contains almost all perception information, we use a set of waypoint querys W t ∈ R N c × T plan × d to extract the information the planning head needed for each action in the proposed action sequence:

in which N t is the number of tokens to represent a scenario. Following the experiment of SSR [25], we set N t = 16 . And d is the dimension of embedding, and N c is the number of high-level command (i.e. left turn, right turn, and go straight). Since autonomous driving vehicles usually require the navigation information to planning, most E2E datasets provide high-level commands for the model.

Then, intuitively, we should generate each action in the sequence step by step, adhering to causal relationships: first sampling a t ∼ π ( a t | s t ) , then a t +1 ∼ π ( a t +1 | s t , a t ) , and so on, until a t + T plan -1 ∼ π ( a t + T plan -1 | s t , a t : t + T plan -2 ) .

However, we argue that humans often plan actions backward: they first identify the goal or endpoint and then reason back to generate the action sequence. In this case, we start from a t + T plan -1 ∼ π ( a t + T plan -1 | s t ) , then sample a t + T plan -2 ∼ π ( a t + T plan -2 | s t , a t + T plan -1 ) , and continue backward until a t ∼ π ( a t | s t , a t +1: t + T plan -1 ) .

To implement the invert chain rule, we adopt the masked attention mechanism with an invert attention mask, which is an inverted version of the causal mask in language model training. We first treat the scene token s t as a sequence of tokens with a horizon T plan : s t = { s t t , s t +1 t , ..., s t + T plan -1 t } , where s t + τ t ∈ R N c × d is used to generate the τ -th action of the proposed action sequence. The masked multi-head attention then enforces the inverted causal dependency during the generation process:

where mask is lower part of a triangular matrix, and mask ∈ R T plan × T plan .

Here, we model each action in the action sequence with gaussian distribution:

where two separate multi-layer perceptrons (MLPs) are used to generate the mean vector and covariance matrix of the Gaussian distribution respectively, for simplicity.

We update the imitation loss in Eq. 3 with a the negative log likelihood (NLL) loss as follows:

where µ is a hyper-parameter to balance the L1 loss and NLL loss.

3.2 Reward Modeling

Inspired by the cost module in LeCun's world model [24], which consists of two components-an intrinsic reward module that estimates the immediate 'energy' of the current state (e.g., pain, pleasure, hunger), and a critic that predicts the expected cumulative future cost-we extend this design by introducing an additional imitation reward that evaluates the similarity between expert trajectories and those proposed by our model.

For the intrinsic reward, we focus solely on the current state to detect unsafe conditions, such as the ego vehicle being too close to surrounding dynamic agents or the road boundary. The critic reward then evaluates whether a proposed action sequence will lead to future collisions by considering the current state, the predicted future states, and the action sequence itself.

Beyond these rule-based components, we incorporate human driving preferences through an imitation 143 reward, which measures the discrepancy between the model's proposed actions and those taken by 144

Ƹ

d

Perception

Latent

World Model

Actor

𝑎𝑡

𝑠𝑡+𝑇𝑓𝑢𝑡

Intrinsic

Reward

Model

Imitation

Reward

Model

Critic

Reward

Model

collision with nearby agents or road boundary

safe distance threshold

driving knowledge and experience

𝑎𝑡

𝑎𝑡

𝑠𝑡+𝑇𝑓𝑢𝑡

Expert Demonstration (near-optimal reward

≈

1)

reward model augmentation

𝑟 𝑡

𝑖𝑚𝑖

𝑟 𝑡

𝑖𝑛𝑡

𝑟 𝑡

𝑐𝑟

supervision

supervision

supervision

Input @

𝑡

Exploration Trajectories (variable reward values)

Ƹ

Thus, our reward model comprises three components (see Fig. 2): 146

We model each reward using a Gaussian distribution and compute ground-truth reward targets with 147 rule-based heuristics. However, due to the open-loop nature of our prediction process, the computed 148 rewards are often near-perfect, which limits the effectiveness of training. To address this, we introduce 149 perturbations to the proposed action sequences, enabling the reward models to learn from both positive 150 and negative samples. 151

We train the reward models using a negative log-likelihood loss: 152

153

154

155

156

3.4 Actor Critic Learning 157

Once the world model is trained via imitation learning, it can serve as a simulated environment for the 158 reinforcement learning (RL) agent to explore. In this work, we adopt an actor-critic framework to op159 timize the actor module, where the critic is implemented as an MLP-based value estimator. However, 160

3.3 End-to-End Imitation Learning

Inspired by the world model training stage of Dreamer [15] and recent works that integrate world models into end-to-end autonomous driving [25, 28], we jointly train the driving policy, world model, and reward model by simply summing their individual losses (see Eq. 14).

161

162

163

164

165

166

167

168

169

170

171

172

The computed advantages are used to update the RL actor via the actor loss. To further encourage 173 exploration, we also add an entropy regularization term to the loss. For the critic, we use the average 174 negative log-likelihood loss over all samples: 175

To enable knowledge sharing between the two actors-one trained via imitation and the other via 176 exploration-we introduce a competitive mechanism that selectively updates their parameters, as 177 described in Alg. 1. 178

Algorithm 1 Competing IL and RL with Knowledge Blending

4 Experiments 179

4.1 Datasets 180

nuScenes [2] is a large-scale autonomous driving benchmark featuring 1,000 20-second urban driving 181 scenes with 1.4M annotated 3D boxes across 23 object classes. It provides 360° imagery from six 182

we observe that the differing optimization objectives of imitation learning (IL) and reinforcement learning can cause unstable training when both attempt to update the same actor. To address this, we decouple the IL and RL components by introducing a separate RL actor module, ensuring that each learning paradigm only updates its own actor.

The RL actor outputs a policy π , similar to IL actor (see Eq. 10). It also contains a critic head to estimate the return value distribution v t = E [ p ( ·| s t )] .We then sample N future action sequence a (1) t : t + T fut -1 , a (2) t : t + T fut -1 , ..., a ( N ) t : t + T fut -1 . The world model acts as a one-step simulator and provides the interact feedback for the RL actor:

After the simulation, the critic estimates the value of the imagine state v ( i ) t + T fut = E [ p ( ·| s ( i ) t + T fut )] , we use the expectation for the λ -return calculation (see Eq. 16) for each one step imagine chain and obtain R ( i ) t . We calculate the advantage value for each sample, where the reward r t = ˆ r int t + ˆ r cr t + ˆ r imi t :

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

cameras and 2Hz keyframe annotations. Following prior works [19, 21], we evaluate planning using L2 placement error and Collision Rate.

Navsim [5] is a compact, filtered version of OpenScenes [4], itself derived from nuPlan [22]. It emphasizes challenging scenarios and contains 120 hours of driving at 2Hz. It includes navtrain and navtest splits for training and testing. To better reflect closed-loop safety and behavior, Navsim evaluates agents with six metrics: No at-fault Collisions (NC), Drivable Area Compliance (DAC), Time to Collision (TTC), Ego Progress (EP), Comfort (C), and Driving Direction Compliance (DDC). These are combined into a weighted Driving Score (PDMS).

4.2 Implementation Details

For experiments on nuScenes, our method builds upon the SSR framework [25]. We train our models using 8 NVIDIA A800-SXM4-80GB GPUs and perform evaluation on a single A800 GPU. The training is conducted with a batch size of 1 using the AdamW optimizer, with a learning rate set to 5 × 10 -5 . All other training settings follow the original SSR configuration. The training process takes approximately 12 hours to complete.

For experiments on Navsim, we adopt the Transfuser [34] model as backbone. Transfuser employs a Transformer-based architecture to fuse front-view camera image and LIDAR data across multiple stages. We train our model on navtrain split and evaluate it on test split, using the same hardware configuration as in nuScenes experiments. The training is performed with a batch size of 16 using the AdamW optimizer, with a learning rare set to 1 × 10 -4 .

4.3 Main Results

We evaluate our method on both the nuScenes [2] and Navsim [5] benchmarks. The results are presented in Tab. 1 and Tab. 2, respectively. For nuScenes, we follow the evaluation protocol of [21], which reports average L2 distance and collision rate over 1s, 2s, and 3s prediction horizons. For Navsim, we adopt the close-loop metrics provided in Navsim. Sepcifically, we use the test split rather than navtest split for evaluation, as the former contains much more scenarios (5044) than the later (885), making it more suitable for comprehensively assessing the model's overall driving performance.

On nuScenes, compared to the recent state-of-the-art SSR [25], which achieves the best L2 error 210 (0.38), our method obtains a slightly higher L2 error (0.43), but significantly improves the collision 211 rate. Ablation studies further confirm that each component of our approach contributes meaningfully 212 to the final performance. Although UAD [10] achieves the lowest collision rate overall (0.06%), it 213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

relies heavily on extensive supervision signals such as detection and tracking. In contrast, our method achieves the best collision rate (0.15%) among methods that do not use any auxiliary supervision beyond expert trajectories.

On navsim, our model obtains a PDMS of 88.2, outperforming recent state-of-art methods, showing notable improvements accoss multiple sub-metrics, including NC (+0.4), DAC (+0.4) and TTC (+0.8). Compared to WoTE, which leverages a world model to evaluate candidate trajectories during testing, our approach achieves a higher overall score. While DiffusionDrive performs competitively on navtest split (88.1), its performance drops significantly on test split (80.3), indicating limited generalization to more diverse scenarios-possible due to the distribution shifts between navtest to test.

4.4 Ablation Study

We conduct an ablation study on the nuScenes dataset [2], evaluating the contribution of three key components: the inverse autoregressive module in the planning head, the reward model in end-to-end imitation learning, and the incorporation of reinforcement learning. Results are summarized in Tab. 3. The experiments demonstrate that each component contributes to improved performance across both planning accuracy and collision rate metrics. And we find that, the import of RL actor enhance the exploration, so that the L2 error becomes higher, but the agent learns to avoid colliding with others.

4.5 Qualitative Results

Through qualitative analysis, we found that the latent BEV features effectively capture distinct driving 231 behaviors. As shown in Fig. 3, due to the end-to-end nature of our model, the planning module 232 directly influences the feature representations. For instance, in the "Turn Left" and "Turn Right" 233 scenarios, the turning direction tends to exhibit higher activation (represented in red), indicating 234 that these regions are more prominently perceived and attended to by the model. Regarding the 235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

Explore Traj 5

Explore Taj 6

xpiore Traj

Explore Traj 9

0t (e4, auojdx]

[xplore Traj 11

xplore Traj 14

lore Traj 15

attention points, we observed that regardless of the driving context-whether going straight, turning left, or turning right-their spatial distribution remains relatively consistent. Interestingly, this pattern persists across all trained models. We attribute this phenomenon to the limitations of the nuScenes dataset, which may be insufficient for training a fully generalized policy, often leading to models that adopt fixed driving strategies. Moreover, we observed that when attention points are more dispersed across the BEV space and less concentrated near the ego vehicle, the L2 error tends to be lower. In contrast, when attention points are concentrated directly in front of and behind the vehicle, the collision rate is often reduced.

As illustrated on the right side of Fig. 3, the proposed method also demonstrates closed-loop behavior by leveraging the world model to simulate and explore diverse potential trajectories. These imagined scenarios are subsequently used to train the model, enabling it to acquire more generalized and robust driving strategies.

5 Conclusion

This work explores how to better utilize latent world models in recent world-model-based end-toend autonomous driving (E2EAD) frameworks. We propose a novel approach that integrates the strengths of both imitation learning (IL) and reinforcement learning (RL) through a shared latent world model-without relying on external simulators. Our results show that latent world models can effectively serve as internal simulators for RL, significantly reducing collision rates and improving planning safety. Furthermore, visualization of the learned trajectories enhances the interpretability of using such latent dynamics for simulation. We believe our method offers new insights into leveraging world models for more robust and generalizable autonomous driving.

Limitations and Future Work : Despite its promising results, our approach has several limitations. Currently, we use relatively simple reward functions, which leads to a dominance of the IL policy in the competitive training framework. This is primarily because the IL agent benefits from dense and structured supervision, whereas the RL agent relies on sparse and delayed reward signals. In future work, we plan to explore more sophisticated reward designs and competition strategies to better balance learning between IL and RL agents. Additionally, we are interested in integrating the latent world model with more expressive generative models to bridge the gap between compact latent dynamics and explicit pixel-level understanding.

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

References

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL https://arxiv.org/abs/2311.15127 .

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning, 2024. URL https://arxiv.org/abs/2402.13243 .

OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving. https://github.com/OpenDriveLab/OpenScene , 2023.

Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking, 2024. URL https://arxiv.org/abs/2406.15349 .

Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, Ying Zhang, Wenyu Liu, Qian Zhang, and Xinggang Wang. Rad: Training an end-to-end driving policy via large-scale 3dgs-based reinforcement learning, 2025. URL https://arxiv.org/abs/2502.13144 .

Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems (NeurIPS) , 2024.

Anant Garg and K Madhava Krishna. Imagine-2-drive: Leveraging high-fidelity world models via multi-modal diffusion policies, 2025. URL https://arxiv.org/abs/2411.10171 .

James Gornet and Matt Thomson. Automated construction of cognitive maps with visual predictive coding. Nature Machine Intelligence , 6(7):820-833, 2024.

Mingzhe Guo, Zhipeng Zhang, Yuan He, Ke Wang, and Liping Jing. End-to-end autonomous driving without costly modularization and 3d manual annotation. arXiv preprint arXiv:2406.17680 , 2024.

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 2455-2467, Red Hook, NY, USA, 2018. Curran Associates Inc.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555-2565. PMLR, 2019.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination, 2020. URL https://arxiv.org/abs/1912. 01603 .

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models, 2022. URL https://arxiv.org/abs/2010.02193 .

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2024. URL https://arxiv.org/abs/2301.04104 .

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving, 2022. URL https://arxiv.org/abs/2210.07729 .

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023. URL https://arxiv.org/abs/2309.17080 .

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision , pages 533-549. Springer, 2022.

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, and Sikai Chen. Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driving, 2024. URL https://arxiv.org/abs/2412.15544 .

Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8350, 2023.

Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, and Holger Caesar. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 629-636, 2024. doi: 10.1109/ICRA57147.2024.10610077.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. , 42(4):139-1, 2023.

Yann LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27, 2022. URL https://openreview.net/pdf?id=BZ5a1r-kVsf .

Peidong Li and Dixiao Cui. Navigation-guided sparse scene representation for end-to-end autonomous driving, 2025. URL https://arxiv.org/abs/2409.18341 .

Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European Conference on Computer Vision , pages 142-158. Springer, 2024.

Xiang Li, Pengfei Li, Yupeng Zheng, Wei Sun, Yan Wang, and yilun chen. Semi-supervised vision-centric 3d occupancy world model for autonomous driving. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/ forum?id=rCX9l4OTCT .

Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model, 2025. URL https: //arxiv.org/abs/2406.08481 .

Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model, 2025. URL https://arxiv. org/abs/2504.01941 .

Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, and Jose M. Alvarez. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation, 2024. URL https://arxiv.org/abs/2406. 06978 .

Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14864-14873, 2024.

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xinggang Wang. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving, 2025. URL https://arxiv.org/abs/2411. 15139 .

Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan, Peng Jia, Xianpeng Lang, Xingang Wang, and Wenjun Mei. Recondreamer: Crafting world models for driving scene reconstruction via online restoration, 2024. URL https://arxiv.org/abs/2411.19548 .

Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems , 35:6531-6543, 2022.

Hang Wang, Xin Ye, Feng Tao, Chenbin Pan, Abhirup Mallik, Burhaneddin Yaman, Liu Ren, and Junshan Zhang. AdaWM: Adaptive world model based planning for autonomous driving. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=NEu8wgPctU .

Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, and Jiwen Lu. Occsora: 4d occupancy generation models as world simulators for autonomous driving, 2024. URL https://arxiv.org/abs/2405.20337 .

Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14749-14759, June 2024.

Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6902-6912, June 2024.

Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15449-15458, 2024.

Tianyi Yan, Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng zhong Xu, and Jianbing Shen. Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation, 2024. URL https://arxiv.org/abs/2411.11252 .

Yunzhi Yan, Zhen Xu, Haotong Lin, Haian Jin, Haoyu Guo, Yida Wang, Kun Zhan, Xianpeng Lang, Hujun Bao, Xiaowei Zhou, and Sida Peng. Streetcrafter: Street view synthesis with controllable video diffusion models, 2025. URL https://arxiv.org/abs/2412.13188 .

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17830-17839, 2023.

Yu Yang, Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving. CoRR , abs/2408.14197, 2024. URL https://doi.org/10.48550/arXiv.2408.14197 .

Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation, 2024. URL https://arxiv.org/abs/2410.13571 .

Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving, 2023. URL https: //arxiv.org/abs/2311.16038 .

Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision , pages 87-104. Springer, 2024.

A Related Work

A.1 End-to-end Autonomous Driving

End-to-End automatic driving methods have recently made significant progress. These learning-based methods replaced traditional modular design by the end-to-end paradigm. [19] first demonstrated the potential of end-to-end by unifying perception and planning tasks within an unified framework. [21] vectorized the scene representation and improved the efficiency of inference. [40] decomposed the traditional end-to-end pipeline and searched for optimal architectures. some works tried to integrate world models into end-to-end automatic driving. [28] predicted future visual latens via world model, improving temporal understanding. [25] utilized sparse tokens to represent dense BEV features and similarly employed a world model to predict next Bird's Eye View (BEV) feature to enhance scene comprehension. [29] leverages a BEV-based world model to predict future agent states, enabling online trajectory evaluation and selection.

In contrast, our approach leverages the world model as an offline simulator for end-to-end automatic driving methods. Specifically, the AD model iteratively interacts with the world model to imagine future scene transitions through a world model, enabling reward-driven policy optimization in a simulated latent space.

A.2 World Model

World model estimates the next state of the environment based on current state and action. It serves as a key component in both human and robot cognitive system to reason, predict and plan.[24]. The concept of world model was first introduced in [11], which designed a recurrent-style world model to predict future states, enabling the agent to learn an effective policy. Later, Yann Lecun proposed an agent architecture [24], including a world model to predict possible future world states given the input from perception, actor, and etc. Dreamer series [12, 13, 14, 15] advanced this idea by employing RSSM to model a world model from experience, helping to train a policy on imagined trajectories within latent space.

In autonomous driving, numerous world model-based methods have emerged. Video-based world models predict future driving videos conditioned on priors. [17] is a generative world model that utilizes various inputs to create realistic driving scenarios. [38] leveraged diffusion model to predict future images and planned based on these predictions. [7] was built on stable video diffusion [1] model and achieved high fidelity and versatile controllability. Occupancy-based world model like [47, 37, 44, 27] predict future occupancy condition on priors. [47] firstly defined the 4D occupancy forecasting task and used an auto-regressive framework to achieve spatial and temporal modeling. [37] further deployed diffusion model to model long-term temporal evolutions.

In contrast, our approach leverages world model in latent space, which greatly saves computational resources.

A.3 Reinforcement Learning in Autonomous Driving

Reinforcement learning (RL) is an important technique for autonomous driving. [45] trained an RL expert to map BEV Input to actions, subsequently served as teacher for the end-to-end student model.

[20] leveraged a Vision-Language-Model (VLM) to generate rewards signals for RL. Some methods 459 explicitly utilized a world model to simulate state-action transition between actor and environment to 460 improve data efficiency, which is called Model-based reinforcement learning (MBRL). [26] integrated 461 DreamV3 to train an expert driving model, becoming the fist agent to finish CARLA v2. [36] analyzed 462 performance degradation sources of driving agents, proposing an adaptive fine-tuning strategy that 463 selectively updates actor or world model. [8] proposed a novel framework by integrating Vista[7] 464 with a multi-modal diffusion-based policy, achieving impressive performance in the CARLA. 465

466

467

In contrast, our fully E2E model conducts MBRL in the latent space and combines with IL policy to reach better performance.

468

469

470

471

472

473

474

B Actor-Critic Preliminary

In model based RL algorithms, the world model is used to imagine future states based on proposed action sequence. More specifically, for each iteration, given an initial state, the world model and actor are invoked for N img imagine horizons. The actor proposes actions based on the state, and the world model predicts the future state based on these actions and the initial state, rolling out in this manner. For each rollout, the imagined state s , action a , and reward r are stored for behavior learning.

There are two main components in the behavior learning process:

Here, π θ represents the policy network, and the action sequence is sampled from the policy distribution. 475 The value output of the critic is used to compute the return value, so here we use the expectation of 476 the value distribution as its predicted value v t = E [ v ψ ( ·| s t )] . 477

The return value is often computed using bootstrapped λ -returns to integrate the predicted rewards 478 and values [15]: 479

where r t is the expectation value of reward distribution output by reward model p ϕ ( r t | s t ) at timestamp 480 t : r t = E [ p ϕ ( ·| s t )] . For the final roll out at timestamp t + T fut · N img , the return value is equal to 481 the reward value. 482

For the actor, the loss is composed of two parts: the reinforce term and an entropy regularizer to 483 encourage exploration: 484

where the sg ( · ) denotes stop gradient. 485

For the critic, we use the negative log likelihood of the return in the value distribution as its loss: 486

C Additional Visualization 487

In this section, we show some additional visualization of our method. 488

000

Explore Traj 1

Explore Traj 5

Explore Traj 12

Explore Traj 7

D

00

o

oo

000

oo

000

o

。

000

6

000

00

。：。

Q

%

o°

oo

FRONT LEFT

FRONT

FRONT_RIGHT

中中甲

BACKLEFT

BACK

BACKRIGHT

C

FRONT_LEFT

FRONT

FRONT_RIGHT

000

000

o

。

。

。

o

BACK LEFT

BACK

BACK RIGHT

0000

000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly define the paper's scope (autonomous driving) and summarize our key contributions at the end of the introduction section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and outline directions for future work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is method and experiment driven; we do not include theoretical results or formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: 1. Our method builds upon well-established and widely-used prior work. 2. We provide comprehensive implementation details in the paper, including architecture, training settings, and hyperparameters. 3. We will release our code and data upon paper acceptance to further support reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will include the code and a README file in the supplementary material (.zip) before the deadline, ensuring reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Due to high computational and time costs, we did not conduct repeated trials for statistical significance. However, we believe our results are reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to the computation cost and time cost, we do not do repetitive experiment to get the statistic significance. But we believe the experiment result is re producible.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details about computing resources used (e.g., GPU type, training time) are included in the Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We believe our research fully adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our primary focus is to improve the performance of autonomous driving systems. We hope this leads to societal benefits such as increased safety and accessibility, though we did not formally analyze broader impacts.

Guidelines: 697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method is not yet ready for deployment and currently poses minimal risk of misuse, so safeguards are not a focus in this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use only licensed assets and properly credit all external data and tools throughout the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will provide comprehensive documentation for all new assets in the supplementary material and code release.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our research does not involve human participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: We only used LLMs for writing assistance, editing, and discussions. They did not contribute to the core methods, experiments, or results.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel model-based approach to self-driving. This work takes inspiration from Dreamer and BEVFormer, and trains a world model in the BEV features space. The model is used for both imitation learning and RL, with two separate policies being trained at the same time. The authors propose a novel blending mechanism to let information flow between IL and RL policies. The authors demonstrate impressive results on nuScenes and navsim, particularly when comparing to other methods that do not use auxiliary tasks.\n\nSTRENGTHS AND WEAKNESSES:\n###### Strengths\n- The proposed method does not use reconstruction loss, and operates fully in the latent space. This enables the representation to disregard irrelevant parts of the observation, such as e.g. foliage patterns in trees, and focus on the more relevant information. Additionally, this makes the method much more compute efficient.\n- The proposed method achieves good results. In particular, collision rate of 15% on NuScenes is very good compared to other methods that don't use auxiliary tasks;\n- Combining the knowledge of obtained from imitation learning and from RL is an exciting direction in the literature, and the idea proposed in this paper is quite interesting.\n\n###### Weaknesses\n- The writing is a bit confusing, some parts of the method are unclear (see questions).\n\nI do think the idea is good, and the results are promising. I'm willing to raise my score if authors answer my questions and help me understand the method.\n\nTypos and nitpicks:\n52: full utilize -> to fully utilize\n71: one step imagined ?\nFigure 1: such large figures look better in the top part of the page\n109: querys - queries\n168: interact feedback -> interactive feedback\n169: imagine state -> imagined state\n\nThese are not all, the paper has many typos, please run it through a spell checker.\n\nQUESTIONS:\n- Q1 What's the reward $r_\\mathrm{int}$? In the text, you say it's hardcoded. How exactly do you implement it? You say that your method doesn't use any auxiliary tasks, but in a way, depending on how this reward is implemented, you can consider this as an auxiliary task;\n- Q2 In the conclusion, you say that the imitation policy often dominates the RL policy due to the nature of the data. Do you have a concrete number for how often RL beats IL throughout training and inference?\n- Q3 How do you mix RL and IL policies during inference? Do you just use one of them? Or do you predict the outcome with the world model and use the actions you think are best according to the critic?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach towards training world models with both imitation learning and reinforcement learning in a stable manner. The approach leverages a reward model that models 3 types of rewards, competitive dynamics, as well as a policy that generates actions in reverse order. The results are promising on autonomous vehicle benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed approach achieved decent results in the nuScenes dataset and strong results on the Navsim test set.\n- The proposed three separate reward model based approach seems intuitive, but ablations are needed to confirm this.\n- The proposed approach is unique and different from existing approaches.\n\n### Weaknesses\n- I found the approach to be generally difficult to understand and that it did not make sense intuitively. I also found the proposed motivation somewhat difficult to follow. The design decision to model actions backwards was not well motivated and contradicts many current paradigms. In fact, the proposed Inv. AR approach worsens performance in L2 error.\n- I would expect an ablation comparing the proposed approach to a more simple and standard approach where the world model is first pre-trained on the imitation learning data (similar to offline world modeling) and then used for MBRL as an initialized world model.\n- It's unclear if the proposed competition aspect of the learning algorithm actually helped with learning improved world models. An ablation confirming whether this is the case would be helpful.\n- The proposed competition based approach does not seem scalable, and although discussions in the limitations section discuss this, there are no experiments to confirm that this was even useful.\n- Ultimately, the proposed approach seems promising but a lack of ablations for a huge amount of (uncommon) design decisions makes me question the success of the proposed approach.\n\n### Other Comments\n- The bolding in Table 2 in the Comf. column is a bit misleading.\n\nQUESTIONS:\n### Questions\n- Line 98-99 \"This allow us to visualize the output of the latent world model directly\" --- how? I am confused about how the authors are able to visualize latents.\n- Why does combining IL and RL become difficult in practice? Simply pretraining a world model with IL and then using it for the initialization of a world model in MBRL seems like a simple and stable baseline worth comparing to.\n- I am still confused on why the proposed approach chooses to plan backwards. More motivation for this intuitively and from a technical perspective would help.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dual-policy framework for end-to-end autonomous driving that integrates imitation learning (IL) and reinforcement learning (RL) through a shared latent world model. Rather than fusing both learning objectives into a single actor, the authors introduce two independent policies—one trained via IL on expert rollouts and another via RL using imagined latent rollouts—and enable them to compete and share knowledge selectively. The world model is trained jointly with the IL actor and used as a one-step simulator for the RL actor. To support training, a hybrid reward model is designed, incorporating intrinsic safety constraints, imitation alignment, and critic-based value estimation. The proposed method achieves strong performance on nuScenes and the Navsim benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The separation of IL and RL into distinct actors that compete and share parameters via performance-based blending is a novel design.\n2. The use of a latent world model as a one-step internal simulator for RL avoids the need for expensive environment rollouts or high-fidelity 3D reconstructions. \n3. The paper proposes a structured reward model combining intrinsic safety, imitation-based similarity, and future cost prediction. This multi-part reward design improves interpretability and supports safe exploration.\n4. The method shows state-of-the-art or competitive performance on two benchmarks, outperforming several recent methods.\n\nWeaknesses\n1. Dual-policy training is creative but, it's unclear whether one actor may eventually dominate or if this setup guarantees consistent improvements.\n2. Although the paper avoids simulator rollouts, the reward signal is composed of three manually designed components: an intrinsic safety score, imitation alignment, and critic-based feedback. Will these rule-based generalize across complex scenarios?\n3. The paper proposes to blend policies based on performance comparisons but provides limited detail on how blending is executed (e.g., soft interpolation vs. hard switches), how the score thresholds are selected, and how stable this process is during training.\n4. The RL actor operates in a latent world model with only one-step imagined rollouts. Will this shallow imagination restrict the agent’s ability to reason over long-term consequences?\n\nQUESTIONS:\n1. How sensitive is the reward model to its designed components? Have you tested alternative reward formulations or considered using learned preference models?\n2. Can the method handle complex scenarios? For example, how would the policy perform in unprotected left turns or with aggressive nearby agents?\n3. Is there a risk of the RL policy exploring unsafe actions that the IL actor avoids?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a _dual-policy framework_ for autonomous driving, where imitation learning (IL) and reinforcement learning (RL) agents are trained independently but share a latent world model. The IL policy leverages supervised latent rollouts from expert data, while the RL policy interacts with the same latent model via Dreamer-style training. Rather than combining IL and RL losses, the agents “compete” during training, with selective knowledge transfer—either expert-driven or exploration-based—depending on the outcome. This design aims to allow specialization while fostering mutual benefit. Experiments in complex driving scenarios show improved robustness and generalization compared to imitation-only baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- **Latent World Model Usage**: The method avoids costly external simulators or scene reconstructions by using a latent model for imagination-based training. This is a compelling and compute-efficient design, particularly relevant to real-world autonomous driving settings.\n- **Modular Dual-Policy Setup**: The separation of IL and RL agents is conceptually clean, and the mutual competition is a novel touch.\n- **Empirical Results**: The experiments are convincing and suggest a tangible benefit over baseline methods.\n    \n**Weaknesses:**\n- **Scope of Contribution**: The algorithmic novelty may be limited for a NeurIPS-style audience; the work feels more suitable for a robotics or autonomous driving venue due to its system-level focus rather than core RL or representation learning advancements.\n- **Evaluation of Core Mechanism**: The proposed dual-policy competition mechanism is underexplored. There is little analysis or ablation of how the competition dynamics impact learning, and no theoretical or empirical justification for why this form of policy interaction should work better than fused or alternating training.\n- **Lack of Theoretical Insight**: No formal analysis or grounding is provided for the knowledge sharing or competitive mechanism. The method remains empirical and heuristic-driven.\n\nQUESTIONS:\n1. **Deeper Ablation**: It would strengthen the paper to include a more detailed ablation of IL vs RL contributions. For instance, what happens when only IL is used with the latent model? Or when the RL policy receives no IL guidance?\n2. **Policy Usage Statistics**: How often is each policy \"trusted\" during learning or inference? Quantitative statistics on policy arbitration would help clarify how the knowledge transfer actually happens in practice.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a hybrid approach that combines imitation learning (IL) and reinforcement learning (RL) for end-to-end autonomous driving. Given the challenges faced by existing end-to-end methods such as generalization,  the method introduces a computationally efficient method by combining  IL and RL in latent space. In particular, it introduces a competitive dual-policy framework  where IL learns from expert demonstrations via supervised latent rollouts, while the other (RL) explores the same latent environment using Dreamer-style imagined rollouts. Rather than jointly optimizing a single policy, the two agents train independently and compete: after each iteration, the better performer “blends” its knowledge into the other, enabling both to specialize and benefit from one another. The paper performs experiments on nuScenes and Navsim benchmarks. It shows improved performance in terms of collision rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1)\tThe paper introduces a novel strategy for end-to-end autonomous driving. In particular, leveraging a dual policy framework where joint training of imitation and reinforcement learning towards enhanced E2E driving is a promising direction\n\n2)\tThe proposed approach operates on latent space which provides computational efficiency over counterpart methods that leverages reconstruction techniques.\n\n3)\tThe proposed method shows improvement in terms of collision rate at expense of L2 error.\n\nWeaknesses\n\n1)\tIntroduction rightfully mentions the generalization problem of E2E models. However, recent E2E methods [1,2, etc] leverages VLMs in an computationally efficient way to address the same problem. The paper lacks mentioning such works and does not provide any comparison.\n\n2)\tIn Section, 3.1. author argues “human often plan actions backward” and then formulate their problem accordingly. What are the analytical/empirical evidence (e.g. ablation study) to back such claims?\n\n3)\tThe paper mentions generalization and long-tail issues, but there is no specific experiment showing how proposed method helps in such specific scenarios. Provided experiments are generic (e.g. collision rate, L2 error on the whole nuscenes dataset). a) How does the proposed method works when it is trained in one city (e.g. Boston-nuscenes) and tested in another one (e.g. Singapore-nuscenes). b) How would the proposed method work on challenging long-tail scenarios (see [2] for reference)\n\n4)\tThe paper mentions the computational efficiency of the method, but fails to provide comparisons with existing methods in terms of computational parameters e.g.  FPS, number of params . \n\n5)\tThe proposed method does not provide impressive performance, e.g. requires trade-off between L2 and Collision Rate. Other methods addressing same problem such as VLM methods provides improved performance across both metrics. \n\n[1] Vlp: Vision language planning for autonomous driving, CVPR 2024 [2] Distilling Multi-modal Large Language Models for Autonomous Driving, CVPR 2025\n\nQUESTIONS:\nI have listed all my concerns in the \"Strength and Weaknesses section\". Please refer to \"Weaknesses\" for the raised concerns and questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-08"}