Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)
Authors: Zhenjie Yang
Xiaosong Jia
Qifeng Li
Xue Yang
Maoqing Yao
Junchi Yan
Venue: NeurIPS 2025 poster ()
Keywords: End-to-End Autonomous Driving
Reinforcement Learning
Primary area: applications
TL;DR: Reinforcement Learning Successfully Attempt for End-to-End Autonomous Driving in CARLA v2

Abstract: Reinforcement Learning (RL) can mitigate the causal confusion and distribution shift inherent to imitation learning (IL). However, applying RL to end-to-end autonomous driving (E2E-AD) remains an open problem for its training difficulty, and IL is still the mainstream paradigm in both academia and industry. Recently Model-based Reinforcement Learning (MBRL) have demonstrated promising results in neural planning; however, these methods typically require privileged information as input rather than raw sensor data. We fill this gap by designing Raw2Drive, a dual-stream MBRL approach. Initially, we efficiently train an auxiliary privileged world model paired with a neural planner that uses privileged information as input. Subsequently, we introduce a raw sensor world model trained via our proposed Guidance Mechanism, which ensures consistency between the raw sensor world model and the privileged world model during rollouts. Finally, the raw sensor world model combines the prior knowledge embedded in the heads of the privileged world model to effectively guide the training of the raw sensor policy. Raw2Drive is so far the only RL based end-to-end method on CARLA Leaderboard 2.0, and Bench2Drive and it achieves state-of-the-art performance.

Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)
Zhenjie Yang!.,, Xiaosong Jia"t, Qifeng Li Xue Yang', Maoqing Yao", Junchi Yan 1.37
1. Sch. CS, Sch. of AIS, Sch. of AI, Shanghai Jiao Tong University
Institute of Trustworthy Embodied AI, Fudan University
3. Asynscale AI 4. AgiBot
https://gi
Abstract
Reinforcement Learning (RL) can mitigate the causal confusion and distribution shift inherent in imitation learning (II.). However, applying RI. to end-to-end autonomous driving (E2E-AD) remains an open problem for its training difficulty, and IL is still the mainstream paradigm in both academia and industry. Recently Model-based Reinforcement Learning (MBRL) have demonstrated promising results in neural plamning: however, these methods typically require privileged information as input rather than raw sensor data. We fill this gap by designing Raw2Drive, a dual-stream MBRL approach. Initially, we efficiently train an auxiliary privileged world model paired with a neural planner that uses privileged information as input. Subsequently, we introduce raw sensor world model trained via our proposed Guidance Mechanism, which ensures consistency between the raw sensor world model and the privileged world model during rollouts. Finally, the raw sensor world model combines the prior knowledge embedded in the heads of the privileged world model to effectively guide the training of the raw sensor policy. Raw2Drive is so far the only RL based end-to-end method on CARLA Leaderboard 2.0, and 2rivh2 and it achieves state-of-the-an performance.
Introduction
Beyond modular systems, end-to-end autonomous driving models [5, 6, 7, 8] are emerging, where a unified model directly uses raw sensor inputs for planning. As shown in Figure 1, most of these models [9, 10, 11, 12] based imitation learning (IL., which trains models
mimic expert demonstrations. However, IL faces fundamental limitations such as poor generalization to unseen situations [2, 13, 14, 15] and causal confusion [1, 16, 17]. which oceur when the model incorrectly associates actions with the wrong causes. These issues become problematic in complex and dynamic driving scenarios where decision-making must account for delicate interactions with environment.
Reinforcement learning (RL) [18, 19, 20) offers a promising alternative by optimizing driving policies through -wwrdd interactions with the dynamic environment [21, 22, 23], beyond the expert demonstration in imitation learning. Recently, RL has been shown to achieve higher performance bounds than IL in various domains, such AlphaZero [24, 19], and OpenAI-O3/DeeppS These successes highlight RL's ability to adapt policies for complex decision-making tasks.
[25].
Cooresppondecce Author. This work was in part supported by by NSFC (62222607,62506222 and Natural Science Foundation of Shanghai (02268.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

a) Imitation Learning or Model-free RL Paradigm
Causal
Dist Shift
Imitation
Policy
Reinforcement
Efficiency
Raw Sensor Input
Convergence
b) Model-based RI. Paradigm
Noise
Convergence X
Redundant
Interaction
Causal
Reinforcement
Occlusion
World Model
Policy
Modeling
Dist Shift
Difficulty
Efficiency
Raw Sensor Input
Dual-stream Model-based RL Paradigm (Ours)
Low Dim
Interaction
Reinforcement
World Model
Policy
Convergence
Structured
Causal Dist Shift Efficiency
Privileged Input
Guidance Mechanism
Interaction
Policy
Reinforcement
World Model
Raw Sensor Input
Figure Comparison of different training paradigms in end-to-end autonomous driving. (a) Imitation Learning suffers from causal confusion [1] and distribution shift [2]. Model-free Reinforvement Learning [3] faces efficiency problem and fails converge. (b) Model-based Reinforcement Learning: There are no reported such works for raw sensor input E2E-AD as the raw data can be noisy and redundant, and Think2Drive [4] assumes the privileged ground truth data is given, which cannot be directly applied in real-world AD. (c) In Raw2Drive, we propose the first feasible model-based reinforcement learning paradigm for end-to-end autonomous driving. By leveraging low-dimen low-dimensional, structured privileged imput, our approach guides the learning of world model from raw sensor data, effectively addressing the issues outlined in (a) and (b).
As shown in Table 1, compared the extensive research systems [26, 27] recent end-to-end approaches [9]. RL for AD has been comparatively less explored. Pioneering work MaRLn [3] faced sigmificant efficiency and convergence issues, requiring approximately 50M steps (57 days of training) with performance lagging far behind contemporary IL based methods [28, 29]. As of today, IL-basee methods achieve saturated performance on CARLA Leaderboard 1.0 [30] (abbreviated CARLA v1), while none of them can achieve satisfying scores on the more challenging CARLA Leaderboard 2.0 [31] (abbreviated as CARLA v2).
for AD, either through modular
This situation recently changed with the seminal work Trink2Drive [4], which introduced model- based RL (MBRL) approach that successfully solved CARLA v2, with the help of the world model [32]. However, Trink2 [4] relies privileged information (ground-truth states of environments) and there is still no success reported the field to apply MBRL with raw sensor inputs. The main challenge lies in the contrast between privileged information, which is compact and facilitates efficient training, and raw sensor data, which is high-dimensional, redundant, and noisy, making the training of world model significantly more difficult. As shown in Table 1, there are no RL based end-to-end methods in CARLA v2, which is the very setting that this paper seeks to challenge. Due to limited space, discuss more details of Related Works in Appendix A.
In this work, we propose Raw2Drive, a dual-stream MBRL method, achieves state-of-the-ee perfor- mance on CARLA v2 and surpasses II based methods by large margin. The training is divided into two stages. the first stage, we leverage the privileged information to train privileged world model and paired neural planner. Then, we further jointly train raw sensor world model and end-to-end

---

Table 1: Comparison of settings of mainstream algorithms. WM means whether to use the world model. SSL means self self-suuer learning. CoRL2017 [33] and CARLA [30] include 4 and 10 standard cases, respectively. CARLA v2 [31, 341, on the other hand, introduces 39 additional real-worke corner cases, which are significantly more difficult. ROM03 is based on CARLA vl including 4 very basic scenes.
Method
Venue
Input Privileged
Scheme
E2E
w/ WM
Benchmark
Corner Case
Chen [35]
ITSC 2019
RL
No
No
Roundabout
No
MaRLn [3]
CVPR 2020
Raw
RL
Yes
No
CoRL 2017
No
Roach [36]
ICCV 2021
Privileged
RL
No
No
Carla v1
No
UniAD [9]
CVPR 2023
Raw
IL
No
Carla v1
No
Thivke [4]
ECCV 2024
Privileged
RL
No
Yes
Carla v2
Yes
1.W [37]
ICT.R 2025
Raw
11 +SSS
Yes
Yes
Carla yi
No
AdaWM [38]
ICI. 2025
Privileged
RI.
No
Yes
ROM03
No
TraneTraan [39]
ICL. 2025
Raw
IL
Yes
No
Carla v2
Yes
Ruw2DriiveOurs))
NeurIPS 2025
Raw
RL
Yes
Carla v2
Yes
Table 2: Notations of Dual Stream World Models in Raw2Drive. WM means the privileged world
model and WM means the raw sensor world model. t denotes the time-step. Both world models
consist of an Encoder, RSSM, and Heads. The privileged world model is similar to DreamerV3 [40]
while raw sensor world model has tailored encoder Enc and only has a tailored decoder head Dec. ! and 2 represent the operation during training and inference respectively.
Dual Stream
Privileged WM
Raw Sensor WM
Observation
0, è, Enc(ór)
Encoder State
e Enc(or)
fo 11(4111 a1-1; 86-1)
Deterministic State
hy = 16-11-100-118)11
he
fo (ha- ar- 1)-1)
RSSM
World Model
Train
8, = 40 (12,, Ee)
8, go (he Ee)
Stochastic State
Infer
S1 Po (l()
$1 po(le)
Reward
11 Reward $1)
Heads
Decoder
d, Decoder (1,, St)
Decoder(hi, $()
Continue
G Continue(he, 8t)
planner whose input is directly the raw video. In the training of the raw sensor world model, instead of reconstructing computationally expensive multi-view videos, it is guided by the alignment with the frame-wise feature of the trained privileged world model, facilitating an otherwise extremely difficult training process. We further propose the Guidance Mechanism to enforce alignment, ensuring consistency in future state predictions during rollouts across both world models. Additionally, leverages the prior knowledge embedded in the heads of the privileged world model to effectively guide the training of the raw sensor policy. The contributions are as follows:
To our best knowledge, Raw2Drive is the first MBRL framework for E2E-AD, i.e. from raw image input to planning, beyond existing IL [9, 6] or privileged input based RL approaches [36, 4].
Raw2Drive achieves state-of-oe saatt-off-hhe-ar the-art performance on the challenging CARLA v2 and Bench2Drive and surpasses II. methods by large margin, validating the power of RI.
We only use 64 H800 GPU days in total to deliver our final planner, and the cost can be further saved to 40 GPU days when Think2Drive is reused which dismisses our phase training. In comparison, IL-based UniAD costs about 30 GPU days yet it only solves even 3-4 corner cases in CARLA v2. We believe it is significantly less than the imitation learning approaches used in industry and the hope is our work could provide an orthogonal reference for industry applications.
2 Problem Formulation & Related Works
In Figure 2, we give a problem formulation where there are two streams: (1) privileged observations with ground-truth bounding boxes, and HD-Map. (II) raw sensor observations â with onboard sensors such as cameras, LiDAR, and IMU. Note that privileged observations are only accessible during training to facilitate learning. During inference, only sensor observations are allowed.
3

---

a) Train Phruse
c) Guidance Mechanism
Interaction
Rollout Guidance
Reinforcement,
Need to otigred
World Model
Policy
Stochastic
Privileged Input
Guildance
Tanoehhide
tnovntio
Igaundin
Interaction
Reinforcement
Rollout Privilaged World Model
Rollout
World Model
Policy
Raw Sensor Input (Multiiviow Video)
Head Guidance
b) Inference Phruse
Privileged World Model
Head
World Model
Policy
Row Sensor
World Model
Policy
Input (Multi-view Video)
Figure 2: The Overall Pipeline of Raw2Drive. (a) During training, privileged input train the privileged world model and paired policy. Then, the privileged world model used to guide the training of the sensor stream. (b) During inference, only sensor inputs available, which aligns with real-world autonomous driving. (c) The guidance mechanism consists of two parts: (I) Rollout Guidance ensure future modeling consistency; (II) Head Guidance ensure the supervision for raw sensor policy is accurate and stable.
3 Method
Instead of directly adopting classic MBRL structure like Dreamer V3 [40], Raw2Drive is dual- stream MBRL framework, as shown in Figure 2. consists four key components: two world models and two corresponding policy models. Detailed symbol definitions are in Table 2.
The rationale behind the two stream design is that: raw sensor data (e-g., high resolution multi-view videos) are high-dimensional and complex, presenting significant challenges to directly train a world model with low error [41, 42]. Thus, rather than building the raw sensor world model from scratch, we first build the privileged stream an auxiliary since the learning of the world model under structured and low-dimensional conditions is much easier and there are already some successes [4]. The raw sensor stream's learning process is guided by the trained privileged stream, which eases the task of extracting the eecisionnmmainng related information from raw sensor data. The Guidance Mechanism is carefully designed to avoid cumulative error and train-val gaps.
3.1 Privileged Stream
Privileged Input: As shown in Figure 2 (a), similar to Roach [36] and Think2Drive [4], we utilize time-sequenced BEV semantic masks o as input. Since it is commonly used in existing works, leave details of the privileged input in Appendix C.I.
As shown in the upper part of Figure 3, the same as Dreamer V3 [40], the privileged world model WM composed of the Encoder, Recurrent State-Space Model (RSSM) [43] and three heads, defined in the left part of Table 2. These components are used (I) for rollout train the privileged policy by RL; (II) to guide the training of the end-to-end raw sensor stream.
Privileged World Model:
Privileged Policy: As shown in the upper part of Figure 3, the privileged policy is composed of actor-critic networks and trained through rollouts, same as Dreamer V3 [32].
3.2 Raw Sensor Stream
As in Figure 2 (a), raw sensor input in this work is multi-view images and IMU. We adopt BEVFormer [44] as the encoder Enc(ô.) of the raw sensor stream, which outputs grid-shaped BEV features for the ease of receiving guidance from privileged streams, The details of the inputs are in Appendix C.I.
Raw Sensor Input:

---

Privileged Input
Privileged Input
Cont
Decoder
Reward
Dc+1
Cont
Decoder
Reward
Oc+2
Cont
Decoder
Reward
Or
Privileged
Privileged
Privileged
Encoder
Encoder
Encoder
Latent State
Latent State
Privileged
Privileged Input
World Model
Privileged Policy
Self-head
From Self-head
It Latent State
C.
Cc+1
F++1
Latent State
RSSM
RSSM
RSSM
a,
2,41
Privileged Eneader
Critic
Actor
Critic
Actor
Rigorous-deduen deduction
Rigorous
Privileged Input
Rollout in Privileged World Model
Figure 3: Training of Privileged World Model and Policy. The privileged world model WM is trained with time-sqquenced BEV semantic masks as inputs. The privileged policy æ is trained through rollouts in the privileged world model. The reward rt and continuous flag et generated by the heads of the privileged world model with the privileged input O2.
Privileged Input 1++
Privileged Input
Oc+2V
Privileged Encoder
Privileged
Privilegedr
Encoder
Encoder
Latent State
Latent State
Privileged World Model
Privileged Input
Intentions
Shared
Alignment
Shared
Alignment
Raw Sensor World Model
Intention
Intention
Latent State
RSSM
Latent State
RSSM
Sensor
Decoder
Sensor Encoder
Decoder
Sensor Encoder
Decoder
Encoder
Input (Multi-view Video at Time
6
uc+2
Raw Sensor Input [Multi-view Video at Time t+1)
Raw Sensor Input (Multi-view Video at Time t+2)
Figure 4: Training of Raw Sensor World Model. During training, the spatial- feature of the privileged world model serves as supervision instead of reconstructing multi-view video so that the learning could focus on decision related information. RSSM parameters are initialized from the privileged world model.
Raw Sensor World Model: As shown in Figure 4, the raw sensor world model has similar architecture the privileged world model except for the encoder Enc and heads. The different encoders are used to process different inputs. As for the heads, we only use the decoder head which provides supervision signals based BEV semantic masks instead of directly reconstructing multi-view videos. Additionally, we find that learning rewards or continuous flag (both only one scalar) could be harmful for the sensor stream. Specifically, observe that two adjacent similar frames could have very different reward and continuous labels, which is confusing and thus hampers convergence. We analyze these effects in Section 4.4.
Raw Sensor Policy: As Figure the raw sensor policy is trained by RL. with the dual-stream world model. During rollouts, the guidance mechanism detailed in Section 3.3 ensures consistency between

---

Privileged
Encoder
Psoudo-deduction
Pseudo deduction
RSSM
RSSM
RSSM
Privileged
Cont
Reward
Head Guidance
Cont
Reward
Head Guidance
Privileged Input
World Model
T++1
Raw Sensor
Ce
Ce+1
Policy
Latent State
Latent State
RSSM
RSSM
RSSM
&
ac+1
Sensor Encoder
Critic
Actor
Critic
Actor
derruus-eeduction
Rigorous deduction
Raw Sensor Input (Multi-view Video Time t)
Rollout Sensor Model
Figure 5: Training of Raw Sensor Policy. The raw sensor policy is trained through RL within the dual-stream world model. The raw model operates under striet deduction, while the reward ry and continuation flag e are derived from the privileged model via pseudo-deduction.
the two world models future predictions. We adopt the heads in the privileged world model to obtain the reward and continuous flag to provide more accurate and stable supervisor signals.
3.3 Guidance Mechanism
the previous section, we introduced Raw2Drive's 's dual-stream MBRL framework. In this section, we give details of the guidance mechanism about (I) how the privileged stream guide the learning process of both the raw sensor world and policy; (II) how to alleviate cumulative errors and train-val gaps of the dual-stream paradigm.
EEncoder State Spatial- Temporal)
Stochartic-Determile State
Rollout
Multi-view Video
Privileged Inputy
Dynamic Intention
Speed up
Slow down
Privllged
h4g
fe (he-
Replay Buffer
a
St
Ce
Turn right
Eliminated Randomness
a
Turn left
Static/ Intention
Sample from Distribution Sensor Stream
Ego-vehider
World Modal
Rawi
ae
Turn Right
action
Map Info
Rollout
Replay Suffer
(4) Encoder State
(b) Stochastic Deterministic Mechaati-Dettrmnnistic
State
(c) Sampling Process Two World Model
Figure 6: State Variables Aligned and Sampling Process in the Rollout Guidance. (a) The encoder state is aligned temporally and spatially. (b) The deterministic state and stochastic state is aligned to maintain dynamic and static intention consistency. (c) Eliminating Cumulative Errors Caused By Sampling. During the rollout process, when deducting next states, we only sample once for the stochastic state from the distribution of the raw sensor stream. The sampled state is fed into both streams eliminate the randomness and thus the alignment is more stable.
Rollout Guidance. In MBRL framework, the rollout of world model is crucial for effective policy learning. Thus, under the proposed dual-stream framework, the first type of guidance Rollout Guidance, which adopts the trained privileged world model provides supervisions during the entire rollout process of the raw sensor world model. Specifically,
shown Figure there are three
components the rollout process: (1) Encoded State é, (2) Deterministic State h, (3) Stochastic State 8. At each timestep I, both stream encode the observation (either privileged or raw sensor) into Encoded State. However, due to the high dimensionality and inherent redundancy multi-view video data, its encoded state é of the raw sensor stream may exhibit instability and errors. To address

---

this, we introduce a loss function to align it with the privileged encoded state e, ensuring spatial consistency each timestep, called Spatial-Tempon Temporal Alignment Loss.
Meanwhile, as illustrated in the upper part of Figure 2 (c), and Figure 6 (b), during the rollout process, the deterministic and stochastic states serve distinct modeling purposes. The deterministic state primarily predicts the ego vehicle's state, while the stochastic state focuses on anticipating the behaviors of other traffic participants, such as sudden braking or deceleration. To ensure the consistency of states in the rollout of the two world models, additional supervisory signals are introduced to ensure alignment with the privileged world model. Specifically, the deterministic state employs L2 loss to maintain prediction consistency, whereas the stochastic state is sampled from an independent one-hot distribution. KL divergence [45] then used to constrain the distributions of the two information streams, ensuring they remain as similar as possible. Furthermore, alignment is enforced across all timesteps, called A Abstract-State Alignment Loss. The overall loss function is:
grid nurn
CRalloal Bee Σ
MSE(e), et) + (BBKKL(s,, 3,) + 3,MSE(h,, h,))
(1)
where grid num means the number of grids under BEV-view and Be, 9r. B, are the loss weights of encoder state, deterministic state, and stochastic state. Notably, in the standard RSSM [40], the deduction of stochastic states employs sampling during rollout. This sampling process can be detrimental for the training of raw sensor world model, as difference caused by randomness during sampling between the two models could accumulate over time. As result, in the later timesteps, using states from the privileged world model as supervision signals is confusing for the raw sensor world model to align with. To this end, shown in Figure 6, we only conduct sampling from the distribution of raw sensor stream and directly feed the sampled variable into the privileged world model to deduct its next state. In this way, the cumulative errors caused by randomness is eliminated and thus is beneficial for the trianing of raw sensor model.
Head Guidance. During the training of the raw sensor world model, as mentioned in Sec 3.2, only the decoder head is trained. We omit the reward head and continuous head learning because directly training the two heads with raw sensor inputs has a convergence issue. Adjacent frames in the video are highly similar while value of reward and continuous flag could fluctuate abruptly, as in Figure 9. As result, it is difficult for the network to learn stable patterns.
For MBRL, reward and continuous flag play crucial role guiding raw policy training. Thus, we use the accurate reward and continuous flag from the privileged world model during the training of raw sensor policy. As shown in Figure 2 (c) Lower Part and Figure 5, at each timestep t, the raw sensor world model executes the action obtained from the raw policy म. transitioning the system to next latent state. At the same time, the privileged world model conducts the same action to rollout. Since Rollout Guidance imposes the consistency between the two world models, the reward r; and the continuation flag a from the privileged world model could be directly used. Finally, the resulting sequence serves the training data for optimizing the raw sensor policy. Notably, also adopt the technique to eliminate randomness mentioned in Section 4.4 so that the reward and continuous flag is accurate. Furthermore, we also use the trained privileged policy to collect to replay buffer and to distill action distributions to raw sensor policy. We summarize our training pipeline in Appendix B.
4
Experiments
4.1
Datasets and Benchmark
We employ the CARLA simulator [33] (version 0.9.15.1) for closed-loop driving performance evaluation. Note that during evaluation, the model only has access to raw sensor observations and is prohibited from utilizing privileged observations. Experimental details are in Appendix C.
Leaderboard 2.0 [33]: It includes two long routes, validation and devtest. Each of which comprises several routes with lengths of 7-10 kilometers and containing a series of complex corner cases. As driving inherently follows a Markov decision process [46], evaluating performance over long routes unnecessary. Moreover, the penalty mechanism employed in scoring [4] fails to accurately reflect the true evaluation capability of the model.
7

---

Table 3: Performance on Carla Official Town13 Validation and Devtest Benchmark. *denotes expert feature distillation. discussed in garla-garage [47] and Section 4.3, long routes evaluation in Leaderboard 2.0 can't reflect the actual driving performance [34].
Closed-loop Metric
Method
Venue
Scheme
Modality
(C))
Validation
Devtest
Validation
Deviest
Validation
Devtest
AD-MLP [13]
Arxiv
State
0.00
0.00
0.00
UniAD-Base [9]
CVPR 2023
TL.
Tmage
0.15
0.00
0.51
0.07
0.23
0.04
VAD [10]
ICCV 2023
Image
0.31
0.04
DriveTrans [39]
ICLR 2025
TL
Tmage
0.85
0.68
1.42
2.13
0.33
0.35
TCP.traj [14]
NeurIPS
Image
ThinkTwice" [16]
CVPR 2023
IL
Image
0.50
0.64
1.23
1.78
0.35
0.43
Drive/dapter [17]
Image
Raw2Drive (Ours)
NeurIPS 2025
RL.
Tmage
4.12
3.56
9.32
6.04
0.43
0.42
[4]
RL Exper
Privilegeed
Table 4: Performance on Bench2Drive Multi-Ability Benchmark. denotes expert feature distillation. IL represents imitation learning. RL represents reinforcement learning. RL expert Think2Drive [4] uses privileged information for training.
Ability
Method
Venue
Scheme
Modality
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
TCP-4a6"
NourTPS 2022
II
Image
8.89
24.29
$1.67
40,00
46.28
34.22
AD-MLP
UniAD-Base (9)
CVPR 2023
II.
Image
14.10
17.78
21.67
10.00
14:21
15.55
ThinkTwize" [16]
CVPR 2023
IL
Image
27.38
18.42
35.82
50.00
54.25
37.17
[10]
ICCV 2023
II.
Image
18
20.00
19:15
18.07
Drive/dapter [17)
Image
DrivelHans [39]
ICLR 2025
I1
Image
17.57
35.00)
48
40.30
52 10
38.60
Rawee (Uus)
NourlS 1025
RL
Image
43.35
51.11
60.00
50,00
02.26
53.34
Expert
Privtleged
Table 5: Results on Bench2Drive Closed-loop Benchmark. *denotes expert feature distillation. denotes expert feature distillation. IL represents imitation learning. RL represents reinforcement learning. RL expert Think2Drive [4] uses privileged information for training.
Closed-loop Metric
Method
Venue
Scheme
Modality
DS
SR(%)
Efficiency
Comfort
TCP-traj" [14]
NeurlPS 2022
II.
Image
59.90
30.00
76.54
18.08
AD-MIPP
Arxiv 2023
II.
State
18.05
0.00
48.45
22.63
UniAD-Base [9]
CVPR 2023
IL
Image
45.81
16.36
129.21
43.58
VAD [10]
ICCV 2023
IL
Image
42.35
15.00
157.94
46.01
ThinkTwice* [16]
CVPR 2023
IL
Image
62.44
31.23
69.33
16.22
DriveAdapter* [17]
ICCV 2023
II.
Image
64.22
33.08
70.22
16.01
GenAD [48]
ECCV 2024
IL
Image
44.81
15.90
Drive Trans [39]
ICLR 2025
IL
Image
63.46
35.01
100.64
20.78
MomAD [49]
CVPR 2025
IL
Image
44.54
16.71
170.21
48.63
Rave (Ours)
NeurlPS 2025
RL
Image
71.36
50.24
214.17
22.42
Thivk2Drive [4]
ECCV 2024
RL. Expert
Privileged
91.85
85.41
269.14
25.97
Bench2Drive [34]: A more comprehensive and fair benchmark, it includes 220 short routes with challenging corner case per route for analysis of different driving abilities. Following Bench2Drive, use 1,000 routes under diverse weather conditions for RL training.
Metric
We employ the official metrics of CARLA [33] for evaluation. Infraction Score (IS) measures the number of infractions made along the route, accounting for interactions with pedestrians, vehicles, road layouts, red lights, etc. Route Completion (RC) indicates the percentage of the route completed by the autonomous agent. Driving Score (DS), is calculated as the product of Route Completion and Infraction Score. In Bench2Drive [34], additional metric Success Rate is used, which measures the proportion of successfully completed routes within the allotted time and without traffic violations.
4.3 Comparison with State-of-the-Art Works
As in Table 3, similar to the conclusions of Bench2Drive, the long-road evaluation in CARLA 2.0 fails to accurately reflect real driving performance due to its cumulative penalty scoring mechanism. In its devtest/validation routes, the first scenario is ParkingExit, where traditional imitation learning

---

methods struggle to solve the task. This is primarily because these models typically rely on L2/L1 loss for training while ParkingExit requires large steering angles, which are challenging learn effectively due to the inherent imbalance in the data distribution. Additionally, when collision causes a blockage, the evaluation stops immediately, preventing the continuation subsequent scenarios. This does not mean that the performance is poor in other scenarios. Therefore, we mainly focus on the short-route closed-loop evaluation results of Bench2Drive. As shown in Table 4, our method achieves SOTA performance in multi-ability benchmark. In Bench2Drive's is closed-loop benchmark, as shown in Table 5, Raw2Drive achieves SOTA performance in raw sensor end-to-end methods.
4.4 Ablation Study
Following Benc [34] and PriveTranffommr [39], the tiny validation set Dev10 comprises 10 carefully selected clips from the official 220 routes. These clips are chosen to be both challenging and representative, with low variance. To avoid overfitting Bench21rrive-221 repeating each experiment 3 times and reporting the average.
use Dev10 for ablations,
Choice of Heads in Raw Sensor World Model. Due to the high-dimensional redundancy of image information and the fact that both the reward and continue flags are represented as scalars, the network struggles achieve stable convergence during training. To further investigate this, we conduct an ablation study on the raw sensor world model heads. As shown in Table 6, the decoder head provides effective supervision, significantly enhancing the learning of the raw sensor world model. contrast, the reward and continuation heads introduce ambiguity that hinders convergence, leading suboptimal world model training and degraded policy performance. We provide detailed visualization in the Appendix H.2.
Table 6: Ablation on the Raw Sensor World Model Heads. Only the decoder head is used.
Table 7: Ablation on the Abstract-State Align- ment in Rollout Guidance.
Decoder
Heads Reward
Continue
DS
SR
Encoder
Deterministic
Latent State
Stochastic
DSt
SR
x
17.4
1,2/10
0.0
0.0/10
x
x
83.5
7,5/10
x
36.4
2.4/10
x
46.6
3.4/10
x
2.8/10
7
34.5
2,2/10
83.5
7.5/10
Abstract-State Alignment Rollout Guidance. We conduct an ablation study to examine the role of Abstract-State Alignment in rollout guidance. As shown in Table 7, since the world model is trained during rollouts, any state misalignment can lead to discrepancies between the predictions of the world models. Our results show that when any component of the rollout guidance is missing, the model can only handle simple tasks, such as moving straight or turning left, while failing to address more complex interactive corner cases. This misalignment significantly hinders the training of the raw sensor-based world model. All three components of rollout guidance components are thus essential for robust learning and performance. Loss comparisons are provided in Appendix
Alignment (based on Dev 10) is shown in Table 8. Spatial Alignment ensures consistency between image repre- sentations and BEV representations-reentee removing it is similar to "driving blind." Temporal Alignment maintains the consistency of future predictions across time. Both components are essential and complementary for a stable world model rollout.
Opatial-Temports Alignment in Rollout Guidance. The results on
Table Ablation on the
Alignment in Rollout Guidance.
Spatial Alignment
Temporal Alignment
DS
SR 3R(%)T ())
X 0.0
0.0/10
13.6
1.2/10
X
9.24
0.8/10
83.5
7.5/10
Head Guidance. For this ablation study, we assume that the raw sensor world model is trained with three heads. As shown Table 9, both Setting and Setting follow this configuration, aligning with the setup in the last row of Table 6 and reflecting the same considerations and conclusions

---

drawn in Ablation 4.4, Our results indicate that head guidance provides some improvement in policy training. However, the additional heads increase training complexity and degrade performance. Thus, Raw2Drive adopts only the decoder head with head guidance.
Table 9: Ablation on head guidance for raw sensor policy learning, D/R/C: decoder, reward, continuation; HG: head guidance.
Method
D
R
C
HG
DSt
SR
Raw2Drive (Ours)
x
83.5
7.5/10
Setting I
V
V
V
34.5
2.2/10
Setting
x
26.4 1.6/I0
Shared Paramefer. To investigate the impact of parameter sharing, we conduct an ablation study on whether the Recurrent State-Space Model (RSSM) and the decoder head should share parameters. As shown in Table 10, we compare different configurations where these components are either shared or kept separate. Our findings indicate that sharing parameters between the RSSM and the decoder head leads to better performance. This suggests that parameter sharing facilitates more efficient representation learning, improving the consistency and generalization of the world model.
Table 10: Ablation Study the Sharing Parameters (RSSM and Decoder Head).
Table 11: Ablation Study on Raw Sensor Policy Training. Compare two strategies: (1) directly using the privileged policy, and (2) fine-tuning the privileged policy on raw sensor inputs.
Method
DS
SRI
Raw2Drive (Ours)
83.5
7.5/10
w/o Shared RSSM
53.2
5,4/10
Method
DS
SR
w/o Shared Head
65.6
6.1/10
Directly Use Privileged Policy
58.4
5.6/10
Fine-tune Privileged Policy (Ours)
83.5
7.5/10
Policy Finetuning. To evaluate the necessity of the raw sensor world model, we conduct an ablation study by comparing policy fine-tuning with directly using the privileged policy. Specifically, analyze the performance difference between policies trained with and without fine-tuning under the raw sensor world model. As shown in Table 11, directly applying the pre-trained privileged policy without fine-tuning results in suboptimal performance, as the policy lacks adaptation to the raw sensor world model's learned dynamics. In contrast, fine-tuning the policy within this world model leads to significant improvements, particularly in handling complex interactive tasks. This result underscores the importance of the raw sensor-based world model for effective policy learning and adaptation.
Real-time Inference. We conducted latency analysis for each module. In end-to-end autonomous driving, the perception backbone (e.g., surround-view image encoder) typically dominates the overall latency. Our world model and policy are highly efficient (each under 2ms), while the raw sensor stream is mainly bottlenecked by the vision encoder (e.g., BEVFormer). The results in Appendix G.
Conclusion
We propose Raw2Drive, the first end-to-end model-based reinforcement lcarning method in au- tonomous driving. Our approach introduces a novel dual-stream architecture and a guidance mech- anism effectively enable MBRL learning. The approach is fulfilled by the careful treatment of the two world models for the raw and privileged information, respectively, and achieves the of the-art performance recently released benchmarks. We hope this work serves stepping stone toward exploring reinforcement learning for end-to-end autonomous driving.
I Limitations: In our setting, the privileged input is ground truth bounding boxes and HD-Map. And in real-world autonomous driving, for industry, the ground truth bounding boxes and HD-Map can be from human annotation advanced perception algorithms. Reinforcement learning in the real world is also a technical issue that would be solved by 3DGS [50, 51, 52] or a diffusion- based simulator [53] in the future. While CARLA remains the only viable closed-loop simulator for RL research at present, our work focuses on policy learning and introduces a dual-stream world model design that is conceptually decoupled from the specific simulator.
Social Impact: Raw2Drive presents an efficient reinforcement learning framework for end-to- end autonomous driving, mitigating key issues of imitation learning such as causal confusion and distribution shift. By learning robust world models from raw sensors, it enhances the safety, reliability, and generalization of autonomous driving systems.
10

---

References
[1] Sidney MeMll, Blair Lehman, Reinhard Pekrun, and Art Graesser. Confusion can be beneficial for learning. Learning and Instruction, 29:153 13:15-777, 2014.
[2] Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33:2564-2555 2575, 2020.
[3] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[4] Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforeement learning by thinking latent world model for quasi-realistic autonomous driving (in carla-v2). In ECCV, 2024.
[5] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-ent autonomous driving: Challenges and frontiers. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[6] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer- based sensor fusion for autonomous driving. TPAMI, 2023.
[7] Penghao Wu, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy pre-training for autonomous driving via self-supervised geometric modeling, 2023.
[8] Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijic Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang. Hanming Deng, et al. Delving into the devils of bird's seye-veew perception: A review, evaluation and recipe. IEEE Transactions Pattern Analysis and Machine Intelligence, 4614)2215- 2023.
[9] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oricnted autonomous driving. In CVPR, pages 17853-17862, 2023.
[10] Bo Jiang. Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang. and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. ICCV, 2023.
[11] Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Conference Robot Learning, pages 1434 1443. PMLR, 2022.
[12] Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-line approach. In Conference on Robot Learning, pages 910-920. PMLR, 2023.
[13] Jiang-Tian Zhai, Ze Feng. Jinhao Du, Yongqiang Mao, Hang-JJang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving nuscenes. arXiv preprint arXiv:2305.106 10430, 2023.
[14] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guide control prediction for end-to-end autonomous driving: simple yet strong baseline. In NeurIPS, 2022.
[15] Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Ide-net: Interactive driving event and pattern extraction from human data. IEEE robotics and automation letters, 6(2):3065-300) -0726 2021.
[16] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023.
[17] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langochuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023.
[18] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness. Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through 2015.
deep reinforcement learning. nature, 518(7540):529-533
[19] Yazhe Niu, Yuan Pu, Zhenjie Yang. Xueyan Li. Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, and Yu Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
11

---

[20] Yuan Pu, Yazhe Niu, Jiyuan Ren. Zhenjie Yang. Hongsheng Li, and Yu Liu. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint arXiv:2406 10667, 2024.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (xXiv:1707.063477 2017.
[22] Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[23] Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, and Yaodong Yang. Rat: Adversarial attacks deep reinforcement agents for targeted hehaviors. Proceedings of the AAAI Conference on Artificial Intelligence, 39(15):15453-15461 Apr. 2025.
[24] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforeement learning algorithm. arXiv preprint 17X1v:1712.018155 2017.
[25] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[26] Mobileye. Mobileye under the hood. https: h/wp:////wwwww/. wwwiwemobleeee com, 2022.
[27] NVIDIA. NVIDIA DRIVE End-to-End Solutions for Autonomous Vehicles. https://developer. nvidia. htt//ddieeere 2022.
[28] Dian Chen, Brady Zhou, Vladien Koltun, and Philipp Krähenbühhl. Learning by cheating. In Conference on robot learning, pages 66-75, PMLR, 2020.
[29] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[30] CARLA Team. Get started with Leaderboard 1.0. http:///dwerbrrrd...rrrrrrr/eeee 2022.
[31] CARLA Team. Get started with Leaderboard 2.0 leaderboard.carfa.org. https://leaderbboard. carla. org/ htrg/eadeebbard/... 2023.
[32] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dreamer: Efficient reinforcement learning through latent imagination. In International Conference on Machine Learning, 2020.
[33] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[34] Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi- ability benchmarking of closed-loop end-to-end autonomous driving. In NeurlPS 2024 Datasets and Benchmarks Track, 2024.
[35] Jianyu Chen, Bodi Yuan, and Masayoshi Tomizuka. Model-free deep reinforcement learning for urban autonomous driving. In 2019 IEEE intelligent transportation systems conference (ITSC), pages 2765-2771. IEEE, 2019.
[36] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool, End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[37] Yingyan Li, Lue Fan, Jiawei He. Yuqi Wang, Yuntao Chen. Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. In The Thirteenth International Conference on Learning Representations, 2025.
[38] Hang Wang, Xin Ye, Feng Tao, Chenbin Pan, Abhirup Mallik, Burhaneddin Yaman, Liu Ren, and Junshan Zhang. Adawm: Adaptive world model hased planning for autonomous driving, 2025.
[39] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Danijar Hafner, Jurgis Pasukonis. Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.041 041000 2023.
12

---

[41] Yiran Qin. Zhelun Shi. Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li. Zhenfei Yin, Xihui Liu, Lu Sheng. Jing Shao, et al. Worldsimbench: Towards video generation models world simulators. arXiv preprint arXiv:2410.18772 2024.
[42] Hongchi Xia, Yang Fu, Sifei Liu, and Xiaolong Wang. Rghd objects the wild: Scaling real-world 3d object learning from rgb-d videos, arXiv preprint arXiv:2401. 12592, 2024.
[43] Andreas Doerr Christian Daniel, Martin Schiegg, Nguyen- Dguyen-Tuoog Duy, Stefan Schaal, Marc Toussaint, and Trimpe Scbastian. Probabilistic recurrent state-space models. In International conference machine learning, pages 1280-1289. PMLR, 2018.
[44] Zhiqi Li, Wenhai Wang, Hongyang Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird' s-yye-view representation from multi-camera images via spatiotemporal trans- formers. ECCV, 2022.
[45] Solomon Kullback and Richard A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
[46] Berhaard Jacger, Daniel Dauner, Jens Beißwenger. Simon Gerstenecker, Kashyap Chitta, and Andreas Geiger. Carl: Learning scalable planning policies with simple rewards. arXiv preprint arXiv:2504. 17838, 2025.
[47] Julian Zimmerlin, Jens BeiBwenger, Berhhard Jaeger. Andreas Geiger, and Kashyap Chitta Hidden biases of end-to-end driving datasets. arXiv preprint arXiv:2412.09002, 2024.
[48] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. arXiv preprint arXiv:2402.11502, 2024.
[49] Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang, and Yadan Luo. Don't 't shake the wheel: Momentum-aware planning in end-to-end autonomous driving. arXiv preprint arXiv:2503.03125, 2025.
[50] Zipci Ma, Junzhe Jiang. Yurui Chen, and Li Zhang. Béziergs: Dynamic urban scene reconstruction with bézier curve gaussian splatting. In ICCV, 2025.
[51] Xianliang Huang. Jiajie Gou, Shuhang Chen, Zhizhou Zhong, Jihong Guan, and Shuigeng Zhou. Iddr-ngp: Incorporating detectors for distractors removal with instant neural radiance field. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1343-1351, 2023.
[52] Shuhang Chen, Xianliang Huang, Zhizhou Zhong, Juhong Guan, and Shuigeng Zhou. A focused human body model for accurate anthropometric measurements extraction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22658 22662222666222 2025.
[53] Jiazhi Yang, Kashyap Chitta, Shenyuan Gao. Long Chen, Yuqian Shao, Xiaosong Jia. Hongyang Li, Andreas Geiger, Xiangyu Yue, and Li Chen. Resim: Reliable world simulation for autonomous driving. arXiv preprint arXiv:2506.0940 099811 2025.
[54] Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint 10Xiv:77090071720 2017.
[55] Xiaosong Jia, Shaoshuai Shi, Zijun Chen, Li Jiang, Wenlong Liao, Tao He, and Junchi Yan. Amp: Autoregressive motion prediction revisited with next token prediction for autonomous driving. arXiv preprint 2433334..333.3. 2024.
[56] Kairui Yang, Zihao Guo, Gengjie Lin, Haotian Dong, Zhao Huang, Yipeng Wu, Die Zuo, Jibin Peng. Ziyuan Zhong, Xin Wang, et al. Trajectory-IIm: A language-based data generator for trajectory prediction in autonomous driving. In The Thirteenth International Conference Learning Representations, 2025.
[57] Yutao Zhu, Xiaosong Jia, Xinyu Yang, and Junchi Yan. Flatfusion: Delving into details of sparse transformer-basee camera-lidar fusion for autonomous driving. arXiv preprint arXiv:2408.06%: 2024.
[58] Han Lu, Xiaosong Jia, Yichen Xie, Wenlong Liao, Xiaokang Yang, and Junchi Yan. Activead: Planning- oriented active learning for end-to-end autonomous driving. arXiv preprint arXiv:2403.02877 2024.
[59] Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey of large language models for autonomous driving. arXiv preprint 201002231, 2023.
13

---

[60] Zhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li. Yuqian Shao, Xuekai Zhu, Haisheng Su, and Junchi Yan. Drivemoe: Mixture-of-experts for tision-anguuage-attion model in end-to-end autonomous driving. arXiv preprint urXiv:2505.1022 162788. 2025.
[61] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transforeer for agent trajectory prediction via scene encoding, IEEE transactions on pattern analysis and machine intelligence, 45(1 (5(11):13888 11)11)66011133355 2023.
[62] Junqi You, Xiaosong Jia, Zhiyuan Zhang, Yutao Zhu, and Junchi Yan. Bench2drive-r: Turning real world data into reactive closed-loop autonomous driving benchmark by generative model. arXiv preprint arXiv:2412.09647 2024.
[63] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicidly differentiable reward learning for preference-basee based reinforcement learning. In Advances in Neural Information Processing Systems (NewrlPS), volume 35, 2022.
[64] Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang, and Bo Xu. Picor: Multi-task deep reinforcement learning with policy correction. Proceedings of the AAAI Conference Artificial Intelligence, 37(6):6728-671 6736, Jun. 2023.
[65] Fengshuo Bai, Rui Zhao, Hongming Zhang. Sijia Cui, Ying Wen, Yaodong Yang. Bo Xu, and Lei Han. Efficient preference- based reinforcement learning via aligned experience estimation. arXiv preprint arXiv:2405. 18688, 2024.
[66] Fengshuo Bai, Yu Li, Jie Chu, Tawei Chou, Runchuan Zhu, Ying Wen, Yaodong Yang, and Yuanpei Chen. Retrieval dexterity: Efficient object retrieval in clutters with dexterous hand. arXiv preprint arXiv:2502. 18423, 2025.
[67] Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: Realignment during test time for personalized preference adaptation of LLMs.. In The Thirteenth International Conference on Learning Representations, 2025.
[68] Lu Li, Jiafei Lyu, Guozheng Ma, Zilin Wang, Zhenjie Yang. Xiu Li, and Zhiheng Li. Normalization enhances generalization in visual reinforcement learning. arXiv preprint arXi::2306.006: 2023.
[69] Ming Zhang, Shenghan Zhang, Zhenjie Yang, Lekai Chen, Jinliang Zheng, Chao Yang, Chuming Li, Hang Zhou, Yazhc Niu, and Yu Liu. Gobigger: A scalable platform for cooperative-compettivy multi-agent interactive simulation. In The Eleventh International Conference on Learning Representations, 2023.
[70] Xuckai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang, Youhang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, al. Flowrl: Matching reward distributions for Ilm reasoning. arXiv preprint arXiv:2509.1520 2025.
[71] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 726-737. PMLR, 2023.
[72] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13723-13733, 2023. 2023.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer Abstract and Introduction Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
A standalone limitation section is provided in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Refer to Section 3.
15

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Refer to Section 3 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
16

---

Justification:
We will release code and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
ceniiiiisincicoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wwtpccccccgppppceeeeceoooccccpce hwtp//uwdee//eeee/eeeee htte///ooSSueiiseeeeeeeeeee ssisnnoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Refer to Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to Section and Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eurri..eeeeee (pppblic//thiccccuuleleeees.
Answer: [Yes]
Justification: Research conducted this paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts are discussed in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our training and evaluation are conducted on publicly licensed datasets and benchmarks and See Section C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and models will released with well-organized documentation and instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdscourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This study does not involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not utilize LLMs an important, original, or non-standard component its core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tte//isn...//eeeee tweeurip.cc/oooffeeeeeeee.e /LLM) for what should or should be described.
21

---

A Related Works
A.1 End-to-End Autonomous Driving Based on Imitation Learning
Imitation Learning (IL) [5, 54, 55, 56, 57, 58, 59, 60] is foundational method for training agents by mimicking expert behavior, especially suitable for end-to-end autonomous driving tasks. IL aims to replicate human driving actions, thus reducing system complexity and simplifying the decision-making process from perception to control.
Some approaches [14, 16, 17, 61] typically involve collecting large amounts of human driving data and using supervised learning to directly map sensor inputs (such as camera, LiDAR data, etc.) to control outputs (such as steering angle, throttle, and brake). The expert driving actions recorded as state-action pairs, and imitation learning is used to train a model that approximates this behavior. The classic TCP model [14] uses single camera input in dual-branch design that integrates trajectory prediction and multi-step control within a single framework. It flexibly combines the outputs of both branches based prior planning to optimize the final control signal. The latest UniAD [9] approach goes further by using aurround-veew camera inputs and leveraging the query feature of transformer architectures, integrating detection, tracking, mapping, trajectory prediction, occupancy grid prediction, and planning into a single differentiable end-to-end pipeline. This design sets new benchmark for integrating tasks in autonomous driving. To accelerate grid-based scene representation in UniAD, VAD [10] proposes a fully vectorized end-to-end framework that uses vectorized representations for agent motion and map elements, significantly improving task efficiency,
Although behavioral cloning methods perform well in simple driving environments, they still cannot successfully complete long-term sequence decisions in complex traffic scenarios [34, 13, 62], which also called corner case. The reason is that its network uses regression loss to generate future trajectories, and scenes with large turns will be averaged by straight road scenes. For example, the recent closed-loop Bench2Drive [34] evaluation showed that supervised learning models are sensitive to shifts in data distribution; they perform well in simple driving interactions but tend to accumulate errors in dynamic traffic settings, causing the model to deviate from the intended driving path.
A.2 End-to-End Autonomous Driving based on Model-free Reinforcement Learning
Model-free reinforcement learning [36, 63, 64, 65, 66, 67, 68, 69, 70] is another key approach for end-to-end autonomous driving, optimizing decision-making through rewards rather than direct imitation. Early RL work, such MaRLn [3], demonstrated the potential of reinforcement learning in autonomous driving. However, it required extensive pre- tre-training, with around 20M (23 days) of training in a single town or 50M (57 days) in multi-towns solve only standard cases in CoRL 2017 [33]. The complexity of perception in autonomous driving makes it difficult to optimize perception and control jointly, increasing the challenge of training end-to-end systems with RI. To reduce the impact of perception on decision training. some classic RL methods, such Roach [36]. use 2D Bird's 's Eye View (BEV) representations as input observations, simplifying the task of learning from raw sensor data and focusing on high-level environmental features. And then use RL methods to collect offline data for training the end-to-end method [14, 71, 72].
A.3 End-to-End Autonomous Driving based on Model-based Reinforcement Learning
Model-free RL suffers challenges in complex driving scenarios corner cases that require long-term memory. Think2Drive [4] combines Roach's 's [36] 2D BEV inputs with a world model, significantly improving sampling efficiency and effectively modeling sequential interactions, leading to strong performance in complex traffic scenarios. To our best knowledge, there is no model-based RI. end-to-end method in autonomous driving.
B Details of Training Pipeline
As shown in Alogrithm 1,the training pipeline of Raw2Drive consists of two stages. In Stage I, use privileged observations for MBRL training. The world model and behavior policy are updated alternately.
In Stage II, we train the raw sensor world with the help of the proposed guidance mechanism. Compared to Stage I, the world model is trained with an additional loss derived from the rollout guidance. For behavior policy training, we fine-tune the privileged policy by interacting with the raw sensor world model with head guidance.
22

---

Algorithm 1: Training Pipeline of Raw2Drive
Privileged Observation World Model WM, Policy
Raw Sensor Observation O, World Model WM. Policy
Privileged Replay Buffer B, Replay Buffer B Training iterations N
Stage 1: Privileged World Model and Policy Training
for = 1 to N do
Collect trajectories (Ot, (t,, Ca2, 0641) with current to interact with simulator and store them in
Buffer B;
Sample trajectory (or:T, Qu:T, fery CapTy 0c+1:T++1) from Privileged Replay Buffer B; Train Priviledged World Model WM with prediction loss Lpred, dynamics loss Edyn and representation loss Grep with weights Spreds Bainn, Brepi
LwM
18
D (PreehCpred + Caya PrepCrep)
Sample trajectory Ou: from Privileged Replay Buffer B;
Rollout in Priviledged World Model to obtain the predicted latent state ( (e:),, as well as the predicted reward and continuous flag. These predictions are then combined to construct the trajectory:
(OET, us.r, Yee,, Carr, 1:41:7411 $141-441):
Train privileged policy æ based the actor-critic algorithm with the above trajectory which rollouts in Priviledged World Model;
Stage 2: Raw Sensor World Model and Policy Training
for 1 1 to N do
Collect trajectories (oe, oe; āe, ft, ce, ōt+1) by using raw sensor policy â to interact with simulator store them in Raw Replay Buffer B:
Sample trajectory (oLe+T, ÕesTT, den+T, fretT, Gea+T, 0++1-4+++) from Raw Replay Buffer B:
Train Raw Sensor World Model WM with additional loss in Rollout Guidance for latent state
alignment;
wwx = LwM + LRollou's
Sample trajectory θer from Raw Replay Buffer
Rollout in Raw Sensor World Model to obtain the predicted latent state (herr, ScT). Using Head
Guidance to obtain the predicted reward and continuous flag. These predictions are then combined construct the trajectory: (ōur, T fer, & f4+1:4+1 1:++::+));
Train Raw Sensor Policy based on the actor-critic algorithm with the above trajectory which rollouts
in Raw Sensor World Model;
Details of the Experiment
C.1 Details of the Dual Stream Input
For the privileged imputs, we utilize BEV semantic segmentation masks E (0, 1) HxxCC as image input and ego vehicle info as vector input. Each channel in the BEV masks represents the presence of a specific type of object. and consists of C' masks of size H x W. Note that the C' 43 channels of semantic segmentation masks correspond to static objects (c.g. roads, lines, lanes, and the ego vehicle) and dynamic objects (signs, lights, pedestrians, vehicles, and obstacles).
generated from the privileged information obtained from the simulator
For the raw sensor inputs, we use BEVFormer [44] as the encoder surrounding RGB images to achieve grid-shape BEV features.
C.2 Action Space in Raw2Drive
To simplify the action space, we design set of 39 discrete actions. The full list of actions provided in Table 12.
23

---

Table 12: The Discrete Actions. The continuous action space is decomposed into 39 discrete actions,
each for specific values of throttle, steer, brake and reverse. Each action rational and legitimate.
Throttle
Brake
Steer
Reverse
Throttle
Brake
Steer
Reverse
Throttle
Brake
Steer
Reverse
0
False
0.3
0
-0.2
False
0
0.1
False
0.7
0
-0.5
False
0.3
0
-0.1
False
0
0
0.3
False
0.7
0
-0.3
False
0.3
0
0
False
0
0
0.6
False
0.7
0
-0.2
False
0.3
0
0.1
False
0
0
1.0
False
0.7
-0.1
False
0.3
0.2
False
0.5
-0.5
True
0.7
0
False
0.3
0
0.3
False
0.5
0
-0.3
True
0.7
0
0.1
False
0.3
0
0.5
False
0.5
0
-0.2
True
0.7
0
0.2
False
0.3
0
0.7
False
0.5
0
-0.1
Truc
0.7
0.3
False
0
-1
False
0.5
0
True
0.7
0
0.5
False
0
0
-0.6
False
0.5
0
0.1
True
0.3
0
-0.7
False
0
0
-0.3
False
0.5
0
0.2
True
0.3
0
-0.5
False
0
0
-0.1
False
0.5
0.3
True
0.3
0
-0.3
False
1
0
False
0.5
0
0.5
True
C.3 Model Configuration
We implement the model using PyTorch. Both world models trained with learning rate of 1e-5, weight decay 0.00, and the Adam W optimizer. The behavior policy is trained with a learning rate of 3e-5, weight decay of 0.00, also using AdamW. The weights Be, Sh, and B, are set to 10, 5, and 10, respectively.
C.4 Reward Design in Raw2Drive
We adopt the reward design and reward shaping approach from Think2Drive [4].
D Details of the losses in World Model
Train the World Model W, using prediction loss Lpred. the dynamics loss Lagns and the representa- tion loss Crep, the loss weights are respectively Spreds Bdym: Srep:
(P) = In po(er se, h4) Impa(ri st, he)
In po(a se, h4)
(2)
(dya(0) = max (1, KL [sg (4o (sx he, ez)) ) po(se ha)l) (0) = max (1, KL (se Sel hes ex) || (po(se) hz))l)
E
Details of the losses in Behavior Policy
The critic network Py estimates the distribution of future returns R while the value function 2 represents the expected value of the return at state $t, train Critic network in the Privileged Behavior Policy Tp by maximum likelihood loss:
((v)= = Σ In (Rp((s))
R ir + 70t ((1 - A)oe + AR(
R) -e
Train the actor network through policy optimization, using entropy regularization H. exponential moving average (EMA) for smoothing, entropy regularization coefficient 17 and the operation for gradient stability.
((0) = Σ sg (R)) - us (())))
m) ( log (acls)))
v/HL [To (aelst)
EMA (Per (R,,, 95) - Per(Rt) 5), 0.99)

---

1.2
Rollout Guidance w/o Rollout Guidance
Loss (x100)
0.6
0.4
0.2
20
100
Training Iter (x1000)
Figure 7: Comparison of the Raw Sensor World Model Loss and w/o Guidance.
Rollout Guidance
As shown Figure 7, the rollout guidance ensures consisteney between the duall-stram world models during rollouts, playing critical role in training the raw sensor world model; without it, the network struggles to converge.
G Inference Latency
Table 13: Latency Comparison between Privileged and Raw Sensor Streams.
Method
Modality
Encoder Latency (ms)
World Model Latency (ms) Policy Latency (ms)
Privileged Stream
BEV State
(5x (onv) 600 (BEVFormer)
Raw Scnsor Stream
Multi-view images
H Visualization
H.1 Decoder Output by the World Model
We visualize the output of the raw sensor decoder to evaluate its reconstruction quality. In our dual-stream architecture, the raw sensor world model replaces expensive video reconstruction with more efficient BEV reconstruction. As shown in the Figure 8, under the guidance mechanism, the raw sensor world model successfully learns to reconstruct the BEV representation with reasonable accuracy. However, due to the inherent limitations of the camera sensor, the model struggles to reconstruct occluded regions areas beyond its direct line of sight.
H.2 Perception Confusion
As shown in Figure 9, adjacent frames exhibit a high degree of similarity. In the upper section, the ego vehicle is in a corner case of ParkingExit. Despite minimal visual differences between adjacent frames, identical actions can yield opposite rewards, causing model confusion. In the lower section, as the ego vehicle nears task completion, the visual change remains subtle. However, the binary nature of the continuation flag (0 or 1) provides limited information, further hindering convergence.
25

---

Ground Truth
Decoder Output
Ground Truth
Decoder 0 Dutput
Case
Case 2
Figure 8: Ground Truth VS Raw Sensor Decoder Output by the World Model (The red circle is the blind spot of the camera).
Tast Progress la 99
Tosk Progress 100
Stear-0.7
Steem- Reward
Spond-6.3
Flag
Figure 9; Perception Confusion in Reward and Continuous Head.
26


Reviews:
[{"review_id": "TtOgwq0fl8", "reviewer": "Reviewer_souv", "summary": "The paper introduces an end-to-end trained RL architecture for the CARLA autonomous driving simulator. The method is evaluated on the CARLA Leaderboard v2 as well as on Bench2drive (also in CARLA).", "questions": "- Clarify what is novel in the method. Is the privileged WM training approach novel? \n\n- How many interactions were needed to train the RL policy, and what RL method did you use exactly? The main paper only describes the WM training. I only found a brief mention in the appendix of the policy training being an actor-critic approach.\n\n- How would this model transfer to outside of CARLA? Even to another simulator? Maybe NAVSIM?", "limitations": "Sim2real transfer is unclear and I can't find a discussion of this. Otherwise ok.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors focus on closed-loop evaluation\n- Getting strong results with end2end RL for AD seems novel. The authors claim it's the first, which I am not sure about, but may at least be the first one with good results.\n- The authors include a long video example of the agent driving\n\nWeaknesses:\n- Scope/Transfer: The main weakness is already admitted in the title, which says \"Autonomous Driving (in CARLA v2)\". The main claim is end2end RL where the authors train from camera in CARLA and they evaluate in CARLA. Although probably different scenarios in CARLA, I presume those still use mostly the same visual assets. This means that it is unclear how the end2end part would transfer to the real world (which looks and behaves differently). \n  - Normally for IL approaches you compete on the amount of data needed to reach a certain performance, and it's also easier to collect in the real world while a human drives the car. Having RL drive a car during training does not seem realistic. I cannot find how many interactions was needed to train this e2e architecture in the main paper, but I presume it was a lot. That leaves sim2real, or how do you envision one would use this type of model?\n- The paper framing, structure and writing could overall be improved. \n  - The intro is rather negative of IL while pushing the virtues of RL, but RL also has weaknesses (like large number of world interactions needed, sim2real). \n  - The related work has been relegated to the appendix, which makes it difficult to evaluate what is novel.\n  - What is novel in the method is rather unclear. The listed contributions only claims to be the first AD e2e RL. The method part contains very few references, it is unclear if this is all novel or from related work. Is the WM alignment part novel? If so I suspect that could have been a contribution on its own.\n  - The intro takes aim at beating the CARLA v2 Leaderboard, but when we get to results it just says this uses bad metrics and focuses on Bench2Drive (also in CARLA) without explaining why. I had to read [40] to understand why, but [40] also introduces a new metric that purports to fix the problems, so why did the authors not use this?", "was_revised": true, "final_justification": "So far the rebuttal clarified some things but the fundamental weaknesses remain: no generalization tests outside of CARLA limits its potential as an AD paper, and while the dual stream RL approach is interesting, it is only tested on CARLA/AD so we do not know how well it would generalize to other problems. I still think that the approach is interesting and it already does have CARLA in the title so if these limitations (and framing of the contributions) were made clearler in the paper I would be OK with this being published. \n\nEDIT: I updated my score to borderline accept after further discussion with the authors and finding the baseline was also just RL in CARLA and still got accepted to ECCV'24, but my reservations about sim2real stand and I hope they can address this in future work.", "rebuttal": "We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.\n\n> **Weakness 1，Question 3, Discussion on Practicality of RL Training and Transfer**\n\n- We appreciate the reviewer’s concerns regarding the practicality of RL training. Recent breakthroughs in reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], have shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though **their readiness for large-scale RL deployment remains limited**.\n\n- Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. **Due to the unavailability of real-world-level authentic simulator**, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.\n\n[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\n[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model\n\n[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold\n\n[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation\n\n[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting\n\n[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving\n\n\n> **Weakness 1，Question 2, How many interactions were needed to train this end-to-end architecture?**\n\n- As noted in Sec. 4.1, we adopt the same 1000 training routes from Bench2Drive to ensure fairness. However, as an online learning paradigm, RL naturally collects a larger number of interactions—**both successful and failed**—to iteratively refine its policy. This **trial-and-error process** is a core advantage of RL over offline imitation learning.\n\n- In Bench2Drive-Base, the 1000 routes yield a total of 250k samples. To further analyze interaction efficiency, we also include additional ablations. Similar to early work such as MaRLn [7], which trained and tested on the same set of routes(albeit with limited generalization), we show that while training on fixed routes is feasible, generalization remains a challenge.\n\n- Notably, Raw2Drive enables better generalization to unseen scenarios by learning to predict future dynamics—this requires more interactions. As the number of interactions increases from 250k to 750k and 1M, we observe performance gains. Both successful and failed trajectories are crucial for learning a robust world model. However, we acknowledge that training time also increases accordingly.\n\n| Method       | Training Route Num | Test Route Num | Interactions | DS$\\uparrow$ | SR(%)$\\uparrow$ | Training Time |\n| ------------ | ------------------ | -------------- | ------------ | ------------ | --------------- | ------------- |\n| Ours(RL)     | 1000               | 220            | 250k         | 58.34        | 30.00           | 8 GPU Days    |\n| Ours(RL)     | 1000               | 220            | 750k         | 68.45        | 47.73           | 30 GPU Days   |\n| Ours(RL)     | 1000               | 220            | 1M           | 71.34        | 50.24           | 40 GPU Days   |\n| Ours(RL)     | 220                | 220            | 250k         | 66.82        | 45.45           | 8 GPU Days    |\n| UniAD(IL)[8] | 1000               | 220            | 250k         | 45.81        | 16.36           | 30 GPU Days   |\n\n\n[7]2020, CVPR, End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances\n\n[8]2023, CVPR, Planning-oriented Autonomous Driving\n\n> **Weakness 1，Limitations 1, Discussion on How to use this paradigm in Real-World?**\n\n- Thanks for your suggestions. Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.\n\n- By leveraging the world model, our paradigm enables **a fully differentiable pipeline while preserving a separation between perception and planning**. \n\n- Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.\n\n> **Weakness 2, Question 1, Contribution and Novelty of Raw2Drive**\n\n- Thank you for the helpful suggestion. Due to page limitations, we moved the related work section to the appendix. Nonetheless, as shown in Table 1 of the Introduction, we compared prior work, including both RL-based and world-model-based approaches. **Our core focus is reinforcement learning in end-to-end autonomous driving**. Early work such as MaRLn [7] (CoRL 2017) encountered significant challenges due to the difficulty of learning directly from raw images, which made policy optimization unstable. \n\n- With recent progress in model-based RL [9, 10], our work introduces two key novelties:\n    - **Successfully achieve end-to-end RL**\n    To our best knowledg, Raw2Drive is **the first successfully achieve end-to-end RL** with performance significantly better than the end-to-end IL of the same period.\n    - **Dual-stream World Model as a Bridge**:\n    We use a dual-stream world model to decouple perception and decision-making. This design simplifies the joint optimization problem, which is often difficult to converge in end-to-end autonomous driving.\n    - **Dual-stream World Model Alignment**:\n    Unlike standard teacher-forcing strategies [11, 12], future prediction in driving involves both stochastic (e.g., other agents’ intentions) and deterministic (e.g., ego state) components over time and space. We propose Spatial-Temporal Alignment and Abstract-State Alignment to ensure consistency between the two streams during rollout.\n\n- We will revise the final version of the paper accordingly to better highlight these contributions.\n\n[9] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination\n\n[10] 2021, ICLR, Mastering Atari with Discrete World Models\n\n[11] 2021, ICCV, End-to-End Urban Driving by Imitating a Reinforcement Learning Coach\n\n[12] 2022, NeurIPS, Teacher Forcing Recovers Reward Functions for Text Generation\n\n> **Weakness 2, Clarification on the Bench2Drive Metric**\n\n- In fact, as noted in Sec. 5.1 of [13], their evaluation metric is based on Bench2Drive, **which is consistent with ours**. Furthermore, [13] modifies the base penalty coefficients for CP (crossing red lights) and CV (vehicle collisions), reducing them specifically to address issues introduced by their early stopping strategy.\n\n[13]2025, Arxiv, Hidden Biases of End-to-End Driving Datasets\n\n> **Question 2, More introduction of RL method**\n \n- We follow DreamerV3 [14], a representative model-based RL method, which employs the REINFORCE[15] algorithm for policy training. As detailed in Our Appendix B and E, we **employ a latent imagination-based actor-critic framework**. Unlike standard actor-critic methods [16] that rely on real environment rollouts, we train the actor and critic entirely within a learned latent space using trajectories imagined by the world model. The actor is optimized to maximize expected long-term reward under these imagined rollouts, while the critic learns to predict value estimates in the same latent space, enabling sample-efficient and stable training.\n\n[14] 2025, Nature, Mastering Diverse Domains through World Models\n\n[15] 1992, Machine Learning, Simple statistical gradient-following algorithms for connectionist reinforcement learning\n\n[16] 2017，NeurIPS, Proximal Policy Optimization Algorithms\n\n> **Question 3, Clarification on NAVSIM simulator**\n\n- Thank you for the question. NAVSIM[17] is a **non-reactive simulator** designed for benchmarking perception and planning, but not for training interactive policies. It does not support closed-loop interactions or provide observations after interaction, making it unsuitable for reinforcement learning, which relies on trial-and-error learning with visual feedback. We will clarify this limitation in the revised version.\n\n[17] 2024, NeurIPS, NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking\n\n> **About Paper Formatting Concerns**\n\n- Thank you for pointing this out. We apologize for the oversight and will carefully correct all typos and formatting issues in the final submission."}, {"review_id": "Kfxbp984Pg", "reviewer": "Reviewer_ps71", "summary": "This paper introduces Raw2Drive, a reinforcement learning algorithm for end-to-end autonomous driving from raw image inputs. A key component of the approach is a world model trained from pixels, which learns under the guidance of a privileged world model trained on Bird’s Eye View (BEV) representations. The core novelty lies in the distillation mechanism that transfers structured knowledge from the BEV-based model to the raw-sensor-based model, enabling more efficient learning and improved driving performance.", "questions": "a.\tIn Section 2 (Problem Formulation & Related Works), the authors mention that raw observations are collected using cameras, LiDAR, and IMU. However, the proposed algorithm relies solely on images, effectively discarding the data from the other modalities. What is the rationale behind this design choice? Would the authors consider incorporating multimodal sensors (e.g., LiDAR or IMU) to further improve performance or robustness?\nb.\tIn the conclusion, the authors claim that “our proposed paradigm is not limited to camera-only systems and is applicable to guiding the learning of any sensor through the privileged input.” While this is an appealing prospect, learning directly from raw point-clouds (e.g., LiDAR) is significantly more challenging. It would be valuable if the authors could elaborate on how their approach could generalize to such modalities, and whether any specific architectural adaptations would be required.\nc.\tThe performance gains reported in Table 4 are substantial, especially when compared to IL-based baselines. Can the authors analyze the underlying causes of these improvements? Specifically, why does MBRL seem to consistently outperform IL approaches in this setting?", "limitations": "Limitations of the proposed method are only briefly addressed in the Appendix and focus mainly on broader issues—such as the general challenges of real-world RL and the nature of the privileged information—rather than on limitations specific to the proposed algorithm. A more transparent and detailed discussion of the method’s limitations (e.g., dependency on privileged BEV data during training, transferability to diverse real-world environments, or failure cases) would greatly improve the overall rigor and self-awareness of the work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper demonstrates strong empirical performance, supported by thorough experiments on well-established benchmarks. The comparisons with baselines are well-structured and convincingly highlight the practical relevance and effectiveness of the proposed method.\nHowever, from an algorithmic standpoint, the novelty is relatively limited. The student-teacher structure at the core of the method is a well-explored paradigm in the literature. For example, similar ideas have been investigated in:\nZhang J., Huang Z., Ohn-Bar E. Coaching a Teachable Student. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7805–7815.\nWhile I acknowledge that many of these prior approaches focus on imitation learning, the current paper would benefit from a more comprehensive Related Work section. This should not only clarify the distinction between IL and RL settings, but also deepen the algorithmic comparison with relevant methods that use similar distillation or privileged information paradigms. A richer contextualization would help better situate the algorithmic contributions of this work.", "was_revised": true, "final_justification": "I appreciate the authors' detailed response, as they have addressed most of my concerns. However, a common issue raised by several reviewers is the ability to generalize beyond CARLA. Overall, I believe the paper is borderline, and to clearly justify acceptance, the approach must be validated to demonstrate its generalizability. I will slightly raise my score.", "rebuttal": "We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.\n\n> **Weakness 1, Discussion and Clarification on Novelty and Student-Teacher Paradigm**\n\n- Thank you for the helpful suggestion. **Our core focus is reinforcement learning in end-to-end autonomous driving**. With recent progress in model-based RL [1,2], our work introduces two key novelties:\n    - **Successfully achieve end-to-end RL**\n    To our best knowledg, Raw2Drive is **the first successfully achieve end-to-end RL** with performance significantly better than the end-to-end IL of the same period.\n    - **Dual-stream World Model as a Bridge**:\n    We use a dual-stream world model to decouple perception and decision-making. This design simplifies the joint optimization problem, which is often difficult to converge in end-to-end autonomous driving.\n    - **Dual-stream World Model Alignment**:\n    Some student-teacher methods such as CaT [3] and DriveAdapter [4] are based on imitation learning. While they achieve strong performance in CARLA v1, they struggle in corner cases under the more challenging Bench2Drive (CARLA v2) benchmark. Moreover，**Unlike standard teacher-forcing strategies**, as shown in Figure 6 in manuscript, future prediction in driving involves both stochastic (e.g., other agents’ intentions) and deterministic (e.g., ego state) components over time and space. We propose Spatial-Temporal Alignment and Abstract-State Alignment to ensure consistency between the two streams during rollout. \n\n- We will revise the final version of the paper accordingly, adding citations to better emphasize these contributions and improving the discussion of related methods.\n\n[1] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination\n\n[2] 2021, ICLR, Mastering Atari with Discrete World Models\n\n[3] 2023, CVPR, Coaching a Teachable Student\n\n[4] 2023, ICCV, DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving\n\n> **Questions a, Clarification on Raw Sensor Inputs**\n\n- Thank you for the question. As stated in Sec. 3.2 (Line 97), the raw sensor input includes multi-view images and IMU data. For visual simplicity, the IMU input is omitted from the figure but is fully incorporated in the model implementation. We will update in the final version.\n\n\n> **Questions b, Additional Results on LiDAR-Based Study**\n\n- Regarding the LiDAR sensor, this highlights **another advantage of our method-extensibility**. Our dual-stream world model decouples perception and decision-making, where the perception module focuses on enhancing the accuracy of BEV-based representations. To further validate this extensibility, we include additional experiments using LiDAR inputs. Specifically, we replace BEVFormer[5] with BEVFusion[6]. Incorporating LiDAR leads to some performance improvement, particularly in reconstructing occluded objects, but the gains are marginal as **perception accuracy is already near saturation**. Moreover, most occluded objects are distant and have limited impact on immediate decision-making.\n\n- Due to the latest NeurIPS submission policy, we are unable to include visualization links, but we will provide them in the final camera-ready version.\n\n| Modality     | Encoder      | DS$\\uparrow$    | SR(%)$\\uparrow$    |\n| ------------ | ------------ | ----- | ----- |\n| Camera       | BEVFormer[5] | 71.36 | 50.24 |\n| Camera+LiDAR | BEVFusion[6] | 72.43 | 50.91 |\n\n[5] BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers\n\n[6] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation\n\n> **Questions c, Analyzing the Performance Gains of MBRL over IL**\n\n- Thank you for raising this important analysis. As noted in the Introduction, imitation learning (IL) relies on expert data and often struggles with out-of-distribution generalization. It is also prone to issues such as causal confusion (e.g., in traffic light scenarios, where the model may mistakenly associate “other vehicles move” with “I should move”) and compounding errors during long-horizon rollouts. In contrast, reinforcement learning (RL) learns through trial and error, optimizing behavior directly from reward signals.\n\n- Recent breakthroughs in RL across domains such as LLMs [7], VLMs [8], and AlphaFold [9] have demonstrated RL’s distinct advantages over supervised fine-tuning or imitation learning, particularly in discovering policies that go beyond demonstrator behavior.\n\n- In summary, while IL provides a strong initialization from expert behavior, it is fundamentally limited by demonstration quality and generalization capacity. Our MBRL framework leverages privileged guidance and learned world models to go beyond imitation, enabling robust and reward-driven policy learning that better handles long-horizon reasoning and distributional shifts. We will incorporate this analysis in the final version to clarify the advantages of MBRL over IL.\n\n[7] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\n[8] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model\n\n[9] 2021, Nature, Highly accurate protein structure prediction with AlphaFold\n\n> **Limitaions 1, Discussion on Method Limitations**\n\n- Thank you for the valuable suggestion. As a reinforcement learning framework, our method requires trial-and-error interaction, which incurs additional computational cost compared to imitation learning that directly learns from expert demonstrations. Moreover, due to the lack of highly realistic real-world simulators, we conduct experiments in CARLA. While CARLA has limitations in fidelity, we believe the core scientific questions and early-stage insights presented in this work are broadly applicable and valuable to the community. We will explicitly discuss these limitations in the final version.\n\n> **Limitation 2, Discussion on Transferability in Real-World Autonomous Driving**\n\n- We appreciate the reviewer’s concerns regarding the practicality of transferability in real-world autonomous driving.\n\n- **About the Transferability of RL**\n    - While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[10] and reconstruction[11,12], though their readiness for large-scale RL deployment remains limited. \n\n    - Our work builds upon the trend of RL, serving as a preliminary study of applying RL to end-to-end autonomous driving. **Due to the unavailablility of real-world-level authentic simulator**, we use CARLA but we argue that **the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the communit**. \n\n- **About the Transferability of Raw2Drive**\n    - Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.\n\n    - By leveraging the world model, our paradigm enables **a fully differentiable pipeline while preserving a separation between perception and planning**. \n\n    - Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.\n\n[10] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation\n\n[11] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting\n\n[12] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving"}, {"review_id": "JN7VbOD2PN", "reviewer": "Reviewer_xxow", "summary": "This paper proposes a novel framework that leverages reinforcement learning with world model pretraining to achieve robust and sample-efficient end-to-end driving in the CARLA simulator. The key contribution lies in the introduction of an aligned world model (AWM) that is jointly trained with a latent policy to ensure representation consistency between pretraining and downstream RL fine-tuning. By enforcing representation alignment through shared encoders and temporal consistency, Raw2Drive effectively bridges the gap between offline world model learning and online policy optimization. The method demonstrates superior performance and generalization across diverse CARLA v2 towns and weather conditions, outperforming existing model-based and model-free baselines in both success rate and driving smoothness.", "questions": "1. Is representation alignment truly necessary for performance gains, or could simpler techniques (e.g., fine-tuning with frozen encoders) suffice?\n\n2. Does the aligned world model retain predictive accuracy throughout policy fine-tuning? As world models may degrade as the policy distribution shifts.\n\n3. Could forcing a single encoder to serve both the world model and RL agent lead to suboptimal representations for either task?\n\n4. How fair are the comparisons to IL baselines in terms of total environment interactions and training time?\n\n5. Does the approach generalize across input modalities or sensor configurations?\n\n6. Is Raw2Drive compatible with safety constraints or explainability requirements for real-world deployment?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The introduction of the Aligned World Model (AWM) with shared encoders and temporal consistency regularization effectively bridges the gap between offline world modeling and online RL, which improves policy stability and sample efficiency.\n\n2. The method achieves state-of-the-art performance in CARLA v2 across challenging generalization settings.\n\n3.By leveraging pretrained world models and aligned representations, the approach reduces the data required during the online reinforcement learning phase.\n\nWeakness:\n\n1. The evaluation is confined to the CARLA simulator, and it remains unclear how well the approach generalizes to real-world driving scenarios with noisy sensors and domain shift. How can an RL trained well in simulation contributed to real world end-to-end?\n\n2. While alignment is emphasized as a core novelty, the paper could benefit from more extensive ablations on the individual contributions of alignment components (e.g., encoder sharing vs. temporal loss).\n\n3. Aligning representation spaces does not guarantee the world model remains useful during online RL updates. If the policy diverges during exploration, the pretrained world model might become misaligned or irrelevant, leading to degraded performance or instability.\n\n4. Lack of comparison with recent RL-based method like [1]. There are no sourc code.\n\n[1] AdaWM: adaptive world model based planning for autonomous driving, ICLR, 2025", "was_revised": true, "final_justification": "Raw2Drive offers interesting methodological insights in transfering the world model capability from perfect BEV perception, to end-to-end Autonomous Driving model. This is one of the most practical problem to be solved in the e2e autonomous driving area. Regarding my previous concerns in baseline comparison, sim2real (alignment) capabilities, and in real-world AD applications, the authors have solved my questions with additional results. Hence, I recommend to remain the Bordeline Accept rating, with confidence towards the acceptance of this paper", "rebuttal": "Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:\n\n> **Weakness 1, Question 6, Discussion on Transferability and Explainability in Real-world Autonomous Driving**\n\n- We appreciate the reviewer’s concerns regarding the Transferability and Explainability in Real-world Autonomous Driving. \n\n- Recent breakthroughs of reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], has shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though their readiness for large-scale RL deployment remains limited. \n\n- Our work builds upon the trend of RL, serving as **a preliminary study of applying RL to end-to-end autonomous driving**. Due to **the unavailablility of real-world-level authentic simulator**, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.\n\n- Regarding explainability(Question 6), as shown in Figure 8, we reconstruct BEV representations instead of expensive RGB images. This not only **provides supervision** for training the world model but also **enhances interpretability** by offering a structured and semantically meaningful intermediate representation.\n\n[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\n[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model\n\n[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold\n\n[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation\n\n[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting\n\n[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving\n\n> **Weakness 2, Additional Ablations and Discussion on Alignment**\n\n- Thanks for your suggestions. In fact, Table 7 presents ablations on the three types of latent states(Abstract-State Alignment). \n\n- **About Spatial-Temporal Alignment**\n    - In addition, we add results on Spatial-Temporal Alignment (based on Dev10). Spatial Alignment ensures consistency between image representations  and BEV representations—removing it is similar to “driving blind.” Temporal Alignment maintains the consistency of future predictions across time. **Both components are essential and complementary for stable world model rollout**.\n\n| Spatial Alignment | Temporal Alignment | DS$\\uparrow$ | SR(%)$\\uparrow$ |\n| ----------------- | ------------------ | ------------ | --------------- |\n| &#10007;          | &#10007;           | 0.0          | 0.0/10          |\n| &#10003;          | &#10007;           | 13.6         | 1.2/10          |\n| &#10007;          | &#10003;           | 9.24         | 0.8/10          |\n| &#10003;          | &#10003;           | 83.5         | 7.5/10          |\n\n- **About Encoder Sharing**\n    - Notably, because **the privileged and raw sensor streams have different inputs**, their encoders cannot share parameters.\n\n> **Weakness 3, Clarification on World Model Alignment during Online RL Updates**\n\n- We appreciate the reviewer’s concern, but we respectfully disagree with this point. During online RL updates, there is **no misalignment between the world models**.\n\n- In the first stage, similar to standard model-based RL approaches [7], the Privileged World Model and Privileged Policy are updated alternately. In the second stage, as shown in Figure 4, **the Privileged World Model is frozen**, ensuring **a stable supervision signal** for optimizing the Raw World Model. Therefore, no misalignment occurs during training.\n\n[7] 2025, Nature, Mastering Diverse Domains through World Models\n\n> **Weakness 4, Clarification on the Comparison with AdaWM**\n\n- In fact, as shown in Table 1, we include a comparison with AdaWM[8]. It is important to note that AdaWM is **not an end-to-end autonomous driving method**. Its benchmark - ROM03, a derivative of CARLA v1, which lacks complex corner cases(CARLA v2, such as Bench2Drive) and is **not open sourced yet**.\n\n[8] 2025, ICLR, AdaWM: adaptive world model based planning for autonomous driving\n\n> **Questions 1, Clarification on the Necessity of Representation Alignment**\n\n- As described in Sec. 3.1 and Sec. 3.2, the dual-stream world model processes two distinct input modalities: privileged input consists of time-sequenced BEV semantic masks, while raw sensor input includes multi-view images and IMU data. Due to **their fundamentally different input formats and encoder architectures**, simple fine-tuning between the two streams is not feasible.\n\n- As shown in Figure 7 (Appendix F), removing rollout guidance causes the Raw Sensor model’s loss to oscillate and fail to converge. Since the world model is used for future prediction, representation alignment is essential to ensure consistency between the two streams during rollout.\n\n> **Question 2, Clarification on World Model Stability During Policy Fine-Tuning**\n\n- In our dual-stream design, the first stage follows a standard model-based RL paradigm[7], where the Privileged World Model and Privileged Policy are updated alternately to ensure stable training. In the second stage, as shown in Figure 5, **both World Models are frozen during raw policy fine-tuning**. As a result, the predictive accuracy of the world model remains unaffected, and there is no degradation due to policy distribution shift.\n\n> **Questions 3, Clarification on Single Encoder to Serve both the World Model and RL Agent**\n \n- **This is a common practice in model-based RL[7,8,9].** During training, the RL agent interacts with the world model instead of the simulator, significantly reducing simulation overhead and improving training efficiency. One of the key roles of the world model is to predict the next observation, which is then used by the RL agent for decision-making. Therefore, using a single shared encoder for both the world model and the RL agent is not only reasonable but necessary to maintain consistency in representation and ensure effective policy learning.\n\n[8] 2020, ICLR, Dream to Control: Learning Behaviors by Latent Imagination\n\n[9] 2021, ICLR, Mastering Atari with Discrete World Models\n\n> **Questions 4, Discussion on Fairness of Comparison with IL Methods and Interaction Nums**\n\n- As noted in Sec. 4.1, we adopt the same 1000 training routes from Bench2Drive to ensure fairness. However, as an online learning paradigm, RL naturally collects a larger number of interactions—**both successful and failed**—to iteratively refine its policy. This **trial-and-error process** is a core advantage of RL over offline imitation learning.\n\n- In Bench2Drive-Base, the 1000 routes yield a total of 250k samples. To further analyze interaction efficiency, we also include additional ablations. Similar to early work such as MaRLn [10], which trained and tested on the same set of routes(albeit with limited generalization), we show that while training on fixed routes is feasible, generalization remains a challenge.\n\n- Notably, Raw2Drive enables better generalization to unseen scenarios by learning to predict future dynamics—this requires more interactions. As the number of interactions increases from 250k to 750k and 1M, we observe performance gains. Both successful and failed trajectories are crucial for learning a robust world model. However, we acknowledge that training time also increases accordingly.\n\n| Method        | Training Route Num | Test Route Num | Interactions | DS$\\uparrow$ | SR(%)$\\uparrow$ | Training Time |     |\n| ------------- | ------------------ | -------------- | ------------ | ------------ | --------------- | ------------- | --- |\n| Ours(RL)      | 1000               | 220            | 250k         | 58.34        | 30.00           | 8 GPU Days    |     |\n| Ours(RL)      | 1000               | 220            | 750k         | 68.45        | 47.73           | 30 GPU Days   |     |\n| Ours(RL)      | 1000               | 220            | 1M           | 71.34        | 50.24           | 40 GPU Days   |     |\n| Ours(RL)      | 220                | 220            | 250k         | 66.82        | 45.45           | 8 GPU Days    |     |\n| UniAD(IL)[11] | 1000               | 220            | 250k         | 45.81        | 16.36           | 30 GPU Days   |     |\n\n[10]2020, CVPR, End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances\n\n[11]2023, CVPR, Planning-oriented Autonomous Driving\n\n\n> **Questions 5, Discussion and Ablation on Generalization Across Input Modalities and Sensor Configurations**\n\n- Thanks for your suggestions. **please refer to the answer: Reviewer #ps71** (Questions b, Additional Results on LiDAR-Based Study)"}, {"review_id": "ygU02bS45d", "reviewer": "Reviewer_qxsa", "summary": "This paper introduces Raw2Drive, the first model-based reinforcement learning framework designed for end-to-end autonomous driving  using raw sensor inputs. It addresses the inefficiencies of imitation learning and the training difficulties of RL with high-dimensional sensory data. The core contribution is a dual-stream architecture where a privileged world model guides the training of a raw sensor world model via a proposed Guidance Mechanism. The method achieves outperforming results on CARLA v2 and Bench2Drive benchmarks.", "questions": "- How does Raw2Drive perform with out-of-distribution data or in other simulators or real-world scenarios? \n- How can the privileged stream be practically realized in non-simulated settings where ground-truth semantics or HD-maps are not perfectly available?\n- How the stochastic state is sampled from a one-hot distribution? What dose the stochastic state contain?\n- The paper says that the multi-head Raw Sensor World Model is difficult to achieve stable convergence during training. Is the Privileged World Model has the same situation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The dual-stream MBRL architecture and the Guidance Mechanism  are novel and straightforward.\n- Experimental results on challenging benchmarks like CARLA v2 and Bench2Drive demonstrate both efficacy and generalizability.\n- Architecture and training pipeline are well-detailed with clear figures.\n\n\nWeaknesses:\n- While innovative, the training phase relies on data (privileged inputs) that may not always be scalable in real-world applications: Now all data are from simulation data, there may be sim2real gap.\n- Some details of method may need more explain: how the stochastic state is sampled from a one-hot distribution? What dose the stochastic state contain?\n- Despite mentioning practical implications, real-world feasibility is only lightly discussed; real-time inference constraints or transfer to real data are not tested.\n- Perhaps more analysis and discussion is needed: The paper says that the multi-head Raw Sensor World Model is difficult to achieve stable convergence during training. Is the Privileged World Model has the same situation?", "was_revised": true, "final_justification": "All in all, I appreciate authors detailed responses. If authors can resolve sim2real gap, it would greatly enhance the value of this work, and I would give it an accepted rating. In its current state, I can only give a borderline rating.", "rebuttal": "Thanks for your acknowledgement and kind advice. Regarding your concerns, we give responses below:\n\n> **Weakness 1,3，Question 1, Discussion on Transferability in Real-world Autonomous Driving**\n\n- We appreciate the reviewer’s concerns regarding the Transferability in Real-world Autonomous Driving. \n\n- **About the Transferability of RL**\n\n    - Recent breakthroughs of reinforcement learning (RL) in domains such as LLMs[1], VLMs[2] and AlphaFold[3], has shown that RL's unique advantages compared to SFT/imitation learning. While real-world deployment of RL remains challenging, the community has made significant efforts in high-fidelity simulation generation[4] and reconstruction[5,6], though their readiness for large-scale RL deployment remains limited. \n\n    - Our work builds upon the trend of RL, serving as **a preliminary study of applying RL to end-to-end autonomous driving**. Due to **the unavailablility of real-world-level authentic simulator**, we use CARLA but we argue that the scientific problem studied and discoveries of the early exploration in this work are general and worth-sharing for the community.\n\n- **About the Transferability of Raw2Drive**\n    - Raw2Drive utilizes a dual-stream world model to effectively decouple perception and decision-making. On the decision side, trajectory prediction is conducted based on structured perception outputs (e.g., BEV or compact representations)—a widely adopted practice in both academia and industry. On the perception side, the emphasis is on enhancing the accuracy of these BEV-based representations.\n\n    - By leveraging the world model, our paradigm enables **a fully differentiable pipeline while preserving a separation between perception and planning**. \n\n    - Furthermore, with recent progress in high-fidelity scene reconstruction[4] and simulation technologies [5,6], a practical deployment path is to pretrain the world model via imitation learning and fine-tune the policy with reinforcement learning—an increasingly feasible strategy in real-world and industrial settings.\n\n[1] 2024, Technical Report, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\n\n[2] 2025, Arxiv, VLM-R1: A stable and generalizable R1-style Large Vision-Language Model\n\n[3] 2021, Nature, Highly accurate protein structure prediction with AlphaFold\n\n[4] 2025，ICCV, HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation\n\n[5] 2025, ICCV, BézierGS: Dynamic Urban Scene Reconstruction with Bézier Curve Gaussian Splatting\n\n[6] 2025, Arxiv, ReSim: Reliable World Simulation for Autonomous Driving\n\n> **Weakness 2, Question 3, Explanation of Stochastic State**\n\n- In our model, the stochastic latent state plays a key role in capturing uncertainty in world model predictions. It consists of **multiple categorical variables(num=32), each represented as a one-hot vector(dim=32)**. These variables are parameterized by logits output from the world model. During training, the stochastic state is sampled using a **Gumbel-Softmax estimator** to maintain differentiability. At inference, the mode (argmax) is used to select one-hot values. The final stochastic state is the concatenation of these one-hot vectors, serving as the discrete latent representation of the environment dynamics.\n\n> **Weakness 3, Results about Real-time Inference**\n\n- Thank you for the suggestion. We have added latency analysis for each module. In end-to-end autonomous driving, the perception backbone (e.g., surround-view image encoder) typically dominates the overall latency. As shown below, our world model and policy are highly efficient (each under 2ms), while the raw sensor stream is mainly bottlenecked by the vision encoder (e.g., BEVFormer).\n\n| Method            | Modality          | Encoder Latency(ms)          | World Model Latency(ms) | Policy Latency(ms) |\n| ----------------- | ----------------- | ---------------- | ------------------- | -------------- |\n| Privileged Stream | BEV State         | 2(5xConv)      | 2                 | 2            |\n| Raw Sensor Stream | Multi-view images | 600(BEVFormer) | 2                 | 2            |\n\n> **Weakness 4 and Question 4, Discussion on Stability of World Model Training(Privileged vs Raw Sensor)**\n\n- We appreciate the reviewer’s concern. The observed training instability mainly affects the Raw Sensor World Model, which learns from **high-dimensional, partial visual inputs**. In contrast, the Privileged World Model is trained on **structured BEV states that are low-dimensional, semantically aligned, and less ambiguous**—leading to much faster and more stable convergence without perceptual noise or occlusion.\n\n- As shown in Figure 7 (Appendix F), removing rollout guidance causes the Raw Sensor model’s loss to oscillate and fail to converge. We further provide a visualization in Appendix G.2, where adjacent frames appear visually similar but correspond to drastically different rewards. This semantic ambiguity makes it hard for the Raw Sensor World Model to capture consistent dynamics, reinforcing the need for privileged guidance during training.\n\n> **Question 2, Discussion on Privileged Input in non-simulated settings**\n\n- Thank you for the question. In non-simulated settings, privileged inputs are typically obtained from perception outputs—either generated by state-of-the-art detection models[7][8] or manually annotated—both of which are common practices in industry. While ground-truth semantics or HD maps may not be perfectly available, this is also the case in many real-world trajectory prediction setups[9], where models are trained on perception outputs that are imperfect but sufficiently reliable. \n\nOur framework follows this convention and can **incorporate such perception-derived privileged inputs** accordingly. This makes our privileged stream design practical and compatible with real-world deployment pipelines.\n\n[7]2024, CVPR, UniMODE: Unified Monocular 3D Object Detection\n\n[8]2024, ICLR, LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving\n\n[9]2025, CVPR, Truncated Diffusion Model for Real-Time End-to-End Autonomous Driving"}]

Author rebuttal:

Meta-review: This paper introduces Raw2Drive, a model-based reinforcement learning framework for end-to-end autonomous driving in CARLA using raw image inputs. The core idea is to leverage a privileged BEV-based world model to guide the training of a raw-sensor world model through a distillation and alignment mechanism. The approach is evaluated extensively on CARLA Leaderboard v2 and Bench2Drive, demonstrating superior performance over baselines.

The main strengths of the paper lie in its emphasis on closed-loop evaluation on established benchmarks and its strong empirical performance, achieving state-of-the-art results. While the scope is limited to CARLA, raising questions about transferability to real-world driving, reviewers recognize that demonstrating robust end-to-end reinforcement learning from raw sensory inputs is an important advance. Overall, the paper is considered a strong contribution and is recommended for acceptance.
Decision: Accept (poster)