Enhancing End-to-End Autonomous Driving with Latent World Model
Authors: Yingyan Li
Lue Fan
Jiawei He
Yuqi Wang
Yuntao Chen
Zhaoxiang Zhang
Tieniu Tan
Venue: ICLR 2025 Poster (2025)
Keywords: end-to-end autonomous driving
world model
self-supervised learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: The latent world model facilitates future prediction tasks, improving scene feature learning and trajectory prediction in end-to-end autonomous driving.

Abstract: In autonomous driving, end-to-end planners directly utilize raw sensor data, enabling them to extract richer scene features and reduce information loss compared to traditional planners. This raises a crucial research question: how can we develop better scene feature representations to fully leverage sensor data in end-to-end driving? Self-supervised learning methods show great success in learning rich feature representations in NLP and computer vision. Inspired by this, we propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving. LAW predicts future latent scene features based on current features and ego trajectories. This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks, improving scene feature learning while optimizing trajectory prediction. LAW achieves state-of-the-art performance across multiple benchmarks, including real-world open-loop benchmark nuScenes, NAVSIM, and simulator-based closed-loop benchmark CARLA. The code will be released.

Published as a conference paper at ICLR 2025
ENHANCING END-TO-END AUTONOMOUS DRIVING WITH LATENT WORLD MODEL
Yingyan L234,3,
Lue Fan 22,3
Jiawei Hel.2,3
Yuqi Wang" 122,3
Yuntao Chen 1,2,3
Zhaoxiang Zhang 22,3,4
Tieniu Tan Tan-.2.3 1,3
Institute Automation, Chinese Academy of Sciences (CASIA)
2 New Laboratory of Pattern Recognition (NLPR)
3 State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)
4 School of Future Technology, University of Chinese Academy of Sciences (UCAS)
ABSTRACT
In autonomous driving, end-to-end planners directly utilize raw sensor data, en- abling them to extract richer scene features and reduce information loss compared traditional planners. This raises crucial research question: how can we develop better scene feature representations to fully leverage sensor data in end-to-end driving? Self-supervised learning methods show great success learning rich feature representations in NLP and computer vision. Inspired by this, we propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving. LAW predicts future scene features based current fea- tures and ego trajectories. This self-supervised task can seamlessly integrated into perception free and perception- berceetion-bbsed frameworks, improving scene feature learning and optimizing trajectory prediction. LAW achieves sate-of-the-aa per- formance across multiple benchmarks, including real-world open-loop benchmark nuScenes, NAVSIM, and simulator- closed-loop benchmark CARLA. The code is released at
INTRODUCTION
End-to-end planners (Hu et al. 2022c Tiang al. 2023 Prakash et al. 2021 Wu al. 2022 Hu
et al. 2022b; ZZhan et al.) 2022 Wu et al. 2023) have garnered significant attention due to their distinct advantages over traditional planners. Traditional planners operate on pre-processed outputs from perception modules, such as bounding boxes and trajectories. In contrast, end-to-ene planners directly utilize raw sensor data to extract scene features, minimizing information loss. This direct use of sensor data raises an important research question: how can we develop more effective scene feature representations to fully leverage the richness of sensor data in end-to-end driving?
In recent years, self-supervised learning has emerged as powerful method for extracting compre- hensive feature representations from large-scale datasets, particularly fields like NI.P (Devlin) 2018) and computer vision (He et al., 2022). Building this success, we aim enrich scene feature learning and further improve end-to-end driving performance through self-supervised learning. Traditional self-supervised methods in computer vision(He et al. 2022 Chen al. 2020b) often focus on static, single- single-frame frame images. However, autonomous driving relies on continuous video input, effectively using temporal data is crucial. Temporal self- self-supervised tasks, such future prediction (Han et al.) 2019) 2020), have shown promise. Traditional future prediction tasks often overlook the impact of ego actions, which play crucial role in shaping the future in autonomous driving.
Considering the critical role of ego actions, we propose utilizing world model predict future states based on the current state and ego actions. While several image-based driving world models (Wang
al. 2023b Hu et 2023at Jia et 2023a exist, they exhibit inefficiencies in enhancing scene
Email: liyingyan2021 @ia.ac.cn

---

Published as a conference paper at ICLR 2025
Input image
Input image
Next Frame
Occupancy Prediction
Motion Prediction
Encoder
Encoder
Supervise
Visual Latents
Pred. Latents
Visual Latents
Optional
Supervise
Depth Estimation
Detection
Decoder
Decoder
Latent World Model
Tracking
Map Seg.
Predicted Trajectory
Predicted
T+1 Trajectory
(a) Annotation-based Perception Tasks
(b) Our Self-supervised Task
Figure 1:
The illustration of our self-supervised method. Traditional methods utilize perception annotations to assist with scene feature learning. In contrast, self-supervised approach uses temporal information to guide feature learning. Pred.: predicted. Seg.: segmentation. The blue part indicates the pipeline of end-to-end planner.
feature representations due to their reliance on diffusion models, which may take several seconds generate images of a future scene. To address this limitation, we introduce latent world model designed to predict future latent features directly from the current latent features and ego actions, as depicted in Fig. [ Specifically, given images, a visual encoder extracts scene features (current state), which are then fed into an action decoder to predict ego trajectory. Based the current state and action, the latent world model predicts the scene feature of the future frame. During training, the predicted future features are supervised using the extracted features from the future frame. By supervising the predicted future feature, this self-supervised method jointly optimizes the current scene feature learning and ego trajectory prediction.
After introducing the concept of the latent world model, we turn our attention to universality across various end-to-end autonomous driving frameworks. In end-to-end autonomous driving, the frameworks can generally be categorized into two types: perception-ffee and perception based.
Perception-free approaches (Toromanoff et 2020 Chen al. 2020a Wu et 2022) bypass
explicit perception tasks, relying solely on trajectory supervision. Prior work Wu et al. (022) in
this category typically extracts perspective- view features to predict future trajectory. In contrast, perception-basee approaches (Prakash et al.) [021] Hu et al. [202cc] Jiang et al.) [023 Hu et al. 2022b) incorporate perception tasks, such as detection, tracking, and map segmentation, to guide scene feature learning. These methods generally use BEV feature maps unified representation for these perception tasks. Our latent world model accommodates both frameworks. perspective-view features in the perception- ffrre setting or predict BEV features in the perception- based setting, showcasing its universality across different autonomous driving paradigms.
either predict
Experiments show that our latent world model enhances performance in both perception- perception-free and perception- frameworks. Furthermore, we achieve state-of oftth--rrt performance on multi- ple benchmarks, including the real-world open-loop datasets nuScenes (Caesar et al.) 2020 and NAVSIM (Dauner et al., 2024) (based on nuPlan (Caesar et al., 2021)) )), as well as the simulator- based closed-loop CARLA benchmark (Dosovitskiy al. 2017). These results underscore the efficacy of our approach and highlight the potential of self-supervised learning to advance end-to-end autonomous driving research. In summary, our contributions are threefold:
Future prediction by latent world model: We introduce the LAtent World model (LAW) predict future scene latents from current scene latents and ego trajectories. This self-
2

---

Published as a conference paper at ICLR 2025
supervised task jointly enhances scene representation learning and trajectory prediction in end-to-end driving.
Cross-framework universality: LAW demonstrates universality across various common au- tonomous driving paradigms. can either predict perspective-viex features in the perception- free framework or predict BEV features in the perception-base praeet framework.
State-of-the-art performance: Our self-supervised approach achieves state-of -fte-of-theearr results on the real-world open-loop nuScenes, NAVSIM, and the Ammultoorbassed close-loop CARLA benchmark.
RELATED WORKS
2.1
END-TO-END AUTONOMOUS DRIVING
We divide end-to-end autonomous driving methods (Hu et al. 2022c Jiang et al. 2023
Renz et al.
2022 Toromanoft al.
2020 Tian al. 2024 Hwang et al. 2024 Pan et al.
2024 Wang al.
2024a b) into two categories, perception- based methods and perception-free methods, depending
on whether performing perception tasks. Perception-basee
methods (Casas al.
2021
Prakash
iet al., 2021 Jaeger et al. 2023 Shao et al., 2023 Hu et al. 2022b Sadat et al.
2020 perform
multiple perception tasks simultaneously, such as detection (Li al. 2022b] Huang al.
2021
et all. 2022a (024a). tracking (Zhou et al., 22200] Wang et al. [2021) map segmentation Hu et al.
2022c Jiang et al. (023) and occupancy prediction (Wang et |l.]|||222222 2023a Huang et al.
2023). As
representative, UniAD (Hu et al. 2022c) integrates multiple modules to support goal-driven planning.
VAD (Jiang et al.
2023 explores vectorized scene representation for planning purposes.
Perception-free end-to-end methods (Toromanoff et al. 2020: Chen et al.
2020a Zhang et al. 2021
Wu ct al. 2022) present a promising direction as they avoid utilizing a large number of perception annotations. Early perception-fiee free end-to-end methods (Zhang al. [02 Toromanoff al. 2020
primarily relied on reinforcement learning. For instance, MaRLn (Toromanoff et al. (0200 des innee reinforcement learning algorithm based on implicit affordances, while LBC (Chen et al. 2020a)
trained a reinforcement learning expert using privileged (ground-truth perception) information. Using trajectory data generated by the reinforcement learning expert, TCP (Wu al. (022) combined
a trajectory waypoint branch with a direct control branch, achieving good performance. However, perception- free end-to-end methods often suffer from inadequate scene representation capabilities. Our work aims to address this issue through the latent world model.
2.2 WORLD MODEL IN AUTONOMOUS DRIVING
Existing world models in autonomous driving can be categorized into two types: image-based world models and occupancy- -aasedd world models. image-based world models (Hu et al. 2022a Wang et al. 20236) Hu et al.] (2023)) aim to enrich the autonomous driving dataset through generative approaches. GAIA-T (Hu et 2023a is generative world model that utilizes video, text, and action inputs to create realistic driving scenarios. MILE (Hu et al. 2022a) produces urban driving videos by leveraging 3D geometry
inductive bias. Drive-Wo MM (Wang et al., 2023b) utilizes diffusion
model to predict future images and then plans based on these predicted images, Copilot4D (Zhang al. 2023) tokenizes sensor observations with VQVAE (Van Den Oord al. 2017) and then predicts the future via discrete diffusion. Another category involves occupancy bosseed based world models (Zheng al. (2023, Min al. (0024). OccWorld (Zheng et al., (002)) and Worve (Min et al.) (024) use the world model to predict the occupancy, which requires occupancy annotations. On the contrary, our proposed latent world model requires no manual annotations.
PRELIMINARY
Vision-based End-to-end Autonomous Driving In the task of end-to-end autonomous driving, the objective is to estimate the future trajectory of the ego vehicle in the form of waypoints. Formally, let I, = {I) I....,, IN be the set of N surrounding multi-vien images captured at time step t. We expect the model predict sequence of waypoints W. (w!, w?,... w w.Y), ), where each
3

---

Published as a conference paper at ICLR 2025
Sup
Encoder
Dacoder
Waypoints
Latent World Model
Encoder
Visual
Adion-aware
Latent
Pred. Latents
Latents
T+1
Motion
Waypoint
Perspective view Encader
Waypoint
Queries
Prediction
Querics
Waypoint
Decoder
BEV Encoder
flttion
K.
DQQ
Q Agent Quarres
Wayport Decoder
DDO
Queries
supervision
Visual
BEV
Visual Latents
Map Construction
Encoder
Latents
Decoder
Encoder
Decoder
Perception-fres with Perspective-view
Poreaption-based with BEV Features
The Overall Framework. The encoder extracts visual latents from the images, while the decoder predicts waypoints based on these latents. Our latent world model predicts future visual latents using the visual latents and waypoints of the current frame. During training, the predicted visual latents are supervised by the extracted latents from the future frame. The latent world model is compatible with both perception-free and perception-baset frameworks, which differ in their encoder and decoder structures. In the perception-based framework, the supervision icon indicates that map annotations supervise the output of the map construction module, while the agent's 's future trajectory supervises the output of the motion prediction module. Pred.: predicted.
Figure 2:
waypoint w/
(xf, vt) represents the predicted BEV position of the ego vehicle at time step + i. M represents the number of future positions of the ego vehicle that the model aims predict.
World Model A world model aims to predict future states based the current state and action. In autonomous driving task, let S, represent the states at time step 4, W, {w] wž...., w:" denote the sequence of predicted waypoints by the planner, the world model predicts state S1+1 at time step + 1 using S, and W..
METHODOLOGY
Our methodology is composed of three key components: Latent World Model: We utilize the latent world model to realize the self-supervised task. This model takes two inputs: latent features extracted by the visual encoder and waypoints predicted by the waypoint decoder. This task is compatible with two common frameworks. Perception-free framework with perspective-view latents: it consists of the perspective- view encoder and perception-free frreeinn--reeeee decoder within this framework. iii) Perception- based framework with BEV latents: it contains the BEV encoder and perception- based decoder within this framework.
4.1 LATENT WORLD MODEL
In this section, utilize the latent world model predict the visual latents of the future frame based on the current visual latents and waypoints.
Visual Latents and Waypoints Extraction The visual encoder processes the images from the current t time step to produce the corresponding visual latent feature set
{v. v?
where L denotes the number of feature vectors, and each vector v E R,, with D representing the number of feature channels. These feature vectors can be derived from various sources, such as a flattened image feature map flattened BEV feature map. Based the V.. the waypoint decoder
4

---

Published as a conference paper at ICLR 2025
.. w("). M represents the number of waypoints, where predicts the waypoints W {w/. wi..... each waypoint w = (a), v)).
Action-aware Latents Construction We produce action-aware latents by integrating visual latents and waypoints. The action-aware latents are then used as imput to the latent world model. Let M represent the number of waypoints, with each waypoint w =xx,, vε). We first reshape W which has the shape [M, 2]. into a one-dimensional vector W, E IR2M Then, we concatenate each visual latent vi with the waypoint vector W, along the feature channel dimension. The resulting concatenated vector is passed through an MLP to produce the action-aware latent aj, which has the same shape as vi. Formally, the action- aware latent for the i-th feature vector is expressed
a MLP((V), w.l).
where [.:]] denotes the concatenating operation. The full set of action-awand latents is denoted as A, [a. .f.....,,f a.") where L denotes the number of feature vectors.
Future Latent Prediction The latent world model utilizes the action-aware latents to predict the future visual latents follows. Given A,, we predict visual latents Ve+1 of the frame + 1 by the latent world model:
Ve+1 LatentWorlddModel (At).
The network architecture of the latent world model consists of transformer blocks. Each block contains a self-attention and feed-forward module. The self-attention performed across the latent feature vectors.
Future Latent Supervision During training, we extract the visual latents V = (V/41 v from the images of frame + 1, which are used as the ground truth to supervise the V using Mean Squared Error (MSE) loss function as
Laacent L
Σ 1
1221
The latent world model is compatible with both perception- and perception- frameworks. In the following sections, we detail the implementation of these two frameworks.
4.2 PERCEPTION-FRES FRAMEWORK WITH PERSPECTIVE-VIER LATENTS
First, we introduce our perception- free framework. Previous perception- -free frameworks (Wu et al.) 2022) typically employ perspective-view encoder for visual latent extraction and perception-free decoder for waypoint prediction. Our framework built upon this established paradigm.
Perspective-view Encoder In the perspective-view encoder, we produce visual latents based on multi-view images. Initially, multi-view images are processed by an image backbone to obtain their corresponding image features. Following PETR (Liu al. 2022). we generate 3D position embeddings for these image features, which are then added to the image features to uniquely distinguish each feature vector. The enriched image features are denoted F {fl f?....,f fN). where represents the number views. The shape of is H, W. D). where H, W represents the height and width of the image feature map and D is the number of feature channels.
To encode the image features into high-level visual latents suitable for planning, we apply a view attention mechanism. To be specific, for N views, there N corresponding learnable view queries Qview = {q hiiew q }. Each view query View undergoes a cross-attention with its corresponding image feature f', resulting in N visual latent Vpr = =[v] v V.. subscript "pf" stands for perception- frree. Formally,
v. where the
v/ CrosssAttention (q'eww F F'r'),
where f serves as the key and value of the cross attention. Vpr are then used as input for the perception-free decoder, which we describe next.
Perception-free Decoder The perception-free decoder deocdes waypoint from Vpr. Specifically, we initialize M waypoint queries, Qup = {aup iup...., 9M), embedding. These waypoint queries interact with Vpr through cross-attention mechanism. The
qwp)) .. where each query is learnable
5

---

Published as a conference paper at ICLR 2025
updated waypoint queries then passed through an MLP head to predict the waypoints W (w') w², ,..,,. w My }. which formulated as
W = MLP(CCrossAttentioon Qup: Vpr, Ppr)).
(5)
Perception-free Supervision In the perception- framework, we rely solely on ground truth waypoints for supervision, as no additional perception annotations are provided. We employ L1 loss to measure the discrepancy between the predicted waypoints W and the ground truth waypoints
W (w' w....., wM]
M
wwnypoint
1 M
Σ
w!
w/
(6)
/=1
Thus, the final loss for the perception free framework is
ppt
Lattcatt
Lowaypoint-
4.3 PERCEPTION-BASHE FRAMEWORK WITH BEV LATENTS
Our latent world model is also compatible with perception- frameworks, which commonly utilize BEV feature maps for perception tasks. We adhere to this paradigm and the perception- based framework is composed of two key components: BEV encoder and a perception-based decoder. The BEV encoder generates BEV feature maps from images, while the perception -based decoder uses these maps for perception tasks such as motion prediction and map construction. The final waypoints are then predicted based on the outputs of these perception tasks.
BEV Encoder We follow Li et al. (2022b) to encode the BEV feature map. First, encode the image features using backbone network. Then, set of BEV queries projects these image features into BEV features. The resulting BEV feature map is flattened into shape of [K, D]. where K represents the number of feature vectors in the BEV feature map and is the number of feature channels. The flattened features denoted Vpb (vp "pb"refers perception-based.
v2 pb:--.g pb where the subscript
Perception-based Decoder Following Jiang et al. (2023), the decoder predicts waypoints with help of perception tasks, namely motion prediction and map construction. For motion prediction, Nagent learnable queries interact with Vpb via cross-attention to generate agent features F Fagent F Fagent then used to predict agent trajectories. Similarly, for map construction, .Nmmp learnable queries perform cross-attention with Vpb to extract map features F Fmp-- F map are then used to predict map vectors. Finally, the learnable waypoint queries perform cross-attention with F Pagent and F map-- The output is then passed through an MLP head to predict the waypoints.
Perception-based Supervision
The perception-basee framework uses the same waypoint supervision as in equation 6 In addition, it includes losses from perception tasks as
Eperception = Lagent + Cmap-
(8)
Here, Lagent is the loss for motion prediction and Lmpp is the loss for map construction, as defined in (Jiang et al. 2023). The final loss for the perception-basee framework is
Lxb
Liatent + Lwayppoit + Leperception--
EXPERIMENTS
5.1 BENCHMARKS
nuScenes (Caesar et al. 2020) The nuScenes dataset contains 1,000 driving scenes. In line with previous works (Hu et al. 2002bb 2023b; Jiang et al. 2023),, we use L2 displacement error and collision rate comprehensive metrics evaluate planning performance. I.2 displacement error measures the L2 distance between the predicted and ground truth trajectories, while collision rate quantifies the frequency of collisions with other objects along the predicted trajectory.
NAVSIM (Dauner et al., 2024) We conducted further experiments using the NAVSIM benchmark. the nuScenes dataset proved to be overly simplistic. The NAVSIM dataset DDuuner et al. 2024

---

Published as a conference paper at ICLR 2025
is built on OpenScene (Contributors 2023 which provides 120 hours of driving logs condensed from the nuPlan dataset (Caesar et al. 2021) NAVSIM enhances OpenScene by resampling the data to reduce the occurrence of simple scenarios, such as straight-line driving. As result, traditional ego status modeling becomes inadequate under the NAVSIM benchmark. NAVSIM evaluates model performance using the predictive driver model score (PDMS), which is calculated based five factors: no at-fault collision (NC), drivable area compliance (DAC), time-to-collision (TTC), comfort (Comf.) and ego progress (EP).
CARLA (Dosovitskly et al., 2017) Closed-loop evaluation is essential to autonomous driving as it constantly updates the sensor inputs based on the driving actions. For the closed-loop benchmark, the training dataset is collected from the CARLA DDssovisskiy et al. (017)) simulator (version 0.9.10.1)
using the teacher model Roach Zhang et al. 2021) following (Wu al., 2022 Jia al. 2023b). resulting in 189K frames. We use the widely-used Town05 Long benchmark (Jia et al. 2023b Shao al. 2022 Hu al., (022a) to assess the closed-loop driving performance. For metric, Route Completion (RC) represents the percentage of the route completed by the autonomous driving model. Infraction Score (IS) quantifies the number of infractions as well as violations traffic rules, A higher Infraction Score indicates better adherence to safe driving practices. Driving Score (DS) is the primary metric used to evaluate overall performance. It is calculated as the product of Route Completion and Infraction Score.
5.2 IMPLEMENTATION DETAILS
nuScenes Benchmark We implement both perception-free and perception-based frameworks. In the perception-free framework, Input images are resized to 800x320. We adopt a Cosine Annealing learning rate schedule[Loshchilov & Hutter 2016), starting at 5e-5. The model is trained using the AdamW optimizer (Loshchilov & Hutter 2017) with a weight decay of 0.01, batch size and 12 epochs across 8 A6000 GPUs For the perception- based framework, following Jiang et al. (2023). we train the model in two stages. In the first stage, we train the encoder and perception head using only perception loss for 48 epochs. In the second stage, we introduce waypoint and latent prediction losses for training another 12 epochs. The network architecture of the latent world model utilizes deformable self-attention for improved convergence.
(Swin-T)(Liu et al. (021) is used as the backbone.
NAVSIM Benchmark The perception framework is implemented on NAV SIM. Specifically, We employ ResNet-34 backbone, training for 20 epochs in line with frthash et al. (2021) to ensure fair comparison. Input images are resized to 640x320. The Adam optimizer is used with a learning rate of 1e-4 and batch size of 32.
CARLA Benchmark We follow Wu et al. 2022) to implement a perception framework on CARLA. To be specific, we use ResNet-34 the backbone and employ the TCP head (Wu et al. 2022) as in Jia et al. (2023b). Input images are resized to 900x256. The Adam optimizer is used with learning rate of le-4 and weight decay of 1e-7. The model is trained for 60 epochs with batch size of 128. After 30 epochs, the learning rate is halved.
5.3 COMPARISON WITH STATE-OF-THE-ART METHODS
For the nuScenes benchmark, we compare our proposed framework with several state-of-the-ar methods, including BEV-Planner (Li et al. (024b) and VAD (Jiang et al. (202)). The results are summarized in [ Our perception -free framework demonstrates competitive performance, while the perception based Framework achieves saat--ofthh--ae results both L.2 displacement and collision rates. For the NAVSIM benchmark, detailed in Table our method achieves state-of-the-arr results in overall PDMS. For the CARLA benchmark, as shown in Table 3 our proposed method outperforms all existing methods. Notably, perception- approach surpasses previous leading methods such as T (Jia et al.) 2023c) and DriveAdapter (Jia et al. 2023b), which incorporate extensive supervision from depth estimation, semantic segmentation, and map segmentation.
5.4 ABLATION STUDY
All experiments conducted within the perception-free framework unless otherwise specified.
7

---

Published as a conference paper at ICLR 2025
Table 1: Performance on the nuScenes (Caesar al. 2020 The overall collision results are computed by the traditional computation way used in Jiang et al. (202). I: The collision results are computed by the way in Li al. 2024b). We do not utilize the historical ego status information.
1.2 (m)
Collision (%)
Method
1s
2s
38
Avg.
2s
38
Avg.
NMP [Zeng et al.
2019
2.31
1.92
SA-NMP Zeng
2019
2.05
1.59
FF (Hu et al. 2021
0.55
1.20
2.54
1.43
0.06
0.17
1.07
0.43
RO (Khurana al. 2022
0.67
1.36
2.78
1.60
0.04
0.09
0.88
0.33
ST-P3 al. 2022h
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
UniAD (Hu et 2022c
0.48
0.96
1.65
1.03
0.05
0.17
0.71
0.31
VAD Jiang et 2023
0.41
0.70
1.05
0.72
0.07
0.17
0.41
0.22
BEV-Planner (L.
2024b
0.30
0.52
0.83
0.55
0.10
0.37$
1.303
0.591
LAW (perception- free)
0.26
0.57
1.01
0.61
0.21
0.54
LAW (perception )
0.24
0.46
0.76
0.10
0.39
0.19
Performance on NAVSIM test set. NC: no at-fault collision. DAC: drivable area compliance. TTC: time-to-collision. Comf.: comfort. EP: ego progress. PDMS: the predictive driver model score.
Table 2:
LAW is in the perception-free setting.
Method
NC
DAC
TTC
Comf. t
EP
PDMS
Human
100
100
100
99.9
87.5
94.8
Constant Velocity
69.9
58.8
49.3
100
49.3
21.6
Ego Status MLP
93.0
77.3
83.6
100
62.8
65.6
TransFuser (Prakash et al.
2021
97.7
92.8
92.8
100
79.2
84.0
UniAD [u al. 2022c
97.8
91.9
92.9
100
78.8
83.4
PARA-Drive (Weng el al.
2024
97.9
92.4
93.0
99.8
79.3
84.0
LAW
96.4
95.4
88.7
99.9
81.7
84.6
Table 3: Performance on Town05 Long benchmark on CARLA. Expert: Imitation learning from the driving trajectories of privileged expert. Seg.: semantic segmentation. Map.: BEV map segmentation. Dep.: depth estimation. Det.: 3D object detection. Latent Prediction: our proposed self-supervised task. RC: route completion. IS: infraction score. DS: driving score. LAW is in the perception-free setting.
Method
Supervision
RCT
IST
DS
CILRS (Codevilla et al.
2019
Expert
10.3+10.0
0.75
7.8
LBC (Chen et 2020a
Expert
31.9=2.2
0.66=10.2
12.3+2.0
Transfuser (Prakash et al.
2021
Expert, Dep., Seg.. Map.. Det.
47.511.33
0.77+0.04
31.011..6
Roach (Zhang et al. 2021
Expert
96.4=2.1
0.43-0.03
41.6111.
LAV (Chen & Krahenbuh
2022
Expert, Seg.. Map., Det.
69.8+2.33
0.73+0.02
46,5+2.3
TCP Wu et 2022
Expert Expert, Map., Det.
80.4+1.5
0.73+0.02
57.2+1.5
MILE Huu
2022a
97.4+0088
0.63+0.03
61.1+3.2
ThinkTwice (Ja al.
2023c
Expert. Dep.. Seg.. Det.
95.512.0
0.69
65.0.11.7
DriveAdapter (Jia et al.
2023b
Expert. Map., Det.
94.4.1-
0.7211-
65.91L-
Interfuser (Shao et al.
2022
Expert. Map., Det.
95.0.12.9
68.3.1
LAW
Expert, Latent Prediction
97,8+10.9
0.72+0.03
70.14:2.6
Ablation Study on Latent World Model In this ablation study, we assess the effectiveness of our proposed latent world model. For the nuScenes benchmark. the results are shown in Table [ We ablate the latent prediction task in both the perception and perception-based frameworks, and further investigate the contribution each input to the latent world model. The findings demonstrate that accurate future latent predictions depend incorporating driving actions, supporting the validity of the latent world model. We also present ablation studies on NAVSIM and CARLA, as detailed in Table 5 In NAVSIM, observed significant improvement in PDMS, mainly driven by the
8

---

Published as a conference paper at ICLR 2025
Table 4: Effectiveness of latent prediction on nuScenes benchmark. The latent world model receives two types of inputs: visual latents and predicted trajectory. No imput refers to not utilizing the world model. Pred.: predicted. Traj.: trajectory. Avg.: average.
Input of Latent World Model
L2 (m)
Collision (%)
Framework
Visual Latent
Pred. Traj.
1s
2s
3s
Avg.
1s
2s
3s
Avg.
0.32
0.67
1.14
0.71
0.20
0.30
0.73
0.41
Perception-fee free
0.30
0.64
1.12
0.68
0.18
0.27
0.66
0.37
0.26
0.57
1.01
0.61
0.14
0.21
0.54
0.30
:
0.30
0.52
0.80
0.54
0.09
0.17
0.48
0.25
Perception-base
0.27
0.49
0.80
0.52
0.08
0.12
0.42
0.21
/ 0.24
0.46
0.76
0.49
0.08
0.10
0.39
0.19
Table 5: Ablation study on latent prediction NAVSIM and CARLA benchmark. NC:
at-fault collision. DAC: drivable area compliance. TTC: time-to-collision. Comf.: comfort. EP: ego
progress. PDMS: the predictive driver model score. RC: route Completion. IS: infraction Score. DS: driving Score. LAW is in the perception- setting.
NAVSIM
CARLA
Latent Prediction
NC 1
DAC 1
TTC1
Comf. f
EP
PDMS
RC1 1
IS t
DS 1
94.4
89.4
84.8
100.0
75.1
77.5
98.6.10.8
0.68.10.02
67.912.1
96.4
95.4
88.7
99.9
81.7
84.6
97.8-0.9
0.72100.3
70.1
enhancements in drivable area compliance (DAC) and Ego progress (EP) metrics. This suggests that our self-supervised task effectively enhances the quality of the driving trajectory. Similarly, in CARLA, we observed notable improvements in the Driving Score.
The Time Horizon of Latent World Model this experiment, the world model predicts latent features at three distinct future time horizons: 0.5 seconds, 1.5 seconds, and 3.0 seconds. This corresponds to the first, third, and sixth future frames from the current frame, given that keyframes occur every 0.5 seconds in the nuScenes dataset. The results, displayed in Table show that the model achieves the best performance at the 1.5-second horizon. In comparison, the 0.5-second interval typically presents scenes with minimal changes, providing insufficient dynamic content improve feature learning. In contrast, the 3.0-second interval often presents scenes that may significantly differ from the current frame, making accurate future predictions more challenging. Moreover, we observe that predicting latents 10 seconds into the future completely diminishes the gains provided by the world model, suggesting that predicting features too far into the future ineffective. This conclusion aligns with observations from MAE (He et al.] (002). where both excessively low and high mask ratios negatively impact the ability of the network.
Table 6: Different time horizons for latent prediction. The time intervals of 0.5, 1.5, 3.0, and 10.0 seconds correspond the first, third, sixth, and twentieth future frames from the current frame, as keyframes occur every 0,5 seconds in the nuScenes dataset.
L2 (m) 1
Collision
(%)
Time Horizon
1s
2s
3s
Avg.
1s
2s
3s
Avg.
0.5s
0.26
0.57
1.01
0.61
0.14
0.21
0.54
0.30
1.5s
0.26
0.54
0.93
0.58
0.14
0.17
0.45
0.25
3.0s
0.28
0.59
1.01
0.63
0.13
0.20
0.48
0.27
10.0s
0.33
0.67
1.16
0.72
0.24
0.28
0.76
0.43
Network Architecture of Latent World Model To validate the impact of the network architecture of the latent world model, we conduct experiments as shown in Table 7 Firstly, it is evident that a single-layer neural network, represented as Linear Projection, is not adequate for fulfilling the functions of the world model, resulting in significantly degraded performance. The two-layer MLP shows considerable improvement in performance. However, it lacks the capability to facilitate interactions among different latent vectors. Therefore, the stacked transformer blocks as

---

Published as a conference paper at ICLR 2025
Ours
CND MUSSP
VAD
Figure 3: Visualization. This figure compares LAW in the perception-based setting with VAD (Jiang et al. 2023 On the right side of this figure, we display the results of ego trajectory prediction, agent motion prediction, and map construction in BEV. indicated by the red circles, our method captures more crucial scene information, which VAD overlooks. Consequently, VAD predicts forward trajectory that results a rear-end collision, as highlighted by the yellow circles.
default network architecture, which achieves the best results among the tested architectures. This indicates that interactions between feature vectors from different positions are important.
Table 7: Different network architecture of the latent world model. Linear Projection means single-layer network.
L2 (m) $
Collision
(%)
Architecture
1s
2s
3s
Avg.
1s
2s
3s
Avg.
Linear Projection
0.31
0.65
1.14
0.70
0.26
0.34
0.66
0.42
Two-layer MLP
0.27
0.58
1.07
0.64
0.17
0.23
0.59
0.33
Transformer Blocks
0,26
0.57
1.01
0.61
0.14
0.21
0.54
0.30
5.5 VISUALIZATION
Figure compares the results LAW the perception- setting with VAD (Jiang et al. 2023). Leveraging our latent world model, our approach acquires more comprehensive scene representations.
CONCLUSION
In conclusion, we present the latent world model to predict future features from current features and ego trajectories, which is novel self-supervised learning method for end-to-end autonomous driving. This method jointly enhances scene representation learning and ego trajectory prediction. Our approach demonstrates universality by accommodating both perception-free and perception- based frameworks, predicting perspective-vies features and BEV features respectively. We achieve state-of-ther afrt results benchmarks like nuScenes, NAVSIM, and CARLA.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work was supported by National Science and Technology Major Project (20222D0116500) This work was also supported in part by the National Key R&D Program of China (No. 2022ZD01 16500), the National Natural Science Foundation of China (No. U21B2042, No. 62320106010), and in part by the 2035 Innovation Program of CAS.
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.1.. 1020066.1882 11810, 2021.
Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14403-14412, 2021.
Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17222- 17231, 2022.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pp. 66-75 PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp- 1597-1607. PMLR, 2020b.
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In ICCV, 2019.
OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving, 2023.
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang
Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. arXiv preprint arXiv:2406. 15349999.539,, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In CoRL, 2017.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. Proceedings of the IEEE/CVE international conference on computer vision workshops, pp. 0-0, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory--aumennted dense predictive coding for video representation learning. In European conference computer vision, pp. 312-329. Springer, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, pp. 16000-16009, 2022.
Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. NeurIPS, 2022a.
11

---

Published as a conference paper at ICLR 2025
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-I: A generative world model for autonomous driving. arXiv preprint arXiv:2309.
17080, 17000, 2023a.
Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with supervisee freespace forecasting. In CVPR, 2021.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022b.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, et al. Goal-oriented autonomous driving. arXiv preprint 1021v:2212.10166, 2022c.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving. In CVPR, 2023b.
Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11796 2021.
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9223-9232, 2023.
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung. Jingwei Ji. Kristy Choi, Di Huang, Tong Hc, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262, 2024.
Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In ICCV, 2023.
Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.1354 2023a.
Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023b.
Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023c.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajic Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, 2023.
Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differen- tiable raycasting for self-supervised occupancy forecasting. ECCV, 2022.
Yingyan Li, Yuntao Chen, Jiawei He, and Zhaoxiang Zhang, Densely constrained depth estimator for monocular 3d object detection. In European Conference on Computer Vision, pp. 718-734. Springer, 2022a.
Yingyan Li, Lue Fan, Yang Liu, Zehao Huang, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Fully sparse fusion for 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima. Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird' v-ewe-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint 2rXiv:2203.172700 2022b.
12

---

Published as a conference paper at ICLR 2025
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to aad-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14864-14873, 2024b.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision, pp. 531-548. Springer, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint a60iv (6080098008 2016.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini arXiv: 1.05 2017.
Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li. Yulan Guo, Junliang Xing, et Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. arXiv preprint arXiv:2405. 04390, 2024.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14760-14769, 2024.
Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.02443; 2022.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021.
Katrin Renz, Kashyap Chitta, Otnicl-Bogdan Mercea, A Koepke, Zeynep Akata, and Andreas Geiger. Plant: Explainable planning transformers via object-level representations. arXiv preprint 1xXiv:2210.14222 2022.
Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV, 2020.
Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. CoRL, 2022.
Hao Shao, Letian Wang, Ruobing Chen, Steven I. Waslander, Hongsheng Li, and Yu Liu. ReasonNet: End-to-End Driving with Temporal and Global Reasoning. In CVPR, pp. 13723-13733, 2023.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024.
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang. Immortal tracker: 2021.
Tracklet never dies. arXiv preprint
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint 2rXiv:2405.01533. 2024a.
13

---

Published as a conference paper at ICLR 2025
Yuqi Wang. Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. Panooce: Unified occupancy representation for camera- based 3d panoptic segmentation. arXiv preprint arXiv:2306. 10013, 2023a.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multíview visual forecasting and planning with world model for autonomous driving. arXiv preprint arXiv:2311.17918, 2023b.
Yuqi Wang, Ke Cheng, Jiawei He, Qitai Wang. Hengchen Dai, Yuntao Chen, Fei Xia, and Zhaoxiang Zhang. Drivingdojo dataset: Advancing interactive and knowledge-enriched driving world model. arXiv preprint urXiv:2410. 10738 2024b.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang. and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15449-15458, 2024.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: simple yet strong baseline. NeurIPS, 2022.
Penghao Wu, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy pre-training for autonomous driving via self-supervised geometric modeling. In ICLR, 2023.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Learning unsuper- vised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01011 2023.
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In European Conference Computer Vision, pp. 111-128. Springer, 2022.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oceworld: Learning 3d occupancy world model for autonomous driving, arXiv preprint arXiv:2311.16033 2023.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In European conference computer vision, pp. 474-490, Springer, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PREDICTING MULTIPLE FEATURES USING MULTIPLE INPUT FRAMES
Predicting multiple future features To better investigate the ability of our latent world model, we utilize the latent world model to predict multiple future frame latents, with the results presented in 8 We conduct this experiment using only the front-view camera to facilitate fast training. In detail, the future frame latents are predicted in auto-regressive manner. For example, we first predicted the latent for 1.5 seconds into the future, then used this predicted latent to further predict the latent for seconds into the future. The latent world model shares the same weights throughout this process. The results demonstrate that predicting multiple future latents further improves performance.
Table 8: Predicting multiple future latents in an auto-regressive manner.
L2
(m)
Collision (%) |
Predicted Future
1s
2s
38
Avg.
2s
38
Avg.
1.58
0.34
0.69
1.17
0.73
0.12
0.22
0.63
0.32
1.5s +
0.31
0.65
1.12
0.69
0.11
0.19
0.57
0.29
Predicting multiple future features using multiple input frames Building on the previous section, we further conduct experiments to predict multiple future frame latents while incorporating multiple input frame latents to leverage temporal information more effectively. Specifically, we adopted two-stage training paradigm for improved convergence, inspired by SOLOFusion (Park et al. 2022). In the first stage, we trained the model using single input frame latents for 12 epochs. This corresponds to the model denoted "1.5s 3s" in Table 8 In the second stage, we fine-tuned the model for an additional epochs, now using two input frame latents, from Os and 1.5s ago. The results summarized in Table(9 The baseline (first row) represents the model fine-tuned using only single input frame latents. In contrast, the second row corresponds to the model fine-tuned with two input frame latents. The latter achieves significantly better performance. This highlights the crucial role of temporal information in autonomous driving.
Table 9: Predicting future latents with multiple history frame inputs.
L2 (m) |.
Collision (%) J
Predicted Future
Input Frames
1s
2s
Avg.
2s
Avg.
1.5s +
0.30
0.64
0.68
0.14
0.23
0.33
1.58
-1.5s. Os
0.26
0.51
0.87
0.55
0.08
0.09
0.33
0.17
A.2 MORE VISUALIZATION
In the appendix, we provide more visualization figures. We also provide demo based on the CARLA simulator in the supplementary materials.
15

---

Published as a conference paper at ICLR 2025
Ours
Stoigh
VAD
Strigt
Ouurs
Worignt
VAD
Figure 4: Visualization. As shown the red circle, our map construction results are noticeably better than those of VAD.
16

---

Published as a conference paper at ICLR 2025
CAL
Tum Right
Ours
6
Tur Right
VAD
BCV
Ours
GEV
VAD
Figure 5: Visualization. As shown in the red circle, our agent motion prediction results are noticeably better than those of VAD.
17

---

Published as a conference paper at ICLR 2025
BLUE HARVES
Straight
Ours
DALPOOT
PERSKE
BLUE HARVES
VAD
Stright
100g
Ours
Straight
VAD
Figure 6: Visualization. As shown in the red circle, our map construction and agent motion prediction results are noticeably better than those of VAD, especially heavily occluded and crowded conditions.
18


Reviews:
[{"review_id": "1jA5gAsMu2", "reviewer": "Reviewer_MgUw", "summary": "This paper introduces LAW (LAtent World model), a self-supervised learning approach for end-to-end autonomous driving. The key innovation is using a latent world model to predict future scene features based on current features and ego trajectories. The method can be integrated into both perception-free and perception-based frameworks, predicting either perspective-view features or BEV (Bird's Eye View) features respectively. The authors demonstrate state-of-the-art performance across multiple benchmarks including nuScenes, NAVSIM, and CARLA simulator.", "questions": "1. What is the computational overhead of adding the latent world model? How does this impact real-time performance?\n2. How does the method perform under challenging weather conditions or poor lighting? Are there specific failure modes?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel integration of world model concepts into end-to-end driving\n2. Comprehensive experimental validation across multiple benchmarks. Demonstrates practical improvements in both closed and open-loop settings", "weaknesses": "1. Limited discussion of computational overhead - no analysis of inference time or model size. Autonomous driving systems must make decisions in real-time, typically requiring processing speeds of at least 10-20 Hz (decisions every 50-100ms). Without inference time analysis, it's unclear if LAW is applicable for real deployment on edge computing devices.\n2. No discussion of robustness to adverse weather/lighting conditions. As in Appendix A.1, the augmentation is claimed to enhance the robustness, but there is no experiments to validate such ability of the model. \n3. Missing references:\n- [1] DriveVLM: The convergence of autonomous driving and large vision-language models\n- [2] EMMA: End-to-End Multimodal Model for Autonomous Driving\n- [3] VLP: Vision Language Planning for Autonomous Driving\n- [4] OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3jKhObJxs", "reviewer": "Reviewer_AFWv", "summary": "This paper introduces LAW, a self-supervised method that enhances end-to-end autonomous driving without the need for expensive annotations. It employs a latent world model to predict future latent features based on current data and actions, along with a view selection strategy to improve model efficiency. LAW outperforms state-of-the-art methods on both open-loop and closed-loop benchmarks.", "questions": "Current View Selection Strategy Adaptability: Currently, the view selection strategy is designed for six cameras. If the number of cameras changes, how much effort is required to adjust the strategy? Is it possible to design the strategy to be adaptive to: 1) the total number of cameras, and 2) the selected number of cameras used at each time step—for example, using two cameras at  t=0  and three cameras at  t=1 ? In that case, how would you modify the selection reward?\n\nTypographical Error in ‘Implementation Details’: On page 7 in the “Implementation Details” section, the last sentence starting with “For the closed-loop benchmark. And we use…” seems to contain a typographical error or is missing information.\n\nEnhancing System with Additional Supervision: In Table 1, the ablation study shows the effectiveness of latent prediction. Additionally, what other forms of supervision could be added to further improve the system?\n\nPerformance with View Selection Strategy: In Tables 1 and 2, the open-loop and closed-loop tests are based on the default LAW, which does not use the View Selection strategy. If that’s the case, could you clarify the performance when the View Selection strategy is included?\n\nPerformance Discrepancy in Table 7: In Table 7, why is the performance using six views worse than using the Front + GT Views—for example, why is six-view performance worse than two-view performance? Could you explain the possible reasons?\n\nMetrics at Other Time Horizons: In Table 4, the time horizons for latent prediction are only 0.5, 1.5, and 3.0 seconds. How does the metric change at other time horizons, such as 1 or 2 seconds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed LAW framework utilized a self-supervised method to significantly reduce the need for heavy annotation tasks, addressing the data scalability challenge of many existing methods. The detailed breakdowns of ablation studies, latency analyses, and visualizations provide readers with clear and comprehensive information to understand and reproduce the work.", "weaknesses": "The view selection strategy is a valuable insight to improve the efficiency of the method, but it adds complexity to the overall framework. Although there is only a minimal performance drop, it seems the view selection strategy hasn’t fully captured the informative scenes in driving scenarios. If there could be more discussion or analysis on what caused the performance drop, or how this issue could be mitigated with the Latent World Model, it would make the work more complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SI1Z07Bj1D", "reviewer": "Reviewer_g24j", "summary": "This paper proposes the Latent World Model (LAW) to predict future features based on current features and ego trajectories, presenting a novel approach for end-to-end autonomous driving.", "questions": "1. What is the shape of the Visual Latents, and can provide ablation studies?\n\n2. Will LAW's use of the unique trajectory of the next frame as ground truth supervision limits the model's capabilities, given that trajectories are often not unique? Since the paper uses a simulation platform, could experiments with multi-trajectory supervision be provided?\n\n3. LAW currently only uses the current frame latent to predict the next frame latent, while the introduction mentions that \"using temporal data is crucial.\"\n\n   3.1. Can experiments be conducted to predict multiple future frame latents?\n\n   3.2. Can the model be trained to predict multiple future frame latents while taking in multiple input frame latents, leveraging temporal information more effectively?\n\n4. Given that LAW currently predicts only the next frame latent based on the current frame latent, how does it handle tasks like predicting multiple future frames in nuScenes? Is this achieved through progressive prediction of the next frame latent?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduction of the Latent World Model (LAW) to predict future scene latents from current scene latents and ego trajectories.\n\n2. Demonstrated universality across various common autonomous driving paradigms, i.e.,  perception-free and perception-based frameworks.\n\n3. Extensive experiments conducted on multiple benchmarks, achieving state-of-the-art performance on real-world open-loop datasets like nuScenes and simulator-based closed-loop CARLA benchmark.", "weaknesses": "See the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQp9JyNfFN", "reviewer": "Reviewer_z3s4", "summary": "This paper proposes a self-supervised learning approach, LAW, using the latent world model for end-to-end planning. The latent world model predicts future scene features based on the current scene features and predicted waypoints to learn better scene feature representation. This module is compatible with both perception-free and perception-based frameworks. The authors demonstrate that the proposed algorithm can beat existing baselines on three different benchmarks.", "questions": "Please consider replying to the points in the Weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed latent world model is a flexible plug-in module, which should be compatible with various end-to-end planning framework. \n2. This paper conducts extensive experiments, showing the great performance on multiple benchmarks. The ablation study is comprehensive to cover different aspects of the latent world model.\n3. The paper is well-written, very easy to understand and follow.", "weaknesses": "1. I think the idea of using self-supervised world model for autonomous driving is not novel, discussed in multiple prior works like GAIA-1, ADriver-I, and Drive-WM. The main difference of this work is the world model in latent space. However, it is not well-motivated why latent world model is better than world model in image space. For example, people can easily evaluate the image space world model by visualizing the prediction but it is hard to do the same thing for latent world model.\n\n2. Following 1, the authors do not show any direct evaluation of world model future prediction itself. I notice that the performance gap of different time horizons are very small (Tab. 6) compared with the gain brought by world model (Tab. 4). Does the future prediction performance similar for different time horizons? If not, why their performances are similar? In my opinion, I do not think it is easy to predict the future if the time horizon is as long as 3s. Maybe the authors can at least provide the L2 error of the latent features on the validation set for different time horizons. \n\n3. I think the settings in the experiment part are not fair. LAW uses Swin-T as the image backbone for nuScenes dataset. However, the baseline VAD and BEV-Planner are using ResNet-50, which is much weaker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)