Enhancing End-to-End Autonomous Driving with Latent World Model

Authors: Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, Tieniu Tan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: end-to-end autonomous driving, world model, self-supervised learning

Abstract:
In autonomous driving, end-to-end planners directly utilize raw sensor data, enabling them to extract richer scene features and reduce information loss compared to traditional planners. This raises a crucial research question: how can we develop better scene feature representations to fully leverage sensor data in end-to-end driving? Self-supervised learning methods show great success in learning rich feature representations in NLP and computer vision. Inspired by this, we propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving. LAW predicts future latent scene features based on current features and ego trajectories. This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks, improving scene feature learning while optimizing trajectory prediction. LAW achieves state-of-the-art performance across multiple benchmarks, including real-world open-loop benchmark nuScenes, NAVSIM, and simulator-based closed-loop benchmark CARLA. The code will be released.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729554975797

thread_type:
official_review

<thread_object>
creator: reviewer_z3s4

<content>
summary:
This paper proposes a self-supervised learning approach, LAW, using the latent world model for end-to-end planning. The latent world model predicts future scene features based on the current scene features and predicted waypoints to learn better scene feature representation. This module is compatible with both perception-free and perception-based frameworks. The authors demonstrate that the proposed algorithm can beat existing baselines on three different benchmarks.

strengths:
1. The proposed latent world model is a flexible plug-in module, which should be compatible with various end-to-end planning framework. 
2. This paper conducts extensive experiments, showing the great performance on multiple benchmarks. The ablation study is comprehensive to cover different aspects of the latent world model.
3. The paper is well-written, very easy to understand and follow.

weaknesses:
1. I think the idea of using self-supervised world model for autonomous driving is not novel, discussed in multiple prior works like GAIA-1, ADriver-I, and Drive-WM. The main difference of this work is the world model in latent space. However, it is not well-motivated why latent world model is better than world model in image space. For example, people can easily evaluate the image space world model by visualizing the prediction but it is hard to do the same thing for latent world model.

2. Following 1, the authors do not show any direct evaluation of world model future prediction itself. I notice that the performance gap of different time horizons are very small (Tab. 6) compared with the gain brought by world model (Tab. 4). Does the future prediction performance similar for different time horizons? If not, why their performances are similar? In my opinion, I do not think it is easy to predict the future if the time horizon is as long as 3s. Maybe the authors can at least provide the L2 error of the latent features on the validation set for different time horizons. 

3. I think the settings in the experiment part are not fair. LAW uses Swin-T as the image backbone for nuScenes dataset. However, the baseline VAD and BEV-Planner are using ResNet-50, which is much weaker.

questions:
Please consider replying to the points in the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547845718

thread_type:
official_review

<thread_object>
creator: reviewer_g24j

<content>
summary:
This paper proposes the Latent World Model (LAW) to predict future features based on current features and ego trajectories, presenting a novel approach for end-to-end autonomous driving.

strengths:
1. Introduction of the Latent World Model (LAW) to predict future scene latents from current scene latents and ego trajectories.

2. Demonstrated universality across various common autonomous driving paradigms, i.e.,  perception-free and perception-based frameworks.

3. Extensive experiments conducted on multiple benchmarks, achieving state-of-the-art performance on real-world open-loop datasets like nuScenes and simulator-based closed-loop CARLA benchmark.

weaknesses:
See the Questions section.

questions:
1. What is the shape of the Visual Latents, and can provide ablation studies?

2. Will LAW's use of the unique trajectory of the next frame as ground truth supervision limits the model's capabilities, given that trajectories are often not unique? Since the paper uses a simulation platform, could experiments with multi-trajectory supervision be provided?

3. LAW currently only uses the current frame latent to predict the next frame latent, while the introduction mentions that "using temporal data is crucial."

   3.1. Can experiments be conducted to predict multiple future frame latents?

   3.2. Can the model be trained to predict multiple future frame latents while taking in multiple input frame latents, leveraging temporal information more effectively?

4. Given that LAW currently predicts only the next frame latent based on the current frame latent, how does it handle tasks like predicting multiple future frames in nuScenes? Is this achieved through progressive prediction of the next frame latent?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638320229

thread_type:
official_review

<thread_object>
creator: reviewer_afwv

<content>
summary:
This paper introduces LAW, a self-supervised method that enhances end-to-end autonomous driving without the need for expensive annotations. It employs a latent world model to predict future latent features based on current data and actions, along with a view selection strategy to improve model efficiency. LAW outperforms state-of-the-art methods on both open-loop and closed-loop benchmarks.

strengths:
The proposed LAW framework utilized a self-supervised method to significantly reduce the need for heavy annotation tasks, addressing the data scalability challenge of many existing methods. The detailed breakdowns of ablation studies, latency analyses, and visualizations provide readers with clear and comprehensive information to understand and reproduce the work.

weaknesses:
The view selection strategy is a valuable insight to improve the efficiency of the method, but it adds complexity to the overall framework. Although there is only a minimal performance drop, it seems the view selection strategy hasn’t fully captured the informative scenes in driving scenarios. If there could be more discussion or analysis on what caused the performance drop, or how this issue could be mitigated with the Latent World Model, it would make the work more complete.

questions:
Current View Selection Strategy Adaptability: Currently, the view selection strategy is designed for six cameras. If the number of cameras changes, how much effort is required to adjust the strategy? Is it possible to design the strategy to be adaptive to: 1) the total number of cameras, and 2) the selected number of cameras used at each time step—for example, using two cameras at  t=0  and three cameras at  t=1 ? In that case, how would you modify the selection reward?

Typographical Error in ‘Implementation Details’: On page 7 in the “Implementation Details” section, the last sentence starting with “For the closed-loop benchmark. And we use…” seems to contain a typographical error or is missing information.

Enhancing System with Additional Supervision: In Table 1, the ablation study shows the effectiveness of latent prediction. Additionally, what other forms of supervision could be added to further improve the system?

Performance with View Selection Strategy: In Tables 1 and 2, the open-loop and closed-loop tests are based on the default LAW, which does not use the View Selection strategy. If that’s the case, could you clarify the performance when the View Selection strategy is included?

Performance Discrepancy in Table 7: In Table 7, why is the performance using six views worse than using the Front + GT Views—for example, why is six-view performance worse than two-view performance? Could you explain the possible reasons?

Metrics at Other Time Horizons: In Table 4, the time horizons for latent prediction are only 0.5, 1.5, and 3.0 seconds. How does the metric change at other time horizons, such as 1 or 2 seconds?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692465207

thread_type:
official_review

<thread_object>
creator: reviewer_mguw

<content>
summary:
This paper introduces LAW (LAtent World model), a self-supervised learning approach for end-to-end autonomous driving. The key innovation is using a latent world model to predict future scene features based on current features and ego trajectories. The method can be integrated into both perception-free and perception-based frameworks, predicting either perspective-view features or BEV (Bird's Eye View) features respectively. The authors demonstrate state-of-the-art performance across multiple benchmarks including nuScenes, NAVSIM, and CARLA simulator.

strengths:
1. Novel integration of world model concepts into end-to-end driving
2. Comprehensive experimental validation across multiple benchmarks. Demonstrates practical improvements in both closed and open-loop settings

weaknesses:
1. Limited discussion of computational overhead - no analysis of inference time or model size. Autonomous driving systems must make decisions in real-time, typically requiring processing speeds of at least 10-20 Hz (decisions every 50-100ms). Without inference time analysis, it's unclear if LAW is applicable for real deployment on edge computing devices.
2. No discussion of robustness to adverse weather/lighting conditions. As in Appendix A.1, the augmentation is claimed to enhance the robustness, but there is no experiments to validate such ability of the model. 
3. Missing references:
- [1] DriveVLM: The convergence of autonomous driving and large vision-language models
- [2] EMMA: End-to-End Multimodal Model for Autonomous Driving
- [3] VLP: Vision Language Planning for Autonomous Driving
- [4] OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning

questions:
1. What is the computational overhead of adding the latent world model? How does this impact real-time performance?
2. How does the method perform under challenging weather conditions or poor lighting? Are there specific failure modes?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed the LAtent World model (LAW), a method of self-supervised learning designed for end-to-end autonomous driving. The primary breakthrough lies in employing a latent world model to forecast upcoming scene characteristics, utilizing existing features and ego trajectories. The manuscript was reviewed by four experts in the field. The recommendations are (2 x "6: marginally above the acceptance threshold", 2 x "8: accept, good paper"). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns (especially additional and important experimental evaluations and ablation studied, needed comparisons with previous literature (clarification regarding technical insights), and together with further polishment of the manuscript) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.