DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving

Authors: Xiaosong Jia, Junqi You, Zhiyuan Zhang, Junchi Yan
Primary area: applications to robotics, autonomy, planning
Keywords: end-to-end autonomous driving

Abstract:
End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field of autonomous driving, promising a data-driven, scalable approach to system design. However, existing E2E-AD methods usually adopt the sequential paradigm of perception-prediction-planning, which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system’s ability to leverage synergies between tasks (for example, planning-aware perception and game-theoretic interactive prediction and planning). Moreover, the dense BEV representation adopted by existing methods brings computational challenges for long-range perception and long-term temporal fusion.  To address these challenges, we present DriveTransformer, a simplified E2E-AD framework for the ease of scaling up, characterized  by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other at each block), Sparse Representation (Task queries direct interact with raw sensor features), and Streaming Processing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. DriveTransformer achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.

Decision: accept (poster)
Reviewer scores: 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730621866025

thread_type:
official_review

<thread_object>
creator: reviewer_fh94

<content>
summary:
This paper proposes a new architecture for end-to-end driving, with three key features: task parallelism, sparse representation, streaming processing. With these key features, the proposed method reduces the complexity of the system and leads to better training stability, achieving SOTA performance on both open-loop and close-loop benchmarks.

strengths:
1. It is a simple and effective idea using a vanilla transformer to unify sensor tokens and task queries.  
2. Benefiting from the simple design, the model can scale up to achieve better performance.  
3. The performance on close-loop benchmark is remarkable, surpassing previous methods by a large margin.  
4. The writing and figures are easy to follow and understand.

weaknesses:
1. The open-loop experiment should exclude ego-status information to prevent short-cut learning on nuScenes.  
2. There seems no evidence to support that BEV Sequential Paradigm has bad training stability.

questions:
1. Curious about the scaling ability, the performance seems not saturate, have you tried on larger model or larger data (Bench2Drive full dataset)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730633672431

thread_type:
official_review

<thread_object>
creator: reviewer_6pt4

<content>
summary:
This paper presents DriveTransformer, a unified Transformer framework for end-to-end autonomous driving. It features three key properties: task parallelism, sparse representation, and streaming processing, and contains three unified operations: task self-attention, sensor cross-attention, and temporal cross-attention, reducing system complexity and improving training stability. The framework achieves good performance in both the simulated closed-loop Bench2Drive and the open-loop nuScenes benchmarks.

strengths:
● Originality: Proposes a new unified Transformer framework, combining sparse representation with parallelized task processing, distinct from existing methods.
● Quality: The design of each module is reasonable and effectively addresses existing problems.
● Clarity: The paper has a clear structure and it is easy to understand the core ideas.
● Significance: Provides new directions and ideas for autonomous driving research.

weaknesses:
● Originality: Lacking enough original design. Because the task parallel concept is not entirely new (ParaDrive), and the sparse representation (core part following PETR & PETRv2) and streaming process lack strong originality. 
● Comparisons: Lacking comparisons with similar methods in sparse and parallel aspects.

questions:
1. In Figure 1, could it be explained why the training stability of b) is poor? And compared to c), why is the task relation of d) okay?
2. The streaming process maintains a FIFO process, retaining and reusing historical information. The BEV temporal fusion also maintains and reuses a fixed-length bev features. If it is a recurrent model, only the previous and next two frames are needed. So the BEV temporal processing of traditional methods can also be regarded as a streaming process. What is special about the streaming process in this paper?
3. In this paper, when comparing performance, training efficiency and inference latency, the comparisons are all made with UniAD and VAD. Is it possible to make a comparison at the same sparse method level with Sparsedrive and SparseAD, also at the same task parallelism method such as ParaDrive?
4. From Table 6c and Table 8, DriveTransformer shows significantly better robustness compared to VAD. Should Table 6c be Table 7?
5. When scaling up in this paper, the largest model used is 0.6b. Would the effect be better if the model were larger?
6. It is mentioned many times in the paper that the sequential model design can cause cumulative errors and training instability. Is it possible to make some comparisons between parallel and sequential methods, especially in terms of cumulative errors and training stability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657978087

thread_type:
official_review

<thread_object>
creator: reviewer_vvtw

<content>
summary:
This paper proposes a new architecture for end to end autonomous driving where all tasks are connected with others. Task queries and raw sensor features are connected. Also, the task queries are passed as history information for temporal cross attention. Experiments show competitive results with other SOTA architectures in the field.

strengths:
The paper proposes a new architecture where all modules interact with each other. This should be able to capture complicated spatio-temporal dependencies among modules. They have also suggested using sensor cross attention and temporal cross attention, this can, at least in theory improve the generalization performance with sensor noise and failures or modules.

weaknesses:
* The motivation for this architecture is not clear. When all modules interact with each other, we ignore the benefits of using modular end to end architectures. This would make the system difficult to debug and maintain. This can only be justified if the performance is significantly better than the modular architectures, however this does not seem to be the case.
* The results for detection, prediction, mapping is not shown. End to end driving is primarily about planning, however, it makes it more convincing if results of related tasks are also shown to be better.
* It would have useful to see the performance of ParaDrive in closed loop setting, as this architecture is closest in concept with ParaDrive. Only open loop comparison is shown.
* This manuscript could use a grammar / spell check, there are quite a few typos.

questions:
* Task parallelism is not the right way to describe this architecture, what is parallel here? 
* The benefit of this architecture needs to be motivated better, open loop results are not considerably better. Closed loop metrics are also mixed. The latency is also comparable. So, what is the benefit of using this architecture?
* What is the criterion for Dev10? 10 scenes out of 220 seem very selective, the criterion needs to be justified, otherwise the results could be prone to selection bias.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732106061757

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear AC and reviewers, 

We express our gratitude to all reviewers for their valuable time and insightful comments. **We share the same excitement to explore the scaling law for E2E-AD and we will open source all our code so that members in the community could try larger models**. Following your kind advice, we added more experiments, results, and explanations to demonstrate the advantages of DriveTransformer from different perspectives. Please check [our updated paper](https://openreview.net/pdf?id=M42KR4W9P5) where the changes are marked in blue.

We hope our responses could solve your concerns. If you have any further concerns or questions to discuss, we are more than willing to address them. Looking forward to your further suggestions to improve the paper!

Thanks,

Authors of Submission 2919
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521391456

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

As the deadline of author-reviewer discussion is approaching, If you have any further concerns or questions to discuss, we are more than willing to address them.

Thanks,

Authors of Submission 2919
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a simplified E2E-AD framework, termed DriveTransformer, to streamline end-to-end autonomous driving. The proposed method leverages attention mechanism to connect different modality, representations, and perception signals, which leads to a scalable and unified architecture. Initially, reviewers are concerned about the completeness of experiments (e.g. comparison to ParaDrive, Sparsedrive, SparseAD),  criticism on BEV Sequential Paradigm, and technical details on nuScenes. During the rebuttal period, authors provide extensive experiments to address these concerns. Reviewers are unanimously positive about this paper. Therefore, I recommend accepting this paper.