DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving
Authors: Xiaosong Jia
Junqi You
Zhiyuan Zhang
Junchi Yan
Venue: ICLR 2025 Poster (2025)
Keywords: end-to-end autonomous driving
Primary area: applications to robotics, autonomy, planning
TL;DR: We develop a unified Transformer based method for end-to-end autonomous driving without complex operators and achieves SOTA performance.

Abstract: End-to-end autonomous driving (E2E-AD) has emerged as a trend in the field of autonomous driving, promising a data-driven, scalable approach to system design. However, existing E2E-AD methods usually adopt the sequential paradigm of perception-prediction-planning, which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system’s ability to leverage synergies between tasks (for example, planning-aware perception and game-theoretic interactive prediction and planning). Moreover, the dense BEV representation adopted by existing methods brings computational challenges for long-range perception and long-term temporal fusion.  To address these challenges, we present DriveTransformer, a simplified E2E-AD framework for the ease of scaling up, characterized  by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other at each block), Sparse Representation (Task queries direct interact with raw sensor features), and Streaming Processing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. DriveTransformer achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.

Published as a conference paper at ICLR 2025
DRIVETRANSFORMER, UNIFIED TRANSFORMER FOR SCALABLE END-TO-EN AUTONOMOUS DRIVING
Xiaosng Jia*, Junqi You*, Zhiyuan , Junchi Yan Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University
* Equal Contributions *Correspondence Author
ht nttps://github com
ABSTRACT
End-to-end autonomous driving (E2E-AD) has emerged as trend in the field of autonomous driving, promising data-driven, scalable approach to system de- sign. However, existing E2E-AD methods usually adopt the sequential paradigm of pnrreppion-predictioo--aaae
which leads to cumulative errors and training instability. The manual ordering of tasks also limits the system's ability to lever- age synergies between tasks (for example, planning-aware perception and game- theoretic interactive prediction and plamning). Moreover, the dense BEV repre- sentation adopted by existing methods brings computational challenges for long- range perception and long-term temporal fusion. To address these challenges, we present Draverrannformerr simplified E2E-AD framework for the ease of scal- ing up, characterized by three key features: Task Parallelism (All agent, map, and planning queries direct interact with each other each block). Sparse Represen- tation (Task queries direct interact with raw sensor features), and Streaming Pro- cessing (Task queries are stored and passed as history information). As a result, the new framework is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention, which significantly reduces the complexity of system and leads to better training stability. TriveTransfrrmer achieves state-of-ther of-the-rrtttt performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.
INTRODUCTION
Sensor Data
Dense BEV Feat
BEV Encoder
Mapping
Planning
Prediction
Planning
Detection
Auxiliary Task Training Stability
Task Relation Efficiency
Auxiliary Task
Task Relation
Planning Paradigm
BEV Sequential Paradigm
Training Stability
X
Efficiency
x
Mapping
Sensor Data
BEV
Mapping
Sensor Data
Unitfed Transformer
Dotection
Detection
Task Query
Prediction
Encoder
Prediction
Planning
Planning
Mapping
Planning
Parallel BEV Paradigm
Auxiliary Task
Task Relation X
Transformer Paradigm (Ours)
Auxiliary
Relation Efficiency
Training Stability
Efficiency
Training Stability
Figure 1: End-to-End Autonomous Driving Paradigm Comparison. The proposed pure Trans- former paradigm avoids the construction of expensive BEV features and allows the tasks to learn their relations with raw sensor inputs, other tasks, and histories all by Transformer attention.
Autonomous driving has been topic of interest (Li et al. 2023 Yang et al.) 2023b) in recent years, with significant progress being made in the field (Hu et al. 2023 Jia et al. 2023b). One of
Correspondence author is also affiliated with Shanghai Innovation Institute. This work was in part sup- ported by by NSFC (62222607) and Shanghai Municipal Science and Technology Major Project under Grant 2021SH2DZX0102.

---

Published as a conference paper at ICLR 2025
Imeges Camera Parametors,
Image Backbone
Sensor Token
Top-K
Push
FIFO:
Pop
Position
K.V
XW
Encoding
Sensor Cross Attention
Past
Motion Compensation
Tomporal Taken
TAXI
K.
10
Detection
Memory Queue
Temporal Cross Attention
Prediction
Ego Transformation
10
Online Mapping
Task Self Attention
Agent Ref
Position Encoding
Map Ref
00000
Planning
|||!!!!!!!
Task
Preprocess
Agent Token
Map Token
Ego Token
Sensor Token
Temporal Token
Figure 2: Overall Framework of DriveTransformer: DriveTransformer features streaming, par- allel, and sparse token interaction. At each layer, task tokens interact with each other by Task Self Attention, extract information from raw sensor inputs by Sensor Cross Attention, and fuse temporal information from history task tokens in memory queue by Temporal Cross Attention.
the most exciting approaches is end-to-end autonomous driving (E2E-AD), which aims to integrate perception (Li al. 2023), prediction et 2023a), and planning (Li at. 2024a) into single, holistic framework. E2E-AD is particularly appealing due to its data-driven (Lu et al., 2024)
and scalable nature, allowing for continuous improvement with more data.
Despite these advantages, existing E2E-AD methods (Hu et al. 2023 Jiang al. 2023 mostly adopt a sequential pipeline dependent upstream queries. This sequential design can lead to cumulative errors and thus training instability. For instance, the training process of UniAD (Hu et al., 2023) necessitates a multi-stage approach: first, pre-training the BEVFormer encoder (Li et al. 2022b); then, training TrackFormer and MapFormer; and finally, training MotionFormer and Planner. This fragmented training approach increases the complexity and difficulty of deploying and scaling the system in industrial settings. Moreover, the manual ordering of tasks can restrict the system's ability to leverage synergies, such planning-awarc perception (Philion al. (2020; Jia al., 2023d) and game-theoretic interactive prediction and planning (Jia et al. 202 Huang et al. 2023).
where downstream tasks are heavily
Another challenge existing methods grapple with is the spatial teemparalttr complexity of the real world. BEV-based representations (Li et al. (002b) encounter computational challenges with long- range detection (Jiang et al.) 2024) due to the dense nature of the BEV grid. Additionally, the image backbone of BEV methods is under-optimized due to weak gradient signals (Yang et al., (023a). hin- dering their ability to scale. For temporal fusion, BEV-based methods typically store history BEV features for fusion, which is computationally extensive BEV-based methods ignore the sparsity of 3D space and drop the task query of each frame, which results in significant waste of computation and thus suffer from efficiency (Li et al. 2024c).
well (Park et al.] (023). In summary,
The latest work ParaDrive (Weng et al., 2024) tries to mitigate the instability issue by removing all tasks' connection. However, it still suffers from the expensive BEV representation and their experiments is limited to open-loop, which could not reflect actual planning ability (Li al. (024c).
To address these deficiencies, introduce TriveTransformer able end-to-end autonomous driving, featuring three key properties as shown in Fig. 2
framework for efficient and scal-
Task Parallelism: All task queries directly interact with each other each block. fostering cross- task knowledge transfer while maintaining system stability without explicit hierarchy.
Sparse Representation: Task queries directly engage with raw sensor features, offering an effi- cient and direct means of information extraction, aligning with end-to-end optimization paradigm. Streaming Processing: Temporal fusion achieved through a first-in-firse
queue that stores task queries in history and temporal cross attention, ensuring efficiency and feature reuse.
DriveTransformer offers a unified, parallel, and synergistic approach to E2E-AD, facilitating easier training and scalability. As result, Drive Transfororrrrr achieves sfate-of-the-an llosed-loop perfor- mance in Bench2Drive (Jia et al. 2024) under CARLA simulation and state-of-ther the-art open-loop planning performance nuScenes (Caesar al. 2020b) dataset.
2

---

Published as a conference paper at ICLR 2025
Learnable
X
(W D
Parameters
1xD
Ng x
Backbone
MLP
Sensor Tokens
Map
Features
Agent
Ego Token
Ego Feats
Ne. x.D
Image Coordinate
Featurize
Coordinate
N.X W
CW D
Ne
Agent 3D PE
MLP
TK
Sensor PE
Map 3D
Zero Ego PE
Position Encoding
Na X
Uniform Points
Figure 3: Initialization Tokenization Process. Sensor inputs are processed by backbone while
their PE are their 3D coordinate. Agent and Map tokens initialized from learnable parameters while their initial PE are uniformly initialized. Ego token initialized from canbus information while its PE is initialized zeros.
RELATED WORKS
The concept of E2E-AD could date back 1980s (Pomerleau 1988). CIL (Codevilla et 2018) trained a simple CNN to map front-view camera images directly to control commands. Refined by CILRS (Codevilla et al.) 2019). it incorporated an auxiliary task predict the ego vehicle's speed, addressing issues related to inertia. Plan'T (Renz et al., 2022) approach suggested leverag- ing Transformer architecture for the teacher model, while LBC (Chen et al.) 2020) focused on initially training teacher model with privileged inputs. Moving forward, studies such as Zhang et al. (00TT): Li al. (2024a) ventured into reinforcement learning to create driving policies. Build- ing these advancements, student models were developed (Wu et al. 2022, Hu al. 2022a). In subsequent research, the use multiple sensors became prevalent, enhancing the models' capabil- ities. Transfuser (Prakash al. 2021 Chitta al. 2022) utilized Transformer for the integration of camera and LiDAR data. LAV (Chen & Krähenbüh]] (022) adopted the PointPainting (Vora al. (220)) technique, and Interfuser (Shao al.] (022) incorporated safety-enhanced rules into the decision-making process. Further innovations included the usage of VectorNet for map encoding by MMFN (Zhang al. 2022) and the introduction of DETR-like scalable decoder paradigm by ThinkTwice (Jia et al. (023d) for student models. ReasonNet (Shao et al. 2023) proposed special- ized modules to improve the exploitation of temporal and global information, while Daeger et al.
2023 suggested a classification-basse approach to student's 's output to mitigate the averaging issue.
In another branch where AD sub-tasks are explicitly conducted, ST-P3 (Hu et al. 2022b) integrated detection, prediction, and planning tasks into a unified BEV segmentation framework. Further, UniAD (Hu et al. 2023) employed Transformer to link different tasks, and VAD (Jiang et al. 2023) proposed vectorized representation space. ParaDrive (Weng al. (024) removes the links among all tasks while BEVPlanner (Li et 2024c) removes all middle tasks. Concurrent to our work, there sparse query based methods (Zhang et al. 2024 Sun et al. 2024 Su et al., (024).. However, they still follow the sequential pipeline while the proposed Drive' Transformer unifies all tasks into parallel Transformer paradigm.
METHOD
Given raw sensor inputs (e.g., multi-view images), DriveTransformer aims to output results for multiple tasks, including object detection (Li et al. 2022a), motion prediction (Jia et al. 2023c), online mapping (Chen et al. 2022), and planning (Li et al., (024a)). Each task handled by its corresponding queries, which directly interact with each other, extract information from raw sensor inputs, and integrate information from histories. The overall framework illustrated in Fig.
3.1 INITIALIZATION & TOKENIZATION
Prior to information exchange in DriveTransformee all inputs are converted into a unified represen- tation tokens. Inspired by DAB-DETE (Liu et al. (022a), all tokens consist of two parts: semantic embeddings for semantic information and position encodings for spatial localization. Fig. 3 we illustrate the process and we give details below.
Sensor Tokens: Multi-view images from surrounding cameras are separately encoded by backbones like ResNet (He al. 2016) into H sensor E X semantic emheddings, where N. the

---

Published as a conference paper at ICLR 2025
Sensor Cross Attention
Task Self-Attention Temporal Cross Attention
Agent-Agent Interaction
Temporal Attention
Agent. Map Ind Interaction,
Ego-Map Interaction
Planning
Ego-Agent Interaction
Map-Mapi Interaction
Detection & Prediction
Temporal
Figure 4: Three Types of Attention in Drive Transformer. Sensor Cross Attention provides a direct way for all tasks access raw inputs in end-to-end way. Task Self-Attention allows the information exchange among tasks. Temporal Cross Attention utilizes the history states as priors.
number of cameras, H and W are the height and width of the feature map after patchification, D is the hidden dimension. To encode the position information of sensor features, we adopt 3D Position Encoding in (Liu et al.) 2022b/c). Specifically, for each patch with pixel coordinate (i, i. its corresponding ray in the 3D space could be represented with K equally spaced 3D points: Ray,, = (77K-|1,3 dx] = 1, 2.., K}. where T and K are the extrinsic and intrinsic matrix of the camera, d₃ is the depth value for the hthh 3D point. Then, coordinates of points in the same rav are concatenated and fed into an MLP obtain the position encoding for each patch (Liu al.) 2022ch and the 3D position encoding for all image patches is denoted as PE E RR Ne xxxxxxxXX XD"
To model the heterogeneous driving scene, three types of task queries are initialized to extract different information: (I) Agent Queries represent dynamic objects (vehicles, pedestrians, etc), which will used to conduct object detection and motion prediction. (II) Map Queries represent static elements (lanes, traffic signs, etc), which will be used to conduct online mapping. (III) Ego Query represents the potential behavior the ego vehicle, which will be used to conduct planning. Following DAB-DETR (Liu et al. 2022a), , both agent queries' and map queries' semantic embeddings are initialized randomly y as learnable parameters H Hagem E R R. x and Happ E IR N.. x where N₂ and Nm the number of agent and map queries pre-defined position encodings PEagent E NRN xD and PEmup E R. are initialized uniformly within pre- defined perception range. For planning query, its semantic embedding initialized from MLP encoding canbus information Hego MLP canbbus) E R. similar to BEVFormer (Li et al. 2022b while its position encoding PEeeo E R2D is initialized as all zeros.
Task Tokens:
apper-parammters. Their
3.2 TOKEN INTERACTION
is established by the vanilla attention mechanisms /Vaswani et al. 2017), ensuring scalability and easy deployment. As a result, the model could be trained under one stage and demonstrate strong scalability, which will be shown in the experiments section. In following sub-sections, we describe the three types of information exchange adopted at each layer of Drive Trivefrransfomer and the illustration is in Fig. 4
All information exchange within
Sensor Cross Attention (SCA) establishes direct pathway between tasks and raw sensor inputs, enabling end-to-end learning without information loss. SCA is conducted as:
H' ogo: Hagent H'
map SCA-Attention(( Q = Hego + PE H Hageut + PEngent: H Hmap
PEmapls
(1)
= Hsensor + PEsensors V Hgensor)
where H' denotes the updated query. this way, raw sensor tokens are matched by task queries based on both semantic and spatial relations to extract task-specifie information in an end-to-end way without information loss. Notably, by adopting 3D position encoding (Liu et al. (022c). Driv-
e'Transormer avoids the construction of BEV feature, which is efficient and has less gradient vanishing issue (Yang et al. 2023a), allowing scaling up.
Task Self-Attention (TSA) enables direct interaction among arbitrary tasks without explicit con- straint, promoting synergy such as planning-aware aware perception (Philion et al.) 2020) and game- theoretic interactive prediction and planning (Huang et al. 2023). TSA is conducted as:
H H H H H'app = TSA-Attention(
Hego + +PEgo Hagen + PEageni, Hmup + PFmup];
ego
agent:
(2)
K = [oo + PE Hngent + PEegent, Hmap + PEEapp]
= [H H Hmpp))
4

---

Published as a conference paper at ICLR 2025
Memory Queues
Tyrurs
Drive Transformer
Tamporal Tokens
Spact
K.
Drive Transformer
Ego Transform
Motion Compensation
T
AC PügenE
Initialization
Push
Initiaiization
l0 +
Figure 5: Streaming Temporal Mechanism in DriveTransformer. Top-K task queries previous timestep from the last layer of DriveTransformer are pushed into FIFO queues. Task queries and positions are transformed into current ego coordinate system and are compensated for potential movement before feeding into Temporal Cross Attention Key and Value.
By eliminating manually designed task dependencies, the interleaved relations among tasks could be flexibly learnt via attention in data-driven way, which eases the scaling up. In contrast, UniAD (Hu al.] (202)) has to adopt a multi-stage training strategy due to the inconsistency at the early stage of training where inaccurate upstream modules influence downstream modules and finally collapse the whole training.
Temporal Cross Attention integrates information from previously observed history. Existing paradigms (Hu et al,, 2023] Jiang et al. 2023) use history BEV features to pass temporal informa- tion, which introduces two drawbacks: (a) Maintaining long-term BEV features is expensive (Park ctt al., 2023) (b) Previous task queries carrying strong prior semantic and spatial information are wastefully discarded. Inspired by StreamPETR (Wang et al. 2023), DriveTranssfrmer maintains First-In-First-Our (FIFO) queues of queries for each task respectively and conducts cross attention history queries in the queue each layer fuse temporal information, as illustrated in Fig. 5
Specifically, denote H ogo H H map with their corresponding position encodings PF PR agent PE map as the ego queries, agent queries, and map queries of the final layer of Drive Transfornsr at time-step t. Suppose current time-step is to and we maintain FIFO queues Queue {(H ago: PF PF )t
ogo agent
to 1, 2, ..., to - TT quene ). Queue
{(H agent, PF )t to 1,40
2,..., T } } and Queue {(H map: pE = 1, 1, 2,..,, Lo Tqueue) where T Tquee is pre-set hyper- hyper-yarameee parameter to control the length of temporal queue. After each time-step. current task queries the final layer are pushed into the queue and the task queries at to
are popped out. Further, since there are redundant queries in DETR style methods (Carion et al., 2020), for agent and map queries, only those with top-K K confidence scores are kept, where a hyper-paaaan pprrr-prrrmttr..
uses the history queries as Key and Value. Since the ego reference point at different time-step step could be different, the history queries' PE is transformed into current coordinate system (Ego Transformation):
Temporal Cross Attention
PE = MLP(TT Pos') where = to - 1, tu 2.... to Tqueue
(3)
where PE is the transformed PE, T is the coordinate transformation matrix from to ty. Besides, since other agents could have their own movement, following (Wang et al. 2023), we conduct DiT (Peebles Xie 2022) style ada-LN for Motion Compensation:
PE agent LayerNorm(PE "ggeet hy, 3] ML x (t--t))) where = to to-to.11.. 1,..., Tqueue (4)
where the layer-norm's weight 7 and bias beta is controlled by the predicted velocity of agents at time-step and the time interval between and current time-step to. Besides, we also set the relative time embedding temb MLP((- to) to indicate different time-steps and Temporal Cross- Attention is conducted as:
H ACA-AAtention(4) = Husk + PE K { misk + PE + tomb to 1,..., to - T Tqueue }.
V = H It = to - 1, .... to T queeee) }) where task=Ego, Map. Agent
(5)
5

---

Published as a conference paper at ICLR 2025
Sensor Cross Attention
Prediction
Local-Coord
Back prop
Shared Feature
Ego-Coard
Instance
Replicate
Detection
Temporal Cross Attention
a) Object Detection & Motion Prodiction
b) Point-level Online Mapping
Mode Embeddings
Multi-Mode Regression
Prod
Traj
GT Tra
áxjay
Transformer Layer
Finer PEs
Offect Prediction
Classification
Coarse PEa
Multi-Mode Planning
d) Coarse-to-Fine Optimization
Figure 6: Task Head Designs. (a) Detection and Motion share the same agent feature, which asso- ciate objects without tracking. Trajectory prediction local coordinate system further disentangles the two tasks, ensuring training stability. (b) Map points in the same polyline have different PEs in Sensor Cross Attention retrieve fine-grained features. (c) Planning head combines the ego query with different mode embeddings to conduct multi-mode prediction. (d) Tasks' positions are gradu- ally refined and their corresponding PEs are gradually more accurate, providing better interaction.
Pure Attention Architecture: In summary, DriveTransfrrmer stack multiple blocks where each block contains the aforementioned three attentions and FFN:
H ego
H H (TCTSSA agent:
H H ago agent:
H mapl: Hgensur) Hiss);
map
(6)
Queee Queue Quuuuempp)
where and + is the layer index, FFN means MLP Transformer (Vaswani et al. 2017) and we omit the PE, residual connection, and pre-layernorm for brevity. Note that the raw sensor tokens Hensor and history information Qucue.go Qucue agent: Queuemap are shared across all blocks.
3.3 DETR-STYLE TASK HEAD
Inspired by DETR CCaroo et al. 2020 Liu et al. 2022a), task heads are set after each block to gradually refine the predictions and PE would be correspondingly updated. following sub-sections, introduce task specific designs and the updating strategy of PE, shown in Fig.
Object Detection & Motion Prediction: Existing E2E methods (Zeng et al. 2022 Hu et al. 2023) still adopt the classic ing due to the inherent difficulty of association (Weng et al. 2022). For example, UniAD, there must be 3D object detection pretrained BEVFormer avoid the divergence of TrackFormer and then the MapFormer and TrackFormer must be primarily trained before the end-to-end training of MotionFormer and planning head, which necessitates its multi-stage training strategy and thus hin- ders the scaling up.
pipeline, which introduces instability to train-
To alleviate this issue, DrineTranffrrmer adopts more end-to-end methodology: conducting ob- ject detection and motion prediction without tracking, by feeding the same agent query into different task heads. The same feature for the same agent naturally establish associations between detection and prediction. For temporal association, since Temporal Cross Attention is conducted between current tokens and all history tokens, the explicit association is avoided, replaced by the learning-based attention mechanism. To further improve the training stability and reduce the inter- ference between these two tasks, the label of motion prediction is transformed the local coordinate system of each agent (Jia et al.) 2022) and thus its loss is not influenced by detection results at all. Only during inference, the waypoints from the prediction are transformed into the global coordinate system based on the detection to calculate motion prediction related metrics.
Online Mapping Recent progress the sub-field Li et al. 2024b Liu et al. 2024 suggests the importance of point-level instead of Instance-level feature retrieval due to the irregular and diverse distribution of map polylines. Thus, when conducting Sensor Cross Attention, we replicate each map query for N Npaim times paired with position encodings for each single point. In this way, for

---

Published as a conference paper at ICLR 2025
Table 1: Planning Performance in Bench2Drive. Avg. L2 is averaged over the predictions in seconds under 2Hz. denotes expert feature distillation. All latency are measured by the averaged inference step-time on CARLA evaluation in A6000.
Open-loup Metric
Closed-loop Metric
Method
Latency
Avg. L2
Driving Score
Success Rate(%)
Efficiency
Comfortness
AD-MLP (Zui et 2023
3.64
18.05
0.00
48.45
22.63
3mn
UniAD-Tiny
40.73
13.18
123.92
47.04
420
UniAD-Rase e 8
0.73
45.81
16.36
129.21
43.58
663 Ams
VAD Jiany
0.91
42.35
15,00
157.94
46,01
278. 3ms
Drivel woneTramsfoomn -Lag (Ours)
63.46
35.01
100.64
20.78
211.7ms
TCPs (Wa al. 2022
1.70
40.70
15.00
54.26
47.80
83ms
TCP-cfi1
30.47
7,27
55.97
$1.51
83ms
TCP-tajj*
1,70
59.90
30,00
76.54
18.08
83ms
TCP.mj distillation
83ms
Think Jia et al. 2023d
0.95
62.44
31,23
69.33
16.22
762ms
DriveAdapter
2025h
64.22
33.08
70.22
16.01
93
those long polylines, each point could retrieve sensor information with better locality. To integrate separate point-level map queries into instance-level ones for other modules, we adopt light-weight PointNet (Qi al., 2017) with max-pooling and MLPs.
Planning: We model the the future movement of the ego vehicle as Gaussian Mixture Model to avoid mode averaging, as widely done in the motion prediction field (Liang al. 2020). Specifi- cally, we divide all training trajectories according to their direction and distance into six categories: Go Straight, Stop, Left Turn, Sharp Left Turn, Right Turn, Sharp Right Turn. To generate trajec- tories of these modes, six mode embeddings are generated by feeding their sine&cosine encoded position (Vaswani et al. (017)) into an MLP and then we add them to the ego feature to predict six mode-specific ego trajectories. During training, only the Ground- Truth mode's trajectory would be used to calculate regression loss, i.e. winner-take-all (Liang et al. 2020), and we also train a classification head to predict current mode. During inference, the trajectory from the mode with the highest confidence score would be used to compute metries or execute.
Coarse-to-Fine Optimization: The success of DETR series (Carion et al. 2020) demonstrates the power of end-to-end learning by coarse-to-fine optimization. In DriveTraastorre rransformer, the Position Encodings (PE) of all task queries updated after each block based current predictions. Specif- ically, the Map and Agent PE are encoded by their corresponding predicted positions and semantic classes with an MLP to capture the spatial and semantic relations among elements. The ego PE is encoded by the predicted planning trajectory with an MLP to capture the ego intentions for the possible interactions. Similar to DETR, During training, losses are applied on task heads at all blocks while during inference we only use the output from the last block.
3.4 LOSS & OPTIMIZATION
DriveTransformer is trained under one single stage, where each tasks could gradually learn to find out their relations in Task Self-Attention, without collapsing each others' basic convergence under Sensor Cross Attention and Temporal Cross Attention. There are detection loss (DETR-style hungarian matching loss (Carion et al.] 2020)), prediction loss (winner-take-all style loss (Liang et al. 2020)), online mapping loss (MapTR (Liao et al. (023) style hungarian matching loss), and planning loss (winner-take- style loss) where we adjust weights to make sure all terms have the same magnitude around 1, in the following equation:
Loverall Wdetection Edenection + Wmotion Cmotion Wmapping mappinng Wplanning Cplanning
4 EXPERIMENTS
4.1 DATASET & BENCHMARK
We use Bench2Drive (Jia et al. 2024), closed-loop evaluation protocol under CARLA Leader- board 2.0 for end-to-end autonomous driving. provides an official training set, where we use the base set (1000 clips) for fair comparison with all the other baselines. We the official 220 routes for evaluation. Additionally, we compare our method with other state-of-the-nn baselines on nuScenes (Caesar et al. 2020a) lpen-loop evaluation. There are three different size of models:
When comparing with SOTA works, we report results of DriveTransformer-arrg For ablation studies, since evaluating on 220 routes of Bench2Drive could take days, select 10 representa-

---

Published as a conference paper at ICLR 2025
Table 2: Multi-Ability Results of E2E-AD Methods. denotes expert feature distillation.
Ability (%)
Method
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
AD-MLP (Zhai et
2023
0.00
0.00
0.00
UniAD-Tiny (Hu 2023
7.04
10.00
21.82
20.00
14.61
14.69
UniAD-Base [Hu
12.16
20.00
23.64
10.00
13.89
15.94
VAD [Tiang et al. 2023
7.14
20.00
16.36
20.00
20.22
16.75
Trasstomee Large (Ours)
17.57
35.00
48.36
40.00
52.10
38.60
TCP* (Wu al. 2022
17.50
13.63
20.00
10.00
6.81
13.59
TCP-ctil*
9.23
5.00
9.10
10.00
6.81
8.03
TCP-traj*
12.50
22.73
52.72
40.00
46.63
34,92
TCP-traj w/o distillation
14.71
7.50
38.18
50.00
29.97
28.03
ThinkTwice* JTa et al. 2023d
13.72
22.93
52.99
50.00
47.78
37.48
DriveAdapter Un 2023b
14.55
22.61
54.04
50.00
50.45
38.33
Table 3: Open-loop planning performance in nuScenes under VAD metric. denotes LiDAR-
based methods. $ denotes the usage of ego-status.
L2 (m)
Collision (%) +
Method
1s
2s
3s
Avg.
1s
2s
3s
Avg.
NMP (Zen et al.
2.31
1.92
SA-NMP (Zeng et
2019)
2.05
1.59
FF (Hu et al. 2021
0.55
1.20
2.54
1.43
0.06
0.17
1.07
0.43
RO
Khurana al.
2022
0.67
1.36
2.78
1.60
0.04
0.09
0.88
0.33
ST-P3 (Hu ct 2022b)
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
UniAD (Hu a..
2023
0.48
0.74
1.07
0.76
0.12
0.13
0.28
0.17
VAD-Tiny UJang èt al.
2023
0.46
0.76
1.12
0.78
0.21
0.35
0.58
0.38
VAD-Base (Jang et
2023)
0.41
0.70
1.05
0.72
0.07
0.17
0.41
0.22
BEVPlaner (i et al.
2024ch
0.27
0.54
0.90
0.57
DriveTransformer Large
(Ours)
0.19
0.34
0.66
0.40
0.03
0.10
0.21
0.11
VAD-Tiny (Jiang et al.
2023
0.20
0.38
0.65
0.41
0.10
0.12
0.27
0.16
VAD-Base (Jiag et al.
2023
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
AD-MLP Zhai al.
2023)
0.20
0.26
0.41
0.29
0.17
0.18
0.24
0.19
BEVPlaner++ (Li et al.
2024c)
0.16
0.32
0.57
0.35
ParaDrive+ WWeng et al. 2024)
0.25
0.46
0.74
0.48
0.14
0.23
0.39
0.25
DriverTranformer-lLaar
(Ours)
0.16
0.30
0.55
0.33
0.01
0.06
0.15
0.07
Table 4: Size Configuration of Drive Transformer. Latency is measured by the averaged model
time for closed-loop evaluation in CARLA. Training batch size is measured by A800 (80G) to the GPU memory. Driving Score is under Dev10 benchmark.
Size
Configuration
#Parameters
Latency
Training Batch Size
Driving Score
Small
Layers, 256 Hidden Layers, 512 Hidden
47.41M
93.8ms
45.04
Base
178.05M
139.6ms
28
60.45
Large
12 Layers, 768 Hidden
646.33M
221.6ms
12
68.22
tive scenes (namely Dev10) balancing behaviors weathers, and towns and report results on it with DriveTrannfoommrrbass for quick validation.
4.2 COMPARISON WITH OFATE-O--TH--AE METHODS
We compare TranSfrraffrmmr with SOTA E2E-AD methods in Table Table 2 and Table We observe that Transforrmr persistently outperforms SOTA methods. From Table Driv- Transformer has a lower inference latency compared to UniAD and VAD. Notably, because of the unified, sparse, and streaming Transformer design, DriveTransformer could be trained with batch size 12 in A800 (80G) while UniAD with batch size and VAD with batch size 4.
4.3 ABLATION STUDIES
In ablation studies, all closed-loop experiments are conducted on Dev10, a subset of Bench2Drive 220 routes, and all open-loop results Bench2Drive official validation (50 clips). Please
8

---

Published as a conference paper at ICLR 2025
L 12-0788
Nodlgg andlernn
Sudin Fruge Boclls
65
FVAD2-G IP-1
L12-D788
LS-D512
34% Faster
72
LS-D512
Pre-Training Still Helps
Large Scale
Driing Scae
18.22 Higher Drivng Score
ResNet101
Similar Trend
EVA02-CLIP4L
LS-0258 ResNa150
ReeNet101
L3-0255
RgdNet50
5.1%%
2003
0.70%
nine
0.00
0:223
0.250
5.3.5
0,200
0.22
0.200
0:73
inferwrce Labneryy Seceadd
(a) Influence on Planning
(b) Influence Perception
Figure 7: Scaling Study of Driving Transformer in Dev10 benchmark. Directly scaling up the unified Transformer structure benefits most on the planning while adopting powerful image back- bones, especially scale pretrained ones brings gains for perception.
Table 5: Paradigm Design Study with Drive Transformer- Base in Dev10.
(a) Attention Mechanism
(b) Training Strategies.
Method
Driving Score
Success Rule
Method
Driving Score
Success Rute
Fall Artention
60.45
30,00
Drive Tramsfommrr
60.45
30.00
wio Sensor-CA
8.41
0.00
Planning Only
$4,22
20.00
w/o Task-SA
52.37
20.00
Pretrain Perception
60.22
30.00
w/o Temporul- CA
56.22
20.00
w/o Middle Supervision
51.67
10.00
Table 6: Task Head Design Study with Drive prriveTransformer-bee
(a) Detection & Prediction
(b) Mapping.
(c) Planning.
Method
minADE 4
Method
MAP
Method
Driving Score 1 Success Rate
Local Prediction
1.34
Point PE
20.25
Multiple Mode
60.45
30,00
Global Prediction
2.68
Line
14.55
Single Mode
49.19
20.00
check Appendix B for details. We use smaller model (6 layers and 512 hidden dimension) for ablation studies save computational resource if not specified. We will open source Dev10 protocol, model code, and model checkpoints.
One attractive characteristic of Transformer- based paradigm is its extremely strong scalability (Radford et al. [019 BBrown et al. 22200). Since formers, we study the scaling behavior of increasing layers and hidden dimension simultaneously and compare the strategy with scaling image backbones similar to (Hu et al.) 2023 Jiang et al.) (023) as shown in Fig. 7 We could observe that scaling up the decoder part, i.e., the number of layers and width for the three attention brings more gains compared to scale up image backbones. It is natu- ral since the former one directly adds more computation to the planning task. On the other hand, for the perception task, scaling up the decoder has similar trend with scaling up image backbones, which demonstrates the generalizing ability of the proposed Transfrrnffommrr.. However, it still falls behind the large scale rision-language pretraining image backbone EVA02-CLIP-LL (Fang al., 2024), aligning with findings in (Wang et al. 2023). 3) It could be an important direction to study how to combine VLLM with autonomous driving Yang al. 2023b).
Scaling Study:
is composed of Trans-
Paradigm Design Study: In Table 5 we ablate the design of Trinsfrranformer.. We conclude that: ① Based on Table 5al all three types of attention are helpful. @ It makes sense that Sensor Cross Attention is especially important since model would drive blindly without sensor information. Temporal information has the least influence, which aligns with the findings in (Chitta et al., 2022). O Task Self-AAtention could improve the driving score since the ego query could utilize the detected objects and map elements to conduct planning. ① Based on Table (5b] we could find that discarding auxiliary tasks leads to performance decay, which may come from the fact that it is rather difficult to fit the single planning output with high-dimensional inputs (surrounding camera images). Actually, it is common practice in the end-to-end autonomous driving community to adopt auxiliary tasks to regularize the learned representations (Chen & Krähenbüh] 2022 Prakash et al. 2021 Jia et al., 2023d). ④ One stage training is enough for convergence and the perception pretraining, L.e, two stage training, does not provide advantages. It comes from the design that there is manual dependency among tasks and thus all tasks could first learn from the raw sensor

---

Published as a conference paper at ICLR 2025
Camera Crash (front left and back right)
Motion Blur
Gassian Noise
Incorrect Calibration
Figure& 8: Visualization of Detection and Planning under Different Robust Challenges.
Table 7: Robustness on Planning (Closed-Loop Evaluation). Driving Score is reported.
Methods
Regular
Camera Crash
Incorrect Calibration
Motion Blur
Gaussian Noise
VAD-Base (Jung al. 2023
53.45
48.54 (9.2%)
38.46 ()).)
45.47 (14.93%))
44.53 (16. 72% 56.94 (6.02%.)
DriveTransfomer (Ours)
60.45
58.67 (2.9%
56.53 (5.94%)
54.04 (10.60%)
Table 8: Robustness on Perception (Open-Loop Evaluation). Object detection NDS1 is reported.
Methods
Regular
Camera Crash
Incurrect Calibration
Motion Blur
Gaussian Noise
VAD VAD-Base (Jiang et al. 2023
53,21
19..3126.1298..
43.01 (21.05%:.)
42.34 (22.315))
(17.6( 63.20(9.26%.)
DriveTransformer (Ours)
69.65
61.84 (11.2%)
66.06(5.15%L)
(1.64())))4%L)
inputs and history information instead of influencing each others' convergence. As complex Transformer based framework, we find that the removal of supervision for middle layers collapse the training. It might need to further explore how to scale up the structure with only final supervisions.
Task Design Study: In Table we ablate the designs of task heads and conclude that: For Table 6a Formulating the prediction output in the local coordinate system decouples the object detection and motion prediction. As a result, the two tasks could optimize their objectives sepa- rately while the shared input agent features naturally associate the same agent. The superior per- formance demonstrates the effectiveness to avoid directly predict in the global coordinate system, aligns with ŞJia et al. 2023c Shi et al., 2024). @ For Table Point level PE: in Sensor Cross Atten- tion leads to significantly better online mapping results, which shows the effectiveness of extending the potential perception range for lane detection (Liu al., 2024; Li al. 2024b). ① For Table 6c multi-mode planning outperforms single mode planning explicitly. By visualization, We observe that
planning achieves better control especially on scenarios requiring subtle steers. comes from the fact that single mode prediction with L2 loss models the output space as one single Gaussian distribution and thus suffers from mode averaging.
4.4 ROBUSTNESS ANALYSIS
Autonomous driving, as an outdoor task, would frequently encounter many kinds of events and failures and thus it is an important perspective to examine the robustness of the system. To this end, We adopt settings in (Xie et al. 2023). Camera Crash. Two cameras are masked as all black. @ Incorrect Calibration. rotation and transition noises are added to camera extrinsic parameters. Motion Blur is applied on images. 4 Gaussian Noise is applied on images. From Table 7 and Table 8 Transffrnsr demonstrates significantly better robustness compared to VAD. It might be because VAD requires the construction of BEV feature, which is sensitive to perception inputs. On the other hand, DriveTrannformer directly interacts with sensor features and thus be able to ignore those failure or noisy inputs and demonstrates better robustness.
5 CONCLUSION
In this work, we present DriveTransformer, unified Transformer based paradigm for end-to-end autonomous driving, featured by task parallel, streaming processing, and sparse representation. achieves state-of-the-art performance on both Bench2Drive in CARLA closed-loop evaluation and nuScenes open-loop evaluation with high FPS, demonstrating the efficiency of those designs.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL attpps:/aae /tt::///axiiv... org/ abs/ 2/1./1100....3
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving, 2020a. URL
11027
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020b.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In CVPR, 2022.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühh. Learning by cheating. In CoRI.,
pp. 66-75. PMLR, 2020.
Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Con- ghui He, Jianping Shi, Yu Qiao, and Junchi Yan. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In European Conference on Computer Vision (ECCV), 2022.
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: imitation with transformer-based sensor fusion for autonomous driving. TPAMI, 2022.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. In ICRA, pp. 4700, 2018.
Felipe Codevilla, Eder Santana, Antonio López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In CVPR, pp. 9329-93323 92388, 2019.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, pp. 105171, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. NeurlPS, 2022a.
Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with self supervieee freespace forecasting. In CVPR, 2021.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022b.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, pp. 17853- 17862, 2023.
Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer- based interactive prediction and planning for autonomous driving. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 3903-3913, 2023.
11

---

Published as a conference paper at ICLR 2025
Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In ICVV,V 2023.
Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Ide-net: Interactive driving event and pattern extraction from human data. IEEE RA-L, 6(2):3065 2021. doi: 10.1 109/RA.. 2021.3062309.
Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. CoRL. pp. 1434-1443. PMLR, 2022.
Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: coarse-to-fine approach. In CoRL, pp. 910-920. PMLR, 2023a.
Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023b.
Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heteroge- neous driving graph transformer for multi-agent trajectory prediction via scene encoding, TPAMI, 2023c.
Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui Hc. Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023d.
Xiaosong Jia. Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In NeurIPS 2024 Datasets and Benchmarks Track, 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. ICCV, 2023.
Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xi- angyu Zhang. Far3d: Expanding the horizon for surround-view 3d object detection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2561-2569, 2024.
Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differen- tiable raycasting for self-supervised occupancy forecasting. In ECCV, 2022.
Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Enze Xie, Zhiqi Li, Hanming Deng. Haonan Tian, Xizhou Zhu, Li Chen, Tianyu Li, Yulu Gao. Xiangwei Geng, Jianqiang Zeng, Yang Li, Jiazhi Yang. Xiaosong Jia, Bo Yu, Y. Qiao, Dahua Lin, Siqian Liu, Junchi Yan, Jianping Shi. and Ping Luo. Delving into the devils of bird' s-eee-view perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:2151-2170, 2022a.
Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, et al. Delving into the devils of bird's-eye sard'--ey--iew perception: A review, evaluation and recipe. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2023.
Qifeng Li. Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2). arXiv preprint 167.0000 2024a.
Tianyu Li, Peijin Jia, Bangjun Wang. Chen, Kun Jiang. Junchi Yan, and Hongyang Li. Laneseg- net: Map learning with lane segment perception for autonomous driving. In ICLR, 2024h.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's s-eye-view representation from multi- camei- images via spa- tiotemporal transformers. ECCV, 2022b.
12

---

Published as a conference paper at ICLR 2025
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and José M. Álvarez. Is ego status all you need for open-loop end-to-end autonomous driving? IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14864-14873 2024c. URL https:
semant
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. ECCV, 2020.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construc- tion. In International Conference on Learning Representations, 2023.
Shilong Liu, Feng Li, Hao Zhang. Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Confer- ence on Learning Representations, 2022a. URL httph:ttt://opp /ttp:ss//ppeereeiiiet www.ne/////// forum?id GMI9P
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for muew 3d object detection. arXiv preprint arXiv:2203.05625, 2022b.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao. Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.01256, 2022c.
Zihao Liu, Xiaoyu Zhang, Guangwei Liu, Ji Zhao, and Ningyi Xu. Leveraging enhanced queries of point sets for vectorized map construction. arXiv preprint arXiv:2402. 2024.
Han Lu, Xiaosong Jia, Yichen Xie, Wenlong Liao, Xiaokang Yang, and Junchi Yan. Activead: Planning-oriented active learning for end-to-end autonomous driving, 2024.
Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Jonah Philion, Amlan Kar, and Sanja Fidler. Learning to evaluate perception models using planner- centric metrics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14055-14064, 2020.
Dean A Pomerlcau. Alvinn: An autonomous land vehicle in neural network. NeurIPS, 1, 1988.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to- end autonomous driving. In CVPR, 2021.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata, and An- dreas Geiger. Plant: explainable planning transformers via leject-level representations. In CoRL., 2022.
Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. CoRL, 2022.
Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li. and Yu Liu. Reason- net: End-to-end driving with temporal and global reasoning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13723-13733, 2023.
13

---

Published as a conference paper at ICLR 2025
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr+: Multi-agent motion prediction with symmetric scene modeling and guided intention querying, 2024. URL https::: arxiv. org/abs/ hat.aab///3....7.7
Haisheng Su, Wei Wu, and Junchi Yan. Difsd: Ego-centric fully sparse paradigm with uncer-
tainty denoising and iterative refinement for efficient end-to-ene autonomous driving, 2024. URL
httpss://ara orgo als// 0..4900.977
Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation, 2024. URL https://axxxx
Brg/abs/24055 19620
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. https:
1lapi.semante www.emnntiicchoooooooooooooooooe cachclacccccooooccccc wwtp.oo....o.oroo.aa:. 13756489
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: sequential fusion for 3d object detection. In CVPR, pp. 4604 4612, 2020.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3621-3631, 2023.
Xinshuo Weng, Boris Ivanovic, Kris Kitani, and Marco Pavone. Whose track is it anyway? improv- ing robustness to tracking errors with affinity- bbaed trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6573-6582, 2022.
Xinshuo Weng, Boris Ivanovic, Yan Wang. Yue Wang. and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guidee control prediction for end-to-end autonomous driving: A simple yet strong baseline. In NeurlPS, 2022.
Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robobev: Towards robust bird's eye view perception under corruptions. arXiv preprint arXiv:2304.06719, 2023.
Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang. Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird s-eee-veew recognition via perspective supervision. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 17830-17839, 2023a.
Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey of large language models for autonomous driving. ArXiv, abs/23 1..1144, 2023b.
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End- to-end multiple-object tracking with transformer. In European Conference on Computer Vision, pp. 659-675. Springer, 2022.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. CVPR, 2019.
Jiang-Tian Zhai, Ze Feng, Jihao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 100.00000 2023.
Diankun Zhang, Guoan Wang. Runwen Zhu, Jianbo Zhao, Xiwu Chen, Siyu Zhang, Jiahao Gong, Qibin Zhou, Wenyuan Zhang, Ningzi Wang, Feiyang Tan, Hangning Zhou, Ziyao Xu, Hao- tian Yao, Chi Zhang. Xiaojun Liu, Xiaoguang Di, and Bin Li. Sparsead: Sparse query-centric paradigm for efficient end-to-end autonomous driving, 2024. URL https: //tp://v/xx..////////// labs/ bbs//404.0 2404.444.. 06892
14

---

Published as a conference paper at ICLR 2025
Qingwen Zhang, Mingkai Tang, Ruoyu Geng, Feiyi Chen, Ren Xin, and Lujia Wang. Mmff:: multi- modal-fusion-nee for end-to-end driving. IROS, 2022.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021.
A IMPLEMENTATION DETAILS
We implement the model with Pytorch. When comparing with state-of-the-ari works, we re- When conducting ablation studies, we report results
port results of TriveTransforme--Lane of DriverTransformrr- Base on Dev10 benchmark if not specified. All models are trained in Bench2Drive (Jia et al.) (024) base set (1000 clips) for 30 epochs on 8 A800 with learning rate le-4, weight decay 0.05, dropout 0.1, AdamW, and cosine annealing schedule. We use ResNet50 as image backbones and the image size of (384, 1056) The temporal length (Iquee) are set as 10 (1 second 10Hz) in Bench2Drive and (2 second 2Hz) in nuScenes. At each time-step, the Top-K queries is pushed into queue, where set K 50. The initial number of queries is set 900 for agent queries following [Wang et al. 2023) and 100 for map queries following (Jiang et al. 2023). We will open source our code and checkpoints.
B DEV10 BENCHMARK
Bench2Drive TTa et al. 2024) is closed-loop evaluation protocol with 220 routes. The abundant number of routes could lower the variance of evaluation while introducing significant computational challenges. For example, could take 2-3 days to evaluate Drive Triveeransformer--.. Large on 8* A8000 To this end, for fast development of ideas, we propose Dev10 Benchmark by the following procedure:
There are kinds of scenarios Bene2Drive where each type has 5 different routes under different locations and weathers. Due to the low variance of Bench2Drive's short routes, selecting one route per scenario could reflect the model's ability in that case.
2. Further, there are some very similar scenarios in these 44 types. For example, as shown in page 15 of Bench2Drive's paper scnenario 2 "ParkingCutln" scenario 3 "ParkingCutin" and scenario 4 "StaticCutIn" all examine the ability of the ego vehicle to slow down or brake for the cut-in vehicle. Thus, after discussing with the Bench2Drive official team, these scenarios could be summarized into 10 high-level types:
03877). 77). the difference between
ParkingExit: requiring the model to drive out of parking lot.
DynamicOhjertrrssingg ControlLoss, PedestraiiCrossingg Vehicle Pureeenau VehicleTurnin Tuniiggueun,,,,ttee,, HardBrake, OppositeVehiilleRumm VehicleRun- ningRedl.ight, Opposite Vehicle TakingPrioritt: requiring the model to conduct emergency brake or slow down drasticly under dangerous situations.
StaticCutIn, HighwayXxit, Invading Turn, ParkingCutIn, HighwayCutln: requiring the model to handle cut-in behaviors of the front vehicle*
ParkedObstacle, Construction, Accident: requiring the model to overtake the blocking obstacles front of it.
EierenTeEeerencc Vehii:: requiring the model give way to emergency vehicles.
ConstractionOOsstteeellwwlw Ways., clesDooropen TwoWays, wwneeAAttideLeeeToooooe requiring the model to drive in the reverse lane for a short distance, complete the overtaking, and then return to the original lane.
AccidentTwwo Vehi-
NunSignaltret/Jnnniinnleffffffr SignalizedJunctioneeffffi InterurbanAActoFFlw,, InteurbannAttvaacddActfffff CrossingBicycleFlov, VinillaVonSignalizeTTuun Vinil- Sonnaizzeduunnnnouuterrreree TJunction: requiring the model to handle the traffic at intersections and complete its turn.
Btersedntt requiring the model to yield for the blocking event within the inter- section until the event is finished.
Sequentiall
Sequentiill.aneChange aneChange: requiring the model continuously change several lanes.
15

---

Published as a conference paper at ICLR 2025
A htw. EnterActorFlows, SignalizedJunctiooRRight Turn, VonSignalized/unctoRRiggg
MergerlntoSlow Traffic, MergerintoSlowTreg Vi: requiring the model enter the dense traffic flow and merge into it.
3. For the 10 high-level types, select one route for each with diverse weathers and towns. We
give the details of Dev10 below:
Table 9: Routes of Dev10 protocol.
Scenario
Route-II
Road-II
Town
ParkingExit
3514
892
13
3255
1237
13
StaticCutIn
26405
137
15
HazardAAtiddLLane
25381
37
05
Emergenyy Vehicle
25378
03
ConstructionObstaale Two Ways
25424
269
11
NonSignaalizeJJunttton.cfffi BlockedInterscction
2091
12
27494
16
04
SequentiallLaneChaagg
16569
1157
12
28198
234
15
We verify the variance of Dev 10 with Driverrannformer--Ba with different seeds. The driving scores are 60.45, 59.20, 58.99 and the success rates 0.3, 0.3, 0.3, demonstrating very low variance.
When comparing with other methods, we stick 220 routes to ensure fairesss. When conducting ablation studies, we use Dev 10 to save computational resource.
When comparing with other methods, we stick 220 routes while conducting ablation studies, we use Dev 10. Additionally, Dev could also serve as a validation set to avoid researchers overfitting Bench2Drive220. It could also avoid the overfit Bench2Drive220 that it could be a test set. We will open source the Dev10 10 benchmark and the Bench2Drive official team plans to integrate it into their official repo to provide a short and economic validation set.
We will open source Dev 10 protocol.
C LIMITATIONS
Similar to existing end-to-end autonomous driving systems, DriveTraanformer entangles the update of all sub-tasks and thus brings challenges for the maintenance of the whole system. An important future direction would make them less coupled and thus easier debug and main separately.
Table 10:
Comparison of Performance on Middle Tasks. i ParaDrive's latency is calculated by their claim that 2.77x speed up compared UniAD when disabling all middle tasks.
Detection
Motion
Online Mapping
Method
Latency
NDS
mAP
minADE
minFDE
IoU-Road
loU-Lane
UniAD Hu et al. 2023
49.8
38.0
0.72
1.05
0.15
0.30
0.67
663.4ms
ParaDrive Weng al. 2024
37.0
0.72
239,5msi
Drive Transtormer
49.9
0.61
0.95
0.13
0.39
0.77
211.7ms
D COMPARISON OF MIDDLE TASKS
We compare the performance of middle tasks in nuScenes validation set as in Table 10
E COMPARISON WITH CONCURRENT PARALLEL AND SPARSE BASED METHODS
Concurrent to DriveTransformer, there are other sparse- sparse-based based methods including SparseAD Zhang et al. (2024) and SpivseDrive Sun et al. (2024). To provide comprehensive overview of existing sparse based methods, we compare DriveTranste Transformer with them together with ParaDrive Weng et al. 2024 most recent efficient modular E2F-AD method:
16

---

Published as a conference paper at ICLR 2025
1.2 (m)
Collision ( (%)
Method
Lateney
1s
2s
38
Avg.
1s
2s
3s
Avg.
ParaDrive
0.25
0.46
0.74
0.48
0.14
0.23
0.39
0.25
239.5mm
SparseAD-B
0.15
0.31
0.57
0.35
0.00
0.06
0.21
0.09
285.7m
SparseAD-L.
0.15
0.31
0.56
0.34
0.00
0.04
0.15
0.06
1428.6m 6ms
SparseDrive-S
0.29
0.58
0.96
0.61
0.01
0.05
0.18
0.08
111.1ms
SparseDrive-B
0.29
0.55
0.91
0.58
0.01
0.02
0.13
0.06
137.0ms
DriveTrrnnsormer-S
0.19
0.33
0.66
0.39
0.01
0.07
0.21
0.10
93.8ms
Branffrrnnoomeer-B
0.16
0.31
0.56
0.34
0.01
0.06
0.16
0.08
139.6mo
ThhveTaanforme--LL
0.16
0.30
0.55
0.33
0.01
0.06
0.15
0.07
221.7ms
We could observe that DriveTTrassfomer achieves good L2 with high efficiency.
F TRAINING STABILITY& & MULTI-STAGE TRAINING
In pioneering work UniAD Hu et al. (2023), they adopt a three step training strategy: (1) Training BEVFormer Li al. (2022b) with 3D object detection task; (2) Training TrackFormer and Map- Former; (3) Training all modules together. They explain in their paper that "We first jointly train perception parts, i.e., the tracking and mapping modules, for few epochs (6 in our experiments), and then train the model end-to-end for 20 epochs with all perception, prediction and planning modules. The two-stage training is found more stable empirically." and " "Joint learning. UniAD trained in two stages which we find more stable.".
We conduct experiments train UniAD with one single stage (loading pretrained BEVForeer) in nuScenes shows that the overall loss (around 54) stops decreasing in early epoch (around epoch 4) while UniAD trained with the official two stage training has the final loss of (around 34), as shwon in Table I
Table 11: Comparison of One-Stage and Two-Stage Trainging for UniAD.
Method
Loss
(Delection)
loU-lane (Mapping)
AMOTA(Tracking)
minADE(MMttion)
Avg. L.2 (Planning)
One-Stage
$4.07
38.1
23.5
20.4
2.52
2.01
Twn-Stage
1.03
We could observe that ste-stages result in underfit of all modules with much higher final loss.
Further, concurrent work Sparse Zhang al. (2024). they mention that The training is divided into three stages in total...stage I and stage 2 can be merged during training for a shorter training time with slight performance degradation in exchange.. In SparseDrive [Snn et al. (2024), they mention that The training of SparseDrive is divided into two stages. In stage-1, we train symmetric spurse perception module from scratch to learn the sparse scene representation. In stage-2, sparse perception module and parallel motion planner trained together.
In contrast, compared to the sequential designs (UniAD, SparseAD, SparseDrive), one significant improvement in DriveTransformer is that all tasks' interaction is learnt via attention instead of manual ordering. As a result, at the early stage of training, each task could access information directly thorough sensor cross attention and temporal self attention, reducing the reliance other under-trained tasks' queries. Such designs are friendly to the scaling up and industrial application of E2E-AD methods. As shown in Table 5 Preiruin Perception does not provide gains for Drive- Transformer, which proves the training stability of DriveTransformer.
17


Reviews:
[{"review_id": "hrrypJDdZH", "reviewer": "Reviewer_VVtW", "summary": "This paper proposes a new architecture for end to end autonomous driving where all tasks are connected with others. Task queries and raw sensor features are connected. Also, the task queries are passed as history information for temporal cross attention. Experiments show competitive results with other SOTA architectures in the field.", "questions": "* Task parallelism is not the right way to describe this architecture, what is parallel here? \n* The benefit of this architecture needs to be motivated better, open loop results are not considerably better. Closed loop metrics are also mixed. The latency is also comparable. So, what is the benefit of using this architecture?\n* What is the criterion for Dev10? 10 scenes out of 220 seem very selective, the criterion needs to be justified, otherwise the results could be prone to selection bias.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a new architecture where all modules interact with each other. This should be able to capture complicated spatio-temporal dependencies among modules. They have also suggested using sensor cross attention and temporal cross attention, this can, at least in theory improve the generalization performance with sensor noise and failures or modules.", "weaknesses": "* The motivation for this architecture is not clear. When all modules interact with each other, we ignore the benefits of using modular end to end architectures. This would make the system difficult to debug and maintain. This can only be justified if the performance is significantly better than the modular architectures, however this does not seem to be the case.\n* The results for detection, prediction, mapping is not shown. End to end driving is primarily about planning, however, it makes it more convincing if results of related tasks are also shown to be better.\n* It would have useful to see the performance of ParaDrive in closed loop setting, as this architecture is closest in concept with ParaDrive. Only open loop comparison is shown.\n* This manuscript could use a grammar / spell check, there are quite a few typos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "31YtfIZrFg", "reviewer": "Reviewer_6PT4", "summary": "This paper presents DriveTransformer, a unified Transformer framework for end-to-end autonomous driving. It features three key properties: task parallelism, sparse representation, and streaming processing, and contains three unified operations: task self-attention, sensor cross-attention, and temporal cross-attention, reducing system complexity and improving training stability. The framework achieves good performance in both the simulated closed-loop Bench2Drive and the open-loop nuScenes benchmarks.", "questions": "1. In Figure 1, could it be explained why the training stability of b) is poor? And compared to c), why is the task relation of d) okay?\n2. The streaming process maintains a FIFO process, retaining and reusing historical information. The BEV temporal fusion also maintains and reuses a fixed-length bev features. If it is a recurrent model, only the previous and next two frames are needed. So the BEV temporal processing of traditional methods can also be regarded as a streaming process. What is special about the streaming process in this paper?\n3. In this paper, when comparing performance, training efficiency and inference latency, the comparisons are all made with UniAD and VAD. Is it possible to make a comparison at the same sparse method level with Sparsedrive and SparseAD, also at the same task parallelism method such as ParaDrive?\n4. From Table 6c and Table 8, DriveTransformer shows significantly better robustness compared to VAD. Should Table 6c be Table 7?\n5. When scaling up in this paper, the largest model used is 0.6b. Would the effect be better if the model were larger?\n6. It is mentioned many times in the paper that the sequential model design can cause cumulative errors and training instability. Is it possible to make some comparisons between parallel and sequential methods, especially in terms of cumulative errors and training stability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "● Originality: Proposes a new unified Transformer framework, combining sparse representation with parallelized task processing, distinct from existing methods.\n● Quality: The design of each module is reasonable and effectively addresses existing problems.\n● Clarity: The paper has a clear structure and it is easy to understand the core ideas.\n● Significance: Provides new directions and ideas for autonomous driving research.", "weaknesses": "● Originality: Lacking enough original design. Because the task parallel concept is not entirely new (ParaDrive), and the sparse representation (core part following PETR & PETRv2) and streaming process lack strong originality. \n● Comparisons: Lacking comparisons with similar methods in sparse and parallel aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgfuwn9y3a", "reviewer": "Reviewer_Fh94", "summary": "This paper proposes a new architecture for end-to-end driving, with three key features: task parallelism, sparse representation, streaming processing. With these key features, the proposed method reduces the complexity of the system and leads to better training stability, achieving SOTA performance on both open-loop and close-loop benchmarks.", "questions": "1. Curious about the scaling ability, the performance seems not saturate, have you tried on larger model or larger data (Bench2Drive full dataset)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is a simple and effective idea using a vanilla transformer to unify sensor tokens and task queries.  \n2. Benefiting from the simple design, the model can scale up to achieve better performance.  \n3. The performance on close-loop benchmark is remarkable, surpassing previous methods by a large margin.  \n4. The writing and figures are easy to follow and understand.", "weaknesses": "1. The open-loop experiment should exclude ego-status information to prevent short-cut learning on nuScenes.  \n2. There seems no evidence to support that BEV Sequential Paradigm has bad training stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)