Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving
Authors: Peidong Li
Dixiao Cui
Venue: ICLR 2025 Poster (2025)
Keywords: End-to-End
Autonomous Driving
Sparse Scene Representation
Primary area: applications to robotics, autonomy, planning

Abstract: End-to-End Autonomous Driving (E2EAD) methods typically rely on supervised perception tasks to extract explicit scene information (e.g., objects, maps). This reliance necessitates expensive annotations and constrains deployment and data scalability in real-time applications. In this paper, we introduce SSR, a novel framework that utilizes only 16 navigation-guided tokens as Sparse Scene Representation, efficiently extracting crucial scene information for E2EAD. Our method eliminates the need for human-designed supervised sub-tasks, allowing computational resources to concentrate on essential elements directly related to navigation intent. We further introduce a temporal enhancement module, aligning predicted future scenes with actual future scenes through self-supervision. SSR achieves a 27.2\% relative reduction in L2 error and a 51.6\% decrease in collision rate to UniAD in nuScenes, with a 10.9× faster inference speed and 13× faster training time. Moreover, SSR outperforms VAD-Base with a 48.6-point improvement on driving score in CARLA's Town05 Long benchmark. This framework represents a significant leap in real-time autonomous driving systems and paves the way for future scalable deployment. Code is available at https://github.com/PeidongLi/SSR.

Published as a conference paper at ICLR 2025
NAVIGATION-GUIDER SPARSE SCENE REPRESENTA- TION FOR END-TO-END AUTONOMOUS DRIVING
Peidong Li, Dixiao Cui
Zhijia Technology, Suzhou, China
(1 ppiddnn, cuidi quidixiaa)ee rtcttttt
ABSTRACT
End-to-End Autonomous Driving (E2EAD) methods typically rely on super- vised perception tasks to extract explicit scene information (e.g., objects, maps). This reliance necessitates expensive annotations and constrains deployment and data scalability in real-time applications. In this paper, introduce SSR, novel framework that utilizes only 16 navigation- -agiidd tokens as Sparse Scene Representation, efficiently extracting crucial scene information for E2EAD. Our method eliminates the need for human-designed supervised sub-tasks, allowing computational resources to concentrate on essential elements directly related navigation intent. We further introduce temporal enhancement module, align- ing predicted future scenes with actual future scenes through self-supervision. SSR achieves a 27.2% relative reduction in L2 error and 51.6% decrease in collision rate to UniAD in nuScenes, with 10.9x faster inference speed and 13x faster training time. Moreover, SSR outperforms VAD-Base with a 48.6- point improvement on driving score in CARLA's Town05 Long benchmark. This framework represents significant laap in real-time autonomous driving sys- tems and paves the way for future scalable deployment. Code available at
INTRODUCTION
000
Training Time vs inference tatency
14
Collis inn Rate vs Freer
VAB,Base
Erue Sml
2:3-
UgAD
Infrencee
VAD,Base
200
G2g40 UAD
0.9
MAD
GeyAD
100
VAD-Tiny
0.8-
20
10
80
80
100
120
14f
160
0.
03
0.5 Collivion E84
0.0
0.6
0.7
0.8
Training Time Shours!
Figure 1: Performance Comparison of Various Methods in Speed and Accuracy on nuScenes.
Eision-based E2EAD (Hu et 2023 Jiang al. 2023 Sima et al. 2023 Zheng et al., 2024b Sun et al. 2024 Weng et al. 2024 Li et al. 2024b Guo et al. 2024) has gained significant atten- tion recent research as a eoft-effective alternative for autonomous driving systems. Traditional architectures typically consist of separate perception and planning modules. While most perception modules are handled by neural networks (NN), planning modules often rely on rule-based pipelines. This separation lead to information loss during the transfer between modules, resulting in sub- optimal performance. E2EAD addresses this by using entire neural networks to predict planning trajectories from images, thereby minimizing information loss and improving overall performance.
However, most current E2EAD approaches build upon complex perception frameworks, often incor- porating additional NN-based planning modules. These approaches typically inherit tasks such as

---

Published as a conference paper at ICLR 2025
object detection (Li et al. 2022b; Philion & Fidler 2020 Li et al. 2024a), mapping Li et al. 2022a Liao et al., 2022). and occupancy prediction (Sima et al. 2023 Huang et al. 2023), resulting in large and computationally intensive neural networks. Despite their integration, these models still maintain modular framework design, requiring independent sub-tasks' supervision. As a result, they remain suffer from scalability issues, and are inefficient for real-time deployment.
While many E2EAD methods continue to mimic the paradigm established by prior BEV perception works, they often overlook a critical question: Do E2EAD systems still require such extensive per- ception tasks? In traditional AD systems, the perception module has to extract all elements for the planning module, as there is no back mack-propagation from planning to perception. Existing E2EAD methods largely ignore the advanced a cascade structure rooted in traditional AD. In contrast, we seek for a more targeted approach that directly identifies driving-relevan elements. This raises key question: How can we efficiently identify and focus on the crucial parts of the scene without auxiliary perception supervision?
caining--orienten insight of E2E paradigms, instead retaining
To address this, we introduce SSR, novel framework that leverages navigation-guider Sparse Scene Representation, learning from temporal context in self-supervision rather than explicit perception supervision. Inspired by how human drivers selectively focus on scene elements based on navigation cues, we find that only a minimal set of tokens from dense BEV features is necessary for effective scene representation in autonomous driving. Since E2EAD methods do not rely on high-definition maps as input, a high-level command (e.g.. "turn left", "turn right", "go straight" following common practices in Hu et al.] (2023) ; Diang al. (2023) is required for navigation. Our method, therefore, extracts scene queries guided by the navigation commands, akin to human attention mechanisms,
As illustrated in Fig. aa), existing methods typically extract all perception elements following previous BEV perception paradigms. These methods rely on Transformer (Vaswani et al. 2017)
to identify relevant ones in the additional planning stage. In contrast, as shown in Fig. 2[b),, SSR directly extracts only the essential perception elements in the guidance of navigation commands, thereby minimizing redundancy. Our approach takes full advantage of the end-to-end framework, breaking the modular cascade architecture in a Navigation-Guidee
Perception manner. While prior works (Sun et al., 2024, Zhang et al., (024) attempt to reduce computation by skipping BEV feature construction, they still depend on hundreds of task-specific queries. However, our method drastically reduces computational overhead by using just 16 tokens guided by navigation commands.
Additionally, SSR capitalizes on temporal context to circumvent the need for perception tasks su- pervision. We hypothesize that if a predicted action aligns with the actual action, the resulting scene should match the real future scene. Specifically, we predict future BEV features, which are then self- supervised by the actual future BEV features. This future feature predictor, which takes current BEV features and the planning trajectory as input to predict future BEV features, offers an alternative for supervising both scene representation and planning trajectories without auxiliary annotations.
By leveraging navigation- -giided perception paradigm and temporal self-supervision, SSR provides an effective and efficient solution for real-time autonomous driving. As illustrated in Fig. [ SSR delivers state-of-the-art performance on the nuScenes (Caesar et al. 2020) dataset, with minimal computational overhead. Specifically, our method decreases average L2 eтror by 0.28 meters (a 27.2% relative improvement) and reduces the average collision rate by 51.6% relatively compared to UniAD (Hu et al.) 2023), even without any annotations. Meanwhile, SSR also achieves supe- rior performance on CARLA'S (Dosovitskiy et al., (017)) Town05 Long benchmark. Remarkably, our method reduces training time to 1/13th of that required by UniAD and is 10.9x faster during inference. Therefore, SSR has the potential to manage large-scale data in real-time applications.
Our contributions are summarized as follow:
We introduce a human-inspired E2EAD framework that utilizes learned sparse query rep- resentations guided by navigation commands, significantly reduces computational costs by adaptively focusing essential parts of scenes.
We highlight the critical role of temporal context in autonomous driving by introducing a future feature predictor for self-supervision on dynamic scene changes, eliminating the need for costly perception tasks supervision.
Our framework achieves state-of-the-an art performance on both open-loop and closed-loop experiments, establishing new benchmark for real-time E2FAD.
2

---

Published as a conference paper at ICLR 2025
Persypice
Rouigatinn
Nonqutoo Corteal
Dutcctes
Magpig
Planrig
Wath mg inege
BGV sukey
WalYev Image fedkrn
BCV Redker
Sond Quili
Dusppay
Speas Pandigg
25A Paseige
(a) Task- Sask-Specffic Supervised Paradigm
(b) Adaptive Unsupervised Paradigm
Figure 2: Comparison of Various End-to-End Paradigms. Compared to previous task-specific pervised paradigms, our adaptive unsupervised approach takes full advantage of end-to-end frame- work by utilizing cavigation-guidee perception, without the need to differentiate between sub-tasks.
2 RELATED WORK
2.1 VISION-BASED END-TO-END AUTONOMOUS DRIVING
Research on End-to-End autonomous driving dates back to 1988 with ALVINN (Pomerleau 1988), which used simple neural network to generate steering outputs. NVIDIA developed proto- type E2E system (Bojarski et al. 2016) based on convolutional neural networks (CNN), bypassing manual decomposition. The recent resurgence in vision-based E2EAD has been driven by rapid advancements in BEV perception (Li et al. 20226 Liao et al., 2022 Liu et al. 2022
Huang et al.,
2023) and modern architectures like Transformer Vaswann
2017).
ST-P3 (Hu et al. 2022) introduced improvements in perception, prediction, and planning modules for enhanced spatial-temporal feature learning. integrating auxiliary tasks such depth estimation and BEV segmentation. UniAD (Hu et al., (023) built on previous BEV perception works to cre- a cascade framework with a variety of auxiliary tasks, including detection, tracking. mapping, occupancy, and motion estimation. VAD (Jiang et al., (023) sought to streamline scene represen- tation by vectorizing it, reducing the tracking and occupancy tasks seen in UniAD. GenAD (Zheng al., (024b) explored the use of generative models for trajectory generation, jointly optimizing motion and planning heads based on VAD. PARA-Drive (Weng et al. 2024) further examined the relationship between auxiliary tasks, reorganizing them to in parallel while deactivating them during inference. In contrast, our approach eliminates all perception tasks, achieving remarkable performance in both accuracy and efficiency.
2.2 SCENE REPRESENTATION IN AUTONOMOUS DRIVING
Most prior works autonomous driving (Hu al. 2022 2023 Jiang et al.) 2023 Zheng et al, 2024b) have inherited approaches from perception tasks, such Li et al. (2022b), which leverages dense BEV features as the primary scene representation. In these frameworks. task specific queries used to extract information from the BEV features under man- ual labels' supervision. While these approaches benefit from rich scene information, they also intro- duce significant model complexity, hindering real-time application, particularly in occupancy-based representations (Sima et al. 2023 Zheng et al. 2024a).
(e.g., for detection and mapping)
Following the trend of sparse paradigms in BEV detection (Lin et al. 2022 Liu al. 2023),
cent sparse E2EAD approaches (Sun et al. 2024 Zhang et al.) (024) directly utilize task-specific queries interact with image features. These methods attempt bypass BEV feature generation al- together by directly interacting with image features through task-specific queries. However, despite the reduction in BEV processing, these models still rely on hundreds of queries, which diminish the promised simplicity and efficiency of the end-to-end paradigm. LAW (Li et al., (024b) proposed the use of view latent queries to represent each camera image with a single query. However, this approach compromises information fidelity, leading to suboptimal performance. UAD (Guo et al.) 2024) attempted to divide the BEV feature into angular wise sectors but still relied on open-set de- tector labels for supervision, maintaining the complexity of task-specife queries. In this work, introduce SSR, a novel approach that represents the scene by a minimal set of adaptively learned queries, enhancing both efficiency and performance.
3

---

Published as a conference paper at ICLR 2025
Varigazon
Nodal Varpoinc Quelies
BEV
Toloen
Scenes
Planring Decoder
Trceastory
Scone Querts
Frmme
Poedicto
BEV EIvoOder
Superisaoo
BEV Festire
PrRDICIO Fiture Feature
Figure 3: Overview of SSR: SSR consists of two parts: the purple part, which used during both training and inference, and the gray part, which is only used during training. In the purple part, the dense BEV feature is first compressed by the Scenes TokenLearnee into sparse queries, which are then used for planning via cross-attention. In the gray part, the predicted BEV feature is obtained from the Future Feature Predictor. The future BEV feature is then used to supervise the predicted BEV feature, enhancing both the scene representation and the planning decoder.
3 METHOD
3.1 OVERVIEW
Problem Formulation: At timestampt, given surrounding N-views camera images I, I and high-level navigation command cmd, the vision-based E2EAD model aims to predict the planning trajectory T, which consists of set of points in BEV space.
BEV Feature Construction: As shown in Fig. N-views camera images are processed by BEV encoder to generate the BEV feature. In the BEV encoder such as BEVFormer (Li et al. 2022b), I, first processed by an image backbone obtain image features F, = [Fi] A BEV then used query temporal information from the previous frame's BEV
query Q "xxCC feature B and spatial information from F; by cross-attention iteratively, resulting in the current BEV feature B, E " l Here, H W represents the spatial dimensions of the BEV feature, and C denotes the feature's channel dimension.
Q Cross Allention Q, B B.-1B,-- B,-1), B,
Q F.).
The key component of our framework is the novel Scenes TokenLearnee module to extract crucial scene information, introduced in Sec. 3.2 Unlike traditional methods that rely on dense BEV features or hundreds of queries, our approach uses small set tokens to effectively represent scene. Leveraging these sparse scene tokens, we generate the planning trajectory, process detailed Sec. 3.3 Additionally, Sec. 3.4 we introduce an augmented future feature predictor designed to further enhance the scene representation through self sele-super learning of BEV features.
3.2 MAVIGATION-GUIDEE SCENES TOKEN LEARNER
BEV features popular scene representation they contain rich perception information. How- ever, this dense representation increases the inference time when searching for relevant perception elements. To address this, introduce sparse representation using adaptive spatial atten- tion, significantly reducing computational load while maintaining high-fidelity scene understanding.
Specifically, propose the Scenes Terner (STL) module extract scene queries [s.] N.C from the BEV feature, where N, is the number of scene queries, to efficiently
4

---

Published as a conference paper at ICLR 2025
Tokeri.asreer
Fature Feature Predictr
Tonpeday
Me a----a spe
Figure 4: Structure of Modules: Scenes TorenLearner and Future Feature Predcitor.
represent the scene. The structure of Scenes Tokenl is illustrated in Fig. To better focus on scene information related to navigation intent, adopt Squeeze-and- (SE) layer (Hu et al. 2018) to encode the navigation command cmd into the dense BEV feature, producing the navigation-aware BEV feature Brndos
gnoes (F,, emd).
(3)
The navigation-aware BEV feature is then passed into the BEV TokenLearner (Ryoo et al. 2021) module TLBEV to adaptively focus the most important information. Unlike previous applica- tions of TokenLearner in image or video domains, we utilize it BEV space to derive a sparse scene representation via spatial attention:
S. = TLBEV (B Bhout).
For each scene query 8;, we adopt tokenizer function M, that maps Bravi into token vector: H XWxxWxXC xC + RC, The tokenizer predicts spatial attention maps of shape x W x 1, and the learned scene tokens obtained through global average pooling:
s; (. p(Baavi (B.ari))
(5)
where ((.) is the spatial attention function and p(-) is the global average pooling function. The multi-layer self-attention Vaswani et al., 2017] applied to further enhance the scene queries:
S. Self (S.entionn).
(6)
3.3 PLANNING BASED ON SPARSE SCENE REPRESENTATION
Since S, contains all relevant perception information, we use set of way point queries W IR No. x Nr xC to extract multi-modal planning trajectories, where N; is the number of future times- tamps and Nm denotes the number of driving commands.
W = Cross Attentiin W,, St).
We then obtain the predicted trajectory from W, using multi-layer perceptron (MLP), and select output trajectory T E N.N.2 x2 based the navigation command cnd::
T = Select cmd))
(8)
The output trajectory is supervised by the ground truth (GT) trajectory Ter using LI loss, defined the imitation loss Limsi
Limi Ter Tll
3.4 TEMPORAL ENHANCEMENT BY FUTURE FEATURE PREDICTOR
We prioritize temporal context enhance representation by self-supervision instead of per- ception sub-tasks. The motivation behind this module is straight: if our predicted actions correspond to real actions, the predicted future scenes should closely resemble the actual future scenes.
As illustrated Fig. we introduce the Future Feature Predictor (FFP) to predict future BEV features. First, the output trajectory T translate the current scene queries into the future
5

---

Published as a conference paper at ICLR 2025
frame using Motion aware Layer Normalization (MLN) (Wang al. 2023) module. The MLN module helps current scene queries encode motion information, producing dreaming queries D;:
D. (S, T).
(10)
We then apply multi-layer self-attention on D, to predict the future scene queries S
S.+1 Sel Attention (D:).
(11)
However, since the autonomous driving system may focus on different regions even in consecutive frames, we do not directly supervise the predicted scene queries S with the future scene queries S.+1- Instead, we reconstruct the dense BEV feature B(+1 using TokenFuser (Ryoo al. 2021
B = Token Puser(S.+1, B,)
(12)
= (B,) @ S1+1.
(13)
where (-(-) is simple MI.P with the sigmoid function to remap the BEV feature B, to a weight tensor: #x + R.. HxN,, After the multiplication @ with S E RNN, we obtain the predicted dense BEV feature B,41 E "xxWx This process aims to recover the BEV feature from the predicted scene queries for further superviinn..
Unlike prior works (Zon et al. 2023 Zou et al. 2024) which utilize predicted BEV features for subsequent object- or pixel-level supervision, we supervise B directly using an L2 loss with the real future BEV feature B,41- This is defined the BEV reconstruction loss Lbee:
Lbe ||a++1
B 1/22-
(14)
In summarr, we apply imitation loss Lims for the predicted trajectory, and BEV reconstruction loss Lren for the predicted BEV feature. The total loss of SSR is:
Lrotal Lim + Lbca-
(15)
4 EXPERIMENTS
4.1 DATASET AND METRIC
Open-Loop We evaluate the proposed SSR framework for autonomous driving using the widely adopted nuScenes dataset (Caesar et al. 2020), following prior works (Hu al. 2023 Jiang et al. 2023]. To assess planning performance, we use displacement error and collision rate (CR), as in previous studies. Displacement error is calculated by L2 error with respect the GT trajectory, measuring the quality of predicted trajectory. Collision rate quantifies the percentage of collisions with other objects when following the predicted trajectory. All metrics calculated in 3s future horizon with a 0.5s interval and evaluated at 1s, 28 and 3s.
We observe that VAD (Jiang et al., 2023) and UniAD (Hu al. 2023) utilize different evaluation methods to calculate results across all predicted frames. VAD computes the average across all previ- ous frames, while UniAD uses the latest result as well as the maximum value. Additionally, UniAD excludes pedestrians from the GT occupancy map, resulting in lower collision rates. We denote the VAD approach with the subscript AVG and the UniAD approach with MAX. For example, the 1.2 error at frame (maximum 3x/ 3s/0.5s = 6) is calculated as T.2'AVG = E L for VAD and I2 L2 for UniAD. We apply MAX metric by default but also calculate AVG metric for comparison with other methods in Tab. [ In our MAX metric, pedestrians are included in the calculation of collision rate.
Closed-Loop We conduct closed-loop experiments using the CARLA simulator (Dosovitskiy et al.) 2017], leveraging the widely adopted Town05 Long benchmark to evaluate performance. The train- ing dataset consists of 189K frames collected by Roach (Zhang et al. (021) at 2 Hz across 4 CARLA towns (Town01, Town03, Town04, and Town06), following previous works (Jia et al. 2023a Wu
et al. 2022). The training data has no overlap with Town05 Long benchmark.
We utilize the official metric provided by CARLA. The Route Completion (RC) is the percentage of the route completed by the autonomous agent. The Infraction Score (IS) quantifies the number of infractions made along the route, with pedestrians, vehicles, road layouts, and traffic signals. The Driving Score (DS) serves the main metric, calculated the product of RC and IS.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of state-of-the-art methods on the nuScenes dataset. The ego status was not utilized in the planning module. o: Lidar- based methods. *: Backbone with 10s (He al., 2016), while others use Res.ett-50 or similar. t: FPS measured on an NVIDIA A100 GPU, while others were tested on an NVIDIA RTX 3090. $: AVG metric protocal as same VAD.
Method
Auxiliary Task
1.2 (m)
Collision Rate (R)
1s
2s
3s
Avg.
Is
2s
3s
Avg.
NMPo Zeng et ml
2019
Det & Motion
0.53
1.25
2.67
1,48
0.04
0.12
0.87
0.34
FF6 Hu al A021
FreeSpace FreeSpace
0.55
1.20
2.54
1.43
0.06
0.17
1.07
0.43
EDc
Khurana et 2022
0.67
1.36
2.78
1.60
0.04
0.09
0.88
0.33
ST-P3
et 2022
Det Map & Depth
1.72
3.26
4.86
3.28
0.44
1.08
3.01
1.51
1.6
UniADr TTu et cim 2023
0.48
0.96
1.65
1,03
0.05
0.17
0.71
0.31
1.81
OccNet+ Sima dral KUA5
Map & Oce
1.29
2.13
2.99
2.14
0.21
0.59
1.37
0.72
VAD-Bas (Jange
2024)
Det & Map & Motion
0.54
1.15
1.98
1.22
0.04
0.39
1.17
0.53
4.5
PARA-Drive ci Film 8U24
0.40
0.77
1.31
0.83
0.07
0,25
0.60
0,30
5.0
GenAD (Zheng
21124b
Map & Motion
0.36
0.91
0.23
1.00
UAD-Tiny (GB0 et i
A0A4
Det
0.47
0.99
1.71
1.06
0.08
0.39
0,90
0.46
18.94
LADs (Gun al 2114
Det
0.39
0.81
1,50
0.90
0.01
0.12
0.43
0.19
7.29
SSR (Ours)
None
0.24
0.65
1.36
0.75
0.00
0.10
0.36
0.15
19.6
ST-P3: (Hu et al. 2022
Det & Map & Depth
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
1.6
CniAD= (Hu et ni 2U23
0.44
0.67
0,96
0.69
0.04
0.08
0.23
0.12
1.8t
VAD-Tiny! Jiang et all
2025
Map & Motion
0.46
0.76
1.12
0.78
0.21
0.35
0.58
0.38
16.8
VAD-Baset Drane
2045
Det & Map & Motion
0.41
0.70
1.05
0.72
0.07
0.17
0.41
0.22
4.5
BEV-Planner, et 20244
None
0.28
0.42
0.68
0.46
0.04
0.37
1.07
0.49
PARA-Drivel (Weng
0.25
0.46
0.74
0.48
0.23
0.39
LAW; [L et al. 20/44b
None
0.26
0.57
1.01
0.61
0.14
0.21
0,54
0.30
19.5
GenAD (Zheny el al.
2024h
Det & Map & Motion
0.28
0.49
0.78
0.52
0.08
0.14
0.34
0.19
6.7
SpivseDrive] (Sun
Map
0.29
0.58
0.96
0.61
0.01
0.05
0,18
LAD=; Guo ét al. 21244
Det
0.28
0.41
0.65
0.45
0.01
0.03
0.14
0.06
7.29
SSR: (Ourg)
None
0.18
0.36
0.63
0.39
0.04
0.12
19.6
4.2 IMPLEMENTATION DETAILS
Settings We build up SSR on VAD Jiang et al. 2023) and follow the setting of VAD-Tiny. We adopt ResNet-50 He et al. 2016) as image backbone operating at image resolution of 640 x 360. The BEV representation is generated at a 100 x 100 resolution and then compressed into sparse scene tokens with shape 16 256. The number of navigation commands remains 3 as prior works (Hu) ett al, 2023) Jiang et al. 2023). Other settings follow VAD-Tiny unless otherwise specified. In closed-loop simulation, we utilize ResNet-34 (He et al.) 2016) the image backbone, resizing the input image size to 900 x 256. The target point is concatenated with driving commands as the navigation information. The TCP head (Wu et al. 2022) is applied for planning module.
Training Parameters Our open-loop model is trained for 12 epochs on 8 NVIDIA RTX 3090 GPUs with a batch size of I per GPU, The training phase costs about 11 hours which is 13x faster than UniAD, We utilize the A (Loshchilov & Hutter 2019) optimizer with a learning rate set to 5x 5x10- The weight of imitation loss and BEV loss is both 1.0. The closed-loop model trained for 60 epochs on 4 NVIDIA RTX 3090 GPUs with a batch size of 32 per GPU. The learning rate is set to 1x10- while being halved after 30 epochs.
4.3 MAIN RESULT
Open-Loop Evaluation Our method outperforms existing E2EAD approaches in nuScenes, achiev- ing superior results in both L2 error and collision rate, as shown in Tab. T For the well-known method UniAD, which employs most auxiliary tasks, our method reduces 0.28m (27.2% relatively) average T.2MAX error and 0.16% (51.6) relatively) average C CRMAX without any auxiliary tasks. When compared to our baseline method VAD-Tiny, SSR not only reduces 0..9mm (50.0% relatively) average AVG error and 0.46% (79.3% relatively) average CRAVG. but also outperforms the VAD- Base with an obvious margin (45.8% average L2 and 70.7% average CRAVG reducation rela- tively). Furthermore, our method demonstrates real-time efficiency, achieving 19.6 FPS (the latency analysis in Appendix [D), which is 10.9x faster than UniAD and 4.3x faster than VAD-Base. Re- markably, it is also 2.2x faster than the prior sparse work SparseDrive (Sun et al. 2024). while reducing the average T.2vVG error by 0.22m.
7

---

Published as a conference paper at ICLR 2025
Table Performance on Town05 Long benchmark.
Table 3: Component Ablation.
Method
Modality
DS1
RCT
ISt
Modales
12 1m0
CR (99)
SIL
BW
15
25
36
Avg
1s
25
34
AVg.
CILRS (Codevilla ct
2019
7.8
10.3
0.75
LBC (Chen
12.3
31.9
0.66
Transtuser (PraKash ct
2
C+L
31.0
47.5
0.77
Roach Zhing cl 2021
41.6
96.4
0.43
ST-P3 cH 2022
11.5
83.2
Table 4: Number of Scene Queries.
TCP Wi et 2122
57.2
80.4
0.73
VAD-Base Tanng cl m
2023
30.3
75.2
(m))
(9))
ThinkTwice Uia al.
20236
C+I.
65.0
95.5
0.69
DriveAdapter
65.9
94.4
0.72
0.24
1.36
1.25
0.25
0.69
0.00
0.14
0.10
0.35
0.43
SSR (Ours)
78.9
95.5
0.83
0.26
0.30
1.38
0.34
0.77
0.12
0.39
0.31
0.66
Table 5: Ablation of navigation guidance. GS means go straight and LR denotes turn left right.
Navigation Guidance
(m)
L2-LR (m)
CR-GS (%)
CR-LR (%)
1s
2s
3s
Avg.
Avg.
1s
2s
3s
Avg.
Avg.
0.24
0.61
1.31
0.72
0.34
0.96
1.98
1.09
0.09
0.38
0.40
0.29
0.00
0.44
1.90
0.78
0.23
0.61
1.28
0.71
0.33
0.91
1.8
1.04
0.00
0.08
0.18
0.10
0.00
0.29
1.70
0.66
When compared to previous approaches that eliminate auxiliary annotations, SSR demonstrates impressive performance across all metrics. LAW (Li al. 2024b), for instance, achieves similar inference speed to SSR but retains a substantial gap in both L2 error and collision rate. The method closest in performance to ours UAD (Guo et al. 2024), using larger ResNet-101 backbone and 1600 x 900 image resolution input, and requires an additional open-set 2D detector to supervise objectness information. Despite these additional resources, UAD still shows 0.155m higher average MAX error compared to SSR, along with 2.7x lower inference speed.
Closed-Loop Evaluation As presented in Tab. our method significantly outperforms existing works in terms of driving score, including those utilizing LiDAR input (Jia al. 2023b a). For camera-based methods, SSR achieves 31.7-point improvement in driving score over TCP and remarkable 2.6x increase VAD-Bas Base. SSR also achieves the highest infraction score, demon- strating its comprehensive capabilities and robust performance in challenging environments.
4.4
ABLATION STUDY
4.4.1
COMPONENT-WIST ABLATION
In Tab. we present an ablation study the proposed components. When the STL is enabled instead of directly interacting waypoint queries with the BEV feature, the collision rate is reduced by more than half. This significant decrease underscores the STL's ability effectively distill criti- cal information from the dense scene data, thereby minimizing the impact of irrelevant features and reducing computational redundancy. Furthermore, when incorporating the Future Feature Predictor, we observe a further reduction in the average collision rate to 0.15%. This improvement highlights the Future Feature Predictor's role in enhancing SSR's 's comprehension of scene dynamics, contribut- ing to more safe trajectory planning and overall performance gains.
4.4.2 NUMBER OF SCENE QUERIES
M For L2 error, using We evaluate the impact of varying the number of scene queries in Tab. queries yields the best performance, with performance declining as the number of queries in- creases. Interestingly, when considering the collision rate, the optimal performance is achieved with 16 queries. Therefore, we select 16 queries as the default setting in SSR to strike balance between minimizing L2 error and reducing collision rate. The poor performance observed with 64 scene queries suggests that an excessive number of queries may overwhelm the model with too much per- ception information, leading to confusion similar directly interacting with dense BEV features.
8

---

Published as a conference paper at ICLR 2025
Scene Query 0
Scene Query
Scene Query 4
Scene Query
FR#10 Sum-Attn
Scene Query 8 Scene Query 10
Scene Query 12
Scene Query 14
FR#65 Sum-Altn
Figure 5: Visualization of BEV Square Attention Map of Scene Queries. Attention maps for of the tokens are displayed. Ego vehicle located at the center while up direction indicates the front of ego. Brighter areas represent higher attention weights. The full set provided in Appendix
Figure 6: Sum- Attention Map in Different Scenes. FR: frame number.
4.5 ANALYSIS AND DISCUSSION
How does scene queries represent the scene? To understand why only a handful of queries can effectively represent the entire scene and even outperform more complex designs, we visualize 8 out of the 16 BEV square attention maps (BRavi) from the STL module in Fig- 5 The results re- veal that each query focuses uniformly distinct region of the BEV space, with different queries attending to different areas. When summing the attention maps of all scene queries, we observe that the aum-attention map surprisingly covers the entire scene in balanced manner, with greater emphasis on the front region than the back. Furthermore, the attention maps remain relatively con- sistent across different frames, shown in Fig. 6 indicating that w (B. offers stable spatial compression guidance for SSR. In essence, the scene queries act as a compressed representation of dense BEV features, where each query concentrates specific spatial region.
What does scene queries learn? In Fig. visualize the navigation-aware BEV features Bhavi as background color map and highlight the most positive activation positions of the scene queries across different scenarios. When overtaking vehicle on the left shown in Fig. [ [), the activation positions primarily focus on the overtaked vehicle and the left rear area, anticipating potential risks. In straightforward driving scenario (Fig. 7()), the scene queries more dispersed, with attention directed towards a front-right vehicle, potentially anticipating cut-in. During right turn at an intersection (Fig. ())) the scene queries not only activate around the right rear vehicle but also pay attention to the left crosswalk, where pedestrians might appear. When we further compare the same command in different scenes between Fig. b)) and Fig. (()), the scene queries focus on different regions adaptively. These observations demonstrate how our sparse scene queries can understand various scenes and manage different driving scenarios.
How does navigation information work? In Tab. 5 We conduct experiments assess the effect of navigation commands in different scenarios, demonstrating that navigation guidance improves planning results across all cases. We further test different commands within the STL module for the same frame at an intersection, and visualize the corresponding scene queries in Fig. 8 Compared to the original command go straight, when the command is changed to turn left, the module shifts its attention to pedestrians on the left. Similarly, with the turn right command, the STL module increases its focus on the front-right area, particularly highlighting a vehicle on the right that is not prominent in the other navigation scenarios. These findings demonstrate that navigation commands effectively guide the STL module to extract relevant information from the scene. Additionally, we conduct experiments evaluate the impact of confusing navigation commands in Appendix F

---

Published as a conference paper at ICLR 2025
(a) Turn Left
(b) Straight (c) Turn Right
(a) Turn Left
(b) Go Straight
(c) Turn Right
Figure 7: Visualization of Scenes Queries in iifferrettSeeaarriee The central green box rep- resents ego vehicle. The red boxes indicate the ground truth object, while the dotted lines de- note ground truth map. The red star marker is the most activated position of each scene query.
Visualization of Scene Queries for Different Navigation Commands Same Scene. Different navigation commands put to the SSR within the same scene to investi- gate their impact scene queries. The original command go straight.
Figure 8:
in-
To summarize above discussions, we revisit the generation of scene query in Eq. where B Brovi encodes the navigation information, and w. (Braon) is responsible for spatial compression:
pl
B naei
B naoi
(16)
NaviGuidance
SpapsalCompression
4.6
VISUALIZATION
Figure 9: Visualization of Planning Results. The perception results are rendered from annotations.
In Fig. we present qualitative result of SSR on planning trajectories, demonstrating strong alignment with the ground truth compared to VAD-Base. Additional visualizations across various scenes, including failure cases, can be found in the Appendix B and C due to space limitations.
5 CONCLUSION
The SSR framework presents significant advancement in the field of E2EAD by challenging the conventional reliance on perception tasks. By utilizing navigation-guice guided sparse tokens and tem- poral self-supervision, SSR addresses the limitations of perception- state-of-the-an performance with minimal costs. Moreover, visualization the sparse tokens en- hances the interpretability and transparency of SSR's navigation-guide process. We hope SSR can provide a strong foundation for scalable, interpretable, and efficient autonomous driving systems.
architectures, achieving
Linitations and Future Work. Despite these advances, current simple navigation commands may constrain SSR's s effectiveness in more complex driving scenarios. Future work will explore integrat- ing more sophisticated navigation prompts, such routing and natural language.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research supported by National Key R&D Program of China (2022YFB4300300,
REFERENCES
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.0030.7 2016.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan. Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In CoRL, pp. 66-75. PMLR, 2020.
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In CVPR, pp. 9329-9338, 2019.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: an open urban driving simulator. In CoRL, pp. 1-16, 2017.
Mingzhe Guo, Zhipeng Zhang, Yuan He, Ke Wang, and Liping Jing. End-to-end autonomous driving without costly modularization and 3d manual annotation. arXiv preprint arXiv:2406, /7680, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-eexctatiio networks. In CVPR, 2018.
Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with self supervised freespace forecasting. In CVPR, 2021.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end rision-based autonomous driving via caatial-tempora feature learning. In ECCV, 2022.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving. In CVPR, 2023.
Yuanhui Huang, Wenzhao Zheng, Yumpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. CVPR, 2023.
Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In ICCV, 2023a.
Xiaosong Jia, Penghao Wu, Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023b.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. In ICCV, 2023.
Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differen- tiable raycasting for self-supervised occupancy forecasting. In ECCV, 2022.
Peidong Li, Wancheng Shen, Qihao Huang, and Dixiao Cui. Dualbev: Unifying dual view transfor- mation with probabilistic correspondences. In ECCV, 2024a.
11

---

Published as a conference paper at ICLR 2025
Qi Li, Yue Wang, Yilun Wang. and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In ICRA, 2022a.
Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406. 200.2406.0881, 2024b.
Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's aidd'seeye-viww representation from multi-camera images via spa- tiotemporal transformers. In ECCV, 2022b.
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. ego status all you need for open-loop end-to-end autonomous driving? In CVPR, 2024c.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construc- tion. arXiv preprint arXiv:2208. 14437, 2022.
Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. 0rXiv:2211.1058) 2022.
Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, and Limin Wang. Sparsebev: High-performance sparse 3d object detection from multi-camera videos. In /CCV,
Yicheng Liu, Yue Wang. Yilun Wang, and Hang Zhao. Vectormapnet: End-to-end vectorized hd map learning. arXiv preprint arXiv:2206.00 220.22006.08920, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NeurlPS, 1988.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to- end autonomous driving. In CVPR, 2021.
Michael Ryoo, AJ Piergiovanni, Anurag Arab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. NeurlPS, 2021.
Chonghao Sima, Wenwen Tong, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, and Hongyang Li. Scene as occupancy. ICCV, 2023.
Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint 1rXi::2405.19620, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. arXiv:2303.11926 2023.
Xinshuo Weng, Boris Ivanovic, Yan Wang. Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. CVPR, 2024.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline. NeurIPS, 2022.
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2025
Diankun Zhang, Guoan Wang, Runwen Zhu, Jianbo Zhao, Xiwu Chen, Siyu Zhang. Jiahao Gong, Qibin Zhou, Wenyuan Zhang, Ningzi Wang, Feiyang Tan, Hangning Zhou, Ziyao Xu, Hao- tian Yao, Chi Zhang. Xiaojun Liu, Xiaoguang Di, and Bin Li. Sparsead: Sparse query-centric paradigm for efficient end-to-end autonomous driving. arXiv preprint arXiv:2404.06892 2024.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021.
Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oce- world: Learning a 3d occupancy world model for autonomous driving. In ECCV, 2024a.
Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. ECCV, 2024b.
Zhuofan Zong, Dongzhi Jiang. Guanglu Song. Zeyue Xue, Jingyong Su, Hongsheng Li, and Yu Liu. Temporal enhanced training of multi-view 3d object detector via historical object prediction. In ICCV, 2023.
Jialy Zou, Bencheng Liao, Qian Zhang, Wenyu Liu, and Xinggang Wang. Mim4d: Masked mod- eling with multi-view video for autonomous driving representation learning. arXiv preprint arX1::2003.08660 2024.
13

---

Published as a conference paper at ICLR 2025
A VISUALIZATION FOR ALL ATTENTION MAP OF SCENE QUERIES
We visualize the BEV square attention maps of all 16 scene queries used in Fig. 10 It can be observed that each scene query captures information from different spatial regions with a uniform attention distribution. When these queries are combined represent the entire BEV feature, it effectively compresses spatial space while applying different attention weights.
Scene Query
Scene Query ]
Scene Query
Scene Query
Scene Query
Scene Query 5
Scene Query
Scene Query
Scene Query
Scene Query 9
Scene Query
Scene Query 11
Scene Query 12
Scene Query 13
Scene Query 14
Scene Query 15
Figure 10:
Visualization of All BEV Square Attention Map Scene Queries Same Scene. The ego vehicle is located the center, with the upward direction indicating the front of the vehicle. Brighter areas represent higher attention weight.
B QUALITATIVE RESULTS IN DIFFERENT SCENES
As illustrated in Fig. II we visualize additional planning trajectories SSR across various scenes. In Fig. a)), SSR achieves even smoother result than the ground truth when turning left. When
14

---

Published as a conference paper at ICLR 2025
BALBO-13H
Turth
(a) Turn Left
(b) Straight
BEV
OM DAC
CALLDAKCPTOT
Turn Hight
(c) Turn Right
Visualization of Planning Trajectories in Different Scenes. The perception informa- tion is rendered from annotations. The dashed lines denote the map of the scene, while red boxes represent for the object detection. Ego vehicle is drawn by the green box in center.
Figure 11:
following bus through an intersection with the go straight command, as shown in Fig. (b), our method also outperforms the baseline method, VAD, by producing better trajectory. Similarly, in the turn right scenario illustrated in Fig. [[]). SSR demonstrates superior performance compared VAD, achieving a smaller turning radius. These results demonstrate that SSR effectively under- stands the scene and produces accurate planning results through imitation learning. Even without explicit perception supervision, the learned scene queries capture the essential elements of complex scenes for autonomous driving.
C FAILURE CASES
We identified some failure cases for SSR, which are visualized Fig. highlighting the two most common reasons. The first common issue is noise in the initial frame of a clip for temporal module, which also discussed in Weng et al. (0024). Due to zero initialization of input features and the ego vehicle's state, the temporal module of E2EAD methods can be negatively impacted, leading to inferior performance, as illustrated Fig. 12 a). The second issue arises from ambiguous navigation
commands in the label generation of nuScenes dataset. For example in Fig. (2)b), the command to go straight, but the ground truth trajectory involves turning left to change lanes. Since SSR heavily relies navigation commands, can be adversely affected when encountering ambiguous
15

---

Published as a conference paper at ICLR 2025
Sae BiLaosRao
Turn Left
(a) Failure Reason: Noisy First Frame for Temporal Module
M
Stright
(b) Failure Reason: Ambiguous Navigation Commands in Dataset
Figure 12: Visualization of Common Failure Cases.
instructions. However, these reasons which heavily related to dataset can be easily avoided in real world scenarios by integrating true navigation command and run image sequences.
D LATENCY ANALYSIS
Inference latency analysis of SSR components presented in Fig. [3] The evaluation con- ducted on an NVIDIA GeForce RTX 3090 GPU with batch size of 1. The image backbone and encoder, responsible for generating dense BEV features, contribute tency. In contrast, our proposed Scenes Token- Learner incurs only 7.8% of the latency, high- lighting its efficiency in extracting useful infor- mation from massive dense BEV feature. The planning decoder, which interacts way point queries with the scene queries and output fi- nal planning trajectory, adds just 1.5% to the latency, as SSR only utilizes 16 tokens to rep- resent the scenes.
90.7% of the total la-
Image Backbone
Encoder
Scene TokenLearner
Pla Planning Decoder
Figure 13: Latency Analysis.
E DISCUSSIONS
Can SSR learn sufficient perception information for AD Task? Although our framework elim- inates the need for perception annotations, we investigate the effect of incorporating perception branches in Tab. 6 We follow approach PARA-Drive (Weng et al. 2024) to integrate auxiliary tasks into SSR, using them to supervise the BEV feature in parallel with the planning task. In addi- tion, to better evaluate the framework in ablation studies, we measure Curb Collision Rate (CCR) as introduced in Li al. 2024c). The results indicate that even without supervised object detection module, our method achieves a lower collision rate. Similarly, the CCR is even lower when SSR operates without map branch. These findings suggest that SSR effectively learn the necessary
16

---

Published as a conference paper at ICLR 2025
Table Perception tasks ablation. CCR calculated in 0.Im resolution following Li al. c024c).
Auxiliary Task
L2 (m)
CR (%)
CCR (%)
Map
Obs
1s
2s
3s
Avg.
1s
28
38
Avg.
2s
38
Avg.
0.24
0.65
1.36
0.75
0.00
0.10
0.36
0.15
0.19
1.01
2.71
1.30
0.29
0.71
1.44
0.81
0.13
0.16
0.68
0.32
0.20
0.90
2.85
1.32
<
0.30
0.70
1.37
0.79
0.06
0.31
0.68
0.35
0.25
1.25
3,40
1.63
/
0.33
0.77
1.50
0.86
0.02
0.16
0.59
0.26
0.21
1.13
2.87
1.40
perception information for autonomous driving tasks without explicit supervision, maintaining high performance across all metrics.
However, we follow the 12-epoch training setup for fairness, which may lead to suboptimal per- ception performance. To investigate further, we conduct additional experiments with SSR trained using pretrained weights from 48 epochs perception learning, as employed in VAD (Jiang et al. 2023). The pretrained model achieves 27.99 mAP and 40.15 NDS for obstacles and 48.78 mAP for mapping on nuScenes. As illustrated in Tab. 7 the pretrained weights does not significantly enhance trajectory prediction performance. This suggests that SSR inherently learns scene understanding in a different way than traditional perception ceappptionttskss-ee -asks-drivenn approaches.
Table 7: Effeet of Pretrained Perception Modules.
1.2 (m) 4
CR (%) 4
CCR (%) 4
Pretrain
18
28
3s
Avg.
18
28
3s
Avg.
18
2s
38
Avg.
X
0.24
0.65
1.36
0.75
0.00
0.10
0.36
0.15
0.19
1.01
2.71
1.30
0.48
1.03
1.82
1.11
0.25
0.31
1.42
0.66
0.67
1.61
4.02
2.10
Impact of confusing driving commands. Since E2EAD models typically lack HD maps, a high- level driving command input is necessary for navigation. To evaluate the effect of confusing com- mands, we test four scenarios: consistently "go straight," consistently "turn left," consistently "turn right," and random commands in Tab. 8 Notably, as the ego vehicle typically operates on the left side of the road, "turn left" and "go straight" commands lead to comparable L2 error and collision rates. However, the "turn right" command shows an obvious increase collision rate, often re- sulting in conflicts with oncoming vehicles. Random commands cause a noticeable degradation in performance but still produce reasonable results, demonstrating the model's resilience to noisy navi- gation inputs. These findings highlight the strengths of our approach while identifying opportunities to improve its handling ambiguous or conflicting commands.
Table 8: Effect of Driving Commands.
L2 (m)
CR (%)
Command
1s
2s
3s
Avg.
Is
2s
3s
Avg.
Original
0.24
0.65
1.36
0.75
0.00
0.10
0.36
0.15
Go Straight
0.25
0.67
1.40
0.77
0.04
0.16
0.51
0.23
Turn Left
0.26
0.68
1.40
0.78
0.00
0.12
0.55
0.22
Turn Right
0.26
0.70
1.44
0.80
0.10
0.23
1.27
0.53
Random
0.26
0.68
1.41
0.78
0.06
0.14
0.72
0.31
17


Reviews:
[{"review_id": "nar3bWDHls", "reviewer": "Reviewer_odD2", "summary": "This paper presents a novel method for end-to-end trajectory prediction for autonomous driving, with a model that takes as input surround view camera images, and outputs a predicted trajectory for the vehicle. The main novelties are:\n- the use of a sparse tokenizer to reduce the typical dense BEV representation into a sparse (in experiments 16) set of feature vectors.\n- the introduction of the driving command into the BEV representation via cross attention.\n- the application of a future prediction module for the BEV features as an auxiliary loss.\nThe proposed method aims to reduce the dimensionality of the BEV space without using pre-defined intermediate signals such as detections, mapping etc. This allows the model to be end-to-end differentiable, with the tokenization learned as a part of the driving task. \n\nExperiments and ablations are performed on the nuScenes dataset, where the proposed method produces a new SOTA in terms of both quality and latency.", "questions": "What is the channel dimension of the scene queries?\n\nHow would the model respond in a very dense scene? Is there an inherent limitation in the number of objects / components that the model can focus on given a fixed number of scene queries?\n\nWhat is the quality of the perception outputs in Table 4?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The main novel contribution of this work lies in the use of the TokenLearner for BEV feature compression. From the experiments, the authors demonstrate that they are able to achieve new SOTA trajectory prediction results without any intermediate perception signals used by previous works. Ablations are also performed demonstrating the improvements provided by both the future prediction auxiliary task as well as the navigation command cross attention into the BEV feature space. Visualizations are also provided demonstrating the the model's attention mechanism does seem to focus on reasonable parts of the scene given a navigation command (i.e. it's possible that the model is not directly overfitting in some way).\n\nOverall the paper is well written and easy to understand, and the overall method seems reasonable and based on sound prior works.", "weaknesses": "The number of optimal scene queries selected by the authors seems surprisingly small, and may be skewed by the difficulty of the average scene encountered in nuScenes. From the provided visualizations, it looks like the model focuses directly on relevant agents and objects in the scene. However, it's unclear whether the model would be able to extend to extremely dense scenes, where a fixed number of scene queries may limit the models ability to handle many other agents / objects.\n\nIn Table 4, the authors ablate the addition of a perception module. However, no results for the perception tasks themselves are provided, and so it cannot be determined whether these tasks were actually trained appropriately (i.e. they may have simply diverged altogether).\n\nOne additional factor to call out is the inherent risk of the nuScenes dataset being skewed in some way (i.e. Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes), but this is not the fault of the authors.\n\nIn Figures 7 and 8, it would be helpful to explain the colormap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z8Gazryn5r", "reviewer": "Reviewer_pPL3", "summary": "This paper proposed a novel approach for trajectory prediction in autonomous driving called SSR. A very straight intuition is that the more comprehensive and precise the information one can extract, the better the downstream tasks will perform. Many recent models try to extract dense BEV features with rich scene information and route the information to multiple auxiliary tasks. On the contrary, the authors argue that for end-to-end autonomous driving, auxiliary supervision from perception tasks might be unnecessary, instead computational resources can be focused on crucial parts of the scene.\nTo achieve this, the paper introduced a scene token learner that learns sparse scene queries from BEV features extracted by a BEV encoder (BEVFormer in particular) and current navigation command. The trajectory is then predicted based on the scene queries. To train the model, two loss terms are used. The first term directly minimizes the L1 loss between the predicted and ground truth trajectory. The second term aims to enhance scene representation by introducing a BEV world model. This term replaces the perception sub-tasks in other models, with more focus on temporal context. The BEV world model takes the BEV features, scene queries and predicted trajectory as input and it tries to predict the BEV features at the next frame. We minimize the L2 distance between the output of the world model and the BEV features directly extracted by the BEV encoder at the next frame. The overall loss is the sum of these two terms.\nThe authors show that a small set of tokens is sufficient to outperform sota models in trajectory prediction on NuScenes dataset.", "questions": "1. There’s a paper “Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes” (https://arxiv.org/abs/2305.10430) claims that a simple MLP model that takes simple ego state (velocity, acceleration, historical trajectory) can also outperform VAD and UniAD. While their claim is that the metrics may not adequately capture the superiority of different methods, in combination with table 2.(b) in this paper being reviewed, where a larger number of scene queries leads to worse performance, is this an indication of overfitting of the larger models (or generally speaking, a case where a larger model is being overperformed by a smaller model due to various reasons)? Considering the trajectories in NuScenes are relatively simple, will this weaken the conclusion of this paper, especially when the navigation tasks become more complex?\n2. Why the visualization in Figure. 5 in a hexagon format instead of square grids?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, with clear and insightful motivation. The proposed model architecture is novel, and the experimental results show significant improvement over existing methods. The experiment section is comprehensive, with insightful analysis.\nIn autonomous driving, multi-head models are very popular, for a number of reasons. Not only because different tasks can serve as auxiliary supervision for each other and contribute to the overall performance improvement, the output of the individual tasks can also be utilized by downstream tasks in a modular autonomous driving system (e.g. occupancy map can be the input of a traditional pathfinding algorithm for parking). However in a fully end-to-end autonomous driving system all those intermediate outputs are unnecessary. It’s interesting to see papers questioning the design choices of existing models and exploring alternative designs.", "weaknesses": "1. The proposed architecture borrows the idea of a world model by predicting future bev features. However it seems that the BEV features are only regularized by the trajectory prediction task. There’s no reconstruction loss and other downstream tasks. In this case it’s questionable to still call it a world model.\n2. The paper lacks theoretical analysis of why the proposed SSR method is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BkRMr1DZp", "reviewer": "Reviewer_rfNp", "summary": "In this paper, the authors propose a navigation-guided sparse scene representation for end-to-end autonomous driving. The proposed solution eliminates the need for supervised sub-tasks, allowing computational resources to concentrate on essential elements directly related to navigation intent. Specifically, the proposed solution utilizes learned sparse query representations guided by navigation commands, and introduces a BEV world model for self-supervision on dynamic scene changes to highlight the critical role of temporal context in autonomous driving. The experimental results show that the proposed solution achieves great performance on nuSccenes dataset with minimal training and inference cost.", "questions": "1. Are the navigation commands open-set or close-set, how many commands were used in this paper, how to pre-processing the high-level navigation commands so that the model can take them as inputs, etc. \n2. What is the performance of the model if the navigation command never appears in the training set (or confusing command, e.g., turn left and right, or turn left or right, etc.)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed solution sounds solid in theory. Specifically, eliminating the need for supervised sub-tasks, e.g., detection, mapping, motion, occupancy, etc., indeed speed up the training and inference time. The proposed navigation-guided concept and BEV world model should extract useful features for the autonomous driving. \n2. The proposed navigation-guided concept is inspring. As mentioned in the paper, it is a human-inspried solution whose intuition is to mimic the behavior of human beings. It is a good research direction. The authors propose a way to combine the information with the special features.\n3. The ablation study is helpful. Readers can get more information from the ablation study results.\n4. The analysis and discussion guide readers think more deeper about the model design and the performance.", "weaknesses": "1. There are some important information is missing. For example, detailed model structure, navigation commands pre-processing operation, etc. \n2. It will be better if the authors could report some failure cases. Especially for those cases which the predictions are totally opposite toward the navigation command.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IO9WqUXUIV", "reviewer": "Reviewer_zc5J", "summary": "The paper introduces a novel SSR framework that requires no auxiliary task outputs or labels, relying instead on just 16 sparse navigation tokens for guidance. This approach achieves SOTA performance on the nuScenes dataset, with both inference and training processes being fast.", "questions": "Refer to the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The model proposed in this paper yields excellent results, achieving SOTA performance on the nuScenes dataset. Additionally, both the inference and training speeds are fast on 3090 GPUs.", "weaknesses": "1. From the ablation studies (Table 2), it can be seen that the core contribution of this paper is the introduction of navigation information as guidance. The addition of the STL and BWM modules does not significantly affect L2, which suggests that navigation information provides significant assistance for lane following, especially when driving straight. It is understandable that the introduction of the BWM module, which incorporates temporal modeling, improves collision rates. In other words, the key contribution of this paper lies in utilizing navigation information, which is not employed by other methods. This kind of approach is relatively common in the industry but is rarely mentioned in academia. Whether merely introducing navigation information as input is worthy of an ICLR paper is something I am somewhat conflicted about.\n2. The STL and BWM modules lack novelty. Furthermore, I'm not particularly fond of naming the BWM module a \"world model.\" Given that it primarily involves predicting the next frame in BEV, the term \"world model\" might be considered too broad. I'd like to reiterate that I think the core contribution of this paper is the introduction of navigation information.\n3. The color scheme used in the figures could be slightly unappealing, with most modules and features using white as the primary color. It would be nicer if the visuals were more aesthetically pleasing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GYV7F1addP", "reviewer": "Reviewer_4p2A", "summary": "This work proposes to extract sparse scene queries to represent the scene and use the scene queries to generate planning results.  Further, they propose an auxiliary task head and loss - predict future BEV feature.", "questions": "None", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The elimination of middle task could indeed reduce the latency and annotation requirements.\n2. The writing is clear", "weaknesses": "1. Lack of closed-loop evaluation:  as pointed in AD-MLP[1] and BEVPlanner[2], it is rather simple to fit nuScenes, as most of its data is going straight.  BEVPlanner only uses BEV feature, even without the need of the proposed scene queries.  (**Strangely, the authors cite [2] but do not compare with [2] in Table 1. I hope the authors could give an explanation**). Not to mention that AD-MLP even does not need any sensor inputs.  Thus,  **it is not convincing to claim that there is no need for middle supervision under nuScenes open-loop evaluation**. It is important to have closed-loop results and ablation studies on it.\n\n[1] Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes.\n\n[2] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? CVPR 2024\n\nDue to the missing of closed-loop results, the argument in the paper is not convincing and thus I give a reject rating. I will increase my rating if closed-loop results and ablations are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)