Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving

## Abstract

Abstract End-to-End Autonomous Driving (E2EAD) methods typically rely on supervised perception tasks to extract explicit scene information (e.g., objects, maps). This reliance necessitates expensive annotations and constrains deployment and data scalability in real-time applications. In this paper, we introduce SSR , a novel framework that utilizes only 16 navigation-guided tokens as S parse S cene R epresentation, efficiently extracting crucial scene information for E2EAD. Our method eliminates the need for supervised sub-tasks, allowing computational resources to concentrate on essential elements directly related to navigation intent. We further introduce a temporal enhancement module that employs a Bird’s-Eye View (BEV) world model, aligning predicted future scenes with actual future scenes through self-supervision. SSR achieves state-of-the-art planning performance on the nuScenes dataset, demonstrating a 27.2% relative reduction in L2 error and a 51.6% decrease in collision rate to the leading E2EAD method, UniAD. Moreover, SSR offers a 10.9 × \times faster inference speed and 13 × \times faster training time. This framework represents a significant leap in real-time autonomous driving systems and paves the way for future scalable deployment. Code will be released at https://github.com/PeidongLi/SSR .

## 1 Introduction

Figure: Figure 1: Performance Comparison of Various Methods in Speed and Accuracy.
Refer to caption: /html/2409.18341/assets/figs/combined_plot.png

Vision-based E2EAD (Hu et al., ; Jiang et al., ; Sima et al., ; Zheng et al., ; Sun et al., ; Weng et al., ; Li et al., ; Guo et al., ) has gained significant attention in recent research as a cost-effective alternative for autonomous driving systems. Traditional architectures typically consist of separate perception and planning modules. While most perception modules are handled by neural networks (NN), planning modules often rely on rule-based pipelines. This separation can lead to information loss during the transfer between modules, resulting in suboptimal performance. E2EAD addresses this by using entire neural networks to predict planning trajectories from images, thereby minimizing information loss and improving overall performance.

However, most current E2EAD approaches build upon complex perception frameworks, often incorporating additional NN-based planning modules. These approaches typically inherit tasks such as object detection (Li et al., ; Philion & Fidler, ), mapping (Li et al., ; Liao et al., ), and occupancy prediction (Sima et al., ; Huang et al., ), resulting in large and computationally intensive neural networks. Despite their integration, these models still maintain modular framework design, requiring independent sub-tasks’ supervision. As a result, they remain annotation-intensive, suffer from scalability issues, and are inefficient for real-time deployment.

While many E2EAD methods continue to mimic the paradigm established by prior BEV perception works, they often overlook a critical question: Do E2EAD systems still require such extensive perception tasks? Existing methods often employ multiple perception branches to supervise task-specific queries, derived from dense BEV features which encapsulate rich scene information. The reliance on auxiliary perception tasks primarily stems from the need to distill relevant information from massive scene elements, limiting data scalability and real-time applicability. Thus, we seek for a more focused approach that directly targets driving-related elements without the need of auxiliary supervision. The challenge, then, becomes: How can we efficiently identify and focus computational resources on the crucial parts of the scene without auxiliary perception supervision?

To address this, we introduce SSR, a novel framework that leverages navigation-guided Sparse Scene Representation, learning from temporal context in self-supervision rather than explicit perception supervision. Inspired by how human drivers selectively focus on scene elements based on navigation cues, we find that only a minimal set of tokens from dense BEV features is necessary for effective scene representation in autonomous driving. Since E2EAD methods do not rely on high-definition maps as input, a high-level command (e.g., “turn left”, “turn right”, “go straight” following common practices in Hu et al. (); Jiang et al. ()) is required for navigation. Our method, therefore, extracts scene queries guided by the navigation commands, akin to human attention mechanisms.

As illustrated in Fig. [2](#S1.F2)(a), existing methods typically extract all perception elements following previous BEV perception paradigms. These methods rely on Transformer (Vaswani et al., ) to identify relevant ones in the additional planning stage. In contrast, as shown in Fig. [2](#S1.F2)(b), SSR directly extracts only the essential perception elements in the guidance of navigation commands, thereby minimizing redundancy. Our approach takes full advantage of the end-to-end framework, breaking the modular cascade architecture in a Navigation-Guided Perception manner. While prior works (Sun et al., ; Zhang et al., ) attempt to reduces computation by skipping BEV feature construction, they still depend on hundreds of task-specific queries. However, our method drastically reduces computational overhead by using just 16 tokens guided by navigation commands.

Additionally, SSR capitalizes on temporal context to circumvent the need for perception tasks supervision. We hypothesize that if a predicted action aligns with the actual action, the resulting scene should match the real future scene. Specifically, we predict future BEV features, which are then self-supervised by the actual future BEV features. This BEV world model, which takes current BEV features and the planning trajectory as input to predict future BEV features, offers an alternative for supervising both scene representation and planning trajectories without auxiliary annotations.

By leveraging navigation-guided perception paradigm and temporal self-supervision, SSR provides an effective and efficient solution for real-time autonomous driving. As illustrated in Fig. [1](#S1.F1), SSR delivers state-of-the-art performance on the nuScenes (Caesar et al., ) dataset, with minimal computational overhead. Specifically, our method decreases average L2 error by 0.28 meters (a 27.2% relative improvement) and reduces the average collision rate by 51.6% relatively compared to UniAD (Hu et al., ), even without any annotations. Remarkably, our method reduces training time to 1/13th of that required by UniAD and is 10.9× faster during inference. Therefore, SSR has the potential to manage large-scale data in real-time applications, further enhancing performance.

Figure: (a) Task-Specific Supervised Paradigm
Refer to caption: /html/2409.18341/assets/figs/paradigm_sup.png

Our contributions are summarized as follow:

- •
We introduce a human-inspired E2EAD framework that utilizes learned sparse query representations guided by navigation commands, significantly reduces computational costs by adaptively focusing on essential parts of scenes.
- •
We highlight the critical role of temporal context in autonomous driving by introducing a BEV world model for self-supervision on dynamic scene changes, eliminating the need for costly perception task supervision.
- •
Our framework achieves state-of-the-art performance on the nuScenes dataset with minimal training and inference costs, establishing a new benchmark for real-time E2EAD.

## 2 Related Work

### 2.1 Vision-Based End-to-End Autonomous Driving

Research on End-to-End autonomous driving dates back to 1988 with ALVINN (Pomerleau, ), which used a simple neural network to generate steering outputs. NVIDIA developed a prototype E2E system (Bojarski et al., ) based on convolutional neural networks (CNN), bypassing manual decomposition. The recent resurgence in vision-based E2EAD has been driven by rapid advancements in BEV perception (Li et al., ; Liao et al., ; Liu et al., ; Huang et al., ) and modern architectures like Transformer (Vaswani et al., ).

ST-P3 (Hu et al., ) introduced improvements in perception, prediction, and planning modules for enhanced spatial-temporal feature learning, integrating auxiliary tasks such as depth estimation and BEV segmentation. UniAD (Hu et al., ) built on previous BEV perception works to create a cascade framework with a variety of auxiliary tasks, including detection, tracking, mapping, occupancy, and motion estimation. VAD (Jiang et al., ) sought to streamline scene representation by vectorizing it, reducing the tracking and occupancy tasks seen in UniAD. GenAD (Zheng et al., ) explored the use of generative models for trajectory generation, jointly optimizing motion and planning heads based on VAD. PARA-Drive (Weng et al., ) further examined the relationship between auxiliary tasks, reorganizing them to run in parallel while deactivating them during inference. In contrast, our approach eliminates all perception tasks, achieving remarkable performance in both accuracy and efficiency.

### 2.2 Scene Representation in Autonomous Driving

Most prior works in autonomous driving (Hu et al., ; ; Jiang et al., ; Zheng et al., ) have inherited approaches from perception tasks, such as Li et al. (), which leverages dense BEV features as the primary scene representation. In these frameworks, task-specific queries (e.g., for detection and mapping) are used to extract information from the BEV features under manual labels’ supervision. While these approaches benefit from rich scene information, they also introduce significant model complexity, hindering real-time application, particularly in occupancy-based representations (Sima et al., ; Zheng et al., ).

Following the trend of sparse paradigms in BEV detection (Lin et al., ; Liu et al., ), recent sparse E2EAD approaches (Sun et al., ; Zhang et al., ) directly utilize task-specific queries to interact with image features. These methods attempt to bypass BEV feature generation altogether by directly interacting with image features through task-specific queries. However, despite the reduction in BEV processing, these models still rely on hundreds of queries, which diminishes the promised simplicity and efficiency of the end-to-end paradigm. LAW (Li et al., ) proposed the use of view latent queries to represent each camera image with a single query. However, this approach compromises information fidelity, leading to suboptimal performance. UAD (Guo et al., ) attempted to divide the BEV feature into angular-wise sectors but still relied on open-set detector labels for supervision, maintaining the complexity of task-specific queries. In this work, we introduce SSR, a novel approach that represents the scene by a minimal set of adaptively learned queries, enhancing both efficiency and performance.

### 2.3 World Model in Autonomous Driving

World models have gained attention in autonomous driving for their generative capabilities, especially in recent years. The importance of world model lies in generating rich training data, including rare corner cases, as demonstrated in early works like Hu et al. (; ). More recently, world models have been integrated into E2EAD frameworks to improve their performance (Wang et al., ; Zheng et al., ; Min et al., ; Li et al., ; Guo et al., ). For instance, Drive-WM (Wang et al., ) employed a diffusion model to predict multi-view videos, using these predictions for planning. OccWorld (Zheng et al., ) simultaneously predicted planning trajectories and occupancy maps, while DriveWorld (Min et al., ) used an occupancy world model for pre-training, requiring costly annotations. LAW (Li et al., ) introduced a latent world model to improve E2EAD training. In our work, we present a BEV world model tailored for our sparse scene representation, effectively encoding temporal context to boost E2EAD performance.

## 3 Method

Figure: Figure 3: Overview of SSR: SSR consists of two parts: the purple part, which is used during both training and inference, and the gray part, which is only used during training. In the purple part, the dense BEV feature is first compressed by the Scenes TokenLearner into sparse queries, which are then used for planning via cross-attention. In the gray part, the predicted BEV feature is obtained from the BEV world model. The future BEV feature is then used to supervise the predicted BEV feature, enhancing both the scene representation and the planning decoder.
Refer to caption: /html/2409.18341/assets/figs/SSR-V2.png

### 3.1 Overview

Problem Formulation: At timestamp $t$, given surrounding N-views camera images ${\mathbf{I}}_{t}=[{\mathbf{I}}_{t}^{i}]_{i=1}^{N}$ and a high-level navigation command $cmd$, the vision-based E2EAD model aims to predict the planning trajectory ${\mathbf{T}}$, which consists of a set of points in BEV space.

BEV Feature Construction: As shown in Fig. [3](#S3.F3), N-views camera images ${\mathbf{I}}_{t}$ are processed by a BEV encoder to generate the BEV feature. In the BEV encoder such as BEVFormer (Li et al., ), ${\mathbf{I}}_{t}$ is first processed by an image backbone to obtain image features ${\mathbf{F}}_{t}=[{\mathbf{F}}_{t}^{i}]_{i=1}^{N}$. A BEV query ${\mathbf{Q}}\in\mathbb{R}^{H\times W\times C}$ is then used to query temporal information from the previous frame’s BEV feature ${\mathbf{B}}_{t-1}$ and spatial information from ${\mathbf{F}}_{t}$ by cross-attention iteratively, resulting in the current BEV feature ${\mathbf{B}}_{t}\in\mathbb{R}^{H\times W\times C}$. Here, $H\times W$ represents the spatial dimensions of the BEV feature, and $C$ denotes the feature’s channel dimension.

$$ $\displaystyle{\mathbf{Q}}=CrossAttention({\mathbf{Q}},{\mathbf{B}}_{t-1},{\mathbf{B}}_{t-1}),$ (1) $\displaystyle{\mathbf{B}}_{t}=CrossAttention({\mathbf{Q}},{\mathbf{F}}_{t},{\mathbf{F}}_{t}).$ (2) $$

The key component of our framework is the novel Scenes TokenLearner module to extract crucial scene information, introduced in Sec. [3.2](#S3.SS2). Unlike traditional methods that rely on dense BEV features or hundreds of queries, our approach uses a small set of tokens to effectively represent the scene. Leveraging these sparse scene tokens, we generate the planning trajectory, a process detailed in Sec. [3.3](#S3.SS3). Additionally, in Sec. [3.4](#S3.SS4), we introduce an augmented BEV world model designed to further enhance the scene representation through self-supervised learning of BEV features.

Figure: Figure 4: Structure of the Proposed Modules: Scenes TokenLearner and BEV World Model.
Refer to caption: /html/2409.18341/assets/figs/STL+BWM-V2.png

### 3.2 Navigation-Guided Scenes Token Learner

BEV features are a popular scene representation as they contain rich perception information. However, this dense representation increases the inference time when searching for relevant perception elements. To address this, we introduce a sparse scene representation using adaptive spatial attention, significantly reducing computational load while maintaining high-fidelity scene understanding.

Specifically, we propose the Scenes TokenLearner (STL) module to extract scene queries ${\mathbf{S}}_{t}=[{\mathbf{s}}_{i}]_{i=1}^{N_{s}}\in\mathbb{R}^{N_{s}\times C}$ from the BEV feature, where $N_{s}$ is the number of scene queries, to efficiently represent the scene. The structure of Scenes TokenLearner is illustrated in Fig. [4](#S3.F4). To better focus on scene information related to our navigation intent, we adopt a Squeeze-and-Excitation (SE) layer (Hu et al., ) to encode the navigation command cmd into the dense BEV feature, producing the navigation-aware BEV feature ${\mathbf{B}}_{t}^{navi}$:

$$ ${\mathbf{B}}_{t}^{navi}=SE({\mathbf{B}}_{t},cmd).$ (3) $$

The navigation-aware BEV feature is then passed into the BEV TokenLearner (Ryoo et al., ) module $TL_{BEV}$ to adaptively focus on the most important information. Unlike previous applications of TokenLearner in image or video domains, we utilize it in BEV space to derive a sparse scene representation via spatial attention:

$$ ${\mathbf{S}}_{t}=TL_{BEV}({\mathbf{B}}_{t}^{navi}).$ (4) $$

For each scene query ${\mathbf{s}}_{i}$, we adopt a tokenizer function $M_{i}$ that maps ${\mathbf{B}}_{t}^{navi}$ into a token vector: $\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{C}$. The tokenizer predicts spatial attention maps of shape $H\times W\times 1$, and the learned scene tokens are obtained through global average pooling:

$$ ${\mathbf{s}}_{i}=M_{i}({\mathbf{B}}_{t}^{navi})=\rho({\mathbf{B}}_{t}^{navi}\odot\varpi_{i}({\mathbf{B}}_{t}^{navi})),$ (5) $$

where $\varpi(\cdot)$ is the spatial attention function and $\rho(\cdot)$ is the global average pooling function. The multi-layer self-attention (Vaswani et al., ) is applied to further enhance the scene queries:

$$ ${\mathbf{S}}_{t}=SelfAttention({\mathbf{S}}_{t}).$ (6) $$

### 3.3 Planning based on sparse scene representation

Since ${\mathbf{S}}_{t}$ contains all relevant perception information, we use a set of way point queries ${\mathbf{W}}_{t}\in\mathbb{R}^{N_{m}\times N_{t}\times C}$ to extract multi-modal planning trajectories, where $N_{t}$ is the number of future timestamps and $N_{m}$ denotes the number of driving commands.

$$ ${\mathbf{W}}_{t}=CrossAttention({\mathbf{W}}_{t},{\mathbf{S}}_{t},{\mathbf{S}}_{t}).$ (7) $$

We then obtain the predicted trajectory from ${\mathbf{W}}_{t}$ using a multi-layer perceptron (MLP), and select output trajectory ${\mathbf{T}}\in\mathbb{R}^{N_{t}\times 2}$ based on the navigation command $cmd$:

$$ ${\mathbf{T}}=Select(MLP({\mathbf{W}}_{t}),cmd).$ (8) $$

The output trajectory is supervised by the ground truth (GT) trajectory ${\mathbf{T}}_{GT}$ using L1 loss, defined as the imitation loss $\mathcal{L}_{imi}$:

$$ $\mathcal{L}_{imi}=\|{\mathbf{T}}_{GT}-{\mathbf{T}}\|_{1}.$ (9) $$

### 3.4 Temporal Enhancement by BEV World Model

We prioritize temporal context to enhance scene representation by world model instead of perception sub-tasks. The motivation behind this module is straightforward: if our predicted actions correspond to real actions, the predicted future scenes should closely resemble the actual future scenes.

As illustrated in Fig. [4](#S3.F4), we introduce the BEV world model (BWM) to predict future BEV features. First, we use the output trajectory ${\mathbf{T}}$ to translate the current scene queries into the future frame using a Motion aware Layer Normalization (MLN) (Wang et al., ). The MLN module helps current scene queries encode motion information, producing dreaming queries ${\mathbf{D}}_{t}$:

$$ ${\mathbf{D}}_{t}=MLN({\mathbf{S}}_{t},{\mathbf{T}}).$ (10) $$

For the dreaming queries ${\mathbf{D}}_{t}$, we apply multi-layer self attention as the world model to predict the future scene queries $\hat{{\mathbf{S}}}_{t+1}$:

$$ $\hat{{\mathbf{S}}}_{t+1}=SelfAttention({\mathbf{D}}_{t}).$ (11) $$

However, since the autonomous driving system may focus on different regions even in consecutive frames, we do not directly supervise the predicted scene queries $\hat{{\mathbf{S}}}_{t+1}$ with the future scene queries ${\mathbf{S}}_{t+1}$. Instead, we reconstruct the dense BEV feature $\hat{{\mathbf{B}}}_{t+1}$ using TokenFuser (Ryoo et al., ):

$$ $\displaystyle\hat{{\mathbf{B}}}_{t+1}$ $\displaystyle=TokenFuser(\hat{{\mathbf{S}}}_{t+1},{\mathbf{B}}_{t})$ (12) $\displaystyle=\psi({\mathbf{B}}_{t})\otimes\hat{{\mathbf{S}}}_{t+1},$ (13) $$

where $\psi(\cdot)$ is a simple MLP with the sigmoid function to remap the BEV feature ${\mathbf{B}}_{t}$ to a weight tensor: $\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W\times N_{s}}$. After the multiplication $\otimes$ with $\hat{{\mathbf{S}}}_{t+1}\in\mathbb{R}^{N_{s}\times C}$, we obtain the predicted dense BEV feature $\hat{{\mathbf{B}}}_{t+1}\in\mathbb{R}^{H\times W\times C}$. This process aims to recover the BEV feature from the predicted scene queries for further self-supervision.

Finally, we supervise $\hat{{\mathbf{B}}}_{t+1}$ using an L2 loss with the real future BEV feature ${\mathbf{B}}_{t+1}$ generated by future surrounding images. This is defined as the BEV reconstruction loss $\mathcal{L}_{bev}$:

$$ $\mathcal{L}_{bev}=\|\hat{{\mathbf{B}}}_{t+1}-{\mathbf{B}}_{t+1}\|_{2}.$ (14) $$

In summary, we apply imitation loss $\mathcal{L}_{imi}$ for the predicted trajectory, and BEV reconstruction loss $\mathcal{L}_{bev}$ for the predicted BEV feature. The total loss of SSR is:

$$ $\mathcal{L}_{total}=\mathcal{L}_{imi}+\mathcal{L}_{bev}.$ (15) $$

**Table 1: Comparison of state-of-the-art methods on the nuScenes dataset. The ego status was not utilized in the planning module. $\diamond$: Lidar-based methods. $\ast$: Backbone with ResNet-101 (He et al., ), while others use ResNet-50 or similar. $\dagger$: FPS measured on an NVIDIA A100 GPU, while others were tested on an NVIDIA RTX 3090. ${\ddagger}$: $\rm{AVG}$ metric protocal as same as VAD.**
| Method | Auxiliary Task | L2 (m) $\downarrow$ | Collision Rate (%) $\downarrow$ | FPS |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. |  |  |  |
| NMP$\diamond$ Zeng et al. () | Det & Motion | 0.53 | 1.25 | 2.67 | 1.48 | 0.04 | 0.12 | 0.87 | 0.34 | - |
| FF$\diamond$ Hu et al. () | FreeSpace | 0.55 | 1.20 | 2.54 | 1.43 | 0.06 | 0.17 | 1.07 | 0.43 | - |
| EO$\diamond$ Khurana et al. () | FreeSpace | 0.67 | 1.36 | 2.78 | 1.60 | 0.04 | 0.09 | 0.88 | 0.33 | - |
| ST-P3 Hu et al. () | Det & Map & Depth | 1.72 | 3.26 | 4.86 | 3.28 | 0.44 | 1.08 | 3.01 | 1.51 | 1.6 |
| UniAD$\ast$ Hu et al. () | Det&Track&Map&Motion&Occ | 0.48 | 0.96 | 1.65 | 1.03 | 0.05 | 0.17 | 0.71 | 0.31 | 1.8† |
| OccNet$\ast$ Sima et al. () | Det & Map & Occ | 1.29 | 2.13 | 2.99 | 2.14 | 0.21 | 0.59 | 1.37 | 0.72 | 2.6 |
| VAD-Base Jiang et al. () | Det & Map & Motion | 0.54 | 1.15 | 1.98 | 1.22 | 0.04 | 0.39 | 1.17 | 0.53 | 4.5 |
| PARA-Drive Weng et al. () | Det&Track&Map&Motion&Occ | 0.40 | 0.77 | 1.31 | 0.83 | 0.07 | 0.25 | 0.60 | 0.30 | 5.0 |
| GenAD Zheng et al. () | Det & Map & Motion | 0.36 | 0.83 | 1.55 | 0.91 | 0.06 | 0.23 | 1.00 | 0.43 | 6.7 |
| UAD-Tiny Guo et al. () | Det | 0.47 | 0.99 | 1.71 | 1.06 | 0.08 | 0.39 | 0.90 | 0.46 | 18.9† |
| UAD$\ast$ Guo et al. () | Det | 0.39 | 0.81 | 1.50 | 0.90 | 0.01 | 0.12 | 0.43 | 0.19 | 7.2† |
| SSR (Ours) | None | 0.24 | 0.65 | 1.36 | 0.75 | 0.00 | 0.10 | 0.36 | 0.15 | 19.6 |
| ST-P3${\ddagger}$ Hu et al. () | Det & Map & Depth | 1.33 | 2.11 | 2.90 | 2.11 | 0.23 | 0.62 | 1.27 | 0.71 | 1.6 |
| UniAD$\ast$${\ddagger}$ Hu et al. () | Det&Track&Map&Motion&Occ | 0.44 | 0.67 | 0.96 | 0.69 | 0.04 | 0.08 | 0.23 | 0.12 | 1.8† |
| VAD-Tiny${\ddagger}$ Jiang et al. () | Det & Map & Motion | 0.46 | 0.76 | 1.12 | 0.78 | 0.21 | 0.35 | 0.58 | 0.38 | 16.8 |
| VAD-Base${\ddagger}$ Jiang et al. () | Det & Map & Motion | 0.41 | 0.70 | 1.05 | 0.72 | 0.07 | 0.17 | 0.41 | 0.22 | 4.5 |
| BEV-Planner${\ddagger}$ Li et al. () | None | 0.28 | 0.42 | 0.68 | 0.46 | 0.04 | 0.37 | 1.07 | 0.49 | - |
| PARA-Drive${\ddagger}$ Weng et al. () | Det&Track&Map&Motion&Occ | 0.25 | 0.46 | 0.74 | 0.48 | 0.14 | 0.23 | 0.39 | 0.25 | 5.0 |
| LAW${\ddagger}$ Li et al. () | None | 0.26 | 0.57 | 1.01 | 0.61 | 0.14 | 0.21 | 0.54 | 0.30 | 19.5 |
| GenAD${\ddagger}$ Zheng et al. () | Det & Map & Motion | 0.28 | 0.49 | 0.78 | 0.52 | 0.08 | 0.14 | 0.34 | 0.19 | 6.7 |
| SparseDrive${\ddagger}$ Sun et al. () | Det & Track & Map & Motion | 0.29 | 0.58 | 0.96 | 0.61 | 0.01 | 0.05 | 0.18 | 0.08 | 9.0 |
| UAD$\ast$${\ddagger}$ Guo et al. () | Det | 0.28 | 0.41 | 0.65 | 0.45 | 0.01 | 0.03 | 0.14 | 0.06 | 7.2† |
| SSR${\ddagger}$ (Ours) | None | 0.18 | 0.36 | 0.63 | 0.39 | 0.01 | 0.04 | 0.12 | 0.06 | 19.6 |

## 4 Experiments

### 4.1 Dataset and Metric

We evaluate the proposed SSR framework for autonomous driving using the widely adopted nuScenes dataset (Caesar et al., ), following prior works (Hu et al., ; Jiang et al., ). To assess planning performance, we use displacement error and collision rate (CR), as in previous studies. Displacement error is calculated by L2 error with respect to the ground truth trajectory, measuring the quality of predicted trajectory. Collision rate quantifies the percentage of collisions with other objects when following the predicted trajectory. In addition, to better evaluate the framework in ablation studies, we measure Curb Collision Rate (CCR) as introduced in Li et al. (). All metrics are calculated in 3s future horizon with a 0.5s interval and evaluated at 1s, 2s and 3s.

We observe that VAD (Jiang et al., ) and UniAD (Hu et al., ) utilize different evaluation methods to calculate results across all predicted frames. VAD computes the average across all previous frames, while UniAD uses the latest result as well as the maximum value. Additionally, UniAD excludes pedestrians from the GT occupancy map, resulting in lower collision rates. We denote the VAD approach with the subscript $\rm{AVG}$ and the UniAD approach with $\rm{MAX}$. For example, the L2 error at frame $t$ (maximum $3s/0.5s=6$) is calculated as $L2_{AVG}^{t}=\frac{1}{t}\sum^{t}_{i=1}L2^{i}$ for VAD and $L2_{MAX}^{t}=L2^{t}$ for UniAD.

We apply $\rm{MAX}$ metric by default but also calculate $\rm{AVG}$ metric for comparison with other methods in Tab. [1](#S3.T1). In our $\rm{MAX}$ metric, pedestrians are included in the calculation of the collision rate. We also adjust the BEV resolution from 0.5m to 0.1m when calculating CCR following Li et al. ().

### 4.2 Implementation Details

Parameters We built up SSR on VAD (Jiang et al., ) and follow the setting of VAD-Tiny. We adopt ResNet-50 (He et al., ) as image backbone operating at an image resolution of 640 $\times$ 360. BEVFormer(Li et al., ) is used as BEV encoder following previous works (Hu et al., ; Jiang et al., ; Zheng et al., ). The BEV representation resolution is 100 $\times$ 100 and then compress into 16 sparse scene tokens. Number of navigation commands is 3 same as prior works.

Training Our model is trained for 12 epochs on 8 NVIDIA RTX 3090 GPUs with a batch size of 1 per GPU. The training phase costs about 11 hours which is 13$\times$ faster than UniAD. We utilize the AdamW (Loshchilov & Hutter, ) optimizer with a learning rate set to 5$\times 10^{-5}$. The weight of imitation loss and BEV loss is both 1.0. Other settings follow VAD-Tiny unless otherwise specified.

### 4.3 Main Result

Our method outperforms existing E2EAD approaches, achieving superior results in both L2 error and collision rate, as shown in Tab. [1](#S3.T1). For the well-known method UniAD, which employs most auxiliary tasks, our method reduces 0.28m (27.2% relatively) average $L2_{MAX}$ error and 0.16% (51.6% relatively) average $CR_{MAX}$ without any auxiliary tasks. When compared to our baseline method VAD-Tiny, SSR not only reduces 0.39m (50.0% relatively) average $L2_{AVG}$ error and 0.46% (79.3% relatively) average $CR_{AVG}$, but also outperforms the VAD-Base with an obvious margin (45.8% average $L2_{AVG}$ and 70.7% average $CR_{AVG}$ reducation relatively). Furthermore, our method demonstrates real-time efficiency, achieving 19.6 FPS (latency analysis in Appendix [A.4](#A1.SS4)), which is 10.9$\times$ faster than UniAD and 4.3$\times$ faster than VAD-Base. Remarkably, it is also 2.2$\times$ faster than the prior sparse work SparseDrive (Sun et al., ), while reducing the average $L2_{AVG}$ error by 0.22m.

When compared to previous approaches that eliminate auxiliary annotations, SSR demonstrates impressive performance across all metrics. LAW (Li et al., ), for instance, achieves a similar inference speed to SSR but retains a substantial gap in both L2 error and collision rate. The method closest in performance to ours is UAD (Guo et al., ), using a larger ResNet-101 backbone and a 1600 $\times$ 900 image resolution input, and requires an additional open-set 2D detector to supervise objectness information. Despite these additional resources, UAD still shows a 0.15m higher average $L2_{MAX}$ error compared to SSR, along with a 2.7× lower inference speed.

**Table 2: Ablation on the nuScenes dataset. STL: Scenes TokenLearner. BWM: BEV world model.**
| Modules | L2 (m) $\downarrow$ | CR (%) $\downarrow$ |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STL | BWM | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. |
|  |  | 0.23 | 0.65 | 1.41 | 0.76 | 0.04 | 0.58 | 0.66 | 0.43 |
| $\checkmark$ |  | 0.23 | 0.64 | 1.39 | 0.75 | 0.02 | 0.10 | 0.47 | 0.20 |
| $\checkmark$ | $\checkmark$ | 0.24 | 0.65 | 1.36 | 0.75 | 0.00 | 0.10 | 0.36 | 0.15 |

**Table 3: Ablation of navigation guidance. GS means go straight and LR denotes turn left / right.**
| Navigation | L2-GS (m) $\downarrow$ | L2-LR (m) $\downarrow$ | CR-GS (%) $\downarrow$ | CR-LR (%) $\downarrow$ |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Guidance | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. |
| $\times$ | 0.24 | 0.61 | 1.31 | 0.72 | 0.34 | 0.96 | 1.98 | 1.09 | 0.09 | 0.38 | 0.40 | 0.29 | 0.00 | 0.44 | 1.90 | 0.78 |
| $\checkmark$ | 0.23 | 0.61 | 1.28 | 0.71 | 0.33 | 0.91 | 1.88 | 1.04 | 0.00 | 0.08 | 0.18 | 0.10 | 0.00 | 0.29 | 1.70 | 0.66 |

### 4.4 Ablation Study

#### 4.4.1 Component-wise ablation

In Tab. [2](#S4.T2)(a), we present an ablation study on the proposed components. When the STL is enabled instead of directly interacting waypoint queries with the BEV feature, the collision rate is reduced by more than half. This significant decrease underscores the STL’s ability to effectively distill critical information from the dense scene data, thereby minimizing the impact of irrelevant features and reducing computational redundancy. Furthermore, when incorporating the BEV world model, we observe a further reduction in the average collision rate to 0.15%. This improvement highlights the BEV world model’s role in enhancing SSR’s comprehension of scene dynamics, contributing to more safe trajectory planning and overall performance gains.

#### 4.4.2 Number of Scene Queries

We evaluate the impact of varying the number of scene queries in Tab. [2](#S4.T2)(b). For L2 error, using 8 queries yields the best performance, with performance declining as the number of queries increases. Interestingly, when considering the collision rate, the optimal performance is achieved with 16 queries. Therefore, we select 16 queries as the default setting in SSR to strike a balance between minimizing L2 error and reducing collision rate. The poor performance observed with 64 scene queries suggests that an excessive number of queries may overwhelm the model with too much perception information, leading to confusion similar to directly interacting with dense BEV features.

Figure: Scene Query 0
Refer to caption: /html/2409.18341/assets/figs/10_296fcfbf2e29489699f1cb5631f38ff5_2_vis_0.png

### 4.5 Analysis and Discussion

How does scene queries represent the scene? To understand why only a handful of queries can effectively represent the entire scene and even outperform more complex designs, we visualize 8 out of the 16 BEV spatial attention maps $\varpi({\mathbf{B}}^{navi}_{t})$ from the STL module in Fig. [6](#S4.F6). The results reveal that each query focuses uniformly on a distinct region of the BEV space, with different queries attending to different areas. When summing the attention maps of all scene queries, we observe that the sum-attention map surprisingly covers the entire scene in a balanced manner, with greater emphasis on the front region than the back. Furthermore, the attention maps remain relatively consistent across different frames, as shown in Fig. [6](#S4.F6), indicating that $\varpi({\mathbf{B}}^{navi}_{t})$ offers stable spatial compression guidance for SSR. In essence, the scene queries act as a compressed representation of dense BEV features, where each query concentrates on a specific spatial region.

What does scene queries learn? In Fig. [8](#S4.F8), we visualize the navigation-aware BEV features and the activation positions of the scene queries across different scenarios. When overtaking a vehicle on the left as shown in Fig. [8](#S4.F8)(a), the activation positions primarily focus on the overtaked vehicle and the left rear area, anticipating potential risks. In a straightforward driving scenario (Fig. [8](#S4.F8)(b)), the scene queries are more dispersed, with attention directed towards a front-right vehicle, potentially anticipating a cut-in. During a right turn at an intersection (Fig. [8](#S4.F8)(c)), the scene queries not only activate around the right rear vehicle but also pay attention to the left crosswalk, where pedestrians might appear. When we further compare the same command in different scenes between Fig. [8](#S4.F8)(b) and Fig. [8](#S4.F8)(b), the scene queries focus on different regions adaptively. These observations demonstrate how our sparse scene queries can understand various scenes and manage different driving scenarios.

Figure: (a) Turn Left
Refer to caption: /html/2409.18341/assets/figs/127_7c3248a9f77341d2ba915b857bd8f287_1_vis.png

How does navigation information work? In Tab. [3](#S4.T3), We conducted experiments to assess the effect of navigation commands in different scenarios, demonstrating that navigation guidance improves planning results across all cases. We further tested different commands within the STL module for the same frame at an intersection, and visualized the corresponding scene queries in Fig. [8](#S4.F8). Compared to the original command go straight, when the command is changed to turn left, the module shifts its attention to pedestrians on the left. Similarly, with the turn right command, the STL module increases its focus on the front-right area, particularly highlighting a vehicle on the right that is not prominent in the other navigation scenarios. These findings demonstrate that navigation commands effectively guide the STL module to extract relevant information from the scene.

To summarize above discussions, we revisit the generation of scene query in Eq. [5](#S3.E5), where ${\mathbf{B}}_{t}^{navi}$ encodes the navigation information, and $\varpi_{i}({\mathbf{B}}_{t}^{navi})$ is responsible for spatial compression:

$$ ${\mathbf{s}}_{i}=\rho(\underbrace{{\mathbf{B}}_{t}^{navi}}_{NaviGuidance}\odot\underbrace{\varpi_{i}({\mathbf{B}}_{t}^{navi})}_{SpatialCompression}).$ (16) $$

**Table 4: Perception tasks ablation. CCR calculated in 0.1m resolution following Li et al. ().**
| Auxiliary Task | L2 (m) $\downarrow$ | CR (%) $\downarrow$ | CCR (%) $\downarrow$ |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Map | Obs | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. |
|  |  | 0.24 | 0.65 | 1.36 | 0.75 | 0.00 | 0.10 | 0.36 | 0.15 | 0.19 | 1.01 | 2.71 | 1.30 |
| $\checkmark$ |  | 0.29 | 0.71 | 1.44 | 0.81 | 0.13 | 0.16 | 0.68 | 0.32 | 0.20 | 0.90 | 2.85 | 1.32 |
|  | $\checkmark$ | 0.30 | 0.70 | 1.37 | 0.79 | 0.06 | 0.31 | 0.68 | 0.35 | 0.25 | 1.25 | 3.40 | 1.63 |
| $\checkmark$ | $\checkmark$ | 0.33 | 0.77 | 1.50 | 0.86 | 0.02 | 0.16 | 0.59 | 0.26 | 0.21 | 1.13 | 2.87 | 1.40 |

Can SSR learn sufficient perception information for AD Task? Although our framework eliminates the need for perception annotations, we investigate the effect of incorporating perception branches in Tab. [4](#S4.T4). We follow approach in PARA-Drive (Weng et al., ) to integrate auxiliary tasks into SSR, using them to supervise the BEV feature in parallel with the planning task. The results indicate that even without a supervised object detection module, our method achieves a lower collision rate. Similarly, the CCR is even lower when SSR operates without a map branch. These findings suggest that SSR can effectively learn the necessary perception information for autonomous driving tasks without explicit supervision, maintaining high performance across all metrics.

### 4.6 Visualization

Figure: Figure 9: Visualization of Planning Results. The perception results are rendered from annotations.
Refer to caption: /html/2409.18341/assets/figs/456_af0f72a7e15c481ab2bb87a561b00241_1_vis.png

In Fig. [9](#S4.F9), we present a qualitative result of SSR on planning trajectories, demonstrating strong alignment with the ground truth compared to VAD-Base. Additional visualizations across various scenes, including failure cases, can be found in the Appendix [A.2](#A1.SS2) and [A.3](#A1.SS3) due to space limitations.

## 5 Conclusion

The SSR framework presents a significant advancement in the field of E2EAD by challenging the conventional reliance on perception tasks. By utilizing navigation-guided sparse tokens and temporal self-supervision, SSR addresses the limitations of perception-heavy architectures, achieving state-of-the-art performance with minimal costs.
Moreover, visualization of the sparse tokens enhances the interpretability and transparency of SSR’s navigation-guided process. We hope SSR can provide a strong foundation for scalable, interpretable, and efficient autonomous driving systems.

Limitations and Future Work. Despite these advances, current simple navigation commands may constrain SSR’s effectiveness in more complex driving scenarios. Future work will explore integrating more sophisticated navigation prompts, such as routing and natural language.

## Appendix A Appendix

### A.1 Visualization for All Attention Map of Scene Queries

We visualize the attention maps of all 16 scene queries used in Fig. [10](#A1.F10). It can be observed that each scene query captures information from different spatial regions with a uniform attention distribution. When these queries are combined to represent the entire BEV feature, it effectively compresses the spatial space while applying different attention weights.

Figure: Scene Query 0
Refer to caption: /html/2409.18341/assets/figs/10_296fcfbf2e29489699f1cb5631f38ff5_2_vis_0.png

### A.2 Qualitative Results in Different Scenes

As illustrated in Fig. [11](#A1.F11), we visualize additional planning trajectories of SSR across various scenes. In Fig. [11](#A1.F11)(a), SSR achieves an even smoother result than the ground truth when turning left. When following a bus through an intersection with the go straight command, as shown in Fig. [11](#A1.F11)(b), our method also outperforms the baseline method, VAD, by producing a better trajectory. Similarly, in the turn right scenario illustrated in Fig. [11](#A1.F11)(c) , SSR demonstrates superior performance compared to VAD, achieving a smaller turning radius. These results demonstrate that SSR effectively understands the scene and produces accurate planning results through imitation learning. Even without explicit perception supervision, the learned scene queries capture the essential elements of complex scenes for autonomous driving.

Figure: (a) Turn Left
Refer to caption: /html/2409.18341/assets/figs/743_56b2752879d247359f620173a94838c3_1_vis.png

### A.3 Failure Cases

Figure: (a) Failure Reason: Noisy First Frame for Temporal Module
Refer to caption: /html/2409.18341/assets/figs/475_88a090c54cc844cc878f00d08274a683_1_vis.png

We identified some failure cases for SSR, which are visualized in Fig. [12](#A1.F12), highlighting the two most common reasons. The first common issue is noise in the initial frame of a clip for temporal module, which is also discussed in Weng et al. (). Due to zero initialization of input features and the ego vehicle’s state, the temporal module of E2EAD methods can be negatively impacted, leading to inferior performance, as illustrated in Fig. [12](#A1.F12)(a). The second issue arises from ambiguous navigation commands in the label generation of nuScenes dataset. For example in Fig. [12](#A1.F12)(b), the command is to go straight, but the ground truth trajectory involves turning left to change lanes. Since SSR heavily relies on navigation commands, it can be adversely affected when encountering ambiguous instructions. However, these reasons which heavily related to dataset can be easily avoided in real world scenarios by integrating true navigation command and run in real-time image sequences.

### A.4 Latency Analysis

Figure: Figure 13: Latency Analysis.
Refer to caption: /html/2409.18341/assets/figs/breakdown.png

Inference latency analysis of SSR components is presented in Fig. [13](#A1.F13). The evaluation was conducted on an NVIDIA GeForce RTX 3090 GPU with a batch size of 1. The image backbone and encoder, responsible for generating dense BEV features, contribute to 90.7% of the total latency. In contrast, our proposed Scenes TokenLearner incurs only 7.8% of the latency, highlighting its efficiency in extracting useful information from massive dense BEV feature. The planning decoder, which interacts way point queries with the scene queries and output final planning trajectory, adds just 1.5% to the latency, as SSR only utilizes 16 tokens to represent the scenes.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: What is the channel dimension of the scene queries?

How would the model respond in a very dense scene? Is there an inherent limitation in the number of objects / components that the model can focus on given a fixed number of scene queries?

What is the quality of the perception outputs in Table 4?
soundness: 3
strengths: The main novel contribution of this work lies in the use of the TokenLearner for BEV feature compression. From the experiments, the authors demonstrate that they are able to achieve new SOTA trajectory prediction results without any intermediate perception signals used by previous works. Ablations are also performed demonstrating the improvements provided by both the future prediction auxiliary task as well as the navigation command cross attention into the BEV feature space. Visualizations are also provided demonstrating the the model's attention mechanism does seem to focus on reasonable parts of the scene given a navigation command (i.e. it's possible that the model is not directly overfitting in some way).

Overall the paper is well written and easy to understand, and the overall method seems reasonable and based on sound prior works.
summary: This paper presents a novel method for end-to-end trajectory prediction for autonomous driving, with a model that takes as input surround view camera images, and outputs a predicted trajectory for the vehicle. The main novelties are:
- the use of a sparse tokenizer to reduce the typical dense BEV representation into a sparse (in experiments 16) set of feature vectors.
- the introduction of the driving command into the BEV representation via cross attention.
- the application of a future prediction module for the BEV features as an auxiliary loss.
The proposed method aims to reduce the dimensionality of the BEV space without using pre-defined intermediate signals such as detections, mapping etc. This allows the model to be end-to-end differentiable, with the tokenization learned as a part of the driving task. 

Experiments and ablations are performed on the nuScenes dataset, where the proposed method produces a new SOTA in terms of both quality and latency.
weaknesses: The number of optimal scene queries selected by the authors seems surprisingly small, and may be skewed by the difficulty of the average scene encountered in nuScenes. From the provided visualizations, it looks like the model focuses directly on relevant agents and objects in the scene. However, it's unclear whether the model would be able to extend to extremely dense scenes, where a fixed number of scene queries may limit the models ability to handle many other agents / objects.

In Table 4, the authors ablate the addition of a perception module. However, no results for the perception tasks themselves are provided, and so it cannot be determined whether these tasks were actually trained appropriately (i.e. they may have simply diverged altogether).

One additional factor to call out is the inherent risk of the nuScenes dataset being skewed in some way (i.e. Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes), but this is not the fault of the authors.

In Figures 7 and 8, it would be helpful to explain the colormap.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. There’s a paper “Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes” (https://arxiv.org/abs/2305.10430) claims that a simple MLP model that takes simple ego state (velocity, acceleration, historical trajectory) can also outperform VAD and UniAD. While their claim is that the metrics may not adequately capture the superiority of different methods, in combination with table 2.(b) in this paper being reviewed, where a larger number of scene queries leads to worse performance, is this an indication of overfitting of the larger models (or generally speaking, a case where a larger model is being overperformed by a smaller model due to various reasons)? Considering the trajectories in NuScenes are relatively simple, will this weaken the conclusion of this paper, especially when the navigation tasks become more complex?
2. Why the visualization in Figure. 5 in a hexagon format instead of square grids?
soundness: 3
strengths: The paper is well written, with clear and insightful motivation. The proposed model architecture is novel, and the experimental results show significant improvement over existing methods. The experiment section is comprehensive, with insightful analysis.
In autonomous driving, multi-head models are very popular, for a number of reasons. Not only because different tasks can serve as auxiliary supervision for each other and contribute to the overall performance improvement, the output of the individual tasks can also be utilized by downstream tasks in a modular autonomous driving system (e.g. occupancy map can be the input of a traditional pathfinding algorithm for parking). However in a fully end-to-end autonomous driving system all those intermediate outputs are unnecessary. It’s interesting to see papers questioning the design choices of existing models and exploring alternative designs.
summary: This paper proposed a novel approach for trajectory prediction in autonomous driving called SSR. A very straight intuition is that the more comprehensive and precise the information one can extract, the better the downstream tasks will perform. Many recent models try to extract dense BEV features with rich scene information and route the information to multiple auxiliary tasks. On the contrary, the authors argue that for end-to-end autonomous driving, auxiliary supervision from perception tasks might be unnecessary, instead computational resources can be focused on crucial parts of the scene.
To achieve this, the paper introduced a scene token learner that learns sparse scene queries from BEV features extracted by a BEV encoder (BEVFormer in particular) and current navigation command. The trajectory is then predicted based on the scene queries. To train the model, two loss terms are used. The first term directly minimizes the L1 loss between the predicted and ground truth trajectory. The second term aims to enhance scene representation by introducing a BEV world model. This term replaces the perception sub-tasks in other models, with more focus on temporal context. The BEV world model takes the BEV features, scene queries and predicted trajectory as input and it tries to predict the BEV features at the next frame. We minimize the L2 distance between the output of the world model and the BEV features directly extracted by the BEV encoder at the next frame. The overall loss is the sum of these two terms.
The authors show that a small set of tokens is sufficient to outperform sota models in trajectory prediction on NuScenes dataset.
weaknesses: 1. The proposed architecture borrows the idea of a world model by predicting future bev features. However it seems that the BEV features are only regularized by the trajectory prediction task. There’s no reconstruction loss and other downstream tasks. In this case it’s questionable to still call it a world model.
2. The paper lacks theoretical analysis of why the proposed SSR method is better.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Are the navigation commands open-set or close-set, how many commands were used in this paper, how to pre-processing the high-level navigation commands so that the model can take them as inputs, etc. 
2. What is the performance of the model if the navigation command never appears in the training set (or confusing command, e.g., turn left and right, or turn left or right, etc.)?
soundness: 3
strengths: 1. The proposed solution sounds solid in theory. Specifically, eliminating the need for supervised sub-tasks, e.g., detection, mapping, motion, occupancy, etc., indeed speed up the training and inference time. The proposed navigation-guided concept and BEV world model should extract useful features for the autonomous driving. 
2. The proposed navigation-guided concept is inspring. As mentioned in the paper, it is a human-inspried solution whose intuition is to mimic the behavior of human beings. It is a good research direction. The authors propose a way to combine the information with the special features.
3. The ablation study is helpful. Readers can get more information from the ablation study results.
4. The analysis and discussion guide readers think more deeper about the model design and the performance.
summary: In this paper, the authors propose a navigation-guided sparse scene representation for end-to-end autonomous driving. The proposed solution eliminates the need for supervised sub-tasks, allowing computational resources to concentrate on essential elements directly related to navigation intent. Specifically, the proposed solution utilizes learned sparse query representations guided by navigation commands, and introduces a BEV world model for self-supervision on dynamic scene changes to highlight the critical role of temporal context in autonomous driving. The experimental results show that the proposed solution achieves great performance on nuSccenes dataset with minimal training and inference cost.
weaknesses: 1. There are some important information is missing. For example, detailed model structure, navigation commands pre-processing operation, etc. 
2. It will be better if the authors could report some failure cases. Especially for those cases which the predictions are totally opposite toward the navigation command.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Refer to the weakness
soundness: 3
strengths: The model proposed in this paper yields excellent results, achieving SOTA performance on the nuScenes dataset. Additionally, both the inference and training speeds are fast on 3090 GPUs.
summary: The paper introduces a novel SSR framework that requires no auxiliary task outputs or labels, relying instead on just 16 sparse navigation tokens for guidance. This approach achieves SOTA performance on the nuScenes dataset, with both inference and training processes being fast.
weaknesses: 1. From the ablation studies (Table 2), it can be seen that the core contribution of this paper is the introduction of navigation information as guidance. The addition of the STL and BWM modules does not significantly affect L2, which suggests that navigation information provides significant assistance for lane following, especially when driving straight. It is understandable that the introduction of the BWM module, which incorporates temporal modeling, improves collision rates. In other words, the key contribution of this paper lies in utilizing navigation information, which is not employed by other methods. This kind of approach is relatively common in the industry but is rarely mentioned in academia. Whether merely introducing navigation information as input is worthy of an ICLR paper is something I am somewhat conflicted about.
2. The STL and BWM modules lack novelty. Furthermore, I'm not particularly fond of naming the BWM module a "world model." Given that it primarily involves predicting the next frame in BEV, the term "world model" might be considered too broad. I'd like to reiterate that I think the core contribution of this paper is the introduction of navigation information.
3. The color scheme used in the figures could be slightly unappealing, with most modules and features using white as the primary color. It would be nicer if the visuals were more aesthetically pleasing.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: None
soundness: 3
strengths: 1. The elimination of middle task could indeed reduce the latency and annotation requirements.
2. The writing is clear
summary: This work proposes to extract sparse scene queries to represent the scene and use the scene queries to generate planning results.  Further, they propose an auxiliary task head and loss - predict future BEV feature.
weaknesses: 1. Lack of closed-loop evaluation:  as pointed in AD-MLP[1] and BEVPlanner[2], it is rather simple to fit nuScenes, as most of its data is going straight.  BEVPlanner only uses BEV feature, even without the need of the proposed scene queries.  (**Strangely, the authors cite [2] but do not compare with [2] in Table 1. I hope the authors could give an explanation**). Not to mention that AD-MLP even does not need any sensor inputs.  Thus,  **it is not convincing to claim that there is no need for middle supervision under nuScenes open-loop evaluation**. It is important to have closed-loop results and ablation studies on it.

[1] Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes.

[2] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? CVPR 2024

Due to the missing of closed-loop results, the argument in the paper is not convincing and thus I give a reject rating. I will increase my rating if closed-loop results and ablations are provided.

Meta-review: The paper proposes navigation-guided sparse scene representation for end-to-end autonomous driving. The motivation is based on the expensive annotations when considering data scalability for real-time production. The core idea is designated for eliminating human-designed sub-tasks. SSR, the proposed method, achieved remarkable results in both performance and efficiency, surpassing previous SOTA (UniAD, e.g.)

There are five reviews for this paper. All the reviewers agree the paper is of sufficient novelty, solid experiments, well-written and easy to follow. Authors did a great job addressing reviewers concern (e.g. adding closed-loop experiments on Carla) and the manuscript is very clear to be ready for ICLR. AC agrees with the consensus.

Decision: Accept (Poster)