Title: End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation
Venue: NeurIPS 2024
Accepted: no
Keywords: End-to-End Autonomous Driving

Unsupervised Pretext Task

Direction-Aware Planning
Abstract: We propose UAD, a method for vision-based end-to-end autonomous driving (E2EAD), achieving the best open-loop evaluation performance in nuScenes, meanwhile showing robust closed-loop driving quality in CARLA. Our motivation stems from the observation that current E2EAD models still mimic the modular architecture in typical driving stacks, with carefully designed supervised perception and prediction subtasks to provide environment information for oriented planning. Although achieving groundbreaking progress, such design has certain drawbacks: 1) preceding subtasks require massive high-quality 3D annotations as supervision, posing a significant impediment to scaling the training data; 2) each submodule entails substantial computation overhead in both training and inference. To this end, we propose UAD, an E2EAD framework with an unsupervised proxy to address all these issues. Firstly, we design a novel Angular Perception Pretext to eliminate the annotation requirement. The pretext models the driving scene by predicting the angular-wise spatial objectness and temporal dynamics, without manual annotation. Secondly, a self-supervised training strategy, which learns the consistency of the predicted trajectories under different augment views, is proposed to enhance the planning robustness in steering scenarios. Our UAD achieves 38.7% relative improvements over UniAD on the average collision rate in nuScenes and surpasses VAD for 6.40 points on the driving score in CARLA's Town05 Long benchmark. Moreover, the proposed method only consumes 44.3% training resources of UniAD and runs 3.4$\times$ faster in inference. Our innovative design not only for the first time demonstrates unarguable performance advantages over supervised counterparts, but also enjoys unprecedented efficiency in data, training, and inference.

Abstract

We propose UAD, a method for vision-based end-to-end autonomous driving (E2EAD), achieving the best open-loop evaluation performance in nuScenes, meanwhile showing robust closed-loop driving quality in CARLA. Our motivation stems from the observation that current E2EAD models still mimic the modular architecture in typical driving stacks, with carefully designed supervised perception and prediction subtasks to provide environment information for oriented planning. Although achieving groundbreaking progress, such design has certain drawbacks: 1) preceding subtasks require massive high-quality 3D annotations as supervision, posing a significant impediment to scaling the training data; 2) each submodule entails substantial computation overhead in both training and inference. To this end, we propose UAD, an E2EAD framework with an unsupervised 1 proxy to address all these issues. Firstly, we design a novel Angular Perception Pretext to eliminate the annotation requirement. The pretext models the driving scene by predicting the angular-wise spatial objectness and temporal dynamics, without manual annotation. Secondly, a self-supervised training strategy, which learns the consistency of the predicted trajectories under different augment views, is proposed to enhance the planning robustness in steering scenarios. Our UAD achieves 38.7% relative improvements over UniAD on the average collision rate in nuScenes and surpasses VAD for 6.40 points on the driving score in CARLA's Town05 Long benchmark. Moreover, the proposed method only consumes 44.3% training resources of UniAD and runs 3 . 4 × faster in inference. Our innovative design not only for the first time demonstrates unarguable performance advantages over supervised counterparts, but also enjoys unprecedented efficiency in data, training, and inference.

1 Introduction

Recent decades have witnessed breakthrough achievements in autonomous driving. The end-to25 end paradigm, which seeks to integrate perception, prediction, and planning tasks into a unified 26 framework, stands as a representative branch [33, 1, 39, 3, 35, 21, 22]. The latest advances in end-to27 end autonomous driving significantly piqued researchers' interest [21, 22]. However, handcrafted and 28 resource-intensive supervised sub-tasks for perception and prediction, which have previously proved 29 their utility in environment modeling [35, 3, 20], continue to be indispensable, as shown in Fig. 1a. 30

Then what insights have we gained from the recent advances? It has come to our attention that one of 31 the most enlightening innovations lies in the Transformer-based pipeline, in which the queries act 32 as a connective thread, seamlessly bridging various tasks. Besides, the capability for environment 33 modeling has also seen a significant boost, primarily due to complicated interactions of supervised 34

(a)

(b)

Input

Planner

Comparison

Tracking

Occupancy

1) Vanilla Design

Training

Cost

14.0%

41.7%

48.0 GPU Days

28.0 GPU Days

21.3 GPU Days

Input

Detection

Motion

Planner

Inference

44.4%

26.4%

2.1 FPS

UniAD

Speed

5.3 FPS

7.2 FPS

Mapping

VAD

一

2) Modularized Design

15.6%

1.03 m

Input

Unsupervised Pretext Task

Planner

L2 Error

Average

26.2%

0.90 m

1.22 m

3) Our Unsupervised Design

UniAD

VAD

Ours

sub-tasks. However, every coin has two sides. In comparison to the vanilla design [33] (see Fig. 1a), 35 modularized methods incur unavoidable computation and annotation overhead. As illustrated in 36 Fig. 1b, the training of the recent method UniAD [21] takes 48 GPU days while running at only 2.1 37 frames per second (FPS). Moreover, modules in existing perception and prediction design require large 38 quantities of high-quality annotated data. The financial overhead for human annotation significantly 39 impedes the scalability of such modularized methods with supervised subtasks to leverage massive 40 data. As proved by large foundation models [24, 31], scaling up the data volume is the key to bringing 41 the model capabilities to the next level. Thus we ask ourselves the question: Is it viable to devise an 42 efficient and robust E2EAD framework while alleviating the reliance on 3D annotation? 43

In this work, we show the answer is affirmative by proposing an innovative U nsupervised pretext task 44 for end-to-end A utonomous D riving ( UAD ), which seeks to efficiently model the environment. The 45 pretext task consists of an angular-wise perception module to learn spatial information by predicting 46 the objectness of each sector region in BEV space, and an angular-wise dreaming decoder to absorb 47 temporal knowledge by predicting inaccessible future states. The introduced angular queries link 48 the two modules as a whole pretext task to perceive the driving scene. Notably, our method shines 49 by completely eliminating the annotation requirement for perception and prediction. Such data 50 efficiency is not attainable for current methods with complex supervised modularization [21, 22]. The 51 supervision for learning spatial objectness is obtained by projecting the 2D region of interests (ROIs) 52 from an off-the-shelf open-set detector [28] to BEV space. While utilizing the publicly available open53 set 2D detector pre-trained with manual annotation from other domains ( e.g. COCO [27]), we avoid 54 the need for any additional 3D labels within our paradigm and target domains ( e.g. nuScenes [2] and 55 CARLA [11]), thereby creating a pragmatically unsupervised setting [30]. Furthermore, we introduce 56 a self-supervised direction-aware learning strategy to train the planning model. Specifically, the 57 visual observations are augmented with different rotation angles, and the consistency loss is applied 58 to the predictions for robust planning. Without bells and whistles, the proposed UAD outperforms 59 UniAD for 0.13m in nuScenes Avg . L2 error, and surpasses VAD [22] for 9.92 points in CARLA 60 route completion score. Such unprecedented performance gain is achieved with a 3 . 4 × inference 61 speed, a mere 44.3% training budget of UniAD, and zero annotations, as illustrated in Fig. 1b. 62

In summary, our contributions are as follows: 1) We propose an unsupervised pretext task to discard 63 the requirement of 3D manual annotation in end-to-end autonomous driving, potentially making 64 it more feasible to scale the training data to billions level without any labeling overload; 2) We 65 introduce a novel self-supervised direction-aware learning strategy to maximize the consistency of the 66 predicted trajectories under different augment views, which enhances planning robustness in steering 67 scenarios; 3) Our method shows superiority in both open- and closed-loop evaluation compared with 68 other vision-based E2EAD methods, with much lower computation and annotation cost. 69

2 Related Work 70

2.1 End-to-End Autonomous Driving 71

End-to-end autonomous driving can be dated back to 1988, when the ALVINN [33] proposed by 72 Carnegie Mellon University could successfully navigate a vehicle over 400 meters. After that, to 73

BEV

Encoder

BEV Feature

OS 2D

Detector

2D Boxes

Dreaming

Decoder

BEV Object Mask

Angular

Partition

Angular Objectness

Label

Angular BEV Feature

Angular Query

Perception

Predict

View

Transform

Objectness

Loss

Cross

Attention

Direction

Predict

Planning

Predict

Direction-Augmented

GT Ego Trajectory

Multi-view Images

Direction

Augment

Direction

Loss

Imitation

Loss

Turn

Right

Driving

Command

F

F

F

F

270

180

90

270

180

90

Angular Perception Pretext

Direction Aware Planning

K

K

improve the robustness of E2EAD, a series of modern approaches such as NEAT [6], P3 [35], 74 MP3 [3], ST-P3 [20] introduce the design of more dedicated modularization, which integrate auxiliary 75 information such as HD maps, and additional tasks like bird's-eye view (BEV) segmentation. Most re76 cently, embracing advanced architectures like Transfromer [37] and visual occupancy prediction [29], 77 UniAD [21] and VAD [22] demonstrate impressive performance in open-loop evaluation. In this work, 78 instead of integrating complex supervised modular sub-tasks, we innovatively propose another path 79 proving that an efficient unsupervised pretext task without any human annotation like 3D bounding 80 boxes and point cloud categories, can achieve even superior performance than recent state-of-the-arts. 81

82

2.2 World Model

In pursuit of understanding the dynamic changes in environments, researchers in the fields of gaming 83 and robotics have proposed various world models [13, 14, 15, 16]. Recently, the autonomous driving 84 community introduces world models for safer maneuvering [32, 18, 12, 38]. MILE [18] considers 85 the environment as a high-level embedding and tends to predict its future state with historical 86 observations. Drive-WM [38] proposes a framework to integrate world models with existing E2E 87 methods to improve planning robustness. In this work, we propose an auto-regressive mechanism, 88 tailored to our unsupervised pretext, to capture angular-wise temporal dynamics within each sector. 89

90

91

3 Method

3.1 Overview

As illustrated in Fig. 2, our UAD framework consists of two essential components: 1) the Angular 92 Perception Pretext, aims to liberate E2EAD from costly modularized tasks in an unsupervised fashion; 93 2) the Direction-Aware Planning, learns self-supervised consistency of the augmented trajectories. 94

Specifically, UAD first models the driving environment with the pretext. The spatial knowledge 95 is acquired by estimating the objectness of each sector region within the BEV space. The angular 96 queries, each responsible for a sector, are introduced to extract features and predict the objectness. 97 The supervision label is generated by projecting the 2D regions of interests (ROIs) to the BEV space, 98 which are predicted with an available open-set detector GroundingDINO [28]. This way not only 99 eliminates the 3D annotation requirement, but also greatly reduces the training budget. Moreover, as 100 driving is inherently a dynamic and continuous process, we thus propose an angular-wise dreaming 101 decoder to encode the temporal knowledge. The dreaming decoder can be viewed as an augmented 102 world model [13] capable of auto-regressively predicting the future states. 103

Subsequently, direction-aware planning is introduced to train the planning module. The raw BEV 104 feature is augmented with different rotation angles, yielding rotated BEV representations and ego 105 trajectories. We apply self-supervised consistency loss to the predicted trajectories of each augmented 106 view, which is expected to improve the robustness for directional change and input noises. The 107 learning strategy can also be regarded as a novel data augmentation technique customized for end-to108 end autonomous driving, which enhances the diversity of trajectory distribution. 109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

2D Boxes

BEV Object Mask

Multi-view Images

Multi-view Image Masks

Sampling

Detect

Angular Objectness

Label

K

(a)

Angular Query

Cross

Attention

μ

σ

Prior Distribution

GRU

Updated Angular Query

Distribution

Sampling

Posterior Distribution

Dreaming Loss

Update

Output

×Planning Steps

Angular BEV Feature

K

μ

σ

(b)

3.2 Angular Perception Pretext

Spatial Representation Learning. Our model attempts to acquire spatial knowledge of the driving scene by predicting the objectness of each sector region within the BEV space. Specifically, taking multi-view images { I i ∈ R H i × W i × 3 } as input, the BEV encoder [25] first extracts visual information into the BEV feature F b ∈ R H b × W b × C . Then, F b is partitioned into K sectors with a uniform angle θ centered around ego car. Each sector contains several feature points in BEV space. Denoting feature of a sector as f ∈ R N × C , where N is the maximum number of feature points in all sectors, we derive angular BEV feature F a ∈ R K × N × C . Zero-padding is applied on sectors with fewer than N points.

Then why do we partition the rectangular BEV feature to angular-wise formatting? The underlying reason is that, in the absence of depth information, the region in BEV space corresponding to an ROI in 2D image is a sector. As illustrated in Fig. 3a, by projecting 3D sampling points to images and verifying their presence in 2D ROIs, a BEV object mask M ∈ R H b × W b × 1 is generated, representing the objectness in BEV space. Specifically, the sampling points falling within 2D ROIs are set to 1, while the others are 0. It is noticed that the positive sectors are irregularly and sparsely distributed in BEV space. To make the objectness label more compact, similar to the BEV feature partition, we uniformly divide M into K equal parts. The segments overlapped with positive sectors are assigned with 1, constituting the angular objectness label Y obj ∈ R K × 1 . Thanks to the rapid development of open-set detection, it's now convenient to obtain 2D ROIs for the input multi-view images by feeding the pre-defined prompts ( e.g., vehicle, pedestrian, and barrier) to a 2D open-set detector like GroundingDINO [28]. Such design is the key in reducing annotation cost and scaling up the dataset.

To predict the objectness score of each sector, we define angular queries Q a ∈ R K × C to summarize F a . Each angular query q a ∈ R 1 × C in Q a will interact with corresponding f by cross attention [37],

Finally, we map Q a to the objectness scores P a ∈ R K × 1 with a linear layer, which is supervised by Y obj with binary cross-entropy loss (denoted as L spat ).

Temporal Representation Learning. We propose to capture the temporal information of driving scenarios with the angular-wise dreaming decoder. As shown in Fig. 3b, the decoder auto-regressively learns transition dynamics of each sector in a similar way of world model [14]. Assuming the planning module predicts the trajectories of future T steps, the dreaming decoder accordingly comprises T layers, where each updates the input angular queries Q a and angular BEV feature F a based on the learned temporal dynamics. At step t , the queries Q t -1 a first grasp environmental dynamics from the observation feature F t a with a gated recurrent unit (GRU) [7], which generates Q t a (hidden state),

In previous world models, the hidden state Q is solely used for perceiving observed scenes. The GRU iteration thus ends at t with the final observation F t a . In our framework, Q is also used for predicting ego trajectories in the future. Yet, the future observation, e.g. , F t +1 a , is unavailable, as the world model [14] is designed for forecasting the future with only current observation. To obtain Q t +1 a , we first propose to update F t a to provide pseudo observations ˆ F t +1 a ,

Then Q t +1 a can be generated with Eq. 2 and inputs of ˆ F t +1 a and Q t a . 146

Following the loss design in world models [14, 15, 16], we respectively map Q t -1 a and Q t a to distri147 butions of { µ t -1 a , σ t -1 a ∈ R K × C } and { µ t a , σ t a ∈ R K × C } , and then minimize their KL divergence. 148

For the prior distribution from Q t -1 a , it's regarded as a prediction of the future dynamics without 149 observation. In contrast, the posterior distribution from Q t a represents the future dynamics with the 150 observation F t a . The KL divergence between the two distributions measures the gap between the 151 imagined future (prior) and the true future (posterior). We expect to enhance the capability of future 152 prediction for long-term driving safety, which is realized by optimizing the dreaming loss L drm , 153

3.3 Direction-Aware Planning

Planning Head. The outputs of angular perception pretext contain a group of angular queries { Q t a ( t = 1 , ..., T ) } . For planning, we correspondingly initialize T ego queries { Q t ego ∈ R 1 × C ( t = 1 , ..., T ) } to extract planning-relevant information and predict the ego trajectory of each future time step. The interaction between ego queries and angular queries is performed with cross attention,

The output ego queries { Q t ego } are then used to predict the ego trajectories of future T steps. Following previous works [21, 22], a high-level driving signal c ( turn left , turn right or go straight ) is provided as prior knowledge. The planning head takes the concatenated ego feature F ego ∈ R T × C from { Q t ego } and the driving command c as inputs, and outputs the planning trajectory P traj ∈ R T × 2 ,

where the PlanHead is the same as UniAD [21]. We apply L 1 loss to minimize the distance between the predicted ego trajectory P traj and the ground truth G traj , denoted as L imi . Notably, G traj is easy to obtain, and manual annotation is not required in practical scenarios.

Directional Augmentation. Observed that the training data is predominated by the go straight scenarios, we propose a directional augmentation strategy to balance the distribution. As shown in Fig. 4, the BEV feature F b is rotated with different angles r ∈ R = { 90 ◦ , 180 ◦ , 270 ◦ } , yielding

BEV Feature/Mask

F

Planner

F

F

F

Rotate

Planner

Pred Trajectory

Pred Trajectory

Consistency

Loss

GT Trajectory

GT Trajectory

Rotate

Direction

Augment

Imitation

Loss

Imitation

Loss

the rotated representations { F r b } . The augmented features will also be used for the pretext and planning task, and supervised by the aforementioned loss functions ( e.g., L spat ). Notably, the BEV object mask M and the ground truth ego trajectory G traj are also rotated to provide corresponding supervision labels.

Furthermore, we propose an auxiliary task to enhance the steering capability. In specific, we predict the planning direction that the ego car intends to maneuver ( i.e., left , straight or right ) based on the ego query Q t ego , which is mapped to the probabilities of three directions P t dir ∈ R 1 × 3 . The direction label Y t dir is generated by comparing the x-axis value of ground truth G t traj ( x ) with the threshold δ . Specifically, Y t dir is assigned to straight if -δ < G t traj ( x ) <δ , otherwise Y t dir = left/right for G t traj ( x ) ⩽ -δ/ G t traj ( x ) ⩾ δ , respectively. We use the cross-entropy loss to minimize the gap between the direction prediction P t dir and the direction label Y t dir , denoted as L dir .

Directional Consistency. Tailored to the introduced directional augmentation, we propose a directional consistency loss to improve the augmented plan training in a self-supervised manner. It should be noticed that the augmented trajectory predictions P t,r traj incorporate the same scene information as the original one P t,r =0 traj , i.e., BEV features with different rotation angles. Therefore, it's reasonable to consider the consistency among the predictions and regulate the noises caused by the rotation. The planning head is expected to be more robust to directional change and input distractors. Specifically, P t,r traj are first rotated back to the original scene direction, then L 1 loss is applied with P t,r =0 traj ,

where Rot is the inverse rotation.

To summarize, the overall objective for our UAD contains spatial objectness loss, dreaming loss from the pretext, and imitation learning loss, direction loss, consistency loss from the planning task,

where ω 1 , ω 2 , ω 3 , ω 4 , ω 5 are the weight coefficients.

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

4 Experiment

4.1 Experimental Setup

We conduct experiments in nuScenes [2] for open-loop evaluation, that contains 40,157 samples, of which 6,019 ones are used for evaluation. Following previous works [20, 21, 22], we adopt the metrics of L2 error (in meters) and collision rate (in percentage). Notably, the intersection rate with road boundary (in percentage), proposed in BEV-Planner [26], is also included for evaluation. For the closed-loop setting, we follow previous works [34, 20] to perform evaluation in the Town05 [34] benchmark of the CARLA simulator [11]. Route completion (in percentage) and driving score (in percentage) are used as the evaluation metrics. We adopt the query-based view transformer [25] to learn BEV features from multi-view images. The confidence threshold of the open-set 2D detector is set to 0.35 to filter unreliable predictions. The angle θ to partition the BEV space is set to 4 ◦ ( K = 360 ◦ / 4 ◦ ), and the default threshold δ is 1 . 2 m (see Sec. 3.3). The weight coefficients in Eq. 8 are set to 2 . 0 , 0 . 1 , 1 . 0 , 2 . 0 , 1 . 0 . Our model is trained for 24 epochs on 8 NVIDIA Tesla A100 GPUs with a batch size of 1 per GPU. Other settings follow UniAD [21] unless otherwise specified.

We observed that ST-P3 [20] and VAD [22] adopt different open-loop evaluation protocols (L2 error and collision rate) from UniAD in their official codes. We denote the setting in ST-P3 and V AD as TemAvg and the one in UniAD as NoAvg , respectively. In specific, the TemAvg protocol calculates metrics by averaging the performances from 0.5s to the corresponding timestamp. Taking the L2 error at 2s as an example, the calculation in TemAvg is

where Avg is the average operation and 0 . 5 s is the time interval between two consecutive annotated frames in nuScenes [2]. For NoAvg protocol, L2@2s = l 2 2 . 0 s .

4.2 Comparison with State-of-the-arts

Open-loop Evaluation. Tab. 1 presents the performance comparison in terms of L2 error, collision rate, intersection rate with road boundary, and FPS. Since ST-P3 and V AD adopt different evaluation protocols from UniAD to compute L2 error and collision rate (see Sec. 4.1), we respectively calculate the results under different settings, i.e., NoAvg and TemAvg . As shown in Tab. 1, the proposed UAD achieves superior planning performance over UniAD and VAD on all metrics, while running faster. Notably, our UAD obtains 39.4% and 55.2% relative improvements on Collision@3s compared with UniAD and VAD under the NoAvg evaluation protocol ( e.g., 39.4%=(0.71%-0.43%)/0.71%), demonstrating the longtime robustness of our method. Moreover, UAD runs at 7.2FPS, which is 3 . 4 × and 1 . 4 × faster than UniAD and VAD-Base, respectively, verifying the efficiency of our framework. Surprisingly, our tiny version, UAD-Tiny, which aligns the settings of backbone, image size, and BEV

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

resolution in V AD-Tiny, runs at the fastest speed of 18.9FPS while clearly outperforming V AD-Tiny and even achieving comparable performance with VAD-Base. This again proves the superiority of our design. More detailed runtime comparisons and analyses are presented in the appendix. We adopt the NoAvg evaluation protocol in the following ablation experiments unless otherwise specified. Recent works discuss the effect of using ego status in the planning module [22, 26]. Following this trend, we also fairly compare the ego status equipped version of our model with these works. It shows that the superiority of our UAD is still preserved, which also achieves the best performance against the compared methods. Moreover, BEV-Planner [26] introduces a new metric named 'interaction' for better evaluating the performance of E2EAD methods. As shown in Tab. 1, our model obtains the average interaction rate of 1.13%, obviously outperforming other methods. This again proves the effectiveness of our UAD. On the other hand, this demonstrates the importance of designing a suitable pretext for perceiving the environment. Only using ego status is not enough for safe driving.

Closed-loop Evaluation. The simulation results in CARLA [11] are shown in Tab. 2. Our UAD achieves better performance compared with recent E2E planners ST-P3 [20] and VAD [22] in all scenarios, proving the effectiveness. Notably, on challenging Town05 Long benchmark, UAD greatly outperforms recent E2E method VAD by 6.40 points on the driving score and 9.92 points on route completion, respectively. This proves the reliability of our UAD for long-term autonomous driving.

4.3 Component-wise Ablation

Loss Functions. We first analyze the influence of different loss functions that correspond to the proposed pretext task and self-supervised trajectory learning strategy. The experiments are conducted on the validation split of the nuScenes [2], as shown in Tab. 3. The model with single imitation loss L imi is considered as the baseline ( ① ). With the enhanced perception capability by the spatial objectness loss L spat , the average L2 error and collision rate are clearly improved to 1.00m and 0.71% from 3.18m and 2.43%, respectively ( ② v.s. ① ). The dreaming loss L drm , direction loss L dir and consistency loss L cons also respectively bring considerable gains on the average L2 error for 1.98m, 1.58m, 1.77m over the baseline model ( ③ , ④ , ⑤ v.s. ① ). The loss functions are finally combined to construct our UAD ( ⑥ ), which obtains the average L2 error of 0.90m and average collision rate of 0.19%. The results demonstrate the effectiveness of each proposed component.

Temporal Learning with Dreaming Decoder. The temporal learning with the proposed dreaming decoder is realized by Circular Update and Dreaming Loss. The circular update is in charge of both extracting information from observed scenes (Eq. 2) and generating pseudo observations to predict the ego trajectories of future frames (Eq. 3). We study the influence of each module in Tab. 4. Circular Update and Dreaming Loss respectively bring performance gains of 0.70m/0.78m on the average L2 error ( ② , ③ v.s. ① ), proving the effectiveness of our designs. Applying both two modules ( ④ ) achieves the best performance, showing their complementarity for temporal representation learning.

Direction Aware Learning Strategy. Directional Augmentation and Directional Consistency are the two core components of the proposed direction-aware learning strategy. We prove their effectiveness in Tab. 5. It shows that the Directional Augmentation improves the average L2 error for considerable

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

0.05m ( ② v.s. ① ). One interesting observation is that applying the augmentation brings more gains for long-term planning than short-term ones, i.e., the L2 error of 1s/3s decreases for 0.01m/0.08m compared with ① , which proves the effectiveness of our augmentation on enhancing longer temporal information. The Directional Consistency further reduces the average collision rate for impressive 0.13% ( ③ v.s. ② ), which enhances the robustness for driving directional change.

Angular Design. We further explore the influence of the proposed angular design by removing the angular partition and angular queries. Specifically, the BEV feature is directly fed into the dreaming decoder to predict pixel-wise objectness, which is supervised by the BEV object mask (see Fig. 2) with binary cross-entropy loss. Besides, the ego query directly interacts with the BEV feature by cross-attention to extract environmental information. The results are presented in Tab. 6.

When discarding the angular design, the average L2 error degrades for 0.47m, and the average collision rate consistently degrades for 1.18%. This demonstrates the effectiveness of our angular design in perceiving complex environments and planning robust driving routes.

4.4 Further Analysis

Planning Performance in Different Driving Scenes. The direction-aware learning strategy is designed to enhance the planning performance in scenarios of vehicle steering. We demonstrate the superiority of our proposed model by evaluating the metrics of different driving scenes in Tab. 7. According to the given driving command ( i.e., go straight , turn left and turn right ), we divide the 6,019 validation samples in nuScenes [2] into three parts, which contain 5,309, 301 and 409 ones, respectively. Not surprisingly, all methods perform better under go straight scenes than the steering scenes, proving the necessity of augmenting the imbalanced training data for robust planning. When applying the proposed direction-aware learning strategy, our UAD achieves considerable gains on the average collision rate of turn left and turn right scenes (UAD v.s. UAD ∗ ). Notably, our model outperforms UniAD and VAD by a large margin in steering scenes, proving its effectiveness.

Visualization of Angular Perception and Planning. The angular perception pretext is designed to perceive the objects in each sector region. We show its capability by visualizing the predicted objectness in nuScenes [2] in Fig. 5a. For a better view, we transform the discrete objectness scores and ground truth to a pseudo-BEV mask. It shows that our model can successfully capture surrounding objects. Fig. 5a also shows the open-loop planning results of recent SOTA UniAD [21], VAD [22] and our UAD, proving the effectiveness of our method to plan a more reasonable ego trajectory. Fig. 5b compares the closed-loop driving routes between Transfuser [34], ST-P3 [20] and our UAD in CARLA [11]. Our method successfully notices the person and drives in a much safer manner, proving the reliability of our UAD in handling safe-critical issues under complex scenarios.

Due to limited space, we present more analyses in the appendix, including 1) the influence of partition angle θ , 2) the influence of direction threshold δ , 3 ) different backbones and pre-trained weights, 4 ) replacing 2D ROIs from GroundingDINO with 2D GT boxes, 5) different settings of GroundingDINO to generate 2D ROIs, 6) the influence of pre-training to previous method UniAD and our UAD, 7) runtime analysis of each module in our UAD and modularized UniAD, 8) more visualizations, etc.

4.5 Discussion

Ego Status and Open-loop Planning Evaluation. As revealed by [26, 40], it's not a challenge to acquire decent performance of L2 error and collision rate (the original metrics in nuScenes [2]) in the open-loop evaluation of nuScenes by using ego status in the planning module (see Tab. 1). The question is: is open-loop evaluation meaningless? Our answer is NO . Firstly, the inherent reason for the observation is that the simple cases of go straight dominate the nuScenes testing dataset. In these

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

(a)

(b)

RIGH

TURN LEFT

T=16

T-IE

Multi-view Images

GroundingDINO ROI

-8

GT Bx

GT Traj

, 003 10

BEV Object Mask

Angular Objectness Label

Predict Angular Objectness

Planning Result

cases, even a linear extrapolation of motion being sufficient for planning is not surprising. However, as shown in Tab. 7, in more challenging cases like turn right and turn left , the open-loop metrics can still clearly indicate the difficulty of steering scenarios and the differences in methods, which is also proved in [26]. Therefore, open-loop evaluation is not meaningless, while the crux is the distribution of the testing data and the metrics. Secondly, the advantage of open-loop evaluation is its efficiency, which benefits the fast development of algorithms. This view is also revealed by a recent simulator design study [9], which tries to transform the closed-loop evaluation into an open-loop fashion.

In our work, we thoroughly compare our model with other methods, which shows consistent improvements against previous works under various driving scenarios (straight or steering), different usage of ego status ( w/. or w/o. ), diverse evaluation metrics (L2 error, collision rate or intersection rate from [26]), and different evaluation types (open- or closed-loop). It thus again proves the importance of designing suitable pretext tasks for end-to-end autonomous driving.

How to Guarantee Safety in Current Auto-Drive System? Safety is the first requirement of autonomous driving systems in practical products, especially for L4-level auto-vehicles. To guarantee safety, offline collision check with predicted 3D boxes is an inevitable post-process under current technological conditions. Then, a question naturally arises: how to safely apply our model to current auto-driving systems? Before answering this question, we reaffirm our claim that we believe discarding 3D labels is an efficient, attractive, and potential direction for E2EAD, but it doesn't mean we refuse to use any 3D labels if the relatively cheap ones are available in practical product engineering. For instance, solely annotating bounding boxes without object identity for tracking is much cheaper than labeling other elements like HD-map, and point-cloud segmentation labels for occupancy. Therefore, we provide a degraded version of our method by arranging an additional 3D detection head.

Then our model can seamlessly integrate into autodrive products, and offline collision check is achievable. As shown in Tab. 8, integrating the 3D detection head doesn't bring additional improvements, which again proves the design of our method has sufficiently encoded 3D information to the planning module.

Table 8: Ablation on the 3D detection head.

#

Detection

Head

L2 (m)

↓

Collision (%)

↓

1s

2s

3s

1s

2s

3s

①

-

0.39

0.81

1.50

0.90

0.01

0.12

0.43

0.19

②

✓

0.37

0.86

1.57

0.93

0.02

0.17

0.55

0.25

In a nutshell, 1) our work can easily integrate other 3D tasks if they are inevitable under current technical conditions; 2) the experiments again prove from the side that our spatial-temporal module has already encoded important 3D clues for planning; 3) we hope our frontier work can eliminate some inessential 3D sub-tasks for both research and engineer usage of E2EAD models. An era of cheap, laboratory-affordable but robust, practical E2EAD design will eventually come!

5 Conclusion

Our work seeks to liberate E2EAD from costly modularization and 3D manual annotation. With this 345 goal, we propose the unsupervised pretext task to perceive the environment by predicting angular346 wise objectness and future dynamics. To improve the robustness in steering scenarios, we introduce 347 the direction-aware training strategy for planning. Experiments demonstrate the effectiveness and 348 efficiency of our method. As discussed, although the ego trajectories are easily obtained, it is almost 349 impossible to collect billion-level precisely annotated data with perception labels. This impedes the 350 further development of end-to-end autonomous driving. We believe our work provides a potential 351 solution to this barrier and may push performance to the next level when massive data are available. 352

353

354

355

356

References

Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911 , 2017.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, 357 Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. 358 In CVPR , 2020. 359

Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. 360 In CVPR , 2021. 361

Jun Cen, Peng Yun, Junhao Cai, Michael Yu Wang, and Ming Liu. Open-set 3d object detection. In 2021 362 International Conference on 3D Vision (3DV) . IEEE, 2021. 363

Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference 364 on Robot Learning , 2020. 365

366

367

Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for end-to-end autonomous driving. In ICCV , 2021.

368

369

370

371

372

373

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In ICCV , 2019.

NAVSIM Contributors. Navsim: Data-driven non-reactive autonomous vehicle simulation. https: //github.com/autonomousvision/navsim , 2024.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical 374 image database. In CVPR . Ieee, 2009. 375

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open 376 urban driving simulator. In Conference on robot learning , 2017. 377

Zeyu Gao, Yao Mu, Ruoyan Shen, Chen Chen, Yangang Ren, Jianyu Chen, Shengbo Eben Li, Ping Luo, 378 and Yanfeng Lu. Enhance sample efficiency and robustness of end-to-end urban autonomous driving via 379 semantic masked world model. arXiv preprint arXiv:2210.04017 , 2022. 380

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. NeurIPS , 2018. 381

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning 382 behaviors by latent imagination. ICLR , 2020. 383

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete 384 world models. ICLR , 2021. 385

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through 386 world models. arXiv preprint arXiv:2301.04104 , 2023. 387

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 388 In CVPR , 2016. 389

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex 390 Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. NeurIPS , 391 2022. 392

Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with 393 self-supervised freespace forecasting. In CVPR , 2021. 394

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end 395 vision-based autonomous driving via spatial-temporal feature learning. In ECCV . Springer, 2022. 396

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei 397 Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR , 2023. 398

Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, 399 Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. 400 arXiv preprint arXiv:2303.12077 , 2023. 401

Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable 402 raycasting for self-supervised occupancy forecasting. In ECCV , 2022. 403

404

405

406

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV . Springer, 2022.

Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In CVPR , 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV . Springer, 2014.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR , 2019.

Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Unsupervised 3d perception with 2d vision-language distillation for autonomous driving. In ICCV , 2023.

OpenAI. Chatgpt [large language model]. https://chat.openai.com , 2023.

Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging 422 noncontrollable visual dynamics in world models. NeurIPS , 2022. 423

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. NeurIPS , 1988. 424

425

426

Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR , 2021.

427

428

429

430

431

432

433

434

435

436

437

Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV . Springer, 2020.

Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In ICCV , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. CVPR , 2024.

Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In CVPR , 2019.

Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing 438 Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. 439 arXiv preprint arXiv:2305.10430 , 2023. 440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

A Appendix

The appendix presents additional designing and explaining details of our U npervised pretext task for end-to-end A utonomous D riving (UAD) in the manuscript.

Different Partition Angles

We explore the influence of different partition angles in angular pretext to learn better spatio-temporal knowledge.

· Different Direction Thresholds

We explore the influence of different thresholds in direction prediction to enhance planning robustness in complex driving scenarios.

· Different Backbones and Pre-trained Weights

We compare the performance of different backbones and pre-trained weights on our method.

· Objectness Label Generation with GT Boxes

We compare the generated objectness label between using the pseudo ROIs from GroundingDINO [28] and ground-truth boxes on different backbones.

· Settings for ROI Generation

We ablate different settings for the open-set 2D detector GroundingDINO, which provides ROIs for the label generation of angular perception pretext.

· Different Image Sizes and BEV Resolution

We compare the performance with different input sizes of multi-view images and BEV resolutions.

· Runtime Analysis

Weevaluate the runtime of each module of UAD and compare with modularized UniAD [21], which demonstrates the efficiency of our method.

· Classification of Angular Perception

We evaluate the objectness prediction in the angular perception pretext, which demonstrates the enhanced perception capability in complex driving scenarios.

Influence of Pre-training We evaluate the influence of pre-training by detailing the training losses and planning

performances with different pre-trained weights.

· More Visualizations

We provide more visualizations for the predicted angular-wise objectness and planning results in the open-loop evaluation of nuScenes [2] and closed-loop simulation of CARLA [11].

A.1 Different Partition Angles

The proposed angular perception pretext divides the BEV space into multiple sectors. We explore the influence of partition angle θ in Tab 9. Experimental results show that the L2 error and inference speed gradually increase with the partition angle. The model with partition angle of 1 ◦ ( ① ) achieves the best average L2 error of 0.85m. And the partition angle of 4 ◦ contributes to the best average collision rate of 0.19% ( ③ ). This reveals that a smaller partition angle helps learn more fine-grained environmental representations, eventually benefiting planning. In contrast, the model with a large partition angle sparsely perceives the scene. Despite reducing the computation cost, it will also degrade the safety of the end-to-end autonomous driving system.

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

A.2 Different Direction Thresholds

The direction prediction that the ego car intends to maneuver ( i.e., left , straight and right ) is proposed to enhance the steering capability for autonomous driving. The label is generated with the threshold δ (see Eq. 7 in the manuscript), which determines the ground-truth direction of each waypoint in the expert trajectory. Here we explore the influence by ablating different thresholds, as shown in Tab. 10. Experimental results show that the L2 error gradually increases with the direction threshold. The model with δ of 0.5m ( ① ) achieves the lowest L2 error of 0.86m. It reveals that a smaller threshold will force the planner to fit the expert navigation, leading to a closer distance between the predicted trajectory and the ground truth. In contrast, the collision rate benefits more from larger thresholds. The model with δ of 2.0m obtains the best collision rate at 2s of 0.08% ( ⑤ ), showing the effectiveness for robust planning. Notably, the threshold of 1.2m contributes to a great balance with the average L2 error of 0.90m and average collision rate of 0.19%.

A.3 Different Backbones and Pre-trained Weights

As a common sense, pre-training the backbone network with fundamental tasks like image classification on ImageNet [10] will benefit the sub-tasks. The previous method UniAD [21] uses the pre-trained weights of BEVFormer [25]. What surprised us is that when replacing the pre-trained weights with the one learned on ImageNet, the performance of UniAD dramatically degraded (see 'Influence of Pre-training' for more details). This inspires us to explore the influence of backbone settings on our framework. As shown in Tab. 11, interestingly, even without any pre-training, our model still outperforms UniAD with pre-trained ResNet101 and VAD with pre-trained ResNet50. This verifies the effectiveness of our unsupervised pretext task on modeling the driving scenes. We also use publicly available pre-trained weights on detection datasets like COCO [27] and nuImages [2] to train our model, which shows better performance. These experimental results and observations demonstrate that a potentially promising topic is how to pre-train a model for end-to-end autonomous driving . We leave this to future research.

A.4 Objectness Label Generation with GT Boxes

As mentioned in the manuscript, the essence of generating the angular objectness label lies in the 2D ROIs, which come from the open-set 2D detector GroundingDINO [28]. Here we explore the influence of using the ground-truth 2D boxes as ROIs, which provide more high-quality samples for the representation learning in the angular perception pretext. Tab. 12 shows that training with GT boxes achieves consistent performance gains on both ResNet50 [17] and ResNet101 [17] ( ② , ④ v.s. ① , ③ ). This reveals that accurate annotation does help to learn better spatio-temporal knowledge and improve ego planning. Considering the cost in real-world deployment, training with accessible

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

pseudo labels is a more efficient way compared with the manual annotation, which also shows comparable performance in autonomous driving ( ① v.s. ② and ③ v.s. ④ ).

A.5 Settings for ROI Generation.

The quality of learned spatio-temporal knowledge highly relies on the generated ROIs by the open-set 2D detector GroundingDINO [28], which are then projected as the BEV objectness label for training the angular perception pretext. We explore the influence of generated ROIs with different settings, as shown in Tab. 13. We take the setting with the confidence score of 0.35, prompt word of vehicle and without the Rule Filter, as the baseline ( ① ). By appending more prompt words ( e.g., pedestrian , barrier ), the planning performance gradually improves ( ③ , ② v.s. ① ), showing the enhanced perception capability with more diversified objects. Filtering the ROIs with overlarge size ( i.e., Rule Filter) brings considerable gains for the average L2 error of 0.07m and average collision rate of 0.10% ( ④ v.s. ③ ). One interesting observation is that decreasing the confidence threshold would slightly improve the L2 error while causing higher collision rate ( ⑤ v.s. ④ ). In contrast, increasing the threshold obtains lower average collision rate of 0.17% and higher average L2 error of 0.98m. This reveals the importance of providing diversified ROIs for angular perception learning as well as ensuring high quality. The model with the confidence score of 0.35, all prompt words and Rule Filter achieves balanced performance with the average L2 error of 0.90m and average collision rate of 0.19%.

A.6 Different Image Sizes and BEV Resolution

For safe autonomous driving, increasing the input size of the multi-view images and the resolution of the built BEV representation is an effective way, which provide more detailed environmental information. While benefiting perception and planning, it inevitably brings heavy computation cost. We then ablate the image size and BEV resolution of our UAD to find a balanced version between performance and efficiency, as shown in Tab. 14. The results show that our UAD with ResNet101 [17], image size of 1600 × 900, BEV resolution of 200 × 200, achieves the best performance compared with previous methods UniAD [21] and VAD-Base [22] while running faster with 7.2FPS ( ⑥ ). By replacing the backbone with ResNet-50, our UAD is more efficient with little performance degradation ( ⑤ v.s. ⑥ ). We further align the settings of V AD-Tiny, which has an inference speed of outstanding 17.6FPS ( ② ), to explore the influence of much smaller input sizes. Tab. 14 shows that our UAD still achieves excellent performance even compared with VAD-Base of high-resolution inputs ( ④ v.s. ③ ). Notably, our UAD of this version has the fastest inference speed of 18.9FPS. This

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

PRCurve

ROCCurve

1.0

1.0

0.9

0.8

0.8

Precision

0.7

0.6

TPR

0.6

0.4

0.5

0.4

Turn Left

0.2

Turn Left

Turn Right

Turn Right

0.3

Go Straight

0.0

Go Straight

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

FPR

(a)

(b)

120

ImageNet Pretrain

BEVFormer Pretrain

ImageNet Pretrain

BEVFormer Pretrain

14

No Pretrain

ImageNet Pretrain

2.50

No Pretrain

ImageNet Pretrain

100

12

2.25

Error

Error

Total Loss

80

Total Loss

10

2.00

Average

4

8

2

Average

1.75

60

3

6

1.50

40

2

4

1.25

20

1.00

0

5000

Iteration

10000

15000

20000

0

5

Epoch

10

15

20

U

5000

Iteration

10000

15000

20000

0

5

Epoch

10

15

20

again proves the effectiveness of our method in performing fine-grained perception, as well as the robustness to fit the inputs of different sizes.

A.7 Runtime Analysis

Tab. 15 compares the runtime of each module between the modularized method UniAD [21] and our UAD. As we adopt the Backbone and BEV Encoder from BEVFormer [25] that are the same in UniAD, the latency of feature extraction is similar with little difference due to different pre-processing. The modular sub-tasks in UniAD consume most of the runtime, i.e., significant 71.8% for Det&Track (31.2%), Map (19.8%), Motion (10.9%) and Occupancy (9.9%), respectively. In contrast, our UAD performs simple Angular Partition and Dreaming Decoder, which take only 14.0% (19.3ms) to model the complex environment. This demonstrates our insight that it's a necessity to liberate end-to-end autonomous driving from costly modularization. The downstream Planning Head takes negligible 1.5ms to plan the ego trajectory, compared with 9.7ms in UniAD. Finally, our UAD finishes the inference with a total runtime of 138.3ms, 3.4 × faster than the 465.1ms of UniAD, showing the efficiency of our design.

A.8 Classification of Angular Perception

The proposed angular perception pretext learns spatio-temporal knowledge of the driving scene 560 by predicting the objectness of each sector region, which is supervised by the generated binary 561 angular-wise label. We show the perception ability by evaluating the classification metrics based on 562 the validation split of the nuScenes [2] dataset. Fig. 6 draws the Precision-Recall (PR) curve and 563 Receiver-Operating-Characteristic (ROC) curve in different driving scenes ( i.e., turn left , go straight 564 and turn right ). In the PR curve, our UAD achieves balanced precision and recall scores in different 565 driving scenes, showing the effectiveness of our pretext task to perceive the surrounding objects. 566 Notably, the performance of go straight scenes is slightly better than the steering ones under all 567 thresholds. This proves our insight to design tailored direction-aware learning strategy for improving 568 the safety-critical turn left and turn right scenes. The ROC curve shows the robustness of our angular 569 perception pretext to classify the objects from complex environmental observations. 570

CAMBACK_LEFT

CAM_FRONI

ACKLEFT

RIGHT

Multi-view Images

GroundingDINOROI

BEV Object Mask

Angular Objectness LabelPredict Angular Objectness

TURN LEFT

=2

CAM_FRONT LEI

CAM BACK_RIGHT

A.9 Influence of Pre-training 571

Pre-training the backbone network with fundamental tasks is a commonly used metric to benefit 572

representation learning. As mentioned in 'Different Backbones and Pre-trained Weights' of Sec. 4.4 573

in the manuscript, the performance of the previous SOTA method UniAD [21] dramatically degrades 574

without the pre-trained weights from BEVFormer [25]. Here we further detail the influence by 575

comparing the training losses and planning performances with different pre-trained weights in Fig. 7. 576

Fig. 7a shows that the training losses increase by about 20 on average when replaced with the 577

pre-trained weights from ImageNet [10]. Correspondingly, the average L2 error is significantly higher 578

than the one with the pre-trained weights from BEVFormer. This reveals that UniAD heavily relies 579

on the perceptive pre-training in BEVFormer to optimize modularized sub-tasks. In contrast, our 580

RNRIGHT

T119

T=13e

Multi-view Images

GroundingDINO ROI

Multi-view Images

GroundingDINO ROI

ST-P3

ST-P3

BEV Object Mask

Angular Objectness Label

Predict Angular Objectness

Planning Result

BEV Object Mask

Angular Objectness Label

Predict Angular Objectness

Planning Result

CAM_FRONT_LEFT

CAM_FRONT

TURN RIGHT

T145

T=160

国

EAMBACKLEF

Multi-view Images

GroundingDINO ROI

GTBox

Multi-view Images

GroundingDINO ROI

=

5T-3

Ours

Bransfuser

Ours

ST-P3

Transfuser

BEV Object Mask

Angular Objectness Label

Predict Angular Objectness

Planning Result

BEV Object Mask

Angular Objectness Label

Predict Angular Objectness

Planning Result

A.10 More Visualizations 583

Open-loop Planning We provide more visualizations about the predicted angular-wise objectness 584 and planning results on nuScenes [2]. Fig. 8 compares the discrete objectness scores and ground 585 truth, proving the effectiveness of our angular perception pretext to perceive the objects in each sector 586 region. The planning results of previous SOTA methods ( i.e., UniAD [21] and VAD [22]) and our 587 UAD are shown in Fig. 9. With the designed pretext and tailored training strategy, our method could 588 plan a more reasonable ego trajectory under different driving scenarios, proving the effectiveness 589 of our work. The third row shows the failure case of our planner. In this case, the ego car is given 590 the ' Turn Right ' command when t = 0 ( i.e., the first frame of the driving scenario), leading to 591 ineffectiveness of our planner in learning helpful temporal information. A possible solution to deal 592 with this is to apply an auxiliary trajectory prior for the first several frames, and we leave this to 593 future work. 594

Closed-loop Simulation Fig. 10 visualizes the predicted objectness and planning results in the 595 Town05 Long benchmark of CARLA [11]. Following the setting of ST-P3 [20] in closed-loop evalua596 tion, we collect visual observations from the cameras of 'CAM_FRONT', 'CAM_FRONT_LEFT', 597 'CAM_FRONT_RIGHT' and 'CAM_BACK'. It shows that the sector regions in which the surround598 ing objects exist are successfully captured by our UAD, proving the effectiveness and robustness of 599 our design. Notably, the missed objects by GroundingDINO [28], e.g., the black car in the camera of 600 'CAM_FRONT_LEFT' at t = 145 , are surprisingly perceived and marked in the corresponding sector. 601 This demonstrates our method has the capability of learning perceptive knowledge in a data-driven 602 manner, even with coarse supervision by the generated 2D pseudo boxes from GroundingDINO. 603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, please see Sec. 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in this paper, please see Sec. 4.5. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the information needed to reproduce the main experimental results of the paper is disclosed, please see Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

Answer: [Yes]

Justification: This paper provides open access to the data and code to reproduce the main experimental results, please see Sec. 4.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are specified in this paper, please see Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This paper reports information about the statistical significance of experiments, please see Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This paper provides sufficient information on the computer resources, please see Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The conducted research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper discusses both potential positive societal impacts and negative societal impacts of the work, please see Sec. 4.5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets used in this paper are credited and the license is respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper handles the costly modularization and 3D manual annotation in current end-to-end autonomous driving, which proposes an unsupervised pretext task to provide necessary environmental information, as well as a direction-aware training strategy to enhance the robustness in safety-critical steering scenarios. \n\nThe authors conduct comprehensive experiments in both open- and closed-loop evaluation benchmarks, which demonstrate the effectiveness in various metrics. Moreover, the improvements are obtained with much less resource cost and faster inference speed, which is surprising and impressive.\n\nIn addition, this paper gives in-depth discussion and performance comparison about the usage of ego status in the open-loop evaluation of nuScenes. The considerable improvement in the intersection rate with the road boundary, which is proposed in recent BEV-Planner, again proves the superiority of the designed pretext task.\n\nSTRENGTHS:\nOverall, I am rather positive on this paper. In particular, I really like the motivation of this work that aims at finding a solution to relieve the heavy annotation and computation overload in current end-to-end autonomous driving. I believe this paper can inspire other works and facilitate this field. The strengths in this work include:\n\n(1)\tEnough novelty. This paper introduces an innovative unsupervised pretext task to perceive the environment, which is completely different from other works that accumulate subtasks requiring massive 3D annotation and computation resources. \n\n(2)\tGood performance. This paper demonstrates excellent performance and fast inference speed in both open- and closed-loop evaluation compared with other end-to-end methods. In specific on the challenging metric, i.e., intersection rate in BEV-Planner, the proposed approach surpasses other methods by a considerable margin. This clearly shows the effectiveness and advantages of the proposed method.\n\n(3)\tInsightful analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module. Notably, the authors observe the different computation ways of open-loop evaluation metrics between ST-P3 and UniAD and provide performance comparison with different settings, showing the comprehensiveness.\n\n(4)\tGood writing and organization. This paper is well-written and organized. Each section has a clear motivation. It’s easy to follow the ideas. I enjoy reading the paper.\n\nOverall, I believe this paper is significant to the autonomous driving community because it shows new insights and directions in designing simple but effective E2EAD framework with SOTA performance.\n\nWEAKNESSES:\n(1) In this work, the 2D ROIs are crucial for the designed pretext task. I noticed that the authors adopt the open-set 2D detector GroundingDINO to generate the ROIs. Then the results and discussion of using other third-party detectors should be presented.\n(2) The proposed method is shown to be efficient with the unsupervised pretext task and self-supervised training strategy, which is nice. It is suggested the authors show the influence of the training data volume (e.g., 25% and 50%).\n\nQUESTIONS:\nIn the appendix, the authors show that the proposed method still achieves comparative performance even without backbone pretraining, while UniAD dramatically degrades without the pretrained weights of BEVFormer. What do you think are the reasons causing this?\n\nLIMITATIONS:\nIt is suggested to provide discussion of limitations and broader impact in the revision.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of current end-to-end autonomous driving models that still rely on modular architectures with manually annotated 3D data. The authors propose an unsupervised pretext task that eliminates the need for manual 3D annotations by predicting angular-wise spatial objectness and temporal dynamics. This is achieved through an Angular Perception Pretext that models the driving scene without the need for manual annotation. A self-supervised training approach is introduced to enhance the robustness of planning in steering scenarios. This strategy learns the consistency of predicted trajectories under different augmented views. UAD demonstrates significant improvements in performance over existing methods like UniAD and VAD in both open-loop and closed-loop evaluations. It achieves these improvements with reduced training resources (44.3% of UniAD) and faster inference speed (3.4× faster than UniAD).\n\nSTRENGTHS:\n1. The paper introduces UAD, an unsupervised autonomous driving framework that eliminates the need for costly 3D manual annotations, which is a significant departure from traditional modular approaches. \n2. The Angular Perception Pretext is an innovative approach to spatial-temporal understanding without manual labeling, offering a new perspective on autonomous driving perception.\n3. The experiments conducted are comprehensive, including both open-loop and closed-loop evaluations, which demonstrate the method's effectiveness across different scenarios. The paper provides a detailed comparison with state-of-the-art methods like UniAD and VAD, showcasing the improvements in performance metrics, which adds to the quality of the research.\n\nWEAKNESSES:\n1. UAD treats an entire sector as occupied when only a part of it contains an object. This seems imprecise. This could potentially lead to less accurate spatial understanding of the environment. In autonomous driving, overly coarse representations might result in the vehicle making less accurate decisions, such as unnecessary braking or incorrect path planning. Have the authors tried some open world segmentation models for more accurate spatial information?\n2.  The paper draft does not provide explicit evidence or analysis on whether UAD can indeed benefit from training on a larger scale of data. The authors could conduct experiments with varying sizes of datasets to empirically evaluate how performance metrics change as more data becomes available. This could provide insights into the benefits of scaling up.\n\nQUESTIONS:\n1. it is not explicitly stated whether UAD and UniAD use the exact same training and test split within the nuScenes dataset.  \n2. As UAD only uses basic obstacles for training.  I wonder how will UAD reacts to traffic lights, lane lines, and policeman's gestures? \n3. What is the BEV area designed in the experiment?\n\nLIMITATIONS:\nIn the current draft, UAD might be limited to basic obstacle detection and does not extend to the interpretation of traffic signals.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to discard the requirement of 3D manual annotation in end-to-end autonomous driving by the proposed angular perception pretext task. Besides, this paper proposes a direction-aware learning strategy consisting of directional augmentation and directional consistency loss. Finally, the proposed method UAD achieves superior performance in both open-loop and closed-loop evaluation compared with previous vision-based methods with much lower computation and annotation costs.\n\nSTRENGTHS:\n1) This paper aims to discard the requirement of 3D manual annotation in end-to-end autonomous driving, which is important and meaningful for training larger end-to-end autonomous driving models at scale. I totally agree and appreciate this.\n2) This paper proposes a direction-aware learning strategy, which will further improve prformance by self-supervised learning.\n3) UAD is evaluated in both open-loop and closed-loop evaluation and different metrics (UniAD, VAD, and BEV-Planner).\n\nWEAKNESSES:\n1) There is a lack of explanation on how to use ego status. Besides, there should be more experiments about the performance of UAD without ego status.\n2) There is a lack of explanation on how many frames are fused and what method is used for temporal fusion (sliding window or streaming).\n3) The angular perception pretext task introduces 2D detection information for perception learning on BEV features. According to Table 6, it seems that angular design is very important for UAD. However, BEV-Planner can achieve a not-so-bad result without any 3D manual annotation and 2D detection information. Therefore, verifying the effectiveness of angular design on BEV-Planner will be more convincing.\n4) For the proposed angular design, I do not think is very novel. Because the effectiveness of the 2D detection auxiliary head has been verified in BEVFormerV2 [1] and StreamPETR [2], the UAD just converts 2D object detection to BEV segmentation.\n5) For the proposed direction-aware learning strategy, although it is useful, it is a method of data augmentation in EBV space, which I do not think is very novel.\n\nQUESTIONS:\n1) I have serious concerns about the use of ego status and temporal fusion. If UAD uses GT ego status when fusing many frames, because there is no cumulative error in the ego trajectory, it will lead to falsely high performance. A better solution is to use predicted ego status instead of GT ego status (such as SparseDrive [3]).\n2) Why does the use of a 3D detection head lead to a decrease in performance? What will be the result if UAD uses the online mapping head?\n3) What is the performance if UAD uses a 2D detection auxiliary head instead of an angular design? For example, angular queries can be obtained by 2D detection auxiliary head and depth prediction (such as Far3D [4]).\n\n\n[1] BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision\n[2] Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection\n[3] SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation\n[4] Far3D: Expanding the Horizon for Surround-view 3D Object Detection\n\nLIMITATIONS:\nSee the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article proposes an end-to-end (E2EAD) autonomous driving method called UAD (Unsupervised Autonomous Driving), which achieves autonomous driving on a visual basis without the need for expensive modular design and 3D manual annotation. UAD aims to overcome the limitations of existing E2EAD models that mimic traditional driving stack module architectures. These models typically require carefully designed supervised perception and prediction subtasks to provide environmental information for planning, which require a large amount of high-quality 3D annotation data and consume significant computational resources during training and inference processes.\n\nSTRENGTHS:\n1. The method is novel and a good direction for exploring the end-to-end model's dependence on 3D manual annotation.\n2. The paper has rich ablation experiments to demonstrate the effectiveness of the method.\n3. The paper has advantages in both speed and accuracy compared to previous articles\n\nWEAKNESSES:\n1. The paper lacks sufficient comparison with other methods(such as Interfuser[1],DriveAdapter[2],DriveMLM[3],VADv2[4]), which acheve better closed-loop performance on carla town 05 long benchmark.\n2. In Table 6, the angular design brings too much gain, especially in terms of collision rate (from 1.37% to 0.19% ). It's strange. The angular design is about how to encode the sensor data and should not have so much impact on collision rate.\n3. Angular design is widely used in BEV-related works, like PolarFormer[5]. The paper lacks citation about these works.And it's not proper to regard it as the main contribution of the work.\n4. The paper lacks a part to introduce the use of 2D tasks as auxiliary tasks.\n\n[1] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 726–737. PMLR, 2023\n[2] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. 2023\n[3] Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023\n[4] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243, 2024\n[5] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, Yu-Gang Jiang. PolarFormer: Multi-camera 3D Object Detection with Polar Transformer. AAAI 2023\n\nQUESTIONS:\nWhy use 2D boxes instead of 2D segmentation of objects? Is it more reasonable to use 2D segmentation labels to link the points in BEV space with the points in image space?\n\nLIMITATIONS:\n1. Incompele comparison on CARLA benchmark. \n2. Lack citation about angular design. \n3. Angular design is not novel.\n4. Some of the experiment results are not that convincing.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new e2e driving model named UAD. In this paper, the authors propose an unsupervised method for effective training and inference of the e2e model. The paper mainly has two contributions: 1. It designs an angular-wise perception module. In this module, the authors directly project 2D GT labels onto the BEV and define a new BEV map label for perception training. This module design can efficiently reduce complexity and preserve effectiveness. 2. The authors propose a direction-aware method to augment the trajectory training and use consistency loss for further supervision. \n\nThe final results show the effectiveness and soundness of the proposed method.\n\nSTRENGTHS:\n1. The idea of the angular-wise perception module is interesting. It can utilize a huge amount of 2D annotated autonomous driving datasets to train the e2e model, which removes the restriction of the limited number of 3D annotations.\n2. The proposed direction-aware method for trajectory prediction is also meaningful since it can add additional consistency loss for better supervision.\n3. The results are promising and the efficiency improvement is impressive.\n\nWEAKNESSES:\n1. The design of the angular-wise perception module is a little bit counter-intuitive to me. From my perspective, it works because (1) It can greatly enlarge the size of the training data. (2) It makes the perception task simpler, thus the model can do it better (knowing an object in a direction is much simpler than detecting the BBox). (3) The efficiency improves because of the light design of the perception task. I think it can be simply treated as a low-resolution object detection task without depth. Except for (1), I do not understand why it can improve the final results.\n\n2. The design of the direction-aware planning training strategy is effective but simple. I cannot see too much insight here. Could the authors provide more insight into this? Or is this just an engineering trick?\n\n3. For experiments, do you use exactly the same training data for both the open-loop and close-loop experiments? If yes, could you provide more analysis about why the results are surprisingly good even if you use a low-resolution detection module? If not, could you provide details about the pertaining data info? Can the impressive results come from leaked data? How to prevent testing data leakage?\n\n4. In real-world applications, when the vehicle plans its path, it needs to \"see\" a lot of things, including objects and some other elements, like traffic lights, traffic signals, or some special marks on the road. How can you handle these elements based on your model? For example, can your model understand traffic signals and see red traffic lights? If not, how to extend your model to real-world scenarios and applications?\n\nQUESTIONS:\nPlease see the weakness part for questions and suggestions.\n\nLIMITATIONS:\nMy main concern with this paper mainly comes from the insight of the angular-wise perception module. I still cannot understand why it works except for the huge amount of additional training data. Provide more details for this. \n\nBesides, how to deploy and extend the model to real-world cases, that requires depth information (e.g., ) or semantic information (e.g., traffic light)? The authors should provide more discussion about this to make the work more promising.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}