End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:37:49

Abstract: We propose UAD, a method for vision-based end-to-end autonomous driving (E2EAD), achieving the best open-loop evaluation performance in nuScenes, meanwhile showing robust closed-loop driving quality in CARLA. Our motivation stems from the observation that current E2EAD models still mimic the modular architecture in typical driving stacks, with carefully designed supervised perception and prediction subtasks to provide environment information for oriented planning. Although achieving groundbreaking progress, such design has certain drawbacks: 1) preceding subtasks require massive high-quality 3D annotations as supervision, posing a significant impediment to scaling the training data; 2) each submodule entails substantial computation overhead in both training and inference. To this end, we propose UAD, an E2EAD framework with an unsupervised proxy to address all these issues. Firstly, we design a novel Angular Perception Pretext to eliminate the annotation requirement. The pretext models the driving scene by predicting the angular-wise spatial objectness and temporal dynamics, without manual annotation. Secondly, a self-supervised training strategy, which learns the consistency of the predicted trajectories under different augment views, is proposed to enhance the planning robustness in steering scenarios. Our UAD achieves 38.7% relative improvements over UniAD on the average collision rate in nuScenes and surpasses VAD for 6.40 points on the driving score in CARLA's Town05 Long benchmark. Moreover, the proposed method only consumes 44.3% training resources of UniAD and runs 3.4$\times$ faster in inference. Our innovative design not only for the first time demonstrates unarguable performance advantages over supervised counterparts, but also enjoys unprecedented efficiency in data, training, and inference.

Main paper: 
 introduction: 
 Recent decades have witnessed breakthrough achievements in autonomous driving. The end-toend paradigm, which seeks to integrate perception, prediction, and planning tasks into a unified framework, stands as a representative branch [33, 1, 39, 3, 35, 21, 22] . The latest advances in end-toend autonomous driving significantly piqued researchers' interest [21, 22] . However, handcrafted and resource-intensive supervised sub-tasks for perception and prediction, which have previously proved their utility in environment modeling [35, 3, 20] , continue to be indispensable, as shown in Fig. 1a .Then what insights have we gained from the recent advances? It has come to our attention that one of the most enlightening innovations lies in the Transformer-based pipeline, in which the queries act as a connective thread, seamlessly bridging various tasks. Besides, the capability for environment modeling has also seen a significant boost, primarily due to complicated interactions of supervised 1 Following [30, 4] , here we consider the methods as "unsupervised" ones as long as no manual annotation is used and required in the target task or domain.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. sub-tasks. However, every coin has two sides. In comparison to the vanilla design [33] (see Fig. 1a ), modularized methods incur unavoidable computation and annotation overhead. As illustrated in Fig. 1b , the training of the recent method UniAD [21] takes 48 GPU days while running at only 2.1 frames per second (FPS). Moreover, modules in existing perception and prediction design require large quantities of high-quality annotated data. The financial overhead for human annotation significantly impedes the scalability of such modularized methods with supervised subtasks to leverage massive data. As proved by large foundation models [24, 31] , scaling up the data volume is the key to bringing the model capabilities to the next level. Thus we ask ourselves the question: Is it viable to devise an efficient and robust E2EAD framework while alleviating the reliance on 3D annotation?In this work, we show the answer is affirmative by proposing an innovative Unsupervised pretext task for end-to-end Autonomous Driving (UAD), which seeks to efficiently model the environment. The pretext task consists of an angular-wise perception module to learn spatial information by predicting the objectness of each sector region in BEV space, and an angular-wise dreaming decoder to absorb temporal knowledge by predicting inaccessible future states. The introduced angular queries link the two modules as a whole pretext task to perceive the driving scene. Notably, our method shines by completely eliminating the annotation requirement for perception and prediction. Such data efficiency is not attainable for current methods with complex supervised modularization [21, 22] . The supervision for learning spatial objectness is obtained by projecting the 2D region of interests (ROIs) from an off-the-shelf open-set detector [28] to BEV space. While utilizing the publicly available openset 2D detector pre-trained with manual annotation from other domains (e.g. COCO [27] ), we avoid the need for any additional 3D labels within our paradigm and target domains (e.g. nuScenes [2] and CARLA [11] ), thereby creating a pragmatically unsupervised setting [30] . Furthermore, we introduce a self-supervised direction-aware learning strategy to train the planning model. Specifically, the visual observations are augmented with different rotation angles, and the consistency loss is applied to the predictions for robust planning. Without bells and whistles, the proposed UAD outperforms UniAD for 0.13m in nuScenes Avg. L2 error, and surpasses VAD [22] for 9.92 points in CARLA route completion score. Such unprecedented performance gain is achieved with a 3.4× inference speed, a mere 44.3% training budget of UniAD, and zero annotations, as illustrated in Fig. 1b . In summary, our contributions are as follows: 1) We propose an unsupervised pretext task to discard the requirement of 3D manual annotation in end-to-end autonomous driving, potentially making it more feasible to scale the training data to billions level without any labeling overload; 2) We introduce a novel self-supervised direction-aware learning strategy to maximize the consistency of the predicted trajectories under different augment views, which enhances planning robustness in steering scenarios; 3) Our method shows superiority in both open-and closed-loop evaluation compared with other vision-based E2EAD methods, with much lower computation and annotation cost.2 Related Work 
end-to-end autonomous driving: 
 End-to-end autonomous driving can be dated back to 1988, when the ALVINN [33] proposed by Carnegie Mellon University could successfully navigate a vehicle over 400 meters. After that, to improve the robustness of E2EAD, a series of modern approaches such as NEAT [6] , P3 [35] ,MP3 [3] , ST-P3 [20] introduce the design of more dedicated modularization, which integrate auxiliary information such as HD maps, and additional tasks like bird's-eye view (BEV) segmentation. Most recently, embracing advanced architectures like Transfromer [37] and visual occupancy prediction [29] ,UniAD [21] and VAD [22] demonstrate impressive performance in open-loop evaluation. In this work, instead of integrating complex supervised modular sub-tasks, we innovatively propose another path proving that an efficient unsupervised pretext task without any human annotation like 3D bounding boxes and point cloud categories, can achieve even superior performance than recent state-of-the-arts.In pursuit of understanding the dynamic changes in environments, researchers in the fields of gaming and robotics have proposed various world models [13, 14, 15, 16] . Recently, the autonomous driving community introduces world models for safer maneuvering [32, 18, 12, 38] . MILE [18] considers the environment as a high-level embedding and tends to predict its future state with historical observations. Drive-WM [38] proposes a framework to integrate world models with existing E2E methods to improve planning robustness. In this work, we propose an auto-regressive mechanism, tailored to our unsupervised pretext, to capture angular-wise temporal dynamics within each sector.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work are discussed in this paper, please see Sec. 4.5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 As illustrated in Fig. 2 , our UAD framework consists of two essential components: 1) the Angular Perception Pretext, aims to liberate E2EAD from costly modularized tasks in an unsupervised fashion;2) the Direction-Aware Planning, learns self-supervised consistency of the augmented trajectories.Specifically, UAD first models the driving environment with the pretext. The spatial knowledge is acquired by estimating the objectness of each sector region within the BEV space. The angular queries, each responsible for a sector, are introduced to extract features and predict the objectness.The supervision label is generated by projecting the 2D regions of interests (ROIs) to the BEV space, which are predicted with an available open-set detector GroundingDINO [28] . This way not only eliminates the 3D annotation requirement, but also greatly reduces the training budget. Moreover, as driving is inherently a dynamic and continuous process, we thus propose an angular-wise dreaming decoder to encode the temporal knowledge. The dreaming decoder can be viewed as an augmented world model [13] capable of auto-regressively predicting the future states.Subsequently, direction-aware planning is introduced to train the planning module. The raw BEV feature is augmented with different rotation angles, yielding rotated BEV representations and ego trajectories. We apply self-supervised consistency loss to the predicted trajectories of each augmented view, which is expected to improve the robustness for directional change and input noises. The learning strategy can also be regarded as a novel data augmentation technique customized for end-toend autonomous driving, which enhances the diversity of trajectory distribution. GroundingDINO [28] . Such design is the key in reducing annotation cost and scaling up the dataset.To predict the objectness score of each sector, we define angular queries Q a ∈ R K×C to summarize F a . Each angular query q a ∈ R 1×C in Q a will interact with corresponding f by cross attention [37] ,EQUATIONFinally, we map Q a to the objectness scores P a ∈ R K×1 with a linear layer, which is supervised by Y obj with binary cross-entropy loss (denoted as L spat ).Temporal Representation Learning. We propose to capture the temporal information of driving scenarios with the angular-wise dreaming decoder. As shown in Fig. 3b , the decoder auto-regressively learns transition dynamics of each sector in a similar way of world model [14] . Assuming the planning module predicts the trajectories of future T steps, the dreaming decoder accordingly comprises T layers, where each updates the input angular queries Q a and angular BEV feature F a based on the learned temporal dynamics. At step t, the queries Q t-1 a first grasp environmental dynamics from the observation feature F t a with a gated recurrent unit (GRU) [7] , which generates Q t a (hidden state),EQUATIONIn previous world models, the hidden state Q is solely used for perceiving observed scenes. The GRU iteration thus ends at t with the final observation F t a . In our framework, Q is also used for predicting ego trajectories in the future. Yet, the future observation, e.g., F t+1 a , is unavailable, as the world model [14] is designed for forecasting the future with only current observation. To obtain Q t+1 a , we first propose to update F t a to provide pseudo observations Ft+1 a ,EQUATIONThen Q t+1 a can be generated with Eq. 2 and inputs of Ft+1 a and Q t a .Following the loss design in world models [14, 15, 16] , we respectively map Q t-1 a and Q t a to distributions of {µ t-1 a , σ t-1 a ∈ R K×C } and {µ t a , σ t a ∈ R K×C }, and then minimize their KL divergence.For the prior distribution from Q t-1 a , it's regarded as a prediction of the future dynamics without observation. In contrast, the posterior distribution from Q t a represents the future dynamics with the observation F t a . The KL divergence between the two distributions measures the gap between the imagined future (prior) and the true future (posterior). We expect to enhance the capability of future prediction for long-term driving safety, which is realized by optimizing the dreaming loss L drm ,EQUATIONPlanning Head. The outputs of angular perception pretext contain a group of angular queries {Q t a (t = 1, ..., T )}. For planning, we correspondingly initialize T ego queries {Q t ego ∈ R 1×C (t = 1, ..., T )} to extract planning-relevant information and predict the ego trajectory of each future time step. The interaction between ego queries and angular queries is performed with cross attention,EQUATIONThe output ego queries {Q t ego } are then used to predict the ego trajectories of future T steps.Following previous works [21, 22] , a high-level driving signal c (turn left, turn right or go straight) is provided as prior knowledge. The planning head takes the concatenated ego feature F ego ∈ R T ×C from {Q t ego } and the driving command c as inputs, and outputs the planning trajectory P traj ∈ R T ×2 ,EQUATIONwhere the PlanHead is the same as UniAD [21] . We apply L 1 loss to minimize the distance between the predicted ego trajectory P traj and the ground truth G traj , denoted as L imi . Notably, G traj is easy to obtain, and manual annotation is not required in practical scenarios. Furthermore, we propose an auxiliary task to enhance the steering capability. In specific, we predict the planning direction that the ego car intends to maneuver (i.e., left, straight or right) based on the ego query Q t ego , which is mapped to the probabilities of three directions P t dir ∈ R 1×3 . The direction label Y t dir is generated by comparing the x-axis value of ground truth G t traj (x) with the thresholdδ. Specifically, Y t dir is assigned to straight if -δ < G t traj (x) < δ, otherwise Y t dir = lef t/rightfor G t traj (x) ⩽ -δ/G t traj (x) ⩾ δ, respectively. We use the cross-entropy loss to minimize the gap between the direction prediction P t dir and the direction label Y t dir , denoted as L dir .Directional Consistency. Tailored to the introduced directional augmentation, we propose a directional consistency loss to improve the augmented plan training in a self-supervised manner. It should be noticed that the augmented trajectory predictions P t,r traj incorporate the same scene information as the original one P t,r=0 traj , i.e., BEV features with different rotation angles. Therefore, it's reasonable to consider the consistency among the predictions and regulate the noises caused by the rotation. The planning head is expected to be more robust to directional change and input distractors. Specifically, P t,r traj are first rotated back to the original scene direction, then L 1 loss is applied with P t,r=0 traj ,EQUATIONwhere Rot is the inverse rotation.To summarize, the overall objective for our UAD contains spatial objectness loss, dreaming loss from the pretext, and imitation learning loss, direction loss, consistency loss from the planning task,EQUATIONwhere ω 1 , ω 2 , ω 3 , ω 4 , ω 5 are the weight coefficients.Table 1 : Open-loop planning performance in nuScenes [2] . † indicates LiDAR-based method and ‡ denotes TemAvg evaluation protocol used in VAD and ST-P3 (see Eq. 9 for details). ⋄ means using ego status in the planning module and calculating collision rates following BEV-Planner [26] . We conduct experiments in nuScenes [2] for open-loop evaluation, that contains 40,157 samples, of which 6,019 ones are used for evaluation. Following previous works [20, 21, 22] , we adopt the metrics of L2 error (in meters) and collision rate (in percentage). Notably, the intersection rate with road boundary (in percentage), proposed in BEV-Planner [26] , is also included for evaluation. For the closed-loop setting, we follow previous works [34, 20] to perform evaluation in the Town05 [34] benchmark of the CARLA simulator [11] . Route completion (in percentage) and driving score (in percentage) are used as the evaluation metrics. We adopt the query-based view transformer [25] to learn BEV features from multi-view images. The confidence threshold of the open-set 2D detector is set to 0.35 to filter unreliable predictions. The angle θ to partition the BEV space is set to 4 • (K=360 • /4 • ), and the default threshold δ is 1.2m (see Sec. 3.3). The weight coefficients in Eq. 8 are set to 2.0, 0.1, 1.0, 2.0, 1.0. Our model is trained for 24 epochs on 8 NVIDIA Tesla A100 GPUs with a batch size of 1 per GPU. Other settings follow UniAD [21] unless otherwise specified.We observed that ST-P3 [20] and VAD [22] adopt different open-loop evaluation protocols (L2 error and collision rate) from UniAD in their official codes. We denote the setting in ST-P3 and VAD asTemAvg and the one in UniAD as NoAvg, respectively. In specific, the TemAvg protocol calculates metrics by averaging the performances from 0.5s to the corresponding timestamp. Taking the L2 error at 2s as an example, the calculation in TemAvg isEQUATIONwhere Avg is the average operation and 0.5s is the time interval between two consecutive annotated frames in nuScenes [2] . For NoAvg protocol, L2@2s = l2 2.0s .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
comparison with state-of-the-arts: 
 Open-loop Evaluation. Tab. Notably, our UAD obtains 39.4% and 55.2% relative improvements on Collision@3s compared with UniAD and VAD under the NoAvg evaluation protocol (e.g., 39.4%=(0.71%-0.43%)/0.71%), demonstrating the longtime robustness of our method. Moreover, UAD runs at 7.2FPS, which is 3.4× and 1.4× faster than UniAD and VAD-Base, respectively, verifying the efficiency of our framework.Surprisingly, our tiny version, UAD-Tiny, which aligns the settings of backbone, image size, and BEV resolution in VAD-Tiny, runs at the fastest speed of 18.9FPS while clearly outperforming VAD-Tiny and even achieving comparable performance with VAD-Base. This again proves the superiority of our design. More detailed runtime comparisons and analyses are presented in the appendix. We adopt the NoAvg evaluation protocol in the following ablation experiments unless otherwise specified.Recent works discuss the effect of using ego status in the planning module [22, 26] . Following this trend, we also fairly compare the ego status equipped version of our model with these works. It shows that the superiority of our UAD is still preserved, which also achieves the best performance against the compared methods. Moreover, BEV-Planner [26] introduces a new metric named "interaction"for better evaluating the performance of E2EAD methods. As shown in Tab. 1, our model obtains the average interaction rate of 1.13%, obviously outperforming other methods. This again proves the effectiveness of our UAD. On the other hand, this demonstrates the importance of designing a suitable pretext for perceiving the environment. Only using ego status is not enough for safe driving.Closed-loop Evaluation. The simulation results in CARLA [11] are shown in Tab. 2. Our UAD achieves better performance compared with recent E2E planners ST-P3 [20] and VAD [22] in all scenarios, proving the effectiveness. Notably, on challenging Town05 Long benchmark, UAD greatly outperforms recent E2E method VAD by 6.40 points on the driving score and 9.92 points on route completion, respectively. This proves the reliability of our UAD for long-term autonomous driving.Loss Functions. We first analyze the influence of different loss functions that correspond to the proposed pretext task and self-supervised trajectory learning strategy. The experiments are conducted on the validation split of the nuScenes [2] , as shown in Tab. 0.05m (②v.s.①). One interesting observation is that applying the augmentation brings more gains for long-term planning than short-term ones, i.e., the L2 error of 1s/3s decreases for 0.01m/0.08m compared with ①, which proves the effectiveness of our augmentation on enhancing longer temporal information. The Directional Consistency further reduces the average collision rate for impressive 0.13% (③v.s.②), which enhances the robustness for driving directional change.Angular Design. We further explore the influence of the proposed angular design by removing the angular partition and angular queries. Specifically, the BEV feature is directly fed into the dreaming decoder to predict pixel-wise objectness, which is supervised by the BEV object mask (see Fig. 2 ) with binary cross-entropy loss. Besides, the ego query directly interacts with the BEV feature by cross-attention to extract environmental information. The results are presented in Tab. 6. When discarding the angular design, the average L2 error degrades for 0.47m, and the average collision rate consistently degrades for 1.18%. This demonstrates the effectiveness of our angular design in perceiving complex environments and planning robust driving routes.Planning Performance in Different Driving Scenes. The direction-aware learning strategy is designed to enhance the planning performance in scenarios of vehicle steering. We demonstrate the superiority of our proposed model by evaluating the metrics of different driving scenes in Tab. 7.According to the given driving command (i.e., go straight, turn left and turn right), we divide the 6,019 validation samples in nuScenes [2] into three parts, which contain 5,309, 301 and 409 ones, respectively. Not surprisingly, all methods perform better under go straight scenes than the steering scenes, proving the necessity of augmenting the imbalanced training data for robust planning. When applying the proposed direction-aware learning strategy, our UAD achieves considerable gains on the average collision rate of turn left and turn right scenes (UAD v.s. UAD * ). Notably, our model outperforms UniAD and VAD by a large margin in steering scenes, proving its effectiveness.Visualization of Angular Perception and Planning. The angular perception pretext is designed to perceive the objects in each sector region. We show its capability by visualizing the predicted objectness in nuScenes [2] in Fig. 5a . For a better view, we transform the discrete objectness scores and ground truth to a pseudo-BEV mask. It shows that our model can successfully capture surrounding objects. Fig. 5a also shows the open-loop planning results of recent SOTA UniAD [21] ,VAD [22] and our UAD, proving the effectiveness of our method to plan a more reasonable ego trajectory. Fig. 5b compares the closed-loop driving routes between Transfuser [34] , ST-P3 [20] and our UAD in CARLA [11] . Our method successfully notices the person and drives in a much safer manner, proving the reliability of our UAD in handling safe-critical issues under complex scenarios.Due to limited space, we present more analyses in the appendix, including 1) the influence of partition angle θ, 2) the influence of direction threshold δ, 3) different backbones and pre-trained weights, 4)replacing 2D ROIs from GroundingDINO with 2D GT boxes, 5) different settings of GroundingDINO to generate 2D ROIs, 6) the influence of pre-training to previous method UniAD and our UAD, 7)runtime analysis of each module in our UAD and modularized UniAD, 8) more visualizations, etc.Ego Status and Open-loop Planning Evaluation. As revealed by [26, 40] , it's not a challenge to acquire decent performance of L2 error and collision rate (the original metrics in nuScenes [2] ) in the open-loop evaluation of nuScenes by using ego status in the planning module (see Tab. 1). The question is: is open-loop evaluation meaningless? Our answer is NO. Firstly, the inherent reason for the observation is that the simple cases of go straight dominate the nuScenes testing dataset. In these still clearly indicate the difficulty of steering scenarios and the differences in methods, which is also proved in [26] . Therefore, open-loop evaluation is not meaningless, while the crux is the distribution of the testing data and the metrics. Secondly, the advantage of open-loop evaluation is its efficiency, which benefits the fast development of algorithms. This view is also revealed by a recent simulator design study [9] , which tries to transform the closed-loop evaluation into an open-loop fashion.In our work, we thoroughly compare our model with other methods, which shows consistent improvements against previous works under various driving scenarios (straight or steering), different usage of ego status (w/. or w/o.), diverse evaluation metrics (L2 error, collision rate or intersection rate from [26] ), and different evaluation types (open-or closed-loop). It thus again proves the importance of designing suitable pretext tasks for end-to-end autonomous driving.How to Guarantee Safety in Current Auto-Drive System? Safety is the first requirement of autonomous driving systems in practical products, especially for L4-level auto-vehicles. To guarantee safety, offline collision check with predicted 3D boxes is an inevitable post-process under current technological conditions. Then, a question naturally arises: how to safely apply our model to current auto-driving systems? Before answering this question, we reaffirm our claim that we believe discarding 3D labels is an efficient, attractive, and potential direction for E2EAD, but it doesn't mean we refuse to use any 3D labels if the relatively cheap ones are available in practical product engineering.For instance, solely annotating bounding boxes without object identity for tracking is much cheaper than labeling other elements like HD-map, and point-cloud segmentation labels for occupancy.Therefore, we provide a degraded version of our method by arranging an additional 3D detection head.  
conclusion: 
 Our work seeks to liberate E2EAD from costly modularization and 3D manual annotation. With this goal, we propose the unsupervised pretext task to perceive the environment by predicting angularwise objectness and future dynamics. To improve the robustness in steering scenarios, we introduce the direction-aware training strategy for planning. Experiments demonstrate the effectiveness and efficiency of our method. As discussed, although the ego trajectories are easily obtained, it is almost impossible to collect billion-level precisely annotated data with perception labels. This impedes the further development of end-to-end autonomous driving. We believe our work provides a potential solution to this barrier and may push performance to the next level when massive data are available. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All the training and test details are specified in this paper, please see Sec. 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: This paper reports information about the statistical significance of experiments, please see Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets used in this paper are credited and the license is respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The appendix presents additional designing and explaining details of our Unpervised pretext task for end-to-end Autonomous Driving (UAD) in the manuscript.We explore the influence of different partition angles in angular pretext to learn better spatio-temporal knowledge.We explore the influence of different thresholds in direction prediction to enhance planning robustness in complex driving scenarios.• Different Backbones and Pre-trained WeightsWe compare the performance of different backbones and pre-trained weights on our method.• Objectness Label Generation with GT BoxesWe compare the generated objectness label between using the pseudo ROIs from Ground-ingDINO [28] and ground-truth boxes on different backbones.We ablate different settings for the open-set 2D detector GroundingDINO, which provides ROIs for the label generation of angular perception pretext.We compare the performance with different input sizes of multi-view images and BEV resolutions.We evaluate the runtime of each module of UAD and compare with modularized UniAD [21] ,which demonstrates the efficiency of our method.We evaluate the objectness prediction in the angular perception pretext, which demonstrates the enhanced perception capability in complex driving scenarios.We evaluate the influence of pre-training by detailing the training losses and planning performances with different pre-trained weights.We provide more visualizations for the predicted angular-wise objectness and planning results in the open-loop evaluation of nuScenes [2] and closed-loop simulation of CARLA [11] .The proposed angular perception pretext divides the BEV space into multiple sectors. We explore the influence of partition angle θ in Tab 9. Experimental results show that the L2 error and inference speed gradually increase with the partition angle. The model with partition angle of 1 • (①) achieves the best average L2 error of 0.85m. And the partition angle of 4 • contributes to the best average collision rate of 0.19% (③). This reveals that a smaller partition angle helps learn more fine-grained environmental representations, eventually benefiting planning. In contrast, the model with a large partition angle sparsely perceives the scene. Despite reducing the computation cost, it will also degrade the safety of the end-to-end autonomous driving system. The direction prediction that the ego car intends to maneuver (i.e., left, straight and right) is proposed to enhance the steering capability for autonomous driving. The label is generated with the threshold δ (see Eq. 7 in the manuscript), which determines the ground-truth direction of each waypoint in the expert trajectory. Here we explore the influence by ablating different thresholds, as shown in Tab. 10.Experimental results show that the L2 error gradually increases with the direction threshold. The model with δ of 0.5m (①) achieves the lowest L2 error of 0.86m. It reveals that a smaller threshold will force the planner to fit the expert navigation, leading to a closer distance between the predicted trajectory and the ground truth. In contrast, the collision rate benefits more from larger thresholds.The model with δ of 2.0m obtains the best collision rate at 2s of 0.08% (⑤), showing the effectiveness for robust planning. Notably, the threshold of 1.2m contributes to a great balance with the average L2 error of 0.90m and average collision rate of 0.19%.As a common sense, pre-training the backbone network with fundamental tasks like image classification on ImageNet [10] will benefit the sub-tasks. The previous method UniAD [21] uses the pre-trained weights of BEVFormer [25] . What surprised us is that when replacing the pre-trained weights with the one learned on ImageNet, the performance of UniAD dramatically degraded (see "Influence of Pre-training" for more details). This inspires us to explore the influence of backbone settings on our framework. As shown in Tab. 11, interestingly, even without any pre-training, our model still outperforms UniAD with pre-trained ResNet101 and VAD with pre-trained ResNet50.This verifies the effectiveness of our unsupervised pretext task on modeling the driving scenes. We also use publicly available pre-trained weights on detection datasets like COCO [27] and nuImages [2] to train our model, which shows better performance. These experimental results and observations demonstrate that a potentially promising topic is how to pre-train a model for end-to-end autonomous driving. We leave this to future research.As mentioned in the manuscript, the essence of generating the angular objectness label lies in the 2D ROIs, which come from the open-set 2D detector GroundingDINO [28] . Here we explore the influence of using the ground-truth 2D boxes as ROIs, which provide more high-quality samples for the representation learning in the angular perception pretext. Tab. 12 shows that training with GT boxes achieves consistent performance gains on both ResNet50 [17] and ResNet101 [17] (②,④ v.s. ①,③). This reveals that accurate annotation does help to learn better spatio-temporal knowledge and improve ego planning. Considering the cost in real-world deployment, training with accessible pseudo labels is a more efficient way compared with the manual annotation, which also shows comparable performance in autonomous driving (① v.s. ② and ③ v.s. ④).A.5 Settings for ROI Generation.The quality of learned spatio-temporal knowledge highly relies on the generated ROIs by the open-set 2D detector GroundingDINO [28] , which are then projected as the BEV objectness label for training the angular perception pretext. We explore the influence of generated ROIs with different settings, as shown in Tab. 13. We take the setting with the confidence score of 0.35, prompt word of vehicle and without the Rule Filter, as the baseline (①). By appending more prompt words (e.g., pedestrian, barrier), the planning performance gradually improves (③,② v.s.①), showing the enhanced perception capability with more diversified objects. Filtering the ROIs with overlarge size (i.e., Rule Filter) brings considerable gains for the average L2 error of 0.07m and average collision rate of 0.10% (④v.s.③). One interesting observation is that decreasing the confidence threshold would slightly improve the L2 error while causing higher collision rate (⑤v.s.④). In contrast, increasing the threshold obtains lower average collision rate of 0.17% and higher average L2 error of 0.98m. This reveals the importance of providing diversified ROIs for angular perception learning as well as ensuring high quality. The model with the confidence score of 0.35, all prompt words and Rule Filter achieves balanced performance with the average L2 error of 0.90m and average collision rate of 0.19%.For safe autonomous driving, increasing the input size of the multi-view images and the resolution of the built BEV representation is an effective way, which provide more detailed environmental information. While benefiting perception and planning, it inevitably brings heavy computation cost.We then ablate the image size and BEV resolution of our UAD to find a balanced version between performance and efficiency, as shown in Tab. 14. The results show that our UAD with ResNet-101 [17] , image size of 1600×900, BEV resolution of 200×200, achieves the best performance compared with previous methods UniAD [21] and VAD-Base [22] while running faster with 7.2FPS (⑥). By replacing the backbone with ResNet-50, our UAD is more efficient with little performance degradation (⑤ v.s. ⑥). We further align the settings of VAD-Tiny, which has an inference speed of outstanding 17.6FPS (②), to explore the influence of much smaller input sizes. Tab. 14 shows that our UAD still achieves excellent performance even compared with VAD-Base of high-resolution inputs (④ v.s. ③). Notably, our UAD of this version has the fastest inference speed of 18.9FPS. This again proves the effectiveness of our method in performing fine-grained perception, as well as the robustness to fit the inputs of different sizes.Tab. 15 compares the runtime of each module between the modularized method UniAD [21] and our UAD. As we adopt the Backbone and BEV Encoder from BEVFormer [25] that are the same in UniAD, the latency of feature extraction is similar with little difference due to different pre-processing.The modular sub-tasks in UniAD consume most of the runtime, i.e., significant 71.8% for Det&Track (31.2%), Map (19.8%), Motion (10.9%) and Occupancy (9.9%), respectively. In contrast, our UAD performs simple Angular Partition and Dreaming Decoder, which take only 14.0% (19.3ms) to model the complex environment. This demonstrates our insight that it's a necessity to liberate end-to-end autonomous driving from costly modularization. The downstream Planning Head takes negligible 1.5ms to plan the ego trajectory, compared with 9.7ms in UniAD. Finally, our UAD finishes the inference with a total runtime of 138.3ms, 3.4× faster than the 465.1ms of UniAD, showing the efficiency of our design.The proposed angular perception pretext learns spatio-temporal knowledge of the driving scene by predicting the objectness of each sector region, which is supervised by the generated binary angular-wise label. We show the perception ability by evaluating the classification metrics based on the validation split of the nuScenes [2] dataset. Fig. 6 draws the Precision-Recall (PR) curve and Receiver-Operating-Characteristic (ROC) curve in different driving scenes (i.e., turn left, go straight and turn right). In the PR curve, our UAD achieves balanced precision and recall scores in different driving scenes, showing the effectiveness of our pretext task to perceive the surrounding objects.Notably, the performance of go straight scenes is slightly better than the steering ones under all thresholds. This proves our insight to design tailored direction-aware learning strategy for improving the safety-critical turn left and turn right scenes. The ROC curve shows the robustness of our angular perception pretext to classify the objects from complex environmental observations. Figure 9 : Visualization of the planning results. The first two rows show the success of our method in safe planning in complex scenarios, while the third row exhibits a failure case of our planner when no temporal information could be acquired when t = 0.Pre-training the backbone network with fundamental tasks is a commonly used metric to benefit representation learning. As mentioned in "Different Backbones and Pre-trained Weights" of Sec. 4.4in the manuscript, the performance of the previous SOTA method UniAD [21] dramatically degrades without the pre-trained weights from BEVFormer [25] . Here we further detail the influence by comparing the training losses and planning performances with different pre-trained weights in Fig. 7 .Fig. 7a shows that the training losses increase by about 20 on average when replaced with the pre-trained weights from ImageNet [10] . Correspondingly, the average L2 error is significantly higher than the one with the pre-trained weights from BEVFormer. This reveals that UniAD heavily relies on the perceptive pre-training in BEVFormer to optimize modularized sub-tasks. In contrast, our UAD performs comparably even without any pre-training (see Fig. 7b ), proving the effectiveness of our designs for robust optimization. Open-loop Planning We provide more visualizations about the predicted angular-wise objectness and planning results on nuScenes [2] . Fig. 8 compares the discrete objectness scores and ground truth, proving the effectiveness of our angular perception pretext to perceive the objects in each sector region. The planning results of previous SOTA methods (i.e., UniAD [21] and VAD [22] ) and our UAD are shown in Fig. 9 . With the designed pretext and tailored training strategy, our method could plan a more reasonable ego trajectory under different driving scenarios, proving the effectiveness of our work. The third row shows the failure case of our planner. In this case, the ego car is given the "Turn Right" command when t = 0 (i.e., the first frame of the driving scenario), leading to ineffectiveness of our planner in learning helpful temporal information. A possible solution to deal with this is to apply an auxiliary trajectory prior for the first several frames, and we leave this to future work.Closed-loop Simulation Fig. 10 visualizes the predicted objectness and planning results in the Town05 Long benchmark of CARLA [11] . Following the setting of ST-P3 [20] in closed-loop evaluation, we collect visual observations from the cameras of "CAM_FRONT", "CAM_FRONT_LEFT", "CAM_FRONT_RIGHT" and "CAM_BACK". It shows that the sector regions in which the surrounding objects exist are successfully captured by our UAD, proving the effectiveness and robustness of our design. Notably, the missed objects by GroundingDINO [28] , e.g., the black car in the camera of "CAM_FRONT_LEFT" at t = 145, are surprisingly perceived and marked in the corresponding sector.This demonstrates our method has the capability of learning perceptive knowledge in a data-driven manner, even with coarse supervision by the generated 2D pseudo boxes from GroundingDINO.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper was reviewed by five experts in the field. The recommendations are diverse (Very Strong Accept, Strong Accept, Weak Accept, Borderline Reject, Reject). Three Reviewers (d489, XcsQ, and UZRG) raise serious concerns regarding the Angular design and angular perception pretext, which share similar insights with previous BEV-related works with limited technical novelty and innovation. Reviewer d489 still has strong concerns about its actual application values in real-world scenes. Reviewer XcsQ argues that the experimental evaluations are not sufficient and convincing, and more detailed ablations and comparisons are needed to validate the effectiveness. Besides more supportive experiments, Reviewer UZRG also claims that a lot of revisions are needed to clearly describe these details. Reviewers 3W73 and YXrs hold positive opinions regarding the manuscript. However, the rebuttal did not assuage all these concerns raised by the former three reviewers. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Reviews: review: paper_summary: paper_summary: The paper proposes a new e2e driving model named UAD. In this paper, the authors propose an unsupervised method for effective training and inference of the e2e model. The paper mainly has two contributions: 1. It designs an angular-wise perception module. In this module, the authors directly project 2D GT labels onto the BEV and define a new BEV map label for perception training. This module design can efficiently reduce complexity and preserve effectiveness. 2. The authors propose a direction-aware method to augment the trajectory training and use consistency loss for further supervision. 

The final results show the effectiveness and soundness of the proposed method.
strength_weakness: strength_weakness: 1. The design of the angular-wise perception module is a little bit counter-intuitive to me. From my perspective, it works because (1) It can greatly enlarge the size of the training data. (2) It makes the perception task simpler, thus the model can do it better (knowing an object in a direction is much simpler than detecting the BBox). (3) The efficiency improves because of the light design of the perception task. I think it can be simply treated as a low-resolution object detection task without depth. Except for (1), I do not understand why it can improve the final results.

2. The design of the direction-aware planning training strategy is effective but simple. I cannot see too much insight here. Could the authors provide more insight into this? Or is this just an engineering trick?

3. For experiments, do you use exactly the same training data for both the open-loop and close-loop experiments? If yes, could you provide more analysis about why the results are surprisingly good even if you use a low-resolution detection module? If not, could you provide details about the pertaining data info? Can the impressive results come from leaked data? How to prevent testing data leakage?

4. In real-world applications, when the vehicle plans its path, it needs to "see" a lot of things, including objects and some other elements, like traffic lights, traffic signals, or some special marks on the road. How can you handle these elements based on your model? For example, can your model understand traffic signals and see red traffic lights? If not, how to extend your model to real-world scenarios and applications?
questions: questions: Please see the weakness part for questions and suggestions.
limitations: limitations: My main concern with this paper mainly comes from the insight of the angular-wise perception module. I still cannot understand why it works except for the huge amount of additional training data. Provide more details for this. 

Besides, how to deploy and extend the model to real-world cases, that requires depth information (e.g., ) or semantic information (e.g., traffic light)? The authors should provide more discussion about this to make the work more promising.
paper_summary: paper_summary: The article proposes an end-to-end (E2EAD) autonomous driving method called UAD (Unsupervised Autonomous Driving), which achieves autonomous driving on a visual basis without the need for expensive modular design and 3D manual annotation. UAD aims to overcome the limitations of existing E2EAD models that mimic traditional driving stack module architectures. These models typically require carefully designed supervised perception and prediction subtasks to provide environmental information for planning, which require a large amount of high-quality 3D annotation data and consume significant computational resources during training and inference processes.
strength_weakness: strength_weakness: 1. The paper lacks sufficient comparison with other methods(such as Interfuser[1],DriveAdapter[2],DriveMLM[3],VADv2[4]), which acheve better closed-loop performance on carla town 05 long benchmark.
2. In Table 6, the angular design brings too much gain, especially in terms of collision rate (from 1.37% to 0.19% ). It's strange. The angular design is about how to encode the sensor data and should not have so much impact on collision rate.
3. Angular design is widely used in BEV-related works, like PolarFormer[5]. The paper lacks citation about these works.And it's not proper to regard it as the main contribution of the work.
4. The paper lacks a part to introduce the use of 2D tasks as auxiliary tasks.

[1] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 726–737. PMLR, 2023
[2] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. 2023
[3] Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023
[4] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243, 2024
[5] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, Yu-Gang Jiang. PolarFormer: Multi-camera 3D Object Detection with Polar Transformer. AAAI 2023
questions: questions: Why use 2D boxes instead of 2D segmentation of objects? Is it more reasonable to use 2D segmentation labels to link the points in BEV space with the points in image space?
limitations: limitations: 1. Incompele comparison on CARLA benchmark. 
2. Lack citation about angular design. 
3. Angular design is not novel.
4. Some of the experiment results are not that convincing.
paper_summary: paper_summary: This paper aims to discard the requirement of 3D manual annotation in end-to-end autonomous driving by the proposed angular perception pretext task. Besides, this paper proposes a direction-aware learning strategy consisting of directional augmentation and directional consistency loss. Finally, the proposed method UAD achieves superior performance in both open-loop and closed-loop evaluation compared with previous vision-based methods with much lower computation and annotation costs.
strength_weakness: strength_weakness: 1) There is a lack of explanation on how to use ego status. Besides, there should be more experiments about the performance of UAD without ego status.
2) There is a lack of explanation on how many frames are fused and what method is used for temporal fusion (sliding window or streaming).
3) The angular perception pretext task introduces 2D detection information for perception learning on BEV features. According to Table 6, it seems that angular design is very important for UAD. However, BEV-Planner can achieve a not-so-bad result without any 3D manual annotation and 2D detection information. Therefore, verifying the effectiveness of angular design on BEV-Planner will be more convincing.
4) For the proposed angular design, I do not think is very novel. Because the effectiveness of the 2D detection auxiliary head has been verified in BEVFormerV2 [1] and StreamPETR [2], the UAD just converts 2D object detection to BEV segmentation.
5) For the proposed direction-aware learning strategy, although it is useful, it is a method of data augmentation in EBV space, which I do not think is very novel.
questions: questions: 1) I have serious concerns about the use of ego status and temporal fusion. If UAD uses GT ego status when fusing many frames, because there is no cumulative error in the ego trajectory, it will lead to falsely high performance. A better solution is to use predicted ego status instead of GT ego status (such as SparseDrive [3]).
2) Why does the use of a 3D detection head lead to a decrease in performance? What will be the result if UAD uses the online mapping head?
3) What is the performance if UAD uses a 2D detection auxiliary head instead of an angular design? For example, angular queries can be obtained by 2D detection auxiliary head and depth prediction (such as Far3D [4]).


[1] BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision
[2] Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection
[3] SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation
[4] Far3D: Expanding the Horizon for Surround-view 3D Object Detection
limitations: limitations: See the weakness and questions.
paper_summary: paper_summary: This paper addresses the limitations of current end-to-end autonomous driving models that still rely on modular architectures with manually annotated 3D data. The authors propose an unsupervised pretext task that eliminates the need for manual 3D annotations by predicting angular-wise spatial objectness and temporal dynamics. This is achieved through an Angular Perception Pretext that models the driving scene without the need for manual annotation. A self-supervised training approach is introduced to enhance the robustness of planning in steering scenarios. This strategy learns the consistency of predicted trajectories under different augmented views. UAD demonstrates significant improvements in performance over existing methods like UniAD and VAD in both open-loop and closed-loop evaluations. It achieves these improvements with reduced training resources (44.3% of UniAD) and faster inference speed (3.4× faster than UniAD).
strength_weakness: strength_weakness: 1. UAD treats an entire sector as occupied when only a part of it contains an object. This seems imprecise. This could potentially lead to less accurate spatial understanding of the environment. In autonomous driving, overly coarse representations might result in the vehicle making less accurate decisions, such as unnecessary braking or incorrect path planning. Have the authors tried some open world segmentation models for more accurate spatial information?
2.  The paper draft does not provide explicit evidence or analysis on whether UAD can indeed benefit from training on a larger scale of data. The authors could conduct experiments with varying sizes of datasets to empirically evaluate how performance metrics change as more data becomes available. This could provide insights into the benefits of scaling up.
questions: questions: 1. it is not explicitly stated whether UAD and UniAD use the exact same training and test split within the nuScenes dataset.  
2. As UAD only uses basic obstacles for training.  I wonder how will UAD reacts to traffic lights, lane lines, and policeman's gestures? 
3. What is the BEV area designed in the experiment?
limitations: limitations: In the current draft, UAD might be limited to basic obstacle detection and does not extend to the interpretation of traffic signals.
paper_summary: paper_summary: This paper handles the costly modularization and 3D manual annotation in current end-to-end autonomous driving, which proposes an unsupervised pretext task to provide necessary environmental information, as well as a direction-aware training strategy to enhance the robustness in safety-critical steering scenarios. 

The authors conduct comprehensive experiments in both open- and closed-loop evaluation benchmarks, which demonstrate the effectiveness in various metrics. Moreover, the improvements are obtained with much less resource cost and faster inference speed, which is surprising and impressive.

In addition, this paper gives in-depth discussion and performance comparison about the usage of ego status in the open-loop evaluation of nuScenes. The considerable improvement in the intersection rate with the road boundary, which is proposed in recent BEV-Planner, again proves the superiority of the designed pretext task.
strength_weakness: strength_weakness: (1) In this work, the 2D ROIs are crucial for the designed pretext task. I noticed that the authors adopt the open-set 2D detector GroundingDINO to generate the ROIs. Then the results and discussion of using other third-party detectors should be presented.
(2) The proposed method is shown to be efficient with the unsupervised pretext task and self-supervised training strategy, which is nice. It is suggested the authors show the influence of the training data volume (e.g., 25% and 50%).
questions: questions: In the appendix, the authors show that the proposed method still achieves comparative performance even without backbone pretraining, while UniAD dramatically degrades without the pretrained weights of BEVFormer. What do you think are the reasons causing this?
limitations: limitations: It is suggested to provide discussion of limitations and broader impact in the revision.

Comments: title: Second Response to Reviewer UZRG
Response to Reviewer UZRG
Post Rebuttal Comments
Thank you so much for the feedback!
Thank you so much for the feedback!
Final comments
Help Check the Rebuttal, Make Discussions, and Update the Final Recommendations
Discussion Invitation
Discussion Invitation
comment: Thanks for your response. I only raise my score to 4. We are still confused about the design of angular perception, and this paper needs a lot of revisions to clearly describe these details, especially the use of ego status and temporal information.
Thanks again for your feedback. We provide the responses below, hoping to address your concerns adequately:

***
***Discussion-re-Q1: Whether ego status is introduced into the BEV feature like BEVFormer? Experiments without ego status under the BEV-Planner metrics.***

**Discussion-re-A1**: (1) The default version of our method in the manuscript follows BEVFormer, which utilizes "ego status" to align BEV features across different frames for temporal fusion.

(2) Based on the reviewer’s suggestion, we removed **all "ego status" usage** from our framework. In this scenario, the temporal fusion module within the view transformer degrades to single-frame variant (#2 in the table below) and simple-concatenate variant (#4 in the table below, where we simply concatenate the BEV features of two adjacent frames without alignment). Despite these modifications, our UAD still achieves superior performance **under all settings and metrics** (i.e., #1 vs. #2 and #3 vs. #4).

We will include the results and analysis in the revision. Again, thank you for your valuable feedback!

**Table 1:** Ablation on the ego status and temporal fusion. Col. and Int. denote collision rate and intersection rate with road boundary, respectively. **All metrics are aligned with BEV-Planner**.
|# | Method | Ego Status ***in BEV*** | Ego Status ***in Planner*** | Temporal Fusion |  | L2-1s | L2-2s | L2-3s | Avg. L2 || Col.-1s | Col.-2s | Col.-3s | Avg. Col. || Int.-1s | Int.-2s | Int.-3s | Int. Col. |
| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
|1| BEV-Planner | No | No | No | |0.27|0.54|0.90|0.57| |0.04|0.35|1.80|0.73| |0.63|3.38|7.93|3.98|
|2| UAD (Ours) | No | No | No | |0.25|0.43|0.76|0.48| |0.02|0.26|1.23|0.50| |0.37|1.53|4.48|2.13|
|3| BEV-Planner | No | No | Yes | |0.30|0.52|0.83|0.55| |0.10|0.37|1.30|0.59| |0.78|3.79|8.22|4.26|
|4| UAD (Ours) | No | No | Yes | |0.22|0.39|0.67|0.43| |0.01|0.21|0.85|0.36| |0.18|1.16|3.51|1.62|
|5| BEV-Planner | Yes | No | Yes | |0.28|0.42|0.68|0.46| |0.04|0.37|1.07|0.49| |0.70|3.77|8.15|4.21|
|6| UAD (Ours) | Yes | No | Yes | |0.21|0.35|0.60|0.39| |0.01|0.17|0.78|0.32| |0.16|1.12|3.46|1.58|
|7| BEV-Planner | Yes | Yes | Yes | |0.16|0.32|0.57|0.35| |0.00|0.29|0.73|0.34| |0.35|2.62|6.51|3.16|
|8| UAD (Ours) | Yes | Yes | Yes | |0.13|0.28|0.48|0.30| |0.00|0.12|0.55|0.22| |0.10|0.80|2.48|1.13|

***
***Discussion-re-Q2: Does this mean that the UAD does not use streaming temporal fusion?***

**Discussion-re-A2**: The temporal fusion in the view transformer of BEVFormer is categorized as streaming temporal fusion, where the BEV features from the past frame are fused into the next frame. We adhere to this pipeline in our UAD method. We will clarify this point in the revision. Thank you for your insightful feedback.

***
***Discussion-re-Q3: (1) 2D detection task is not necessarily applied for downstream tasks. In fact, StreamPETR utilizes the 2D auxiliary head to improve the performance. UAD projects the 2D boxes into the BEV and applies a mask prediction task. It seems to me that only the form of the prediction has changed. (2) And I'm not clear about the difference in performance for angular perception and 2D auxiliary head, since authors do not provide the experiments in Q8 of the previous rebuttal.***

**Discussion-re-A3**: Thanks for the comment.

(1) We agree that the output of auxiliary tasks is not necessary for downstream ones. However, the key difference between the mentioned 2D auxiliary head and our angular perception lies in our primary objective: we designed the model to extract compact environmental knowledge using angular queries instead of just improving performances with auxiliary tasks. In other words, our spatial module provides a new approach to summarize environmental information in the E2E framework.

(2) The angular queries in our method effectively summarize the environmental information, ensuring lossless information transmission. When using 2D auxiliary heads and the sparse queries derived from these heads, we do not observe significant differences compared to existing 3D detection heads. This is why we argue that the results would likely be worse than those achieved with a 3D auxiliary detection head (please refer to Table 8 of the manuscript). Moreover, our temporal representation learning is specifically tailored to the angular design, enabling the prediction of future dynamics within each sector region: a task that cannot be performed effectively with 2D object queries.

Again, we emphasize that our core philosophy is to eliminate the need for costly modularized designs and manual annotations.

***
We sincerely thank the reviewer for the time and efforts, and remain committed to clarifying further questions.
Thanks for your reply. 

For Q1, I think there should be the results of experiments without ego status under the BEV-Planner metrics for a clearer comparison. Besides, UAD uses the structure of BEVFormer, so whether ego status is introduced into the BEV feature like BEVFormer?

For Q2, does this mean that the UAD does not use streaming temporal fusion?

For Q3, 2D detection task is not necessarily applied for downstream tasks. In fact, StreamPETR utilizes the 2D auxiliary head to improve the performance. UAD projects the 2D boxes into the BEV and applies a mask prediction task. It seems to me that only the form of the prediction has changed. And I'm not clear about the difference in performance for angular perception and 2D auxiliary head, since authors do not provide the experiments in Q8 of the previous rebuttal.
Thank you for taking the time to offer suggestions amidst your busy schedule, yet we still need to point out several misunderstandings by the reviewer about our rebuttal. The detailed responses are provided below, hoping to address your concerns adequately:

***
***Discussion-Q1: Experiments without ego status are required. The BEV-Planner contains relevant experiments without ego status.***

**Discussion-A1**: As clarified in #Rebuttal-Q1-A1, **only the rhombus mark** in Tab.1 of the manuscript denotes the version using ego status. The other experiments are **all** conducted **without ego status** for fair comparisons. Our planner achieves the best open-loop performance in nuScenes for both w/. and w/o. ego-status settings, as shown in Tab.1 of the manuscript. We will clarify this in the revision. Thanks.


***
***Discussion-Q2: How many frames are fused? Is streaming time fusion used to fuse many frames?***

**Discussion-A2**: As clarified in #Rebuttal-Q2-A2, we use the view transformer from BEVFormer[1] to encode BEV features, which performs temporal fusion with the BEV feature of the last frame through deformable attentions. In other words, there are total 2 frames for temporal fusion, i.e. t-1 and t. For more details, please refer to [1]. We will clarify this in the revision. Thanks.

[1] Li Z, et al. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV 2022.

***
***Discussion-Q3: I don't think angular perception is very different from 2D detection and depth estimation. Angular perception is just projecting the 2D detection box into BEV, which only involves the additional operation of extrinsic parameter transformation.***

**Discussion-A3**: In #Rebuttal-Q4-A4, we have detailed the fundamental differences between our angular perception and 2d detection/depth estimation, as well as the core idea and contributions of our work. It should be noted that the projection of 2D boxes is only used to generate the pseudo labels for training our pretext task, instead of 'detecting' the object areas for downstream tasks. **The processes of projection and objectness prediction are not performed during inference, since the angular queries have provided compact and comprehensive environmental knowledge**. We will clarify this in the revision. Thanks.

***
Again, we thank the reviewer for the time and efforts, and remain committed to clarifying further questions.
Thanks for your response.  However, after reading the author's rebuttal, some issues still need to be addressed, especially the experiments.
1. Experiments without ego status are required. The BEV-Planner contains relevant experiments without ego status.
2. How many frames are fused? Is streaming time fusion used to fuse many frames?
3. I don't think angular perception is very different from 2D detection and depth estimation. Angular perception is just projecting the 2D detection box into BEV, which only involves the additional operation of extrinsic parameter transformation.
Thanks the authors for their detailed feedback. All my concerns have been addressed. I very like this paper due to its novelty, great performance and simplicity. I believe this is a potentially right way for end-to-end autonomous driving. 

I also see other reviewers' comments and the authors' rebuttals, and think that the authors have done a good job to explain their work better. 

Thus, I keep my original score (Very Strong Accept).
Dear Reviewer 3W73,

Thank you again for your kind review and constructive comments. Your suggestions greatly strengthen our work and improve our manuscript's quality. We will include them in the revision.

Best regards,

Authors
Dear Reviewer d489,

We are sincerely grateful for your decision to raise the score. Your suggestions have greatly enhanced our paper and inspired our future research direction.

Regarding the application of our work in the real world, (1) we share the same expectation as the reviewer in extending our work to autonomous vehicles. And We are also actively working on implementing our approach in real-world auto-vehicles.  (2) We provided a transition plan in the paper to integrate our approach into current autonomous driving systems. Specifically, our framework can easily accommodate a typical perception model for post-processing. (3) We believe our paradigm will unlock more potential from data. Given that billions of data points are used in real-world scenarios to ensure the robustness of autonomous driving, it is nearly impossible to precisely annotate all the data. Therefore, it is essential to advance efficient and unsupervised end-to-end research. In addition to addressing real-world application needs, we aim to set a potential direction for unsupervised methods in the future. We are grateful that the reviewer recognizes our contribution.

Once again, thank you for your positive feedback and valuable insights! Your suggestions regarding the deeper exploration of working mechanisms, practical considerations, and experimental details will greatly strengthen our work and improve the quality of our manuscript. We will incorporate them in the revision. Code and models will also be released to facilitate future research.

Best regards,

Authors
Thank the authors for the detailed response and information. Some of my concerns are addressed, but I still have problems with the design and working principles of the angular-wise perception module. I have strong concerns about its actual application values in real-world scenes. I will slightly raise the score, and the authors are encouraged to further refine the paper and make it more solid.
Thank you for answering the questions. All my concerns have been addressed. I will keep my initial rating.
Dear Reviewers,

Thanks for serving as a reviewer for the NeurIPS. The rebuttal deadline has just passed.

The author has provided the rebuttal, could you help check the rebuttal and other fellow reviewers' comments, make necessary discussions, and update your final recommendations as soon as possible?

Thank you very much.

Best,

Area Chair
Dear Reviewer UZRG,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

Sincerely,

Authors.
Dear Reviewer XcsQ,

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work.

Sincerely,

Authors.