Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?

## Abstract

Abstract Rapid advancements in Autonomous Driving (AD) tasks turned a significant shift toward end-to-end fashion, particularly in the utilization of vision-language models (VLMs) that integrate robust logical reasoning and cognitive abilities to enable comprehensive end-to-end planning.
However, these VLM-based approaches tend to integrate 2D vision tokenizers and a large language model (LLM) for ego-car planning, which lack 3D geometric priors as a cornerstone of reliable planning.
Naturally, this observation raises a critical concern: Can a 2D-tokenized LLM accurately perceive the 3D environment? Our evaluation of current VLM-based methods across 3D object detection, vectorized map construction, and environmental caption suggests that the answer is, unfortunately, NO .
In other words, 2D-tokenized LLM fails to provide reliable autonomous driving.
In response, we introduce DETR-style 3D perceptrons as 3D tokenizers, which connect LLM with a one-layer linear projector. This simple yet elegant strategy, termed Atlas, harnesses the inherent priors of the 3D physical world, enabling it to simultaneously process high-resolution multi-view images and employ spatiotemporal modeling.
Despite its simplicity, Atlas demonstrates superior performance in both 3D detection and ego planning tasks on nuScenes dataset, proving that 3D-tokenized LLM is the key to reliable autonomous driving. The code and datasets will be released.

## 1 Introduction

Figure: Figure 1: Comparision among end-to-end methods. (a) Modular BEV-based methods have three sequential modules for perception, prediction, and planning, but they cannot provide multiple potential trajectories and environment reasoning. (b) 2D-tokenized VLM projects 2D distorted images into tokens, which lack 3D prior for reliable autonomous driving. (c) Our 3D-tokenized LLM-based methods utilize 3D perceptions as 3D tokenizers, which provide potential trajectories and rich 3D priors for reliable driving.
Refer to caption: x1.png

Autonomous Driving (AD) is a sophisticated system that integrates perception, reasoning, and planning ; ; .
Perception serves as the initial stage, capturing details of the surrounding environment. This information then feeds into the reasoning component, facilitating a deeper understanding, and ultimately guiding informed decision-making through the planning process.
Recently, the incorporation of perception, reasoning, and planning to construct end-to-end models has become prevalent.
It can be broadly categorized into two distinct methodologies: modular bird’s-eye view (BEV) based approaches and large vision-language model (VLM) based methods.

The modular BEV-based approaches are meticulously engineered, comprising custom-tailored modules, including 3D perception, trajectory prediction, and ego-car planning ; ; ; ; ; ; , as shown in Figure 1(a).
While BEV representation enhances environmental perception, these methods may encounter difficulty stemming from their limited reasoning abilities.
Specifically, these models tend to mimic established expert trajectories and struggle to predict multiple potential motion trajectories when confronted with novel scenarios.
To tackle this challenge, VLM-based methods mark a significant turning point.
They usually employ a 2D vision tokenizer (e.g., ViT-CLIP ) with a Large Language Model (LLM) to interpret distorted images and produce navigational commands ; ; ; ; ; . Benefiting from the robust logical reasoning and cognitive abilities of the VLM agent, the model can generate rational decisions and dialogues.

Despite the success of VLM-based algorithms, the perceptual capabilities within this paradigm is barely studied. While we argue that the perception sub-task may not be essential for end-to-end driving, the capacity to perceive the environment remains a cornerstone of reliable planning. Since VLM-based methods rely on 2D vision tokenizers for environmental perception without incorporating 3D geometric priors, an intuitive question arises:
Can a 2D-tokenized LLM accurately perceive the 3D environment? To answer this question, we specially design experiments to evaluate the perception performance of prevalent VLM-based systems in three tasks: 3D object detection, 3D lane detection, and environmental captioning.
Our findings reveal that despite extensive pre-training and expansive parameters, mainstream VLM solutions typically lag in precision when compared to specialized models designed for these tasks. This glaring gap highlights the limitations of 2D tokenizers in perceiving 3D environments.

To address this issue, we wonder if 3D vision tokenizers hold the key to Pandora. We discover that the existing DETR-style BEV framework can naturally serve as a 3D visual compression tokenizer.
Therefore, we opt for the advanced StreamPETR  and TopoMLP  as our 3D visual tokenizers, forgoing the traditional use of ViT-CLIP .
This strategy brings three advantages: 1) The innate priors of the 3D physical world are naturally encoded within visual tokens by introducing the position encodings. 2) It is capable of handling high-resolution images with any aspect ratio without the risk of distorting the images. 3) Video frames can be processed in a streaming manner, benefiting from DETR-style query propagation. Through evaluation of the nuScenes dataset, we demonstrate that our 3D-tokenized LLM approach achieves performance on par with specialized algorithms in tasks such as 3D object detection and lane detection.

Beyond that, we need to answer another question: Is a 3D-tokenized LLM the key to reliable autonomous driving? Following BEV-Planner , we extend our exploration to the open-loop planning on the nuScenes dataset. By leveraging the 3D tokenizers for enhanced perception capabilities, our model not only comprehends the environment around the vehicle but also utilizes the LLM to formulate driving recommendations and plan the ego-car trajectory in an end-to-end manner. Remarkably, this approach eschews hand-crafted designs and achieves state-of-the-art performance on the nuScenes planning task.

In summary, our work highlights the importance of proper vision tokenizers in VLM-based AD and introduces the 3D-tokenized LLM as a solution. We showcase its superiority in adeptly addressing challenges across multiple tasks such as 3D perception, vectorized map construction, environmental caption, and planning within autonomous driving systems. Our model demonstrates superior performance in both benchmark evaluations and practical downstream applications, proving its reliability and versatility. Furthermore, our framework paves the way for pioneering end-to-end LLM-driven solutions in autonomous driving, potentially transforming how these systems are developed.

## 2 Can a 2D-Tokenized LLM Accurately Perceive 3D Environment?

Current VLM-based methods ; ; ; ; in AD tend to employ 2D vision tokenizers.
They operate without incorporating geometric 3D priors, raising concerns about their capability to accurately perceive and describe 3D environments, which is crucial for reliable planning.
In this section, we provide insightful analysis and reveal the limitations of relying solely on 2D tokenizers for understanding 3D driving scenes, including 3D perception and visual captioning.

### 2.1 2D-Tokenized LLM for Perception

To investigate the 3D understanding capability of current VLM-based approaches, we first conduct experiments on traditional perception tasks: 3D object detection and 3D lane detection.
In this part, we introduce datasets, models, and metrics.

Figure: Figure 2: Brief answer format of datasets. It transforms several tasks, such as 3D object detection, map perception, environment caption, and ego-car planning, into a uniform text format. We discretize the bird’s-eye view (BEV) space, spanning from -50 meters to +50 meters, into 1,000 bins.
Refer to caption: x2.png

Datasets.
We design datasets tailored for VLM methods built upon popular multi-view benchmark nuScenes , as shown in Figure 2.
For the 3D detection task, we construct question-and-answer (QA) pairs that focus on pinpointing the locations of objects surrounding the ego vehicle.
Each question prompts the model to extract spatial information about the target objects from six views. The corresponding answers require the model to identify both the category and the 3D coordinates of objects.
Similarly, the dataset for 3D lane detection also comprises QA pairs, whose answers are lane points borrowed from OpenLane-V2 subset-B . Here, each road is depicted using four consecutive points describing the road centerline.
More details can be found in the supplementary.

Models.
All 2D-tokenized LLMs in our study adhere to a uniform architecture, which consists of three main components: 2D tokenizer, projector, and large language model.
The 2D tokenizer follows ViT-CLIP  to extract visual features from multiple perspectives of images.
For the projection module, we incorporate a single convolutional layer to bridge the 2D tokenizer and LLM.
Besides, we utilize diverse pre-trained LLMs, such as LLaMA , LLaVA , Vicuna , which are comprehensive processing of complex visual information to generate the perception of the environment, to prove consistency and fairness in our exploration.
Additionally, another available VLM-based model pre-trained on 2D object detection Merlin  is also evaluated.

Metrics.
In this study, we employ the F1 score as the main evaluation metric.
The choice of the F1 score is motivated by two primary considerations:
First, VLMs cannot deliver the necessary predictive confidence for metrics such as mean Average Precision (mAP).
Second, traditional perceptual metrics commonly encourage numerous redundant predictions, which can clutter the model output. In contrast, VLMs are designed to generate more targeted and focused predictions, making the F1 score a better fit for assessing these models.
In this work, for 3D detection, we choose threshold distances of 0.5, 1.0, 2.0, and 4.0 meters to define positive predictions, similar to the discrimination levels used in detection mAP calculations.
As for 3D lane detection, we follow OpenLane-V2 evaluation protocol  to compute the F1 score.

**Table 1: Comparisons with task-specific and VLM-based methods for 3D object detection tasks using our proposed dataset. The bold numbers represent the highest accuracy achieved in each category. The Pk, Rk, and F1k represent the Precision, Recall, and respective F1 score ultimate $k$ as threshold distances to define positive prediction. The Spe. represents task-specialist model.**
|  | Method | Tokenizers | P0.5 | R0.5 | F10.5 | P1.0 | R1.0 | F11.0 | P2.0 | R2.0 | F12.0 | P4.0 | R4.0 | F14.0 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Spe. | PETR | - | 12.4 | 21.5 | 15.8 | 20.0 | 30.5 | 24.1 | 27.5 | 37.7 | 31.8 | 33.8 | 42.6 | 37.7 |
| StreamPETR | - | 22.7 | 41.3 | 29.3 | 31.6 | 49.5 | 38.6 | 38.1 | 54.2 | 44.7 | 42.5 | 56.9 | 48.7 |  |
| VLM | LLaMA | 2D | 0.3 | 1.1 | 0.4 | 0.6 | 2.6 | 1.0 | 1.5 | 5.8 | 2.4 | 3.5 | 12.8 | 5.5 |
| LLaVA | 2D | 2.0 | 20.3 | 3.0 | 3.6 | 35.7 | 6.5 | 6.5 | 50.3 | 11.6 | 10.9 | 62.8 | 18.9 |  |
| Vicuna | 2D | 2.0 | 20.1 | 2.5 | 2.9 | 35.6 | 5.4 | 5.9 | 51.1 | 10.1 | 9.4 | 63.8 | 16.4 |  |
| Merlin | 2D | 3.0 | 22.5 | 5.3 | 4.1 | 36.1 | 7.4 | 6.6 | 52.6 | 11.7 | 12.1 | 64.3 | 20.4 |  |
| Atlas(Ours) | 3D | 15.0 | 61.2 | 24.1 | 27.2 | 74.0 | 39.8 | 36.2 | 79.2 | 49.7 | 41.2 | 81.2 | 54.6 |  |

3D Object Detection.
In this study, we conduct extensive experiments to evaluate the performance of VLMs on 3D detection, as listed in Table 1.
As a comparison, Table 1 also includes task-specific models such as PETR  and StreamPETR . Among these, the state-of-the-art detector StreamPETR achieves an F14.0 score of 48.7.
Despite the rich contextual knowledge and extensive parameters, 2D-tokenized LLM methods exhibit a considerable performance drop in both precision and recall, leading to surprisingly low F1 scores.
These methods struggle with detecting objects in the vicinity of the ego vehicle, highlighting a considerable disparity in 3D object detection capabilities between VLM-based methods and dedicated task-specific approaches.

**Table 2: 3D lane detection.**
| Method | Tokenizers | P | R | F1 |
| --- | --- | --- | --- | --- |
| TopoMLP | - | 50.6 | 55.7 | 53.0 |
| LLaVA | 2D | 10.4 | 9.8 | 10.0 |
| Vicuna | 2D | 11.7 | 10.3 | 10.9 |
| Merlin | 2D | 22.1 | 22.4 | 22.2 |
| Atlas(ours) | 3D | 45.7 | 39.1 | 42.2 |

3D Lane Detection.
Vectorized maps provide a driving route for ego car, serving as a crucial perception task for autonomous driving.
We present experiments of a state-of-the-art task-specific model TopoMLP  and several aforementioned 2D-tokenized LLM methods on lane detection.
The main results are shown in Table 2.
Similarly, the performance of 2D-tokenized LLM methods is far away from the task-specific model, struggling to deal with 3D lane detection.

### 2.2 2D-Tokenized LLM for Captioning

Figure: Figure 3: Comparsion between 2D-tokenized and our 3D-tokenized VLMs on driving caption. The 2D-tokenized VLM sometimes generates "hallucinated" descriptions, while our 3D-tokenized VLM is able to produce accurate and comprehensive captions for driving environment.
Refer to caption: x3.png

In addition to basic environmental perception tasks, LLMs can be adapted to perform more complex tasks like extracting and interpreting key features from visual for captioning environments. This capability extends the utility of LLMs in practical applications, and leverages world knowledge and reasoning ability, particularly in scenarios requiring detailed environmental understanding.

To explore whether a 2D-tokenized LLM could serve as an effective perceptron, we develop a specialized version of the model for environmental captioning.
This variant utilizes Vicuna  as its underlying LLM, tasked with capturing and describing the operational environment of a vehicle.
This description includes various elements such as the location and quantity of nearby vehicles and pedestrians, traffic dynamics, concerning surrounding lanes of pedestrian crossing and road.

Despite the advanced capabilities of VLMs in generating natural language descriptions, as illustrated in Figure 3, our findings indicate that the 2D-tokenized LLM struggles with accurate environmental perception. The model frequently produces erroneous or "hallucinated" descriptions, which suggests that it still falls short of reliable perception in practical applications. This underscores the challenges and limitations inherent in deploying LLMs for complex perceptual tasks in dynamic environments.

Remark.
To sum up, the experiments above reveal a significant limitation in the perception capabilities of LLMs that rely on 2D visual tokenizers. This limitation poses serious challenges for reliable ego vehicle planning.
We claim that the primary reason for this limitation lies in the inability of 2D visual tokenizers to effectively integrate 3D spatial priors.
To address the limitation, we introduce advanced pre-trained 3D perception models as 3D tokenizers in the following section.

## 3 3D-Tokenized LLM for Reliable Autonomous Driving

### 3.1 3D-Tokenized LLM

Distinct from 2D-tokenized LLM, we introduce 3D tokenizers founded upon a DETR-inspired architecture into LLM, formulating a 3D-tokenized LLM framework, named Atlas.
In specific, Altas consists of three primary components. Initially, the model employs 3D tokenizers, StreamPETR  and TopoMLP , to process multi-view images into DETR-style query representations. Following this, these queries are streamlined through a single linear layer, functioning as a projector, to align with the LLM. The final component of Atlas is an LLM, designed as Vicuna .
This approach brings significant benefits in incorporating 3D innate prior, achieving high resolution, and facilitating temporal propagation, as previously elaborated.

3D Environment Perception.
The performance of Atlas is evaluated on standard datasets tailored to the tasks of 3D object detection and 3D lane detection, as reported in Table 1 and Table 2.
The results demonstrate that 3D-tokenized LLM achieves remarkable performance across both tasks.
Besides, 3D-tokenized LLM performs better than 2D-tokenized LLM on driving environment captioning, as shown in Figure 3, thereby affirming the significant advantages of utilizing 3D tokenizers.
In addition to representing 3D environment, our ultimate goal is to achieve reliable autonomous driving.
In the following, we will evaluate the performance of 3D-tokenized LLM on ego-car planning.

### 3.2 Implementation

The whole model trains with 8 Tesla A100 GPUs, with training times of approximately 100 hours.

Dataset.
We employ the nuScenes planning dataset  in our experiments of reliable autonomous driving.
As illustrated in Figure 2, we have reformatted the planning data into a question-answer format to facilitate our analysis.
Previous research  has established that the "ego states"—sensor-provided data on the autonomous vehicle such as velocity, acceleration, yaw angle, and historical trajectory—play a crucial role in open-loop planning. Additionally, to aid in navigation, especially at intersections, it is essential to incorporate a high-level command (e.g., go straight, turn left, turn right) which provides directional guidance.
Building on these insights, we propose the question-and-answer pairs demand the models to predict future velocity and acceleration based on the current state and to subsequently generate planning waypoints for the ego-car prompting by a high-level command. This processing called chain-of-thought , not only enhances the interpretability of the model’s reasoning process but also its reliability. A typical example is shown in Figure 2, and additional details about the dataset are available in the supplementary materials.

Metrics. We adhere to standard practices by utilizing the implementation provided by ST-P3  to assess planning over time horizons of 1s, 2s, and 3s. We assess the performance with two widely accepted metrics: the L2 error calculated by comparing the predicted trajectories of the ego vehicle with the ground-truth trajectories at corresponding waypoints, and the collision rate calculated by checking for any intersections between the ego vehicle and other entities within the scene.

### 3.3 Main Results

**Table 3: Comparisons on the planning. For a fair comparison, we refer to the reproduced results in BEV-Planner . The bold numbers represent the highest accuracy.**
| Method | High-level | Ego States | L2 (m) | Collision (%) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Command | Bev | Planner | 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. |  |
| FF | ✘ | ✔ | ✔ | 0.55 | 1.20 | 2.54 | 1.43 | 0.06 | 0.17 | 1.07 | 0.43 |
| ST-P3 | ✔ | ✘ | ✘ | 1.59 | 2.64 | 3.73 | 2.65 | 0.69 | 3.62 | 8.39 | 4.23 |
| ✔ | ✔ | ✔ | 1.33 | 2.11 | 2.90 | 2.11 | 0.23 | 0.62 | 1.27 | 0.71 |  |
| UniAD | ✔ | ✘ | ✘ | 0.59 | 1.01 | 1.48 | 1.03 | 0.16 | 0.51 | 1.64 | 0.77 |
| ✔ | ✔ | ✔ | 0.20 | 0.42 | 0.75 | 0.46 | 0.02 | 0.25 | 0.84 | 0.37 |  |
| VAD-Base | ✔ | ✘ | ✘ | 0.69 | 1.22 | 1.83 | 1.25 | 0.06 | 0.68 | 2.52 | 1.09 |
| ✔ | ✔ | ✔ | 0.17 | 0.34 | 0.60 | 0.37 | 0.04 | 0.27 | 0.67 | 0.33 |  |
| Ego-MLP | ✔ | ✘ | ✔ | 0.15 | 0.32 | 0.59 | 0.35 | 0.00 | 0.27 | 0.85 | 0.37 |
| BEV-Planner | ✔ | ✘ | ✘ | 0.30 | 0.52 | 0.83 | 0.55 | 0.10 | 0.37 | 1.30 | 0.59 |
| ✔ | ✔ | ✔ | 0.16 | 0.32 | 0.57 | 0.35 | 0.00 | 0.29 | 0.73 | 0.34 |  |
| LLaVA | ✔ | ✘ | ✘ | 1.04 | 1.74 | 2.57 | 1.79 | 0.58 | 1.17 | 1.74 | 1.16 |
| Vicuna | ✔ | ✘ | ✘ | 1.06 | 1.80 | 2.54 | 1.80 | 0.60 | 1.21 | 1.78 | 1.20 |
| Merlin | ✔ | ✘ | ✘ | 1.03 | 1.71 | 2.40 | 1.71 | 0.48 | 1.05 | 1.77 | 1.10 |
| Atlas | ✘ | ✘ | ✘ | 1.69 | 1.89 | 2.25 | 1.94 | 0.51 | 0.85 | 1.44 | 0.93 |
| ✔ | ✘ | ✘ | 0.52 | 0.97 | 1.53 | 1.00 | 0.15 | 0.31 | 0.70 | 0.38 |  |
| ✔ | ✔ | ✔ | 0.18 | 0.21 | 0.26 | 0.21 | 0.12 | 0.13 | 0.16 | 0.13 |  |

In this section, we evaluate the performance of our proposed method, Atlas, by comparing it against existing state-of-the-art (SoTA) BEV-based planners, as detailed in Table 3. Our experimental results reveal that Atlas achieves substantial improvements over the SoTA methods, reducing the average L2 metric by 40.0% and the average Collision metric by 60.6%. These significant enhancements corroborate the effectiveness of the 3D-tokenized LLMs, which we consider as the key to reliable autonomous driving.

Further, to ascertain whether the performance improvements are solely attributable to the inclusion of ego state information—a frequent topic of discussion within the community—we conduct additional experiments by removing the ego state data during both training and testing.
In this experimental setting, compared to the prevailing VLM-based methods, our Atlas demonstrates superior performance and robustly validates the effectiveness of 3D tokenizers.
Despite this, Atlas continues to outperform other BEV-based methods in terms of collision rates. However, the performance on the L2 metric is comparable to other methods. We hypothesize that this outcome may stem from the inherent capabilities of the LLM to predict multiple potential motion trajectories and make rational decisions, which, while confronted with novel scenarios, deviate from the ground truth.

### 3.4 Ablation Study

To avoid unnecessary misunderstandings, our ablation does not introduce any ego states.

**Table 4: A set of ablative studies on 3D object detection and ego-car planning. The adopted algorithm designs and hyper-parameter settings are marked in bold. See §3.4 for details.**
|  | 3D Detection | Planning |  |  |
| --- | --- | --- | --- | --- |
| F11.0 | F12.0 | Avg. L2 | Avg. Col. |  |
| Vicuna | 5.4 | 10.1 | 2.19 | 2.75 |
| $+$QR | 30.7 | 41.2 | 1.22 | 0.62 |
| $+$ RP | 34.6 | 46.5 | 1.10 | 0.44 |
| $+$ MQ | 39.8 | 49.7 | 1.00 | 0.38 |

Component Effect.
We conduct ablation studies to analyze our proposed 3D-tokenized LLM, considering several key aspects: query representation, reference point embedding, and memory queue, all decoupling from StreamPETR on 3D detection and planning.
The results are summarized in Table 4(a).
Our experiments demonstrate that each component progressively enhances performance in both tasks. Furthermore, we observe a synergistic effect where improvements in one task appear to amplify accuracy in the other, strongly proving that the capacity to perceive the environment remains a cornerstone of reliable planning.

3D Tokenizers.
We investigate the effectiveness of various 3D tokenizers for ego-car planning, which are the central enhancements introduced in our study. The results are shown in Table 4(b). The tokenizers we evaluate include PETR (PT) , StreamPETR (SP) , and TopoMLP (TM) . Our incorporation of progressively advanced 3D perceptrons into LLM demonstrates a notable improvement in planning performance, underscoring the significance of 3D perception in achieving robust autonomous driving. Furthermore, we integrate TopoMLP to provide supplementary lane line information. This addition results in a modest enhancement in performance, suggesting the potential benefits of incorporating contextual roadway features into the motion planning process.

Resolution.
Our approach integrates 3D tokenizers with adjustable image resolution capabilities, which aligns well with real-world applications in autonomous driving.
As Table 4(c) presents, we observe that increasing the image resolution leads to a noticeable improvement in performance. This evidence indicates that our method holds significant advantages over traditional VLM techniques, particularly in terms of flexibility and efficacy in handling diverse image resolutions.

Reference Point Embeddings.
Our Atlas introduces an important concept: 3D tokenizers equipped with reference point embeddings, following the setting of StreamPETR  and TopoMLP .
Here, we evaluate the model performance of decoupling reference point embedding and query embedding.
Our initial approaches relied solely on query representations (i.e., "none" in Table 4(d)), which overlooks the crucial 3D spatial context—termed as the reference point. However, as shown in Table 4(d), simply applying conventional embedding techniques , like sin-cos position embedding and learned position embedding, to 3D queries do not markedly influence performance. This outcome underscores the unique advantages of reference points.
To effectively utilize this, we incorporate offset mappings from the reference points via a single layer projector aka reference point embeddings to the 3D query representation (i.e., "RP" in Table 4(d)). Notably, this method achieves remarkable improvements in accuracy, underscoring its effectiveness.

Pretrained LLMs.
In our experiments, we evaluate different LLMs that varied in their pre-training methodologies, as detailed in Table 4(e). Our results show that LLMs pre-trained with methods that align text and images significantly outperform others in planning tasks. We attribute this enhanced performance to the multimodal nature of their training.
Additionally, our analysis reveals that models pre-trained with various visual-language data exhibited no significant differences in planning performance. We believe this is due to the absence of 3D data in their pre-training processes, suggesting that the inclusion of 3D data in pre-training, as 3D tokenizers do, is necessary.

**Table 5: Effective of the chain of thought.**
| chain | Avg. L2 | Avg. Col. |
| --- | --- | --- |
| P | 1.33 | 0.79 |
| V-P | 1.21 | 0.61 |
| V-A-P | 1.00 | 0.38 |
| V-A-Y-P | 1.15 | 0.55 |
| V-A-T-P | 1.40 | 0.81 |
| P-V-A | 1.01 | 0.40 |

Chain of Thought.
In the realm of autonomous driving, recent works ; converge on a key insight: the state of the ego vehicle is a pivotal factor in shaping open-loop planning strategies.
To this end, we delineate the ego states into four distinct yet interrelated dimensions: velocity (V), acceleration (A), yaw angle (Y), and the historical trajectory (T).
To evaluate the influence of each dimension, we conduct ego planning based on the predicated of these parameters. The experimental findings, as outlined in Table 5, where "P" denotes "Planning". Notably, our results diverge from prevailing research, indicating that the yaw angle and historical trajectory do not enhance the efficacy of the planning process. This counterintuitive outcome is likely a consequence of the inherent difficulties in the precise forecasting of these variables ; .
Moreover, we discover an interesting aspect of our model’s robustness: the sequence in which these parameters are predicted does not impact the performance. This suggests that altering the order of prediction (e.g., reversed) does not increase computational time.

### 3.5 Qualitative Results

Figure: Figure 4: Qualitative results with diverse planning from Atlas. The five planning trajectories presented here are generated through five iterations of utilizing our 3D-tokenized LLM. It is obvious that Atlas is able to output different potential planning trajectories thanks to LLM.
Refer to caption: x4.png

We also conduct a qualitative analysis by visualizing the trajectory predictions made by Atlas, as shown in Figure 4.
We execute the 3D-tokenized LLM five times to produce five depicted planning trajectories.
The results demonstrate that Atlas is capable of generating multiple feasible plans for autonomous driving that are not only practical but also adhere to safety standards. Specifically, Atlas successfully devises various potential routes tailored to distinct driving scenarios, including following other vehicles, lane changing, and overtaking. Importantly, Atlas effectively identifies and avoids pedestrians and cars, showcasing its robust capability in ensuring road safety.

## 4 Related Work

DETR-style BEV Perception.
DETR  is initially proposed to address the challenge of end-to-end detection, and further extensively applied in BEV perception ; ; ; , thereby significantly advancing its development. DETR3D  is a pioneering work that introduces the concept of 3D object queries, which interact with multi-view image features to produce sparse yet informative object representations. Further, PETR ; introduces the concept of 3D position encoding, and BEVFormer  brings BEV temporal modeling.
StreamPETR  and Sparsev2  use object queries as a vessel for temporal modeling, effectively propagating temporal information while achieving SoTA performance with commendable efficiency.
In a notable finding within StreamPETR, the inclusion of additional multi-frame image feature interactions does not enhance performance, suggesting that the highly compressed object queries are sufficiently expressive to encapsulate all necessary information for BEV perception.
Moreover, the application of DETR framework has been expanded to map queries by works such as MapTR , TopoNet  and TopoMLP , which are instrumental in the construction of vectorized map representations.

BEV-based End-to-end Driving.
Traditional autonomous driving systems have often relied on manual rules for planning, which can be cumbersome and complex, struggling to cover the numerous corner cases. In recent years, there has been a pronounced shift towards end-to-end autonomous driving approaches, which have demonstrated significant progress in simplifying and streamlining the autonomous driving pipeline. UniAD  is a pioneering work that introduces an end-to-end framework encompassing tasks such as perception, prediction, and planning, with these tasks executed sequentially to ultimately produce control outputs. Building upon this framework, VAD  further streamlines the pipeline, enhancing efficiency and reducing complexity. However, AD-MLP  and BEV-Planner  have observed that existing end-to-end methods can achieve high performance on open-loop benchmarks like nuScenes  by simply fitting to the ego status of the autonomous vehicle. This finding suggests that the integration of planning and control in these models may not yet fully capture the complexities of real-world driving scenarios. Subsequent works, such as Think-Twice  and VADv2 , have made substantial advancements in closed-loop simulators like Carla . Following BEV-Planner , we present results with and without the ego status to address the open-loop challenges on the nuScenes  benchmark.

VLM-Agent for Autonomous Driving.
The visual-language model (VLM) domenstrates promising results in the fields of visual-language understanding and logical reasoning, and has been extended to autonomous driving  ; ; ; ; ; . DriveGPT4  employs a VLM model to predict driving commands and provide rational explanations for its decisions. DriveLM  excels at conversing about environmental information, while ADriver-I  focuses on predicting low-level vehicle signals. Furthermore, DriveMLM  and LMDrive  have implemented end-to-end autonomous driving solutions and validated effectiveness on CARLA  closed-loop benchmarks, showcasing the potential of VLM-based agents.
Despite impressive progress, no work explores how the 3D-tokenized LLM influences real-life autonomous driving.

## 5 Conclusion and Limitations

In this paper, we explored VLM-based methods increasingly used in autonomous driving, focusing first on perception. We found large gaps between task-specific and 2D tokenized LLM-based methods in environmental perception, which is essential for reliable autonomous driving.
To address these gaps, we introduced Atlas, a system combining DETR-style 3D perceptrons with LLMs. This approach integrates 3D priors for better depth perception and supports high-resolution, multi-view images, and temporal modeling through query propagation.
Our evaluation of Atlas on nuScenes dataset revealed substantial improvements in 3D detection and planning, surpassing established methods. This confirms our belief that 3D-tokenized LLM is the key to reliable autonomous driving.

Limitations. This paper aims to demonstrate the effectiveness of the 3D tokenizer for VLM-based autonomous driving. Although our method has demonstrated outstanding performance in open-loop planning, it has not yet been tested on a closed-loop dataset. However, existing close-loop benchmarks (e.g., CARLA ) lack reality, which fails to verify our motivation. Moreover, this paper lacks of performance comparison with VLM-based AD methods. This omission is due to the proprietary codes for these methods.

## Appendix A Datasets Details

To align with the requirements of VLM-based models, the necessary step is to transform all evaluation datasets into a textual format, specifically structured as question-answer pairs.
In this section, we will delve deeper into the specifics of various datasets, including 3D object detection (§A.1), 3D lane detection (§A.2), driving captioning (§A.3), and ego planning (§A.4).

Figure: Figure 5: Our constructed question-answer pairs for VLM-based methods. It transforms several critical driving reasoning tasks, such as 3D object detection, map perception, environment caption, and ego-car planning, into a uniform text format.
Refer to caption: x5.png

### A.1 3D Object Detection

The 3D object detection utilized in the VLM-based method (2D/3D visual tokenizers with LLM) evaluation is based on nuScenes .
To adapt to the inputs and outputs of LLM, we convert the detection task into a text-format question-answer task.
Here, the question is randomly sampled from a pool that is listed in Table 6.
As seen, we set a special token ‘<query>’ to accept tokens from 3D tokenizers.
If the inputs are six-view images, we replace the text ‘They are uniformly represented as queries embeddings<query>’ in question with ‘They represent left rear image<query>, left front image<query>, direct front image<query>, right front image<query>, right rear image<query>, and direct rear image<query>.’.
As for the answer, we choose the category name and 3D center points of each bounding box, as shown in Figure 5.
To facilitate more efficient localization, we discretize the bird’s-eye view (BEV) space ranging from -50 meters to +50 meters into 1,000 bins.

Table: Table 6: Question pool of 3D object detection for VLM-based methods.

### A.2 3D Lane Detection

We formulate a 3D lane detection dataset with question-answer pairs based on the OpenLane-V2 Subset-B , which itself originates from the nuScenes dataset.
A representative is shown in Figure 5.
Their questions are sampled from Table 7, and the corresponding answer comprises a set of four lane points. Analogous to the 3D object detection dataset, we discretize the BEV space, spanning from -50 meters to +50 meters, into 1,000 bins.

Table: Table 7: Question pool of 3D lane detection for VLM-based methods.

### A.3 Driving Captioning

Our driving captioning dataset is created through the annotation of nuScenes, leveraging the capabilities of GPT-4V. The specific prompt utilized in GPT-4V is detailed in Table 8, while an illustrative example is presented in Figure 5. It is worth mentioning that, to harness the full potential of GPT-4V, we request a unique description for each individual view, resulting in a total of approximately 180k question-answer pairs.

Table: Table 8: Prompt used in GPT-4V for caption generation.

### A.4 Ego Planning

Similar to 3D object and lane detection, we adapt the nuScenes dataset into a question-answer pairs format. Following the chain-of-thought approach, we prompt our model to generate safe driving plans and to describe various ego states, such as velocity and acceleration. The specific questions used are sampled from Table 9. For the answers, the model predicts the current state’s velocity and acceleration and then generates the ego-car’s planning waypoints for the next 3 seconds at 0.5-second intervals. This approach mirrors our methods in 3D object detection and 3D lane detection, where we discretize the BEV space, which ranges from -50 to +50 meters, into 1,000 bins. Similarly, we discretize both velocity and acceleration across a range from -50 m/s (m/s^2) to +50 m/s (m/s^2) into 1,000 bins each.

Table: Table 9: Question pool of ego planning for VLM-based methods.

## Appendix B Model Details

### B.1 3D Tokenizers Pre-training

We pre-train two distinct 3D tokenizers: StreamPETR  and TopoMLP .
StreamPETR  is designed for multi-view 3D object detection. We utilize a ViT-L backbone  and process images at a high resolution of 800x1600. Moreover, we follow the official training schedule established for the nuScenes dataset.
TopoMLP  focuses on constructing vectorized maps from multiple views. To maintain methodological consistency with StreamPETR, we employ the same ViT-L backbone and resolution. The training strategy for TopoMLP also mirrors the official.

### B.2 3D-tokenized LLM

Query Representation. For the innate priors of the 3D physical world, the query-based BEV framework is introduced. These DETR-style methods, StreamPETR and TopoMLP, extract target-aware query embeddings aka query representations (content) with reference points (localization) to represent objects from multi-view images.

Reference Point Embeddings.
As previously mentioned, a target is characterized by both its content and location. We integrate the query embeddings by adding reference point embeddings, which are generated from reference points via a single linear layer, to formulate the 3D tokens that represent target information. A notable aspect of our setup is we initialize the weight of the reference point projector to zero.

Memory Queue.
Taking inspiration from StreamPETR, our approach involves the storage of historical queries to preserve continuity in time, as memory queues. Specifically, we concatenate these memory queries with current queries for temporal modeling.
To elaborate, our method includes storing queries from three additional frames that exhibit the highest confidence—specifically, the top-K queries, where in our implementation, K is set to 256. The management of these queues adheres to a first-in, first-out (FIFO) principle.

Our 3D-tokenized LLM, Atlas, integrates the 3D tokenizers described earlier with an LLM, specifically the Vicuna-1.5. This LLM has been pre-trained on a diverse open-world data corpus, providing a robust foundation for understanding and processing spatial-temporal. Atlas follows most of the basic settings in Merlin, with a batch size of 1, a learning rate of 2e-5, and the AdamW optimizer with a weight decay of 1e-4. We implement a linear warm-up phase consisting of the first 3% steps in total. Following the warm-up, we transition to a cosine learning rate strategy. The maximum length of prompt tokens is 4096.

## Appendix C 3D Detection Results

Figure: Figure 6: Compare of 3D detection with various thresholds. We provide the precision-recall curves of PETR and StreamPETR. As the predictions from Atlas do not include confidence scores, we calculate the precision and recall across all predicted samples.
Refer to caption: x6.png

Precion-Recall Curve. In the paper text, we present a comparison of the F1 scores between task-specific models and Atlas in 3D detection, focusing on predictions with a confidence score above $0.3$, which yielded the highest F1 score. Additionally, we illustrate the performance variations of PETR, StreamPETR, and Atlas through the Precision-Recall curves at different positive thresholds, as shown in Figure 6.
It’s important to note that Atlas does not generate confidence scores; therefore, we treated all its predictions as positive samples for the purpose of calculating precision and recall. Although Atlas shows slightly weaker performance in making fine-grained predictions (specifically at a threshold of 0.5 meters), it excels in scenarios with larger thresholds. This observation suggests that large language models like Atlas might struggle with highly precise numerical predictions but perform well when broader tolerances are acceptable.

## Appendix D More Qualitative Results

### D.1 Qualitative Results of 3D Detection

Figure: Figure 7: Qualitative results of Atlas on 3D object detection. The red circles represent the predicted objects and green circles represent the ground truth.
Refer to caption: x7.png

We visualize the prediction results of the Atlas model in 3D detection tasks, as shown in Figure 7. The results align well with our performance metrics, demonstrating a notably high recall rate. This high recall is particularly important in practical applications of autonomous driving, where accurately detecting every potential obstacle, like pedestrians, is critical. Furthermore, the model maintains its accuracy even in complex scenarios characterized by high pedestrian density or closely packed targets.
Moreover, Atlas proves robust under challenging environmental conditions. For instance, even on rainy days, the model continues to perform strongly. This resilience is essential for the reliability needed in real-world applications, ensuring consistent performance regardless of weather conditions.

### D.2 Qualitative Results of 3D Lane Detection

We showcase the visualization outcomes of Atlas in its application to 3D lane detection, depicted in Figure 8. While the quantitative performance does not surpass task-specific models, Atlas demonstrates noteworthy qualitative performance.
As seen, our model performs well in challenging road situations because it accurately recognizes road crossings and dividings.

Figure: Figure 8: Qualitative results of Atlas on map detection. The red represent the predicted lane and green represent the ground truth.
Refer to caption: x8.png

### D.3 Qualitative Results of Planning

Figure: Figure 9: Qualitative results of Atlas on ego-car planning. Atlas outputs multiple potential planning trajectories within diverse weather and scenarios.
Refer to caption: x9.png

We also demonstrated the adaptability of Atlas’s driving plans across various weather conditions in Figure 9. Notably, even during rain, Atlas effectively plans its future travel trajectories with considerable diversity. This capability underscores the model’s robustness in challenging environments.
Furthermore, Atlas impressively maintains compliance with traffic signals, such as stopping at red lights, without having undergone specific training for traffic light recognition. This aspect highlights the model’s inherent understanding and application of world knowledge relying on LLM.
Additionally, the model’s diverse planning strategy enables it to effectively balance the decisions between maintaining its current lane and executing lane changes for overtaking. This flexibility greatly enhances the variety of possible travel routes, adapting dynamically to the flow of traffic and road conditions.

## Appendix E Failure Cases

Figure: Figure 10: Overly conservative
Refer to caption: x10.png

Discussing error examples in our model, Atlas, provides valuable insights that can guide future improvements. In this section, we analyze two primary types of failure observed during our experiments:

Overly Conservative Behavior. Atlas tends to make overly conservative decisions, favoring caution even when the path ahead is clear, as shown in Figure 11. This behavior results in a lower travel efficiency as the model opts to prioritize safety excessively. Our analysis suggests that this conservatism is likely rooted in the sampling bias of the nuScenes dataset. This dataset predominantly includes safer driving examples and favors lower-speed scenarios, which may have influenced Atlas’ decision-making strategy. To address this issue, incorporating a substantial amount of closed-loop data could be beneficial. This would provide Atlas with more dynamic and varied driving scenarios, potentially reducing its overly conservative tendencies.

Violation of Traffic Regulations. Despite Atlas having learned to adhere to several traffic rules, it occasionally fails to comply with traffic light signals, as shown in Figure 11. Specifically, Atlas may proceed through intersections during a red light. This error stems from the model’s lack of explicit traffic light information in its current framework. To mitigate this issue, integrating enhanced traffic-related data queries could be crucial. By providing Atlas with more explicit and detailed traffic signal information, we can improve its compliance with traffic laws and overall decision-making accuracy.

These findings highlight critical areas for further research and development. Enhancing the dataset and incorporating explicit models of traffic elements such as lights and signs are promising avenues for improving Atlas’ performance and reliability.

## Appendix F Border Impact

Our work focuses on designing a VLM-based solution for autonomous driving.
Training our model using a large language model is often computationally demanding, which may result in significant environmental impacts.
Also, researchers with constrained computational resources may find it challenging to follow our work.
Except for these, there are no typical border impacts for our work.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: - The authors may also consider evaluating their method on language-based benchmarks such as BDD-X.
soundness: 2
strengths: - The paper introduces Atlas, a novel 3D-tokenized LLM framework that integrates 3D geometric priors, addressing the limitations of existing 2D-tokenized VLMs in autonomous driving.

- Through comprehensive evaluations on the nuScenes dataset, Atlas demonstrates superior performance in 3D object detection, lane detection, and ego-car planning, validating its effectiveness over traditional methods.

- The paper clearly explains the need for 3D tokenization in autonomous driving, providing structured comparisons and detailed visuals to convey the model’s advantages.

- By advancing reliable end-to-end autonomous driving, this work highlights the essential role of 3D tokenization, with implications for future research in multimodal LLM applications across complex environments.
summary: The paper addresses the limitations of current vision-language models (VLMs) that rely on 2D tokenizers, which struggle with 3D perception essential for reliable autonomous driving. To overcome this, the authors introduce Atlas, a novel framework using 3D tokenizers based on DETR-style 3D perceptrons, allowing enhanced 3D object detection, lane detection, and planning. Evaluated on the nuScenes dataset, Atlas outperforms traditional 2D-tokenized approaches by integrating 3D geometric priors, showcasing its potential for more accurate and reliable end-to-end autonomous driving.
weaknesses: - Using 3D perception as tokens will directly benefit from the perception labels, significantly increase the model size, and thus make it **unfair** to compare with those VLMs with images as inputs. 

- The paper uses StreamPETR as the 3D perception backbone. However, from Table 1 joint training with LLMs significantly harms the perception performance, which indicates the ineffectiveness of the proposed method.

- Table 2 also shows a performance degradation in lane detection.

- Open-loop planning is not sufficient for evaluating LLM's planning capability. The authors need to evaluate their method in CARLA. Although it is not a real-world simulator, it is still sufficient for just evaluating the closed-loop planning capabilities of the methods 

- The paper lacks technical novelty as it just includes explicit 3D perception and lane detection as inputs to LLMs.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: - The paper introduces a 3D-tokenized framework that effectively bridges the gap between 2D perception in current VLMs and the 3D requirements of autonomous driving. This integration is both novel and practical for enhancing perception and planning tasks.
- The paper conducts thorough experiments across multiple perception and planning tasks using the nuScenes dataset, with Atlas outperforming previous VLM-based approaches in 3D perception and open-loop planning tasks.
-  The chain-of-thought design for planning is interesting and effective.
summary: The paper proposes a novel approach for autonomous driving by introducing a 3D-tokenized LLM framework, named Atlas, which integrates a 3D perception mechanism using DETR-style 3D tokenizers (StreamPETR for detection&tracking and TopoMLP for mapping). Traditional VLMs have primarily relied on 2D visual tokenizers, which lack the spatial depth essential for autonomous driving tasks such as 3D object detection, lane perception, and environmental reasoning. Atlas addresses these limitations by leveraging 3D perception models, allowing it to incorporate spatiotemporal data, which significantly enhances environmental understanding and planning tasks. Empirical results on the nuScenes dataset demonstrate that Atlas outperforms previous VLM-based approaches, especially in 3D perception and open-loop planning tasks.
weaknesses: - While this work introduces advancements in spatial perception by incorporating 3D tokenizers, it would benefit from a comparison with recent open-source studies that explore large language models’ spatial understanding in driving scenarios, such as [1].
- The impressive performance in open-loop planning is acknowledged, yet the detailed aspects contributing to this strength remain unclear. Further explanation of the performance-driving factors would enhance the clarity and reliability of these results.
  - Such as, if we use a non-pretrained transformer to replace the LLM, will the results being same?
- Since the 3D tokenizers are trained on labeled driving datasets, solely relying on them could limit the model's adaptability to novel or unseen scenarios, potentially constraining its utility in unpredictable real-world environments.
- To more comprehensively assess environmental understanding, incorporating question-answering tasks, as demonstrated in datasets like DriveLM[2], would provide a well-rounded evaluation of the model’s interpretive abilities.
- In the original 3D tokenizers, object detection relied only on some tiny MLP layers, while the current setup incorporates a more complex LLM, Vicuna. The application of Vicuna following StreamPETR and TopoMLP slightly reduces 3D perception performance.
- Currently, separate tokenizers are used for object detection and lane recognition. For an autonomous driving system that requires multi-tasking, adding tokenizers for each task could pose challenges. Given that each tokenizer includes an image backbone and maybe transformer encoder/decoder, scaling this approach may affect the model's real-time capabilities.

[1] Zhou, Yunsong, et al. Embodied understanding of driving scenarios. In ECCV 2024.
[2] Chonghao, Sima, et al. DriveLM: Driving with Graph Visual Question Answering. In ECCV 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: The primary questions are in the weaknesses part noted above. Additional questions include:

1. In Line 157, it is stated, “First, VLMs cannot deliver the necessary predictive confidence for metrics such as mean Average Precision (mAP).” Could the authors clarify the specific challenges associated with generating confidence scores directly for bounding boxes using VLMs? Since VLMs should be capable of producing scores for bounding boxes, are the authors suggesting that these scores lack reliability?
2. To support the claim in Figure 3, are there *quantitative* metrics that substantiate the superior performance of 3D-tokenized VLMs over 2D-tokenized VLMs in driving caption tasks?
3. In Table 3, regarding the collision metric, the authors mention, “We adhere to standard practices by utilizing the implementation provided by ST-P3 (Hu et al., 2022b).” However, the reported results for VAD and UniAD follow the collision metric from BEV-Planner, which yields significantly different values. Could the authors clarify which metric was actually used?
soundness: 2
strengths: 1. The paper is well-written and easy to understand.
2. The work highlights the shortcomings of 2D-tokenized LLMs and demonstrates how a 3D tokenizer can enhance performance.
summary: This paper addresses the limitations of current vision-language models (VLMs) used in autonomous driving, which rely primarily on 2D vision tokenizers paired with large language models (LLMs) but lack 3D geometric insights essential for accurate planning. To overcome this gap, the authors propose Atlas, an approach that integrates 3D perception networks as 3D tokenizers with an LLM. Experimental results on the nuScenes dataset indicate that Atlas enhances 3D detection and ego planning, supporting the value of 3D tokenization in advancing reliable autonomous driving.
weaknesses: 1. About the framework design: The necessity of connecting a 3D tokenizer to an LLM is questionable. A baseline experiment could clarify this by showing whether a simple combination of a 3D tokenizer and MLP could achieve similar planning results. The main advantage of incorporating an LLM seems to be interpretability, as LLMs may generate explanations, but whether they can control the trajectory as effectively remains unclear. Additionally, the LLM introduces limitations, such as reduced speed and difficulty in producing accurate trajectory scores. The paper would benefit from discussing the necessity of connecting an LLM to the 3D tokenizer.
2. About the performance: The performance when using the LLM is subpar, especially when ego states are not incorporated. As shown in Table 3, models without ego states underperform significantly compared to BEV-Planner. Conversely, experiments that include ego states are of limited value, as the network may rely on shortcuts derived from ego states rather than genuinely learning to predict trajectories.
3. About the benchmark: The paper lacks experiments on the CARLA simulator, which is critical as the nuScenes dataset alone is inadequate for validation. Although the authors mention that CARLA may lack realism, nuScenes predominantly consists of straightforward driving scenarios, heavily influenced by ego status, making it a limited and potentially biased metric for this work.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My concern is the lack of novelty in the methodology of this paper. I hope the authors can compare this work with existing methods to highlight the contributions of this paper.
soundness: 3
strengths: 1. The exploration of VLM-based methods in autonomous driving is interesting. This paper highlights the significant differences in perception between task-specific models and 2D tokenized VLM-based approaches.

2. The paper combines DETR-style 3D perceptrons with VLMs, leveraging 3D priors for improved depth perception and supporting various image types and temporal modeling.

3. The evaluation on the nuScenes dataset demonstrates enhancements in 3D detection and planning tasks, showcasing the role of 3D-tokenized VLMs as the key to ensuring reliable autonomous driving.
summary: This paper combines modular bird’s-eye view (BEV) approaches with vision-language models, integrating 3D priors for enhanced depth perception and supporting multi-view images and temporal modeling via DETR-liked query propagation. Evaluation on the nuScenes dataset showcases notable advancements in 3D object detection, lane detection, and E2E planning, affirming the pivotal role of 3D-tokenized LLMs in ensuring reliable autonomous driving systems.
weaknesses: 1. Overclaiming: The paper argues that existing 2D VLMs struggle in achieving 3D perception, but it has been shown that by pre-training a 2D tokenizer, 2D VLMs can outperform traditional 3D detection methods [1]. The authors are suggested to compare the performance curves of the 2D and 3D tokenizers as the amount of traing data grows. Additionally, other works [2] utilize 3D encoders like depth maps and extrinsics for 3D scene understanding. How does DETR-style tokenizer compare to or improve upon the mothods in [2]?

2. Lack of Innovation: 
a) In terms of model designs, some studies [3] have already explored spatial understanding tasks using 3D tokenizers from a BEV perspective. Sec. 3.1's content appears superficial and lacks novelty, making it challenging to discern the paper's contributions. How does Atlas differs from the work in [3], particularly regarding the use of 3D tokenizers. 
b) Data preparation, task design, and evaluation metrics mentioned in [1] cover topics discussed in the paper. Could the authors provide a more detailed comparison with [1] to highlight any unique aspects of their work.

3. Based on Fig. 4 and supp Fig. 5, it is not clear whether the model can accurately follow instructions to generate reasonable driving trajectories. Instead, it appears to list all potential trajectories, raising doubts about the model's capability to achieve end-to-end planning tasks. How does the model select the most appropriate trajectory from the multiple options? The authors should provide specific examples or metrics that demonstrate the model's ability to follow instructions.

[1] Zhou, et al. "Embodied Understanding of Driving Scenarios".

[2] Zhu, et al. "LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness".

[3] Choudhary, et al. "Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving".