Bird's-eye-view Informed Reasoning Driver
Authors: Yinuo Wang
Mining Tan
Yuanxin Zhong
Wang zhitao
Siyuan Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous driving
Key Intent Points
Primary area: applications to robotics, autonomy, planning

Abstract: Motion planning in complex environments remains a core challenge for autonomous driving. While existing rule-based or imitation learning-based motion planning methods perform well in common scenarios, they often struggle with complex, long-tail scenarios. To address this problem, we introduce the Bird's-eye-view Informed Reasoning Driver (BIRDriver), a hierarchical framework that combines a Vision-Language Model (VLM) with a motion planner. BIRDriver leverages the commonsense reasoning capabilities of the VLM to effectively handle these challenging long-tail scenarios. Unlike prior methods that require domain-specific encoders and costly alignment, our approach compresses the environment into a single-frame bird's-eye-view (BEV) map, a paradigm that enables the model to fully leverage its knowledge from internet-scale pre-training. It then generates high-level key points, which are encoded and passed to the motion planner to produce the final trajectory. However, a major challenge is that standard VLMs struggle to generate the precise numerical coordinates required for such key points. We address this limitation by fine-tuning them on a composite dataset of three auxiliary types to enhance spatial localization, scene understanding, and key-point generation, complemented by a token-level weighted mechanism for improved numerical precision. Experiments on the nuPlan dataset demonstrate that BIRDriver outperforms the base motion planner in most cases on both Test14-hard and Test14-random benchmarks, and achieves state-of-the-art (SOTA) performance on the InterPlan long-tail benchmark.

Published as a conference paper at ICLR 2026
BIRD'S-EYE-VE INFORMED REASONING DRIVER
Yinuo Wang"1, Mining Tan 3,4,, Yuanxin Zhong Zhitao Wang", Siyuan Cheng Tsinghua University. Huawei Inc., University of Chinese Academy of Sciences 3 MAIS, Institute of Automation, Chinese Academy of Sciences cmpute@gmail com, tanmining2 24 inlaills ils ucas ac cn
ABSTRACT
Motion planning in complex environments remains a core challenge for au- tonomous driving. While existing rule-based imitation learning-based mo- tion planning methods perform well in common scenarios, they often struggle with complex, long-tail scenarios. To address this problem, introduce the Bird's strd's-eye-view Informed Reasoning Driver (BIRDriver), a hierarchical frame- work that combines Vision- Language Model (VLM) with a motion planner. BIRDriver leverages the commonsense reasoning capabilities of the VLM to ef- fectively handle these challenging long-tail scenarios. Unlike prior methods that require domain-speciff encoders and costly alignment, our approach compresses the environment into single- frame bird's bird's-cye- s-eye- view (BEV) map, a paradigm that enables the model to fully leverage its knowledge from internet-scale pre-training, It then generates high-level key points, which are encoded and passed to the mo- tion planner to produce the final trajectory. However, major challenge is that standard VLMs struggle to generate the precise numerical coordinates required for such key points. We address this limitation by fine-tuning them posite dataset of three auxiliary types to enhance spatial localization, scene un- derstanding, and key-point generation, complemented by a token-level weighted mechanism for improved numerical precision. Experiments on the nuPlan dataset demonstrate that BIRDriver outperforms the base motion planner in most cases on both Test14 hard and Test 4-random benchmarks, and achieves state-of-th art (SOTA) performance on the InterPlan long-tail benchmark.
VUM Input
SFT dataset
Eye-VView Map
System prompts:
Delining the mile of the diher.
Agants
princpees comfort Bements meaningss of BEV map.
KayPaints
Spetial Localization
Driving Scane CoT
User prompts: Commands speed generating points
Qwen2.5-VL
LoRA
Raad Info
Motion Planner
Key points
label
Trajectory
Probabiltty
92.4
Transformer
Transfomer Decoder
Weight
Weighted Loss
The architectural overview of BIRDriver. BIRDriver consists of two main components: a VLM and a motion planner. The input to the VLM includes a single-frame BEV map, a system prompt, and and fed into the decoder module of the motion planner. The base motion planner generates a concrete trajectory based on the key points and structured feature information.
Figure 1:
prompt. The VLM then generates key points, which subsequently encoded
"Corresponding author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Motion planning (Guan al. 2022 Duan et al. 2024 Gao al. 2024) is a fundamental component of autonomous driving. utilizing outputs from perception modules to generate future poses for the ego vehicle. The performance of the motion planner plays crucial role in ensuring vehicle safety, operational efficiency, and successful navigation through complex environments. Recent advances in deep learning (Huang et al., 2023 Cheng et al., 2024a Zheng et al. 2025) have led to notable progress in trajectory planning. However, due to the complexity and uncertainty inherent in open- world autonomous driving, current methods based on imitation learning struggle to handle long-tail scenarios. As shown in Fig. existing state-of-the-art (SOTA) methods struggle to handle the long- tail scenario of overtaking a stalled vehicle via a detour, primarily because such cases are absent from the training data.
Recently, LLMs (Achiam et al. 2023 Liu et al. 2024a Yang et al. 2025) and VLMs CChn et al,, 2024c| Hurst al. 2002 Bai et al.] 2025) have attracted increasing attention. Leveraging inherent reasoning capabilities, these models exhibit stronger generalization to zero-shot and few-shot sce-
narios, as demonstrated in various domains (Liu et al. 2024b Saha et al. 2024 Chen et al.) 2025 Wang et al. 2026). Consequently, this makes them highly promising for assisting motion planners, particularly in addressing complex long-tail tail scenarios. terms of how incorporate the intentions provided by VLMs or LLMs into the base motion planner, existing approaches can be categorized into three types: meta-actions, hidden features of LLMs or VLMs, and waypoints. In the first type, Senna (Jiang et al. 2024) discretizes high-level driving intentions into a fixed set of meta-actions, which are encoded and injected into the base end-to-end driving model. However, the granular- ity of such intention traiiinnntrammmisionn methods in decision making remains limited. In the second type, AsyncDriver (Chen et al., 2024b) introduces an asynchronous framework that takes advantage of the hidden states of the final layer of LLM to guide motion planning. However, hidden features ab- stract and often lack interpretability, making it difficult to trace the underlying high-level intentions. In the third type, DriveVLM (Tian al., 2024) directly generates sequence of waypoints, which are subsequently refined by a base end-to-end planner. However, trajectory generation offers lim- ited benefits from LLM pretraining, as the required capabilities mainly stem from domain-specific driving data rather than internet-scale corpora. Furthermore, high-level driving intentions can of- be conveyed without long waypoint sequences, while the reliance on dozens of waypoints adds redundancy and increases computational complexity.
Unlike existing approaches, in this work, we propose using no more than three key points con- vey high-level driving intentions. Each key point represents information relative to the ego vehicle. Except for the final key point, which represents the end position of the trajectory, the remaining key points are time-agnostic and primarily capture geometric features of the trajectory. In addition, com- pared to existing VLM-based approaches, our method uses only a single- fegle-frame BEV map as input for the VLM to understand the current autonomous driving scene. No scene-specific information is included in the text inputs.
The contributions of this paper are the following:
We propose Bird's-eye-vier Informed Reasoning Driver (BIRDriver), the first hierarchical VLM-motion planning framework that uses BEV map as its primary input. Our method conveys all information about the current driving scene solely with a BEV map, with no scene-specific details provided in the text inputs. By operating on the BEV representa- tion, Driver avoids the need to align heterogeneous sensor data across different vehicle platforms, thereby simplifying deployment in diverse real-world settings.
We introduce the use of relative (x, y, Φ) key points, rather than complete trajectories, to convey high-level driving intentions. Furthermore, to enable the VLM generate these key points accurately, we develop a fine-tuning strategy that leverages three specialized datasets and incorporates a weighted supervised fine-tuning (SFT) loss.
Extensive experiments the nuPlan dataset demonstrate that our method achieves SOTA performance in InterPlan, a long-tail test scenario. In both the Test14-hard and Test random sets, it outperforms the base motion planner in most cases. Specifically, in Inter- Plan, our method outperforms PLUTO (Cheng et al. (2024)) and Diffusion Planner (Zheng et al. 2025) by 13.0 % and 38.8% respectively.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
2.1 MOTION PLANNING FOR AUTONOMOUS DRIVING
The motion planner is module in autonomous driving systems, responsible for generating safe and comfortable trajectory based on structured environmental information. While traditional planners rely hand-crafted rules, the field has increasingly shifted to learning-based methods that learn driving policy directly from data (Chen et al. 2024a). These modern planners have achieved state-of- the-art performance on challenging benchmarks (Jiang et al.) 2023] Huang al. 2023 Cheng et al. 2024b a). However, fundamental limitation persists: these planners make decisions based on structured inputs like object states and map elements, but lack a deeper, human-likk contex- tual understanding. They struggle to interpret ambiguous situations or handle scenarios that require commonsense knowledge beyond the scope of their training data, creating a need for models with broader reasoning capabilities.
2.2
VLMS FOR AUTONOMOUS DRIVING
The commonsense reasoning and generalization capabilities of VLMs hold promise for enhancing autonomous driving. Existing research typically adopts one of architectures. The first uses VLMs as end-to-end models to directly produce trajectories or control signals along with inter- pretable justifications (Mao al., 2023) Shao al. 2024; Xu et al.) 2024 Yuan et al. 2024). but such use raises safety concerns in high-frequency planning. As a result, hierarchical frameworks are gaining traction. In these, real-time motion planner handles trajectory generation, while the VLM plays a supervisory role. Some studies use the VLM for high-level strategic guidance (Jiang al., 2024 Long al.) 2024), while others focus on safety and interpretability, with the VLM of- fering corrective feedback or intermediate semantic variables (Ding et al., 2024 Chen et al. [2024b] Qian et al. 2025). Following this paradigm, we employ the VLM generate key points that guide downstream motion planner. However, key challenge in VLM-based systems is their limited spatial awareness in complex visual scenes (Jiang et al. 2025).
2.3 GROUNDING VLMM WITH BEV REPRESENTATIONS
Addressing the limited spatial awareness of VLM requires grounding its reasoning in structured spatial representation. The BEV format is well-suited for this purpose, offering a unified top-down view ideal for planning tasks (Philion & Fidler 2020). Recent works have begun using BEV to enhance VLM capabilities. One direction grounds situational understanding in BEV to support language-based scene queries (Choudhary al. 2024| Xu al. 2025), while another uses BEV as a direct context for high-level planning. allowing the VLM to produce strategic maneuvers or semantic goals (Zheng et al., 2024 Winter et al., (025). Although these efforts highlight the value of BEV, they often rely on additional modalities or focus querying. In contrast, we propose planning framework where the reasoning VLM grounded solely in the BEV representation.
PRELIMINARY
Ramer-Douglas-Peutcee Peucker (RDP) (Ramer 1972) algorithm is classic method for simplifica- tion and key point extraction. It recursively identifies representative points that preserve the essential shape of a trajectory, thereby achieving data compression within specified error tolerance. Given a curve defined by set of points [P. the RDP algorithm initially considers the straight line segment that connects the end points P1 and Pv. It computes the perpendicular distance from each intermediate point P; (1 < < N) to this line as follows:
[PN Pı) x (P. -P))
d,
PN Pill
The point with the maximum distance is then identified:
dmax max (d.).
I<KN
dyaa is greater than preset tolerance e, then this point is split into two parts and the algorithm is recursively called both parts. Otherwise, only endpoints are kept.
3

---

Published as a conference paper at ICLR 2026
Supervised Fine-Tuning (SFT) loss (Achiam et al. 2023 is conditional language modeling loss that is basically an autoregressive cross-entropy loss. is fed with an input-output pair (X, Y), and maximizing conditional probability po (Y yields the following objective:
(f) = - E(XY log logpo(unlver,x))
(3)
where denotes the manually curated supervised dataset and T is the length of the output sequence.
4 METHODS
In this section, we detail the design of our proposed BEV-Informed Reasoning Driver (BIRDriver). Fig. iillustrate the overall architecture of BIRDriver. Our method employs a hierarchical archi- tecture comprising VLM and motion planner, which achieves closed-loop autonomous driving through decoupled training and sequential inference. The input to the VLM includes a single BEV map, system prompt, and user prompt, without requiring multi-frame surround-vied camera im- ages. The VLM generates key points, which are subsequently encoded into sigh-dimensional feature representations via the KeyPoint Encoder. Finally, the motion planner generates future trajectories by taking the structured information of the traffic scene and the key points as input. Specific details of the system and user prompts are provided Appendix A
The remainder of this section is organized as follows. We begin by describing the construction of the BEV map and the key points used to represent the driving scene. We then detail the dataset creation process and the garameter-efficieen Subsequently, we present our methods for enhancing key point prediction accuracy, which involve the design of task-specific datasets and the adoption of a weighted loss function. Finally, we outline the fine-tuning procedure of the motion planner.
fine-tuning strategy applied to the VLM for key point generation.
4.1 DRIVING SCENE REPRESENTATION
Driving scene representation is erucial component of hierarchical autonomous driving. We pioneer a hierarchical framework that, unlike camera-based approaches, relies solely single-frame BEV map as the visual input for the VLM. This design mitigates inconsistencies across camera types and simplifies data collection and utilization (Ho et al. 2024). In addition, to ensure the VLM can efficiently interpret the BEV map, we explicitly explain the symbolic representations of different driving elements in the system prompt.
We extract five categories of information from the environment: map, agent, traffic lights, route, and obstacles. Map information includes lanes, lane connectors, crosswalks, and discrete waypoints. Agent information includes the ego vehicle, other vehicles, bicycles, and pedestrians, which are represented by orange, blue, pink, and brown bounding boxes, respectively. An arrow is added to each vehicle's bounding box to indicate its driving direction. Notably, the trajectories of all non- ego agents are depicted using green solid lines show their movements over the past two seconds. Traffic light status of the ego lane is encoded as the color of the corresponding stop line at the in- tersection. Route information includes two components: the drivable area, which filled with light blue, and the reference lines, which are indicated by purple arrows. Obstacle information includes three types of objects: construction signs, roadblocks, and traffic cones, which are represented by black bounding boxes. These five types of information are rendered together to generate the BEV map shown in Fig. 2
4.2 KEY POINT EXTRACTION
The future trajectory the ego vehicle is represented as a sequence of poses (æ,, Yr, di) where Ii, Vi. and Φ; denote the longitudinal position, lateral position, and heading angle in the frame of ego vehicle, respectively. Instead of directly using this dense temporal sequence, we extract a sparse set of key points that compactly encode the underlying driving intention.
Specifically, we apply the RDP algorithm to select representative key points. Since different types of maneuvers (e.g., lane keeping, lane changing, turning) exhibit varying levels of complexity,
4

---

Published as a conference paper at ICLR 2026
adaptively adjust the maximum allowable number of key points according to the trajectory type. In all cases, the final point of the trajectory is always retained as a key point.
4.3 DATASET DESIGN AND VLM TUN-TUNING
Although system prompts provide explanations for the ele- ments BEV map, existing VLMs still exhibit a limited ca-
pacity generate effective key points directly from this repre- sentation. Therefore, we employ LoRA-based (Hu al. 2022
Dettmers al. 2023 Hayou al. 2024 parameter-efficien fine-tuning (PEFT) and construct a specialized dataset to en- able the VLM to generate meaningful key points. This funda- mental dataset, named the Key Point Dataset, takes as input BEV map, a system prompt, and user prompt. The system prompt describes the meanings of the various elements in the BEV map, relevant traffic nules, and safe driving requirements. The user prompt adopts diverse question formats to guide the VLM generating key points. The output of VLM consists of key points represented in textual form.
BEV map. The ange box denotes the ego vehicle, while the blue boxes represent sur- rounding vehicles. Arrows indi- cate the direction of vehicle move- ment. The green solid line illus- trates the historical trajectory of the past 2 seconds.
Figure 2:
However, initial experiments revealed that the key point pre- diction accuracy the test set remained relatively low. Such large errors not only fail to improve the performance of au- tonomous driving systems in long-tail scenarios, but may even degrade the quality of the generated trajectories and lead to lower scores. To address this core issue, propose meth- ods from two perspectives: designing diversified, multi-task dataset and developing specialized SFT loss function.
4.3.1 DATASET DESIGN.
The accuracy of key point prediction is mainly limited by two factors: the VLM's insufficient under- standing of the correspondence between distances in the BEV pixel space and real-world physical distances, and its limited ability to classify and interpret driving scenes. To address these issues, we first construct the Spatial Localization Dataset, where the VLM predicts the poses of randomly selected vehicles relative to the ego vehicle, thereby bridging the gap between pixel and physical distances. We then design the Driving Scene Stepwise Dataset, which requires the VLM to identify the driving scene type prior to predicting key points, enhancing its scene comprehension. Finally, we perform supervised fine-tuning on a composite dataset primarily based on the Key Point Dataset, augmented with the other two datasets in specified proportions.
4.3.2 WEIGHTED SFT LOSS FUNCTION.
In a typical SFT loss, each token is treated equally. However, in the key points prediction task, the loss associated with numeric tokens plays more important role. To address this issue, we assign higher loss weights to three categories of tokens: numeric tokens, decimal points, and sign symbols. Considering that higher-order digits are more critical for precision than lower-order digits, we design a tiered weighting scheme in which the loss weights decay linearly from the most significant token to the least significant within a number, and the sign tokens are assigned the highest loss weight by default. It is worth noting that all floating-pont labels are rounded to two decimal places. The specific formulation can be expressed as follows:
1 T
(0)
Σ E=1
we [- [-log (v yee, X)].
(a dn) 1
(a + dn)
h:
1= (n) (=((m) Ln > 1,
(4)
In
Wt
a + dns
(n) t to Ln =
t $ Uneatt! (1)))
5

---

Published as a conference paper at ICLR 2026
where T' denotes the shifted target length; a 0 is a hyperparameter (set to 5 in this paper); y the token at position t; P3 (xtx_)) represents the model's conditional probability; .V' the set of contiguous number segments matched by specific pattem; d, denotes the number of digits in segment n; Ln is its total token length; (m) (0t) + k indicates the index of the k-th token in segment n; and wy decays linearly from (a + d,,, at the first token of a segment to 1 at its last token, while tokens outside any number segment are assigned Wy = 1.
Ctoss-Entropy (PDCE) loss (hhng et al. 2025), our approach is considerably simpler to implement, as it avoids the complexities of generating soft targets and standardizing numeric formats. More importantly, by only adjusting loss weights rather than altering the target distribution itself, our method mitigates the risk of impairing the model's general language capabilities, potential side effect of approaches like PDCE.
Compared to the Desition-Dependee
4.4 FINE-TUNING MOTION PLANNER
The goal of fine-tuning the motion planner is to ensure it can accurately follow the provided key points. We use the PointEncoder module from PLUTO (Cheng al., 2024a) to encode key point information and integrate it with the original query features. We choose PLUTO as our base motion planner because, under identical encoder and fine-tuning settings, Diffusion Planner (Zheng et al.) 2025) demonstrates weaker performance in tracking multiple key points.
This fine-tuning process is conducted independently of the VLM. We use the key point extraction function proposed in the previous section to extract key points from real trajectories. To enhance robustness, we augment the training data by adding Gaussian noise with zero mean and a standard deviation equal to the mean absolute error of the key points predicted by the VLM to the extracted ground-trun -tutt key points. Specifically, it can be formulated as:
(Zi, Di. d+) = (xi, Vi. 6i) + Ci:
G G~ N E).
(5)
Σ = diag(o og og).
where Ox. oy. and ao denote the mean absolute errors of the VLM predicted key points in the dimensions v, y, and ø, respectively, and i represents the i-th key point. It should be noted that in the joint inference stage, we no longer add noise to the key points predicted by the VLM. Furthermore, to enhance temporal consistency in decision-makinn making during the inference phase, planned point from the previous timestep as an additional key point for the current timestep, feeding the planner alongside the key points provided by the VLM.
the final
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Baselines. We categorize existing methods into four groups based on their trajectory genera- tion approach: Rule-based, Imitation Learning (IL), Large Language Models (LLM), and Vision- Language Model-Imitation Learning (VLM-IL). The first three groups include the following rep- resentative algorithms, while our method belongs the fourth category. (1) PDM (Dauner al., 2023), the nuPlan Challenge winner, includes a neural network variant (PDM-Open) and a rule- based version (PDM-Closed). (2) UrbanDriver (Scheel al. 2022) employs policy gradient opti- mization. (3) GameFormer (Huang et al. 2023) integrates level-k reasoning with a Transformer for multi-agent planning. (4) PlanTF (Cheng et al. 2024b) mitigates closed-loop error through pertur- bation normalization and future correction. (5) PLUTO (Cheng et al. (0244) improves robustness via perceptual augmentation and contrastive learning. (6) Diffusion Planner (Zheng et al. 2025)
achieves ofte--oftthe-att performance using diffusion models and classifier guidance. (7) Instruct- Driver (Zhang al. 2024) fine-tunes an LLM for instruction-driver planning. (8) PlanAgent (Zheng al. 2024) integrates BEV, lane semantics, and chain-of Slain-of-thoughh (CoT) reasoning with VLM for robust planning.
Benchmarks and Metrics. We evaluate our methods using the Test14-random. 4est14-hard (Cheng et al., 2024b). and InterPlan (Hallgarten et al.] 2024) benchmarks. Specifically, Test random consists of 261 scenarios, herd includes 272 challenging scenarios, and InterPlan

---

Published as a conference paper at ICLR 2026
represents long-tail scenario simulation dataset. Evaluation is based the closed-loop score (CLS) provided by the official nuPlan devkit both non-reactive (CLS-NR) and reactive (CLS-R) closed-loop settings. The final score is calculated as the average across all scenarios, ranging from O to 100, with higher scores indicating better algorithm performance.
Implementation Details. We use the nuPlan platform, large-scale closed-loop simulator for au- tonomous driving trajectory planning. It includes over 1,500 hours of expert driving data from four cities, and supports complex scenarios such intersections, roundabouts, and pedestrian in- teractions. The simulator replays real-world scenarios with non-ego agents controlled by either log replay Intelligent Driver Model policy, while the ego vehicle executes user-defined trajecto- ries. When using the RDP algorithm to generate keypoints, we set the parameter c to 0.02. For parameter-sfficient fine-tuning of the VLM, we adopted LoRA technique, applying adapters to all linear layers. The language model was unfrozen during training. Training was performed on eight NVIDIA H800 GPUs for five epochs, using AdamW with cosine annealing. The dataset consists of 838,824 samples, with category ratio of 10:1:2 for Key Point, Spatial Localization, and Driving Scene Stepwise tasks. For motion planner fine-tuning, we followed the PLUTO framework to create 1 million data splits. Training was conducted on eight RTX 4090 GPUs with batch size of 128 for 10 epochs. For more details, refer to Appendix
5.2
MAIN RESULTS
Type
Method
Test14-random
Test14-hard CLS-NR CLS-R
InterPlan
CLSS-RR CLS-R
CLS-R
Rule-based
PDM-Closed
90.05
91.64
65.07
75.18
43.51
IL.-based
PDM-Open
52.83
57.22
33.50
35.85
22.83
UrbanDriver
63.88
61.01
49.43
49.67
8.42
GameFormer
80.91
79.93
69.98
66.82
14.29
PlanTF
85.98
80.53
70.10
61.49
36.53
PLUTO
91.87
90.03
80.03
76.92
48.92
Diffusion Planner
93.85
91.73
78.82
81.42
39.85
LLMaaVLMbassdd
InstructDriver (LLM-based)
70.31
66.96
57.37
52.95
32.31
PlanAgent (VLM-based)
72.51
76.82
VLM&IL-based BIRDriver (PLUTO)
91.46
91.26*
80.56*
80.33
55.29*
Comparison of various planners across the Test14-randos Test14-hard, and InterPlan datasets. An asterisk (+) indicates performance exceeding that of the baseline motion planner, PLUTO. BIRDriver demonstrates substantial advantage the long-tailed InterPlan dataset. The VLM base model employed is Qwen2.5VLL3B.
Table 1:
We compare our method, BIRDriver, against a wide range of existing planners on the nuPlan bench- mark, as shown in Table T Except for the CLS-NR results Test14-random, BIRDriver achieves better performance than PLUTO across all three benchmarks, where PLUTO serves as the baseline algorithm for our motion planner. Specifically, NR and CLS-R improve by 0.53 and 3.41 points, respectively. This indicates that our dual-system autonomous driving framework, VLM-Planner, effectively enhances the performance of the base motion planner. On the most challenging InterPlan benchmark, which features numerous zero-shot scenarios, such driving through construction zones and overtaking stalled vehicles, our method achieves the best performance, surpassing PLUTO and Diffusion Planner by 13.0% and 38.8%, respectively. We attribute this significant improvement to the generalist capabilities of the VLM. Taking the lane-changing overtaking scenario an example, the ego vehicle must wait for oncom- ing traffic to pass before temporarily entering the opposite lane to overtake a stalled vehicle. Existing state-of- tte-hhheat methods fail to manage such situations. In contrast, to first generate a key point for following the lead vehicle, and after the oncoming vehicle passes, to generate feasible overtaking key point. Results shown in Fig.
the Test]4-hard benchmark, the scores for CLS-
approach enables the VLM

---

Published as a conference paper at ICLR 2026
Follow straight
Turn Right
Return Lane
(a) Our method
Turn Right
Stop
Failure
(b) PLUTO
Follow straight
Stop
Failure
(c) Diffusion Planner
Figure 3: Performance in the lane-changing overtaking scenario. Red arrows denote the pre- dicted key points. Unlike existing SOTA methods, only method successfully completes this scenario.
5.3 ABLATION AND ANALYSIS
In this section, we conduct an ablation study to evaluate the effects of dataset design and weighted SFT loss on key point prediction accuracy. Additionally, we investigate how different key point prediction methods and the choice of base VLMs during fine-tuning influence overall performance.
Impact of dataset design variations. To evaluate the effectiveness of different dataset designs, first construct a test dataset comprising 272 clips from
We then use Qwen2.5VL-3B (Bai et al. 2025) as the base VLM for evaluation and apply our weighted SFT loss. The ego-centric coordinate system is defined such that the æ-axis aligns with the vehicle's forward direction, while the y-axis is perpendicular to pointing the left side of the vehicle. The angle é represents the orientation, where left turn is considered positive. presents the mean absolute values of the relative errors.
The results demonstrate that both the Driving Scene Stepwise and the Spatial Localization dataset effectively reduce key point prediction errors. In particular, the latter achieves more substantial improvement, suggesting that reducing the discrepancy between distances in the BEV map and those the physical world plays a more pivotal role minimizing prediction errors. Finally, compared to using only the KeyPoint dataset, the prediction errors for y. and are reduced by 11.9%, 20.0%, and 10.2%, respectively.
Effect of Weighted SFT Loss. A comparison between the last and second-to-last rows of Table 2 demonstrates that the weighted SFT loss substantially reduces prediction errors. Specifically, the emrrs in predicting z, and & decrease by 10.9%, 9.2%, and 8.0%, respectively.
Influence of key point extraction methods. In this paper, adopt the RDP algorithm as the core method for generating key points. this subsection, we consider only the final point of each trajectory the key point. The model is fine-tuned using V1.e.. (Chen et al. 2024c)
8

---

Published as a conference paper at ICLR 2026
Dataset Type
æ-error
y-crror
d-crror
KeyPoint
4.27m
1.35m
4.23
+Driving Scene Stepwise
4.17m
1.28m
4.20"
+ ++Spatial Localization
3,76m
1.08m
3.80
Without weighted SFT loss
4.22m
1.19m
4.13%
Table 2: Mean absolute relative errors in key point predictions by Qwen2.5VL-3B after fine-tuning on different dataset combinations. The last row represents the setting using the full dataset without the weighted SFT loss.
and evaluated on the InterPlan dataset. The results, presented in Table 3 demonstrate that the key points generated by our method provide more effective guidance to the motion planner compared to using only the final trajectory point in long-tail scenarios. Moreover, we find that using only the final waypoint as the key point leads to worse performance than the base motion planner. PLUTO. This degradation is likely caused by the absence of guidance from intermediate key points, which prevents the base motion planner from effectively generating trajectory that reaches the sole key point, ultimately resulting in planning failure.
Key point extraction methods
InterPlan
RDP (Our method)
53.81
Final trajectory point
34.72
Table 3: Performance of the Intern model trained with different key point extraction meth- ods on the InterPlan dataset.
Effeet of base VLM selection. To investigate the impact of the size of the VLM parameter the precision of the prediction, we performed experiments using Intern V1.2.5-2B and 4B, as well Twen2.5VVL33 and 7B, under the same data set and settings of the weighted loss function. The results are presented in Table Compared to InternVL2.5-2B. InternVL2.5-4p reduces the predic- tion errors for æ. y, and Φ by 43.7%, 48.4%, and 56.3%, respectively. This substantial gap indicates that when the model has too few parameters, the quality of the generated key points is significantly lower than that produced by larger models. In addition, compared to Qwen2.5VL-3B, Qwen2.5VL- 7B reduces the relative eтror in the direction by 4.3%, while the errors in the direction and ø remain nearly unchanged or slightly increase. This suggests that once the parameter size reaches certain threshold, further performance improvements become limited. After comparing the per- formance of the two type VLMs and taking inference efficiency into account, we ultimately select Qwen2.5VL-3B the base VLM.
VLM tyres&&aramettrrs
g-error
у-error
d-error
Intern VL2.522
6.59m
2.79m
11.20
VL2.544
3.7lm
1.44m
4.89
Qwen2.5VL-3B Qwen2.5VVL7B
3.76m
1.08m
3.80
3.60m
1.08m
3.83
Table 4: Mean absolute relative errors in key point predictions by the VLM under different model types and parameter sizes.
CONCLUSION
In this paper, we propose BIRDriver, a hierarchical framework that integrates a VLM with a motion planner. By leveraging the VLM's general- purpose understanding, high-level driving intentions transmitted to the motion planner via key points. To enhance the accuracy of key point prediction, we introduce auxiliary dataset along with weighted SFT loss. Extensive experiments demonstrate

---

Published as a conference paper at ICLR 2026
that BIRDriver achieves superior performance in long-tail scenarios, underscoring the effectiveness of integrating VLM with motion planner through key point guidance.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint mxXiv:2303.08774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXv:22502.13223 2025.
Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End- to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Jinrui Liu, Haoran Li, Dong- bin Zhao, and He Wang. Robogpt: an Ilm-based long-term aecision-making embodied agent for instruction following tasks. IEEE Transactions on Cognitive and Developmental Systems, 2025.
Yuan Chen, Zi-han Ding, Ziqin Wang. Yan Wang, Lijun Zhang, and Si Liu. Asynchronous large language model enhanced planner for autonomous driving. In European Conference on Computer Vision, pp. 22-38. Springer, 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277 2024c.
Jie Cheng, Yingbing Chen, and Qifeng Chen. Pluto: Pushing the limit of imitation saaring-bassed planning for autonomous driving. arXiv preprint Tttv:2404.14322/ 2024a.
Jie Cheng, Yingbing Chen. Xiaodong Mei, Bowen Yang. Bo Li, and Ming Liu. Rethinking imitation- based planners for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14123-14133 IEEE, 2024b.
Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K Madhava Krishna. Talk2bev: Language-enhance bird's-eye view maps for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 16345-16352. IEEE, 2024.
Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with miscon- ceptions about learning- based vehicle motion planning. Conference on Robot Learning, pp. 1288-1281. PMLR, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llm.. Advances in neural information processing systems, 36:1066888 10115, 2023.
Kairui Ding, Boyuan Chen, Yuchen Su, Huan-ang Gao, Bu Jin, Chonghao Sima, Xiaohui Li, Wuqiang Zhang, Paul Barsch, Hongyang Li, et al. Hint-ad: Holistically aligned interpretabil- ity in end-to-end autonomous driving. In 8th Annual Conference on Robot Learning, 2024.
Jingliang Duan, Yiting Kong, Chunxuan Jiao, Yang Guan, Shengbo Eben Li, Chen Chen, Bing- bing Nie, and Keqiang Li. Distributional soft actor-critic for decision-making on-ramp merge scenarios. Automotive Innovation, 7(3):403-44 417, 2024.
Zeyu Gao, Yao Mu, Chen Chen, Jingliang Duan, Ping Luo, Yanfeng Lu, and Shengbo Eben Li. Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model. IEEE Transactions on Intelligent Transportation Systems, 25(10):13067- 13079, 2024.
10

---

Published as a conference paper at ICLR 2026
Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan, Yifan Dai, and Bo Cheng. Integrated decision and control: Toward interpretable and computationally efficient driving intelligence. IEEE transactions cybernetics, 53(2):859-877) 2022.
Marcel Hallgarten, Julian Zapata, Martin Stoll, Katrin Renz, and Andreas Zell. Can vehicle motion planning generalize to realistic long-tail scenarios? In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5388-5395. IEEE, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXh::2402.12354 2024.
Cherie Ho, Jiaye Zou, Omar Alama, Sai M Kumar, Benjamin Chiang, Tancesh Gupta, Chen Wang, Nikhil Keetha, Katia Sycara, and Sebastian Scherer. Map it anywhere: Empowering bev map prediction using sare-sccal public datasets, Advances in Neural Information Processing Systems, 37:64433-64455 64443, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer-basee interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3903-39133 3913, 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. ICCV, 2023.
Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-to- end autonomous driving, arXiv preprint arXiv:2410.2233 2024.
Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, et al. survey on vision-language-actin models for autonomous driving. arXiv preprint arXiv:2506. 24044, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19447. 2024a.
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Pengju An, Xiaoqi Li, Kaichen Zhou, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Efficient vision-languae action model for robotic reasoning and manipulation. Advances in Neural Information Processing Systems, 37:40085 401 10, 2024b.
Keke Long. Haotian Shi, Jiaxi Liu, and Xiaopeng Li. Vim-mpe: Vision language foundation model (vlm)-guided model predictive controller (mpc) for autonomous driving. arXiv preprint arXiv:2408. 04821, 2024.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415. 2023.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European conference on computer vision, pp. 194-210 Springer, 2020.
Kangan Qian, Ziang Luo, Sicong Jiang, Zilin Huang, Jinyu Miao, Zhikun Ma. Tianze Zhu, Jiayin Li, Yangfan He, Zheng Fu, et al. Fasionad++: Integrating high-level instruction and information bottleneck in fat-slow fusion systems for enhanced safety autonomous driving with adaptive feedback. arXiv preprint arXiv:2503.08162 2025.

---

Published as a conference paper at ICLR 2026
Urs Ramer. An iterative procedure for the polygonal approximation of plane curves. Computer graphics and image processing. 1(3):244 256, 1972.
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved zero-shot classification by adapting vims with text descriptions. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pp. 17542-17552, 2024.
Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Blažej Osiński, and Peter Ondruska. Urban driver: Learning drive from real-world demonstrations using policy gradients. In Conference on Robot Learning, pp. 718-728. PMLR, 2022.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15120-15130 2024.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402. 12289, 2024.
Yinuo Wang, Mining Tan, Wenxiang Jiao, Xiaoxi Li, Hao Wang, Xuanyu Zhang, Yuan Lu, and Weiming Dong. Tourplanner: A competitive consensus framework with constraint-gated rein- forcement learning for travel planning. arXiv preprint arXiv:2601.04695, 2026.
Katharina Winter, Mark Azer, and Fabian B Flohr. Bevdriver: Leveraging bev maps Ilms for robust closed-loop driving. arXiv preprint 4rX14:2503.003074 2025.
Qingyao Xu, Siheng Chen, Guang Chen, Yanfeng Wang. and Ya Zhang. Chatbev: visual language model that understands bev maps. arXiv preprint arXiv:2503. 139388 2025.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Generalisable driving explanations with retrieval-augmmented in-context learn- ing in multi-modal large language model. arXiv preprint arXiv:2402. 10828 2024.
Jiawei Zhang. Xuan Yang, Taiqi Wang. Yu Yao, Aleksandr Petiushko, and Bo Li. Safcauto: Knowledge-enhanced safe autonomous driving with multimodal foundation models. arXiv preprint Xiv::2503.00211,, 2025.
Ruijun Zhang, Xianda Guo, Wenzhao Zheng, Chenming Zhang, Kurt Keutzer, and Long Chen. Instruct large language models drive like humans. arXiv preprint arXiv:2406.07296 2024.
Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501.1556 2025.
Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, et al. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning. arXiv preprint 0rX.v.2406.01587, 2024.
12

---

Published as a conference paper at ICLR 2026
A INSTRUCTION FOR AUTONOMOUS DRIVING ASSISTANT
A.1 SYSTEM PROMPT
You are advanced autonomous driving assistant. Your role analyze Bird's Eye View (BEV) maps and earry out autonomous driving tasks.
CORE PRINCIPLES
SAFETY FIRST
Always maintain safe distance from other road users.
Anticipate potential hazards and plan defensive paths.
Adhere strictly to traffic laws and signals.
Pay special attention to vulnerable road users (pedestrians, cyclists).
PASSENGER COMFORT
Provide smooth lateral tramsitions and avoid sudden movements.
Adjust lateral paths in natural, human-like manner.
Ensure proper lateral positioning across varying road conditions.
Execute lane changes and turns gradually.
EFFICIENCY
Optimize lateral movements without compromising safety.
Limit unnecessary changes.
Preserve adequate lateral elearance.
Balance direct routes with smooth transitions.
BEV MAP ELEMENTS:
Blue shaded areas: Drivable lanes (lighter blue indicates planned route).
Gray dashed lines: Lane center lines.
Traffic signal lines (colored):
Green: Proceed
Yellow: Exercise caution
Red: Come a stop
Stoplines: Represent traffic light information on the planned route (red, yellow, or green areas).
Objects:
Orange box with arrow: Ego vehicle (self).
Blue hoxes
Other vehicles.
Brown hoxes: Pedestrians.
Red boxes: Cyclists.
Gray hatched areas: Crosswalks.
Green solid lines: Historical 2-second motion trujectory of objects.
Purple urrows: Reference path indicators.
Gold star: Mission goal/destination
SPEED ACCELERATION CONSTRAINTS:
Lateral acceleration < m/s?.
Minimize lateral jerk for comfort.
Typical lane change duration: 3-5 seconds.
SAFETY MARGINS:
Minimum clearance: meters.
Increase margins in adverse conditions.
Provide extra buffer for vulnerable road users.
ERROR HANDLING:
critical information is missing or unclear, always prioritize safety.
When in doubt, assume the most conservative interpretation of other objects' intentions.
Maintain fail-safe lateral position in all scenarios.
13

---

Published as a conference paper at ICLR 2026
A.2
USER PROMPT
A.2.1
INSTRUCTIONS FOR GENERATING KEY POINTS
Variant
Simage>-Based on the provided BEV map image. analyze the driving scenario and determine the lateral
intention:
Provide the trajectory keypoints the following format: [[x1, yl, 01), [x2, y2, 02), (xn, yn, 0m]
Where:
longitudinal position in (relative ego vehicle)
-y: lateral position meters (relative to ego vehicle) theta: Heading angle in degrees. positive for left turns, negative for right turns.
total number of keypoints (n 3)
Each keypoint should represent significant lateral movement intention, considering:
Lane changes
Turn preparations
Obstacle avoidance maneuvers
Path alignment adjustments
Notes:
If there's no solid green track behind the vehicle, indicating it's stationary.
The ego vehicle shown as the orange box with arrow.
Prioritize safety (collision avoidance) and autonomy (self-recovery from blocked paths) in your planning.
Variant
<image"> Predict the ego vehicle's immediate lateral driving path based on the BEV map scenario. Output the planned trajectory as keypoints using this format: [[x1, y1, 01), [x2, y2, 02), .. [xn, yn, 0m]]
Details:
-Cooriinates (x, y) relative to the ego vehicle in meters (x: forward, left).
Heading angle (theta) degrees (positive left, negative right).
Maximum keypoints (n S 3).
Keypoints should mark critical lateral actions like lane changes, turns, or avoidance.
Follow the reference path (purple arrows) us the default, deviating only when necessury for sufety
maneuvers, then smoothly return.
Ensure path safe and avoids collisions.
Variant
Ainagee>Analyze the BEV map. Plan safe lateral maneuver sequence for the ego vehicle (orange box). Provide the sequence trajectory keypoints:
[[x1, y1, 01], ... [xn, yn, @n]]. max points.
Parameters:
(meters): forward distance from ego.
y (meters): lateral distance from ego (left positive).
theta (degrees): heading relative ego's current heading (left positive).
Instructions:
Prioritize collision avoidance above all.
Follow purple reference path unless a deviation (lane change, obstacle avoidance) required.
Ensure smooth realignment with the reference path after any deviation.
Keypoints should capture the essence of the lateral plan.
14

---

Published as a conference paper at ICLR 2026
Variant 4
Banagg:>Evvllaae the current driving situation from the BEV map. Decide the optimal lateral course of action for the ego vehicle. Represent this decision sequence trajectory keypoints:
lxx1, yl, #1), [x2, y2, 62], (xn, yn, /n]] (n S 3)
Coordinate System & Constraints:
longitudinal (m). y: lateral (m), relative ego.
theta: heading change (deg), positive left.
Must generally follow the reference path (purple arrows).
Deviate only for essential maneuvers (lane change, avoidance).
Prioritize safety and smooth recovery to the intended path.
Keypoints define significant lateral shifts adjustments.
A.2.2 INSTRUCTIONS FOR GENERATING COORDINATE POINTS
Variant
Timage>-Based on the provided BEV mup image, identify the three vehicles labeled with numbers 1, 2, and 3.
Determine output their longitudinal position (x), lateral position (y), and heading angle (0) relative to the ego vehicle, strictly in the order of their labels (1, then 2, then 3).
Provide the information in the following format: [[x1, y1, 811, [x2, y2, 62), [x3, y3,
Where:
longitudinal position in meters (relative ego vehicle)
lateral position meters (relative ego vehicle)
Heading angle degrees, positive left turns, negative for right
Variant
<image>>From the given BEV scene representation, extract state parameters for the vehicles labeled 1, 2, and
For each vehicle, provide its relative (forward/hackward), (left/right), and 0 (heading) values refer- enced from the ego vehicle. Ensure the output follows numerical order of the labels 3).
Format the output strictly as: [[x1, y1, 01), [x2, y2, 62), [x3, y3, 83]]
Parameter definitions:
X: Relative longitudinal position (m).
y: Relative lateral position (m).
8: Relative heading angle (deg), positive left turn.
Variant
<image>>Analyze the provided Bird's-Eye View (BEV) perception image. Locate the three distinct targets marked with numerical labels 1, 3.
For each target, compute its pose relative to the ego vehicle's coordinate system. Specifically, deter- mine its longitudinal distance (x), lateral distance (y), and yaw angle (#).
Output results sequentially according to target labels (1 first. then then using the exact format below: [[x1, y1, 01), [x2, y2, 02), [x3, y3, 03]]
15

---

Published as a conference paper at ICLR 2026
Where:
Longitudinal distance (meters) from ego vehicle origin.
y: Lateral distance (meters) from the ego vehicle origin.
0: Yaw angle (degrees), where counter-clockwise rotation relative to the ego vehicle's forward direction is positive.
Variant
<image>>Consider the ego vehicle as the origin (0.0) with heading aligned with positive x-axis in this BEV map. Identify the traffic participants labeled 1, 2, and 3.
Determine the ego-centric coordinates (x, y) and relative orientation (0) for each labeled participant.
Structure the output as a nested containing the [x, y, #] for participant 1, followed by participant 2, and finally participant 3:
[[x participant1, participant1, 9 participant1),
Ix.partie) y-particcpant2, 6-participant2),
[x xx-participant,, y parrticipant3, 6 e-participant3]]
Specifications:
y units: meters.
units: degrees (positive counter-clookkwss///ee)..
A.2.3 INSTRUCTIONS FOR SCENE TYPE RECOGNITION
Instruction Variants (The following prefixes are used with the shared list below):
Variant 1:
<image>>Basce Bassd the BEVMap shown in the image, please determine the current driving scenario of the autonomous vehicle. Choose the most appropriate scenario from the following list 57 predefined categories:
Variant 2:
Feoage>-From From the information in the BEVMap image, infer what type of driving situation the self-driving is currently Use the predefined scenario categories listed below:
<image>>
Variant 3:
<'imag>>GGveen Given the BEVMap in the image, identify which of the 57 labeled driving scenarios best describes the current situation of the autonomous vehicle. Here all possible categories:
Variant
<image:> Please analyze the BEVMap provided in the image and select the most appropriate driving scenario type (from of 57 categories) that describes the present condition of the autonomous
Shared Category List (Applies to all variants above):
accelerating at crosswalk
oet.ol/raiiiggatttteei.ii.iooeoe behind.bike ahhiinddpedestriannon..did
chengingg.lane.too.lffff
clangiing_lane.torrightt
changiing_lane.vvth_leeddd
crossed.by.bike
crossed.by.vehiccle
tollowiing.anne.wihhh.edddd
fo1.orkigg.aaewwthhhssooo.dld
fol folloxingg.laeewtthhnt...eee
aigh
high magnitude jerk
16

---

Published as a conference paper at ICLR 2026
low Low.magnitude.speed
meciumm.mammttuudessedd
ber.baarreer...dddvveebelllll
max..constrrottioon.oo...ss..s
near wear:hiigh.seed..eehiilee
near.Jong.vebicle
neer:multijple..ehhhleeess
near.trafficcon..oo driveable
en.carpark
on.intersectier:
on.pickup.dropoff
oa.stoppino.ercsseallk
ea.stoplinne.ttop.sim
en.stoppl.ne.trafffcc....gg
ptartiing cross turn
starting nonoross turn
tun
sbeeriaiggaaagiaaarrrrr....oooor stationary
stattingguuprrcccceeecccoss.uuu
stationaary.t..rrassalkk
stationaay.in.ttaaffii
stogpingg.at.crossaaall
stoppl traffis git wf thout lead
stopping ng lead
traveraing crosswalkk
craversingiintersection
craversiingnnarrovv.aee
Leaverssinggp.icuppddrooop.
wattinggfoorposstrriantt...orrr.
B IMPLEMENTATION DETAILS
Dataset and Simulator. The nuPlan platform is a large-scale, closed-loop system for autonomous driving trajectory planning, comprising over 1,500 hours of expert driving data collected from four cities. features a wide range complex scenarios, including car following, lane changes, turns, intersections, bus stops, roundabouts, and pedestrian interactions. The platform includes simulator that initializes from real-world scenarios. During simulation, non-ego agents are controlled either by log replay (non-reactive) or Intelligent Driver Model policy (reactive), while the ego vehicle executes user-defined planned trajectories. Each 15-second simulation runs at 10 Hz, with Linear Quadratic Regulator controller used track the trajectory and issue control commands.
Training Details. We adopted the LoRA technique for parameter-efficcent fine-tuning of the VLM. Specifically, we set the LoRA rank to 16, the scaling factor to 32, and the dropout rate to 0.05, ap- plying LoRA adapters to all linear layers. The part of language model was unfrozen during training. For optimization, we used the AdamW optimizer with an initial learning rate of x 10-4 and weight decay of 0.1. The learning rate scheduled using cosine annealing strategy. Training was conducted on eight NVIDIA H800 GPUs over five epochs, taking approximately three days to complete. The proportions of the three dataset categories, namely Key Point, Spatial Localization,
17

---

Published as a conference paper at ICLR 2026
and Driving Scene Stepwise, 10:1:2. The entire dataset contains 838,824 samples. During the process fine-tuning the motion planner, we followed the PLUTO framework to construct mil- lion data splits. Training conducted eight RTX 4090 GPUs with batch size of 128 over 10 epochs, taking approximately 6 hours to complete. We employed the Adam W optimizer with an initial learning rate of 1 x 10-4 and weight decay of 10-1 Unlike PLUTO, however, did not employ the warm-up up technique.
C LIMITATIONS
Despite the numerous advantages of the hierarchical framework that combines VLM with a motion planner, the inference efficiency of the VLM remains key concern. In offline or simulation-based settings, where our method can be used to generate data or serve as smart agent, this limitation generally acceptable. For onboard deployment, the inference burden can be alleviated through techniques such as model quantization, asynchronous inference, or training lightweight model to identify long-tail scenarios. We leave the exploration of real-world deployment strategies future work.
D LLM STATEMENT
Large Language Models (LLMs) were employed solely for language refinement in this paper. Specifically, we used them to polish grammar, improve clarity, and enhance the academic style of our writing. The role of LLMs was limited to editing and improving the presentation of the text, without contributing to the technical content.
18


Reviews:
[{"review_id": "gOhDv5lOsV", "reviewer": "Reviewer_2yL1", "summary": "The paper proposes BIRDriver (Bird’s-eye-view Informed Reasoning Driver) that contains  hierarchical framework that integrates a Vision-Language Model (VLM) with a motion planner to improve autonomous driving to address long-tail scenarios. Unlike prior approaches that rely on multi-frame camera inputs or large waypoint sequences, BIRDriver uses a single-frame Bird’s-Eye-View (BEV) map and generates a small set of key points (≤3) to convey high-level driving intentions to the motion planner.\nBIRDriver key contributions are:\n\n- In BIRDriver, perception input tokens along with BEV maps are passed as input to VLM which interprets and outputs key points; motion planner generates trajectories based on these points.\n- In BIRDrivier, it predicts relative key points small set of points instead of full trajectories for efficient intention transmission.\n- BIRDriver uses three auxiliary datasets (Key Point, Spatial Localization, Driving Scene Stepwise) and a weighted SFT loss to improve numeric precision.", "questions": "- How does BIRDriver handle ambiguity in BEV maps, such as occluded agents or missing traffic signals?\n- Could temporal context (multi-frame BEV) improve reasoning without significantly increasing complexity?\n- How sensitive is the system to BEV map quality (e.g., sensor noise, map inaccuracies)?\n- It would be better to see how weighted SFT loss improves compared to reinforcement learning  GRPO like in Diffusion Planner or geometry-aware objectives?\n- What are the most common failure modes observed in InterPlan scenarios?\n- How does the system perform under real-time constraints? Any latency benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- BIRDriver combines VLM reasoning with BEV representation for motion planning, that results in reducing reliance on multi-modal alignment.\n- In BIRDrivier, it predicts relative a small set of key points that are  compact, interpretable interface between reasoning and planning.\n- In BIRDrivier, is good at spatial localization and scene understanding that addresses numeric token importance, improving precision in key point generation.", "weaknesses": "- BIRDriver latency or inference times remains still, it concerns whether BIRDriver can be applied to real-time applications: VLM inference of BIRDriver remains heavy for real-time onboard deployment; paper only briefly mentions quantization and asynchronous inference as future work.\n- BIRDriver experiments are conducted on limited real-world validation: Experiments are simulation-based  Test14-random, Test14-hard, and InterPlan datasets. It would better to see zero-shot comparisons from simulation to real-world like nuScenes or waymo.\n- Paper lacks detailed discussion on failure cases or robustness under sensor noise and occlusions.\n- BIRDriver mostly focuses on nuPlan benchmarks; does not compare against recent RL-based or hybrid planning approaches like Diffusion Planner.\n- BIRDriver highly relies on BEV maps omiting temporal context beyond 2-second history; may limit reasoning in highly dynamic scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pSjGAecKP5", "reviewer": "Reviewer_BksK", "summary": "This paper proposes BIRDriver, an hierarchical framework that integrates Vision-Language Models (VLMs) with a motion planner to address the motion planning challenges in complex, long-tail autonomous driving scenarios. Unlike existing methods, BIRDriver utilizes only a single-frame Bird's-Eye View (BEV) as the primary input for the VLM, avoiding the reliance on domain-specific encoders and the complexity of multi-modal data alignment. The framework leverages the VLM's common-sense reasoning capabilities to generate keypoints, which are subsequently passed to the motion planner to produce the final trajectory. To address the difficulty of standard VLMs in generating precise coordinates, the authors designed three auxiliary datasets and a token-level weighting mechanism, significantly improving keypoint prediction accuracy.", "questions": "Refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1、VLM Integration Paradigm: The paper introduces a hierarchical architecture where a Vision-Language Model (VLM) generates high-level keypoints, while a conventional planner executes low-level trajectories. This approach elegantly balances the advanced reasoning capabilities of VLMs with the real-time performance and safety of traditional planners.\n\n2、Dataset Innovation: To address the limitations of Vision-Language Models (VLMs) in spatial localization and scene understanding, the authors meticulously designed three auxiliary datasets. This multi-task learning strategy significantly improved the accuracy of keypoint prediction.\n\n3、Comprehensive Experimental Validation: The method was extensively validated on the nuPlan dataset across three benchmarks: Test14-random, Test14-hard, and InterPlan. Notably, on the InterPlan long-tail test set, BIRDriver significantly outperforms PLUTO and Diffusion Planner, demonstrating its effectiveness in handling complex and rare scenarios.", "weaknesses": "1. BEV-Only Input: Relying solely on BEV as input is overly idealistic and impractical. While the dataset currently in use contains BEV information, other datasets may not. More critically, ground-truth BEV information is not available during real-world driving. This would necessitate the model to predict the BEV, which in turn would introduce errors. Such a practical setting is far more complex than one with standard multi-view images, raising concerns about the model's generalizability and practical deployability.\n\n2. Inference Efficiency: The authors' acknowledgement of the VLM's inference efficiency issues in the limitations section is commendable for its honesty. A clarification is needed: What is the inference time for a single VLM call? Motion planning typically requires high-frequency operation (e.g., 10Hz or higher). Is the current framework executed synchronously (with the VLM being called at every step) or asynchronously? If synchronous, the VLM's latency would directly impact the system's reaction time.\n\n3. Keypoints: While the authors advocate for using no more than three keypoints, there is a lack of sufficient theoretical and experimental justification for why three is the optimal number. The experiment in Section 5.3 only compares the RDP algorithm with a method using only the final trajectory point, failing to explore the performance differences among using 2, 3, or 4 keypoints. In certain highly complex scenarios, three keypoints may be insufficient to fully express the complete driving intent, yet the paper does not investigate the relationship between the number of keypoints and performance. Additionally, more details regarding the setting of ε are required.\n\n4. Minor Performance Degradation in Simple Scenarios and Insufficient Analysis: As shown in Table 1, on the Test14-random benchmark, BIRDriver (91.46) performs slightly worse than its base model, PLUTO (91.87), on the CLS-NR metric. Although the margin is small, this raises an important question: Does the intervention of the VLM introduce unnecessary complexity or superfluous decisions in simple, routine scenarios, thereby causing this slight performance degradation? Does this imply the need for an arbitration mechanism between the VLM and the base planner, which would activate the VLM only when a difficult scenario is detected? The authors should provide an analysis and discussion of this phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Ag8QDWeaz", "reviewer": "Reviewer_ScDp", "summary": "This paper proposes a hierarchical motion planning framework that integrates a Vision-Language Model (VLM) with a motion planner. Specifically, the VLM interprets the driving scene from a BEV map and outputs keypoints to represent high-level driving intentions. The motion planner PLUTO is then extended to incorporate these keypoints for trajectory planning.", "questions": "1. The paper extends the PLUTO architecture to implement the hierarchical framework. Can the proposed architecture be adapted to other existing advanced motion planners as well?\n\n2. Can the proposed framework meet the real-time requirements of autonomous driving motion planning, given that VLM inference may be relatively slow?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The VLM in BIRDriver requires only the BEV map as input to convey the complete driving scene, eliminating the need for textual descriptions.\n\n2. BIRDriver expresses driving intentions through keypoints, which offer advantages over meta-action, hidden-state, and waypoint-based approaches.\n\n3. BIRDriver introduces a weighted SFT loss and three specialized datasets to enhance the accuracy of keypoint prediction.\n\n4. BIRDriver achieves state-of-the-art performance in closed-loop evaluations on long-tail scenarios of the InterPlan benchmark.", "weaknesses": "1. The contribution of this work is limited: using BEV maps as VLM inputs has already been explored in prior work [1], and some studies [4] have even gone further by feeding BEV features directly into LLMs for autonomous driving tasks. Moreover, the authors claim that relying solely on BEV maps eliminates the need to align heterogeneous sensor data across different vehicle platforms. In fact, many existing works [2,3,4,5] have already achieved the same goal by using only map features or multi-view images as inputs.\n\n2. The paper conducts experiments only on nuPlan and its long-tail test set InterPlan, lacking evaluation on broader datasets such as NuScenes and Waymo. If time and computational resources permit, extending experiments to more datasets and comparing with additional VLM-based autonomous driving methods[2,4,5,6,7,8,9,10] would be valuable.\n\n[1] PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning\n[2] Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving\n[3] Asynchronous Large Language Model Enhanced Planner for Autonomous Driving\n[4] Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models\n[5] OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning\n[6] VLP: Vision Language Planning for Autonomous Driving\n[7] Making Large Language Models Better Planners with Reasoning-Decision Alignment\n[8] VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision\n[9] EMMA: End-to-End Multimodal Model for Autonomous Driving\n\n[10] DRIVEVLM: The Convergence of Autonomous Driving and Large Vision-Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQKMhjb8oi", "reviewer": "Reviewer_ctTg", "summary": "This work proposes improvement over PLUTO hybrid planner by utilizing extra features generated by a VLM using BEV image. They fine tune the VLM to produce key points from BEV scene image using LoRA. These key points summarizes the trajectories of the agents in a compressed way. These are fed into the point encoding module of PLUTO planner to generate the trajectory. Their experiments show improvement on NuPlan and InterPlan over PLUTO.", "questions": "1. Key point generation could be possible from deterministic or classical learning methods as well, instead of using VLM. That would make the inference latency more practical. Did the authors consider this?\n2. As the authors mention the PDCE loss, did they perform any comparison with their weighted SFT loss?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. BEV representation is quite generic and it's generalizable over sensor types, orientation and platforms. This can be applicable to a lot of different architectures and vehicle platforms.\n2. Using key points, instead of dense trajectory representation, is more efficient.\n3. Using weighted SFT loss and spatial localization learning for BEV are also generalizable techniques, that can benefit many works.", "weaknesses": "1. The approach is generalizable, but it's only shown with PLUTO planner. It would have made the case stronger if the authors also showed benefit with at least one more planner with BEV based feature augmentation.\n2. The results are shown only using nuPlan, not using reactive simulator like CARLA. This work would benefit from adopting Bench2Drive for benchmarking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)