CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
Authors: Dapeng Zhang
Fei Shen
Rui Zhao
Yinda Chen
Peng Zhi
Chenyang Li
Rui Zhou
Qingguo Zhou
Venue: NeurIPS 2025 poster ()
Keywords: VLA
VLM
LLM-AD
Domain Transfer
Autonomous Driving
End-to-end.
Primary area: applications
TL;DR: A VLM method for vehicles and robots.

Abstract: Autonomous driving represents a prominent application of artificial intelligence. Recent approaches have shifted from focusing solely on common scenarios to addressing complex, long-tail situations such as subtle human behaviors, traffic accidents, and non-compliant driving patterns. Given the demonstrated capabilities of large language models (LLMs) in understanding visual and natural language inputs and following instructions, recent methods have integrated LLMs into autonomous driving systems to enhance reasoning, interpretability, and performance across diverse scenarios. However, existing methods typically rely either on real-world data, which is suitable for industrial deployment, or on simulation data tailored to rare or hard case scenarios. Few approaches effectively integrate the complementary advantages of both data sources. To address this limitation, we propose a novel VLM-guided, end-to-end adversarial transfer framework for autonomous driving that transfers long-tail handling capabilities from simulation to real-world deployment, named CoC-VLA. The framework comprises a teacher VLM model, a student VLM model, and a discriminator. Both the teacher and student VLM models utilize a shared base architecture, termed the Chain-of-Causality Visual–Language Model (CoC VLM), which integrates temporal information via an end-to-end text adapter. This architecture supports chain-of-thought reasoning to infer complex driving logic. The teacher and student VLM models are pre-trained separately on simulated and real-world datasets. The discriminator is trained adversarially to facilitate the transfer of long-tail handling capabilities from simulated to real-world environments by the student VLM model, using a novel backpropagation strategy. Experimental results show that our method effectively bridges the gap between simulation and real-world autonomous driving, indicating a promising direction for future research.

CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
Dapeng Zhang 1.2 Fei Shen'; Rui Zhao', Yinda Chen', Peng Zhi', Chenyang Li',
Rui Zhou', Qingguo Zhou
Lanzhou University. China
National University of Singapore, Singapore
University of Science and Technology of China, China
Corresponding authors
/zhangdp22, zhaorui, zhip21, Ichenyang2024, zr, zhougg) @ @lzu.edu.cn, Seanfei2@@nus.cem edd.udu.sg cyd0806 mail, uswi.luuut.eunn
Abstract
Autonomous driving represents a prominent application of artificial intelligence. Recent approaches have shifted from focusing solely on common scenarios addressing complex, long-tail situations such as subtle human behaviors, traffic accidents, and non-compliant driving patterns. Given the demonstrated capabilities of large language models (LLMs) in understanding visual and natural language inputs and following instructions, recent methods have integrated LLMs into au- tonomous driving systems to enhance reasoning, interpretability, and performance across diverse scenarios. However, existing methods typically rely either on real- world data, which is suitable for industrial deployment, tailored to rare or hard case scenarios. Few approaches effectively integrate the complementary advantages of both data sources. To address this limitation, we propose a novel "LM-guided, end-to-end adversarial transfer framework for au- tonomous driving that transfers long-tail handling capabilities from simulation real-world deployment, named CoC-VLA. The framework comprises a teacher VLM model, student VLM model, and discriminator. Both the teacher and stu- dent VLM models utilize a shared base architecture, termed the Chain-of- of-Causalit Visual-Language Model (CoC VLM), which integrates temporal information via an end-to-end text adapter. This architecture supports chain-of-though reason- ing infer complex driving logic. The teacher and student VLM models are pre-trained separately on simulated and real-world datasets. The diseriminator is trained adversarially to facilitate the transfer of long-tail handling capabilities from simulated to real-world environments by the student VLM model, using a novel backpropagation strategy. Experimental results show that our method effectively bridges the gap between simulation and real-wort autonomous driving, indicating a promising direction for future research.
simulation data
Introduction
Autonomous driving has advanced significantly over the past decades, attracting interest from both commercial and academic sectors. has evolved from simple trajectory tracking into complex, integrated system. Typically, an autonomous driving system consists of several modules, including environmental perception and cacisionnmaking.. These systems may rely on rule-based methods for navigating familiar roads or imitate human behavior to manage wide range of driving scenarios. Prevailing autonomous driving approaches primarily focus on achieving breakthroughs on challenging benchmarks and are typically trained and evaluated datasets collected in real-world
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

End-to-end prompt teacher model
step1:
End-to-end prompt student model
Discriminator Loss
Train slep2: All data backward
Simulation image fonward vision adapter
Train slep2: All data back propagate without update model parameters
weal-world image forward in vision adapter
Discriminator
VLM Loss
End2end Answer
End2end Answer
Filter
LLM
LLM
Filter
Projection
Projection
Text Embedding
Text Embedding
Vision Encoder
Vision Encoder
CoC Text Adapter
Vision Adapter
Vision Adapter
CoC Text Adapter
Text Adapter
Text Adapter
Temporal Info
End2end
End2end Prompt
Temporal
T-1
Prompt
Info
Teacher VLM
Student VLM
Figure 1: Overview of the proposed framework. The framework comprises teacher VLM model (based on the simulation CoC VLM baseline), student VLM model (based on the real-world CoC VLM baseline), and discriminator. The teacher and student VLM models share the same architecture are trained separately on simulation and real-world datasets, respectively. The discriminator employed facilitate the adversarial transfer of capabilities from the simulation domain to the real-world domain. Train stepl refers to the adversarial training stepl. Train step2 refers to the adversarial training step2.
environments. However, such systems may fail previously unseen cases, such as rare accidents or unexpected human behaviors, prompting the development of simulated dataset benchmarks to evaluate performance under rare or hard case scenarios.
To ensure robust performance across diverse environments, researchers have adopted data-driven end-to-ee autonomous driving methods. These methods enhance system integrity by eliminating accumulated errors. Moreover, the widespread adoption of end-to-end models enables faster inference and lower resource consumption 2 By processing input from surrounding images and/or LiDAR data, end-to-end systems generate final trajectories and/or control signals. However, these systems operate as black boxes, offering no interpretability or explanation for their decisions, thereby raising ethical and legal concerns. Furthermore, they lack mechanisms for human interaction, limiting their applicability advanced autonomous driving.
To address the black-box nature of end-to-end autonomous driving, numerous studies have explored the integration of large language models (LLMs) enhance vehicle interpretability, controllability, and robustness. LLMs have demonstrated remarkable capabilities when trained on large-scale datasets, which are highly desirable autonomous driving systems. Additionally. LLMs exhibit strong generalization capabilities, enabling them to handle unseen scenarios and unfamiliar environments. Among L.L.-based autonomous driving approaches, several trained and evaluated real- col-world datasets for industrial deployment. Inspired by 3 4 DriveGPT4 (5) represents a pioneering effort to leverage LLMs for interpretable end-to-end autonomous driving. It accepts multi-frame sequences and instruction prompts, and outputs vehicle control signals. Experimental results on real-world datasets demonstrate the effectiveness of this method. LLMs offer capability absent in traditional autonomous driving systems: they can interpret textual descriptions and translate driving instructions into actionable control signals. Moreover, LLM-based approaches aim to address long-tail challenges in autonomous driving by employing diverse modeling strategies and utilizing various sensor modalities (6-8), However, LLMs trained on real-world datasets often fail to capture critical uncommon cases such traffic accidents, non-compliant driving behaviors, and pedestrian intrusions. To address this limitation, some approaches are trained on simulated data and evaluated in virtual environments. For example, [9 trains language-bass based autonomous driving model using

---

data collected in the CARLA simulator [10]. and performs closed-loop evaluations within simulated environments. However, these methods are tested on simulators, where the generated control signals cannot be executed in real-world conditions. In summary, the gap between real-world and simulated datasets remains an underexplored area of research.
Furthermore, to incorporate temporal information from historical frames, existing methods typically feed image sequences directly into the LLM [9]. This significantly increases token length and computational resource consumption. Therefore, an efficient and lightweight temporal aggregation strategy essential.
In this paper, we propose a novel VLM-based model for explainable end-to-end autonomous driving that not only effectively transfers capabilities learned from simulated hard cases to real-world applications, but also integrates temporal information and end-to-end outputs via a chain-of-causalle policy. The model takes surrounding image pairs and driving instructions in natural language, and predicts CoC answers. The overall architecture is illustrated in Fig. [ Our method comprises two base models: a teacher VLM model and a student VLM model, as well a discriminator. The teacher VLM model is trained on synthetic data to acquire the ability to handle rare and challenging scenarios, and transfers this knowledge to the student VLM model via adversarial learning with the discriminator. The training process is non-trivial, requiring both a pre-trained VLM model and multiple stages of training. During inference, only the student VLM model is deployed in real- world scenarios. Furthermore, we reproduce several existing methods and compare them with ours using public benchmark dataset. Experimental results demonstrate that our method achieves an excellent result.
The contributions of this work are summarized as follows:
We present the first VLM-based autonomous driving model capable of transferring the ability to handle uncommon scenarios from simulation to real-world, thereby bridging the gap between simulated and real-world environments.
We introduce a novel discriminator that learns the domain gap between simulation and real-world data, enabling effective knowledge transfer from the teacher to the student VLM model.
We propose a back prockprrpaggtion strategy that enhances the convergence stability of the adversarial training process.
We develop chain-of-causall policy that connects temporal information to CoC answers, enabling chain-of-thon -f-thughht reasoning to model deep driving logic.
We conduct extensive experiments the nuScenes-VLMM dataset, demonstrating that our approach significantly outperforms existing methods.
2
Related Works
2.1 End-to-End Autonomous Driving.
Traditional autonomous driving methods suffer from complex module designs and limited interaction between modules, often resulting in error accumulation. To address these challenges, researchers have proposed end-to-end autonomous driving methods that unify previously separate modules into a cohesive framework. Typically, these methods integrate perception, mapping, prediction, and planning sub-tasks into single model. For instance, UniAD [I] integrates features from the perception and prediction modules and generates ago-vehicle planning trajectories using a transformer architecture. Building on BEVFormer [1I] VAD (2 extracts BEV features and regresses planning trajectories through multiple interactions and constraints. FusionAD [2) also predicts ego trajectories
on BEV maps constructed from both camera data and LiDAR point clouds. VAD2 KK] introduces vectorized encoding approach to model the probabilistic distribution of trajectories, demonstrating superior closed-loop performance. Additionally, Ego-MLP and BEV-Planner [14] 15) extensively explore the impact of ego-vehicle status to enhance trajectory planning accuracy. ThinkTwice [16] retrieves encoder features near the predicted coordinates and refines coarse-graines positions and actions. Inspired by these advancements, ReasonNet [17] utilizes global and temporal representations of driving scenarios enhance feature extraction. Notably, TCP [8] integrates trajectory and control action predictions into dedicated branches to improve overall driving robustness. Roach [19 employs reinforcement learning to distill final agent capable of interacting effectively with dynamic environments. In contrast to prior methods, SparseDrive (20) employs sparse feature sampling alongside hierarchical planning strategy to generate rational and efficient planning outputs. Since open-loop evaluations do not account for dynamic responses from surrounding vehicles, researchers
3

---

have introduced closed-loop evaluation metrics, such as Driving Score, Route Completion, and Infraction Score, to more accurately assess and optimize their models [0]
2.2 LLM for Autonomous Driving
In recent months, the emergence of large language models (LLMs) 22 4 has led researchers to extend them into vision-language models (VLMs) (3 24 which integrate textual and visual data for richer content representation. In the field of autonomous driving, researchers have begun using L.MSs and VL enhance overall system performance. For example, DriveGPT4 [5] uses multi-modal input data to generate expected control signals for the vehicle. Another method integrates LLMs into autonomous driving frameworks to generate action recommendations along with detailed explanations 19. However, the control actions predicted by these methods often fall short of real-world navigation demands. Consequently, researchers are increasingly focusing on generating textual descriptions of driving actions to offer interpretable and contextually appropriate explanations. For instance, CM [6] incorporates diverse environmental contexts to enhance robustness across a range of scenarios. Instead of relying camera data, LiDAR-LLM [7] utilizes raw LiDAR inputs and employs a three-stage training strategy to align 3D modalities with the L.LM embedding space, thereby enhancing spatial understanding for autonomous driving. DriveMM S processes diverse inputs, such as images and multi-view videos, to pre-train a baseline model, which is then refined to improve generalization in vehicle control. To improve temporal representation, La VidaDrive [25] introduces a Query-aware Token Selection module, a Spatial-Temporal Token Recovery and Enhancement module optimize both efficiency and performance. To address VLM limitations in spatial reasoning. DriM (26) integrates specialized reasoning modules for scene understanding and hierarchical planning. Additionally, DriveMLM 27 introduces a behavior planning module to generate optimal driving decisions with interpretable justifications. Notably, recent approaches planning capabilities (2]).
incorporate reinforcement learning to enhance
2.3 Domain Transfer
Domain transfer learning aims to build models capable of performing tasks in target domain by
leveraging knowledge learned from a source domain. The method proposed in (29), implemented using deep learning strategy, achieves effective domain adaptation across several classification datasets. To explore domain-specific characteristics, (30] explicitly extracts image representations
partitioned into two subspaces: one private to each domain and the other shared across domains. Given the importance of pre-training in transfer learning, some methods leverage it to enhance adversarial robustness compared to other approaches (3I. Most existing methods align the fully connected layers in neural networks, while convolutional layers, which typically encode critical low-level domain knowledge, are often left unmodified. This limitation restricts the effectiveness of domain discrepancy reduction. To address this, 332 proposes an attention alignment mechanism on convolutional layers to better minimize discrepancies between domains. MetaAlign 33) introduces novel meta- optimization strategy that maximizes gradient-based learning during training. Additionally, some researchers have proposed asymmetric training schemes that align target domain features more closely with those from the source domain 34
[35] leverages object attributes to facilitate robotic grasping and rapid adaptation across domains. Inspired by [36], which simultaneously trains generative model to learn data distribution and a discriminative model to estimate sample likelihoods, Pix2Pix (37 extends the GAN-based strategy to image-to-image translation tasks, demonstrating the effectiveness of the discriminator in image generation. Subsequently, numerous methods have emerged to address diverse application scenarios, including PatchGAN, DTSGAN, and REGAN 38 40].
3 Methods
As illustrated in Fig. 1, the proposed architecture consists of three components: a Teacher Visual Language Model (Teacher VLM), a Student Visual Language Model (Student VLM), and Visual Language Model Diseriminator. Both the Teacher and Student VLMs share a common base architec- ture, referred to as the -faussaliyyy Visual Language Model (CoC VLM), which processes multi-view image pairs, end-to-end prompts, and historical instructions from previous frames to generate end-to-end outputs. The Teacher VLM is trained on simulated data to address diverse and rare scenarios, such pedestrian trespassing, driving violations, and traffic accidents. In contrast, the Student VI.M is trained on real-world data and serves as the final inference model, enabling the transfer of knowledge from simulated to real- wol-w contexts.

---

3.1 Chain-of-Causalitt Visual Language Model
Since the Teacher VLM primarily designed to transfer the capability of handling hard and challeng- ing cases to the Student VLM, both VLMs are constructed using the same base architecture but with different parameter sets. We design the Chain-of-Causalle
Visual Language Model as the shared backbone for both the Teacher and Student VLMs. Based on extensive experimentation, LL-VA-v1.5 41 was selected as the pre-trained VLM. The CoC VLM is primarily based on LLaVA 41 and comprises four modules: Text Adapter, Vision Adapter, LLM Brain, and CoC Answer. Compared to the original LLaVA, the CoC VLM introduces several enhancements: (1) A novel Chain-of ofaii--f-Causality Text Adapter aggregates simplified answers from the previous frame and current instruction prompts, thereby incorporating historical context and enhancing temporal causal reasoning. (2) A filter is em- ployed to simplify the -LM--eenrated responses, effectively reducing token length. (3) A dedicated CoC answer generation module is introduced to streamline output formatting.
3.1.1 Text Adapter
The Text Adapter comprises two components: the
(CoC) Text Adapter and the Text Embedding module. The CoC Text Adapter aggregates temporal information from the previous frame along with the current end-to-end prompt. Unlike existing methods such as LMDrive [9]. which utilize all historical sensor data to encode temporal information, our method selectively incorporates simplified LLM outputs from the previous frame to enhance temporal consistency. This approach significantly reduces token length and computational resource consumption. The proposed Text Adapter both simple and efficient. An illustrative example is provided below:
In the previous frame, white car ahead moving away from the ego vehicle, and the vehicle continues in its current driving state.
Temporal Instruction:
End-to-End Prompt:
What are the objects around the ego vehicle? What is the moving status of this object? What is the next action the closest object? Will thread ego vehicle's safety? What are safe actions to take for the ego vehicle? Predict the future motion of the ego vehicle.
Given the Temporal Instruction and End-to-End Prompt, we tokenize and concatenate the texts into textual tokens, which are then embedded using the same embedding module as LLaVA 41).
3.1.2 Vision Adapter
The Vision Adapter transforms image data into tokens, as illustrated in Fig. I To emulate human driving behavior, which relies solely 2D visual perspective, we directly stack six surrounding camera images as input. No extrinsic or intrinsic parameters are utilized, nor are the images transformed into BEV features, human drivers also do not rely on such representations. This approach contrasts with several existing end-to-end autonomous driving methods. Following the guidance of LLaVA-V1.5 [4), we adopt the pre-trained CLIP image vision tower our vision encoder and apply a projection module to convert image features into tokens, which are subsequently integrated into the textual prompt.
3.1.3 LLM Brain
The LLM Brain employs a specialized
reasoning mechanism to infer deep driving logic. This module processes image tokens generated by the vision adapter and combines them with text instructions from the text adapter to comprehend driving scenarios and generate tokens for the next step. Similar to LLaVA-v1.5 [41], we adopt the LLaMA [4] language model as our LLM Brain. Leveraging its pre-trained weights and fine-tuned components, our model achieves convergence as we expected. The LLM Brain outputs end-to-end autonomous driving answers to manipulate the ego vehicle.
3.1.4 CoC Answer
After processing by the LLM Brain, the output token sequences are decoded using a tokenizer decoder to generate CoC answers. Inspired by 421, our CoC answer integrates perception, prediction, and planning into a causal structure, which we represent as a causal chain. Specifically, perception determines the future actions of surrounding objects, which subsequently influence the ego vehicle's motion. Causal reasoning is applied throughout the entire CoC answer. Furthermore, we design a filter to summarize and cache the CoC answer, converting it into a concise instruction for the subsequent frame. We provide an example below. This filter removes the reasoning component of the CoC answer and extracts only the final action instruction for the ego vehicle-for example, and reduces the input token size while preserving the most critical information from the previous frame.
5

---

CoC Answer: There is white car in front of the ego vehicle, with coordinates <CAM FRONT, 1009, 486, 1074. 527>. The white car <CAM FRONT, 1009, 486, 1074, and there are two pedes- trians .... In front of the ego vehicle, there no safety threat front of the ego vehicle, the ego vehicle should continue moving at the same speed. The future trajectory is (4.7, -0.7), (7.4.
527> is accelerating and moving away. There is a traffic light
Cached Temporal Information: There is no safety threat, the vehicle maintains its current speed.
3.2
Discriminator
3.2.1
Structure Design
This module aims to address the distributional gap between real-world and simulation domains by transferring the performance of the teacher VLM model to align with the expectations of the student VLM model. The discriminator is implemented using transformer architectures. processes features from both the teacher and student VLM models. Its objective is to minimize the domain gap and adversarially align feature representations for the student VLM model.
3.2.2 Discrepancy Analysis
Let the data spaces of the simulation and real-world domains (referred to the teacher and student domains, respectively) be denoted as X S and X We denote the distributions that collected data samples from these two domains as {-t p" (yy [=])] E X and {x,, (uf [xr]] X", x and æt are the data samples of simulation and real-world domains, p² p"( (=() and p" (w[ (a]) represent the conditional label distributions in the source and target domains, respectively. i is the sample index. The discriminator is designed to learn representations that capture the domain shift between the
two domains. Since both the teacher and student VLM models project data into feature spaces, the corresponding transformations are:
( Z (x R = gr gr(x
o E X of E X
where, 2 and R are the feature space representation of two inputs. Is and Gr are the corresponding functions of two base models, tasked with preserving rich information relevant to autonomous driving. The discriminator extracts representations in hypothesized domain space:
HS (2 h. (g)( H
2 X
(x())
of X
H and H denote the output distribution spaces induced by the compositions h2 Is and h, gr. h2 and h are hypothesis functions to unify the feature spaces. Given the definable difference between domains, we introduce a transformation matrix Ter to minimize the distributional distance between the simulation and real-world domains. Accordingly, the transformation function between the two domains is defined as:
H R = T22r H = Tø2r (h, 9a)
With the relationship between the two domains established, the hypothesis function for the real-world domain is defined as:
0. = T22r lr, (x( R
here, the real-world output 1/. is defined by has Is and transfer matrix Ts2r- This method aims to quantify the discrepancy between the two domains. Given the two equations defined above, the discrepancy Dí. between two domains X and X can be expressed by space samples of the two domains. An invariant representation is expected to satisfy D (Ix = Hence our algorithms learning representations and minimizing the domain error when label distributions differ between source and target domains.
(I( = E (x"h)) exx
(5)
E
(1. (L. )),""
rEx

---

These results demonstrate that the hypothesis function h captures domain data distributions and directly affects the measured discrepancy. As indicated by these equations, an upper bound is required to characterize the simulation domain, which can be defined as:
sup E (ho y), = 2"EXS
(6)
Here, l, is the bounded distance loss, the expected parameter is represented with p. Given the transformation matrix Te2r linking the two domains, we further explore the formula for discrepancy Do:
(X*( <= sup( E (h>(gs(x*)) v²)
x*EXa
E x"Ex"
(he= y')))
where sup is supremum, T.2r- denotes the Fenchel conjugate of a lower semi-continuous convex function. This discrepancy Dj is variational formulation of the f-divergence for the convex function 5, thus, D (X51) (X$IIX serves as a lower bound estimation of the f-divergence function. In our method, the hypothesis can clearly explain the foundation of our hypothesis, since the final function is a continuous convex function, which can be optimized with appropriate solvers.
3.2.3 Adversarial Optimization
In our VLM-based autonomous driving model, two components must be optimized to effectively transfer knowledge from the simulation domain the real-world domain. These components are the VLM autonomous driving regression and the domain discrepancy discriminator, denoted as D (X* The discriminator estimates and minimizes the feature distribution discrepancy between domains X and X The total optimization problem is formulated as:
min( {."
v)((v(>))))))))) p) + Di (x
(8)
where, IVLM is the VLM autonomous driving loss, hvL. is the VLM autonomous driving optimiza- tion function. Furthermore, we define an upper bound for this discriminator:
Li pise= S (ais(haassgass(""" (Idia(=""")) "), """"""""""""" "p("""">"""" "1=""
(x"")(" p(a"" "yy dx
"
where hais Idis represents the discriminator networks responsible for extracting features from the input data. From the two equations above, we derive the following inequality:
Drie (x () <= max ddis
(10)
The final optimized function is given by:
min I IV Las (o)
(11)
min mar 7-"
(z*)), ").y" yd "ply"" |x" )((p")(()))))))))))"))) " Jdy dx
As shown in Eq. I1] our model first minimizes the VLM loss functions using finite samples, the second components correspond to discriminator losses, which are optimized using adversarial strategy with a min-max formulation.
Experiments
4.1
Model Training
The training procedure comprises two stages: pre-training and adversarial training.
4.1.1
Pre-training
During pew-tprriinnee training, the teacher VLM model and student VLM model are trained separately using simulation and real-world datasets, respectively. To expedite training, checkpoints from LLaVA-v1.55
41 are loaded as initialization. Both models accept image frames, temporal instructions, and end-to-end prompts as inputs to fine-tune their respective Shain-of-CCusaliit (CoC) VLMs, thereby effectively aligning instruction, visual, and temporal information. Efficient training is achieved by sampling frames at fixed intervals and applying temporal augmentation through random temporal shifts.
7

---

4.1.2 Adversarial Training
As illustrated in Fig. T the adversarial training process involves multiple steps. Initially, the pre-trained teacher and student VLM models are loaded, and then the following steps are executed. Step I: The teacher VLM processes simulation data, while the student VLM processes real-world data. Features extracted from both CoC VLMs are subsequently fed into the discriminator, After calculating the discriminator loss, backpropagation is performed to only update exclusively the discriminator parameters.
Step 2: The student VLM model is forward propagated using the real-world dataset, with concur- rent involvement of the discriminator. This step optimizes the student VLM model using both its autonomous driving loss and the discriminator loss. During backpropagation, the discriminator propagates gradients without updating its parameters. The student VLM model is updated through backpropagation based on the combined loss.
4.2 Main Comparison
We conducted a VQA-related experiment to compare our method with existing LLM-based au- tonomous driving approaches. As illustrated Table our method demonstrates significant advan- tages in VQA performance on the nuScenes- VLM dataset. The 4LM-based autonomous driving task is similar traditional L.MM tasks. Our method significantly outperforms the well-known DriveLM 42 in terms of BLEU scores, achieving BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 74.06, 69.33, 63.77, and 58.84, respectively, using the VA-V backbone. We also trained our student VLM model mixed dataset comprising both simulation and real-world data. Besides, we conduct another experiment, we using a two-stage fine-tuning process: first on the simulation dataset, followed by fine-tuning on the real-world dataset. Both approaches yielded inferior results compared to our method. The ROUGE-L performance aligns with our expectations, achieving improvements of 3.38 and 1.01 over DriveLM and our proposed-Fine Finetune Twice model, respectively. Notably, our method achieves nearly f-fold improvement over the competing method in terms of CIDEr score. Specifically, our -----bbed method shows 1.05x improvement in accuracy compared to D [42]. Furthermore, it yields a Match score of 45.30 and a SPICE score of 51 $1.88, both of which significantly surpass competing methods.
In our autonomous driving quantitative experiments, we aim to evaluate our method's ability to perform open-loop driving the nuScenes-VLLE dataset, focusing transfer learning from simula- tion to real-world scenarios and addressing the scarcity of challenging cases in real-world data, As shown in Table 2 our approach demonstrates superior performance the nuScenes- VLM dataset. Specifically, our method with the LLaVA-7b backbone outperforms the competing method [42] by more than 15.8% and 15.5% in ADE and collision rate, respectively. Notably, our final proposed model achieves improvements of 0.16 and 0.15 over the proposed-Finetus Twice model. These findings strongly validate the effectiveness of our proposed model architecture.
Table The general performance on our nuScenes- VLM dataset is evaluated using various language metrics. * Indicates reproduced results. Proposed-Mii refers to the approach in which the student VLM model mix-trained using both simulated and real-world data. Sepposed-FnnetueeTVVe de- notes the strategy where the student VLM model is first fine-tuned simulated data and subsequently fine-tuned on real-world data to obtain the final results.
BLEU 1
Methods
ROUGE-L CIDEr 1 GPT1 Accurucy Match SPICE
2
3
4
DriveL.M 22
72.64 66.77 61.02 55.13
68.32
3.62
58.21
36.67
34.59
48.95
Propsedd-Mix(LLLaVA-
71.98 67.31
61,57
56.11
68.60
21.07
60.22
73.13
40.38
49.98
72.18 68.07 61.77 58.36
70.69
21.21
61.72
73.74
41.51
49.87
Proposed(LLaVA-b))
74.06 69.33 63.77
58.84
71.70
23.21
62.64
75.31
45.30
$1.88
4.3 Ablation Study
4.3.1
Key Components Elfectiveness
We conduct several ablation studies to evaluate the effectiveness of key component designs, with results presented in Table Five experiments are conducted using different combinations of model components. As shown Index-1 of Table 3 the teacher VL.M model trained solely on the simulated dataset and evaluated on the nuScenes- VLM dataset. This configuration results in poor
8

---

The open-loop evaluation of planning performance conducted on our nuScenes- VLM dataset. This evaluation is based on Average Displacement Error (ADE) and Collision Rate metrics. * Indicates results reproduced on our nuScenes-Vil VL.M dataset. Proposed-Mi Mix refers to the approach in which the student VLM model is mix-trained using both simulated and real-world data. Proposed- Finetune'Twic denotes the strategy where the student VL.M model first fine-tuned on simulated data and subsequently fine-tuned on real-world data to obtain the final results.
Table 2:
Methods
ADE +
Collision Rate (%)
DriveLM* 42
1.71
1.87
Proposed-Mix (LLaVA-1
1.84
1.96
Proposed-Finctund Twice (LLVA-70
1.60
1.73
Proposed (LLaVA. VA-7b)
1.44
1.58
performance. Similarly, as shown in Index-2, training the student VLM model exclusively on the nuScenes- VLM dataset yields suboptimal results, achieving an ADE of 1.66 and collision rate of 1.80%. In Index-3, the student VLM model is trained on mixture simulated and real-world datasets. A slight performance degradation is observed compared to Index-2, which can be attributed the model's attempt generalize across two distinct domains, reducing its effectiveness in a single- scenario evaluation. In Index-4, the student VLM model is first fine-tuned on simulated data and then fine-tuned on real-world data. This sequential fine-tuning strategy yields slight improvements compared to Index-2. The final experiment corresponds to our proposed model, which achieves an ADE of 1.44 and a collision rate of 1.58, representing reductions of 10.0% and 8.7%, respectively, compared to Index-4.
Table 3: Ablation studies are conducted key design elements of the proposed method using the nuScenes-VI. VI.M and simulation datasets. The results demonstrate the effectiveness of the proposed design. The baseline large language model (LLM) used for comparison is LLaVA-v1.5 (7b).
Index
Module Component Description
ADE |
Collision Rate (%)
teacher VLM model baseline:
17.33
Simulation dataset trained, real-world dataset tested
student VLM model baseline:
2
1.66
1.80
Real-world dataset trained, real-world dataset tested
student VLM model baseline:
3
1.84
1.96
Simulation dataset and Real-world dataset mixed trained
student VLM model baseline:
4
Simulation dataset finetune firstly,
1.60
1.73
then finetune with Real-world dataset
5
Proposed (LLaVA-7b)
1.44
1.58
4.4 Qualitative Evaluation
We present additional qualitative results further substantiate the superior performance of our model. Qualitative results are shown in fig. 2 and fig. [3] fig. 2 depicts a scenario in which the vehicle continues to move while the traffic light is green. fig. iillustrates a scenario involving the avoidance of potential risks. Guided by our end-to-end prompt instruction, the method generates the expected CoC answers and corresponding planning trajectories.
5 Conclusion
This paper introduces an end-to-end autonomous driving method based on vision-language model (VLM) that transfers long-tail and challenging cases handling capabilities from simulated data to real-world deployment. The method comprises two baseline models and one discriminator. The two baseline models separately incorporate text instructions and sensor data from simulated and real-world datasets, respectively, to output CoC answers and future trajectories. The discriminator employs adversarial learning to enhance the handling of uncommon scenarios and transfers this knowledge to the student VLM model. Moreover, the overall training process accelerates convergence and achieves promising results. Finally, the effectiveness of the proposed approach validated on the nuScenes-ViL VLM dataset.

---

Temporal Instruction: In the previous frame, white car ahead is moving away from the ego vehicle, and the vehicle continues in its current driving state. End-to-end Prompt: What are the objects around the ego vehicle? What the moving status of this object? What the next action of the closest object? Will thread ego vehicle's safety? What safe actions to take for the ego vehicle? Predict the future motion of the ego vehicle.
End-to-end Answer: There is white
front of ego <CAM_FRONT,
vehicle, the coordinate is <CAM_ FRONT, 1009, 486, 1074, 527>. The write car <CAM. FRONT, 1009, 486, 1074, 527> is accelerate moving away. There is traffic light ... There are two people .... In frant ego vehicle, there is no safety threat, ego vehicle should keep going at same speed. The future trajectory (4.7, -0.7), -1.3)....
Cache Temporal Information: There is safety threat, the vehicle maintains its current speed.
Figure 2: Qualitative results. Ego vehicle passing through a traffic light.
Temporal Instruction: From frame, a white car decreasing front ego vehicle, ego vehicle slow down.
End-to-end Answer: There is white front ego vehicle, coordinate <CAM_ FRONT, 551, 409, 909, 739>. The write braking, front of ego vehicle, there is threat in front of ego vehicle, ego vehicle should slow down and proceed with caution. The future trajectory (0.3, -0.0), (0.9, -0.1)... Cache Temporal Information: There is a threat in front of ego vehicle. ego vehicle should slow down and proceed with caution.
S0AM_ROON,, 551, 409, 909, 739>
End-to-end Prompt: What are the objects around ego vehicle? what is the moving status of this object? What the next action of the closest object? will thread vehicle's safety? What are safe actions to take for the ego vehicle? Predict the future motion of ego vehicle.
Figure 3: Qualitative results. A scenario in which a car brakes in front of the ego vehicle.
References
Y. Hu, J. Yang, Chen, K. Li. C. Sima. X. Zhu, S. Chai, S. Du. T. Lin. W. Wang. L. Lu. X. Jia, Q. Liu. Dai, Qiao, and H. Li, PPlanning-orienter autonomous driving." in 2023 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2023, pp. 17 878-17 862.
[2] B. Jiang, S. Chen, Q Xu, B. Liao, J. Chen, H. Zhou, Q Zhang, W. Liu, C. Huang, and X. Wang, "Vad: Vectorized scene representation for efficient autonomous driving." in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8306-8316.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,
A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler,
10

---

J. Wu, C. Winter. C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner. S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models few-shot leameer," 2020. [Online]. Available: https://arxv..ogg//a//22.. 14165
[4] H. Touvron, T. Lavril. G. Izacard, X. Martinet, M.-A. Lachaux. T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. "Llama: Open and efficient foundation language models," 2023. [Online]. Available: httpss//arxiv.orr////222.......1
[5] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K.-Y. K. Wong, Z. Li, and H. Zhao, "Drivegpt4; Interpretable end-to-end autonomous driving via large language model," IEEE Robotics and Automation Letters, 2024.
[6] S. Sural, Naren, and R. Rajkumar, "Contextvlm: Zero-shot and few-shot context understanding for autonomous driving using vision language models," 2024. [Online]. Available: https V/axiv.org/bb///440.00033
[7] S. Yang. J. Liu, R. Zhang, M. Pan, Z. Guo, X. Li, Z. Chen, P. Gao, Y. Guo, and S. Zhang, "Lidar-llm: Exploring the potential of large language models for 3d lidar understanding," 2023. [Online]. Available:
https///arivoogg/bs2311.....
[8] Z. Huang, C. Fen, F. Yan, B. Xiao, Z. Jie, Y. Zhong, X. Liang, and L. Ma, "Drivemm: All-in-one large multimodal model for autonomous driving," arXiv preprint urXiv:2412.07689 2024.
[9] H. Shao, Y. Hu, L. Wang. G. Song. S. L. Waslander. Liu, and H. Li, "Lmdrive: Closed-loop end-to-end driving with large language models," in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. pp. 15 120-15 130.
[10] A. Dosovitskiy, G. Ros, Codevilla, A. Lopez, and Koltun, "CARLA: An open urban driving simulator," in Proceedings of the 1st Annual Conference on Robot Learning, ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR. 13-15 Nov 2017, pp. 1-16. [Online]. Available: https//proccedinngs.oo
[11] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q Yu, and J. Dai, "Bevformer: Learning bird's bird's-eye-Vee s-ee-vveew representation from lidar- camera via spatiotemporal transformers." JEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 3, p. 2020-2036, Dec. 2024. [Online]. Available: https://doi.org/ 10. 209/TPA
[12] T. Ye, W. Jing, C. Hu, S. Huang, L. Gao, F. Li, J. Wang. K. Guo, W. Xiao, W. Mao, H. Zheng, K. Li, J. Chen, and K. Yu, "Fusionad: Multi-modalit modality fusion for prediction and planning tasks of autonomous driving," 2023, *Equal Contribution.
[13] S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q Zhang, C. Huang, W. Liu, and X. Wang, "Vadv2: End-to-end vectorized autonomous driving via probabilistic planning," 2024. [Online]. Available: https:/aaniv.or/////200..2.22
[14] J. Zhai, Z. Feng, J. Du, Y. Mao, J.J. Liu, Z. Tan, Zhang, X. Ye, and J. Wang. "Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes," 2023. [Online]. Available: https://aniiv.rr/ab///000 10430
[15] Z. Li, Z. Yu, S. Lan, Li, J. Kautz, T. Lu, and M. Alvarez, "Is ego status all you need for open-loop end-to-end autonomous driving?" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 14 14
[16] X. Jia, Wu, L.. Chen, Xie, C. He, Yan, and H. L.i, "Think twice hefore driving: Towards scalable decoders for end-to-end autonomous driving," in CVPR, 2023.
[17] H. Shao, Wang. R. Chen, L. Waslander, H. Li, and Y. Liu, "Reasonnet: End-to-end driving with temporal and global reasoning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 723-133333
[18] P. Wu, Jia L. Chen, Yan, Li, and Y. Qiao, "Trajectory- control prediction for end-to-end autonomous driving: A simple yet strong baseline," in NewrlPS, 2022.
[19] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. V. Gool, "End-to-end urban driving by imitating reinforcement learning coach," " 2021. [Online]. Available: https://ariivorr//////00.00
[20] W. Sun, X. Lin. Y. Shi, C. Zhang, H. Wu, and S. Zheng. "Sparsedrive: End-to-end autonomous driving via sparse scene representation," 2024. [Online]. Available: httpss//arxiv.orr/as///// 19620
[21] X. Jia, Y. Gao, L. Chen, J. Yan, P. L. Liu, and H. Li. "Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving," in 2023 IEEE/CVF International Conference Computer Vision (ICCV), 2023, pp. 7919-7929.

---

[22] G. Yenduri, R. C. S. G. S. Y. Srivastava, P. K. R. Maddikunta, D. R. G, R. H. Jhaveri, P. B. W. Wang. A. V. Vasilakos, and T. R. Gadekallu, "Generative pre-trained transformer: comprehensive review on enabling technologies. potential applications. emerging challenges, and future directions," 2023. [Online]. Available: httpss//arxxv.orr/ab////03 10435
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and 1. Sutskever. "Learning transferable visual models from natural language supervision." 2021. [Online]. Available: httpss//ariiv.orraaa///10...0000
[24] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing rision-languse understanding with advanced large language models," arXiv preprint arXiv:2304. 10592, 2023.
[25] S. Jiao, Y. Fang, B. Peng. W. Chen, and B. Veeravalli, "Lavida drive: Vision-text interaction vlm for autonomous driving with token selection, recovery and enhancement," 2025. [Online], Available: https://arxiv.rr//bb//224 2411 12980
[26] X. Tian, J. Gu, B. Li, Y. Liu, Y. Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, "Drivevlm: The convergence of autonomous driving and large ussio-language models," " 2024. [Online]. Available:
https://axiv.orr/ab///4400
[27] W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y. Wen, S. Wu, H. Deng, Z. Li, H. Tian, L. Lu, X. Zhu, X. Wang, Y. Qiao, and J. Dai, "Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving," 2023. [Online]. Available:
[28] B. Jiang, S. Chen, Q. Zhang, W. Liu, and X. Wang, "Alphadrive: Unlcashing the power of vims in autonomous driving via reinforcement learning and reasoning," 2025. [Online]. Available: https://arii..orgaab//555...0.
[29] Y. Ganin and V. Lempitsky. "Unsupervised domain adaptation by backpropagation." in Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ser. ICML'15. JMLR.org. 2015, p. 1180-1189)
[30] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, "Domain separation networks," in Proceedings of the 30th International Conference on Neural Information Processing Systems, ser. NIPS' 16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 343-351.
[31] D. Hendrycks, K. Lee, and M. Mazeika, "Using pre-training can improve model robustness and uncertainty," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov. Eds., vol. 97. PMLR, 09-15 Jun 2019. pp. 2712-2721. [Online], Available: https::/proceedings.. mir. press/1977eeeddyysslll.....
[32] G. Kang. L. Zheng, Yan, and Yang, "Deep adversarial attention alignment for unsupervised domain adaptation: The henefit of target expectation maximization," in Computer Vision - ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XI. Berlin, Heidelherg:
Opringer-Verlag. 2018, p. 420-436. [Online]. Available: http://doi.oorg wttp///..000///000 1007/978 --077788----00000000000 3-030001152----50 6_2
[33] G. Wei, C. Lan, W. Zeng, and Z. Chen, "Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation," in 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. pp. 16 638 648.
[34] J. Yang, H. Zou, Y. Zhou. Z. Zeng. and L. Xie. "Mind the discriminability: Asymmetric adversarial domain adaptation." " in Computer Vision ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 589-606.
[35] Y. Yang, H. Yu, X. Lou, Y. Liu, and C. Choi, "Attribute-based robotic grasping with data-efficient adaptation," " IEEE Transactions on Robotics, vol. 40, pp. 1566-1579, 2024.
[36] I. J. Goodfellow, J. Kouget-Abadice M. Mirza, B. Xu, D. Warde-Farlcy, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," 2014. [Online]. Available:
[37] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image- to image translation with conditional adversarial networks." 2018. [Online]. Available: https://arxxvvor//bb/////// 1.07004
[38] W. Ma, O. Karakus, and P. L. Rosin, "Patch-gan transfer learning with reconstructive models for cloud removal," 2025. [Online]. Available: httpss//ariiv.rr/aas//55...00 205200
[39] X. Li, X. Wang, Z. Qi, H. Cao. Z. Zhang, and A. Xiang, "Dtsgan: Learning dynamic textures via spatiotemporal generative adversarial network," 2024. [Online]. Available: https//arri.oon//b/2221...6
12

---

[40] Z. Xie. J. Cao. Y. Zhang, Q. Zhang, and R. Xu, "A dual-agent adversarial framework for robust generalization in deep reinforcement learning." 2025. [Online]. Available: hugps//axiw.orr//b22550.....
[41] H. Liu, C. Li. Q. Wu, and Y. J. Lee, "Visual instruction tuning." 2023.
[42] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li, "Drivelm: Driving with graph visual question answering." " arXiv preprint arXiv:2312. 14150, 2023.
[43] J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models," in Proceedings of the 40th International Conference on Machine Learning, ser. ICML'23. JMLR.org, 2023.
[44] W. Ljungbergh. A. Tonderski, J. Johnander, H. Caesar, K. Aström, M. Felsberg, and C. Petersson, "Neuroncap: Photorealistic closed-loop safety testing for autonomous driving," " European Conference on Computer Vision (ECCV), 2024.
[45] A. Tonderski, C. Lindström, G. Hess, W. Ljungbergh.. L. Svensson, and C. Petersson, "NeuRAD: Neural rendering for autonomous driving." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 895-14 904.
[46] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," " 2021. [Online]. Available: https::/arxiv..rr/bb////00000 09685
[47] P. Anderson, B. Fernando. M. Johnson, and S. Gould, "Spice: Semantic propositional image caption evaluation." " in ECCV, 2016.
[48] J. Fu, K. Ng. Z. Jiang, and P. Liu, "Gptscore: Evaluate as you desire," 2023. [Online], Available: https:/aaxii.orrraa//222222222
[49] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: method for automatic evaluation of machine translation." " in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2002.
[50] C. Y. Lin, "Rouge: A package for automatic evaluation of summarics," " 2004.
[51] R. Vedantam, C. L.. Zitnick, and D. Parikh, "Cider: Consensus- image description evaluation," in 2015 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2015, pp. 4566-4575.
[52] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: A multimodal dataset for autonomous driving," in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. pp. 11 618 628.
[53] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q Tian, "Centernet: Keypoint triplets for object detection," in 2019 IEEF/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6568-6577
[54] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, "Pointpillars: Fast encoders for object detection from point clouds," in 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019, pp. 12 689 12 697.
[55] Y. Wang, V. Guizilini, T. Zhang. Y. Wang, H. Zhao, and J. Solomon, "Detr3d: 3d object detection from multi-view images via 3d-to-2d queries," 2021. [Online]. Available: https://arxxv..rr//bbb//2 10.06922
[56] J. Philion and S. Fidler, "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d," in Computer Vision - ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J.-M Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 194-210.
[57] Y. Liu. T. Wang, X. Zhang, and J. Sun, "Petr: Position embedding transformation for multi-view 3d object detection," 2022. [Online]. Available: hhtps://arriv.ogg/////222......0
[58] Z. Wang. Ren, and Q Qiu, "Lanenet: Real-time lane detection networks for autonomous driving." 2018. [Online], Available: httpp://aariv.nggad/////8... httpg///a/a/////7722///1
[59] A. Hu, Z. Murez, N. Mohan, S. Dudas, Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall, "Fiery: Future instance prediction in bird's-eye view from surround monocular cameras," in 2021 IEÉE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15 253-15 262.
[60] Q Li. Y. Wang, Y. Wang, and H. Zhao, "Hdmapnet: An online hd map construction and evaluation framework." in 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 4628 4634.
13

---

[61] Y. Liu. T. Yuan, Y. Wang. Y. Wang, and H. Zhao, "Vectormapnet: end-to-end vectorized hd map learning," in Proceedings the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
[62] B. Liao, S. Chen, X. Wang, T. Cheng, Q. Zhang, W. Liu, and C. Huang, "Maptr: Structured modcling and learning for online vectorized hd map construction." 2023. [Online]. Available: hutps//aaxiv.orr/aa//2220
[63] D. Zhang. D. Chen, P. Zhi, Y. Chen, Z. Yuan, C. Li, Sunjing, R. Zhou, and Q. Zhou, "Mapexpert: Online hd map construction with simple and efficient sparse map element expert." 2024. [Online]. Available:
[64] W. Luo, B. Yang, and R. Urtasun, "Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net," 2020. [Online]. Available: https://arxxv.or//bbb/// 12395
[65] S. Casas, W. Luo, and R. Urtasun, "Intentmet: Learning predict intention from raw sensor data," in Proceedings of The 2nd Conference on Robot Learning, ser. Proceedings Machine Learning Research, A. Billard, A. Dragan, J. Peters, and J. Morimoto, Eds., vol. 87. PMLR, 29-31 Oct 2018, pp. 947-956. [Online]. Available: https://rroceedinggs.s mwt.aggsss..srrrrrrrer pttpspp/7/sssaaaaaaa. 88.html
[66] T. Phan-Minhy E. C. Grigore, F. A. Boulton, Beijbom, and M. Wolff, "Covernet: Multimodal behavior prediction using trajectory sets," 2020. [Online]. Available: httpss//aaxiv.org/aa///1111111 10298
[67] Y. Liu, J. Zhang. I.. Fang, Q. Jiang, and B. Zhou, "Multimodal motion prediction with stacked transformers." in 202/ / IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 7573-7582.
[68] Y. Chai, B. Sapp. M. Bansal, and D. Anguelov, "Multipath: Multiple probabilistic anchor trajectory
hypotheses for behavior prediction," in Proceedings of the Conference Robot Learning, ser. Proceedings of Machine Learning Research, L. P. Kaelbling. D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct-01 Nov 2020, pp. 86 99. [Online]. Available: https://proccedings prrpee/c 100cha//
[69] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, "Vectomet: Encoding hd maps and agent dynamics from vectorized representation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[70] Y. Liu, J. Zhang, L. Fang, Q. Jiang, and B. Zhou, "Multimodal motion prediction with stacked transformers." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June 2021. pp. 7577-7586.
[71] J. Ngiam, B. Caine, Vasudevan, Z. Zhang, H.-T. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. Weiss, B. Sapp. Z. Chen, and J. Shlens, "Scene transformer: A unified architecture for predicting multiple agent trajectories," 2022. [Online). Available: https://anxivorr/abb//00../
[72] Y. Hu, Shao, B. Jiang, Chen, S. Chai, Z. Yang. J. Qian, H. Zhou, and Q. Liu, "Hope: Hierarchical spatial-temporal network for occupancy flow prediction," 2022. [Online]. Available: https://arxiv.orggabaae 101188
[73] Gu, Hu, Zhang. X. Chen, Y. Wang, Y. Wang, and H. Zhao, "Vip3d: End-to-end visual trajectory pre- diction via 3d agent queries," in 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 5496-5506.
[74] B. Jiang, S. Chen, X. Wang. Liao, T. Cheng, Chen, H. Zhou, Zhang, W. Liu, and C. Huang. "Perceive, interact, predict: Learning dynamic and static clues for end-to-end motion prediction," " 2022. [Online]. Available: https://axii..rrr//////222220..0
[75] M. Liang, B. Yang, W. Zeng, Y. Chen, R. Hu, S. Casas, and R. Urtasun, "Popnet: End-to-end perception and prediction with tracking in the loop." in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11 110011559
[76] D. A. Pomerleau, "Alvinn: an autonomous land vehicle in a neural network." in Proceedings of the Ist International Conference on Neural Information Processing Systems, ser. NIPS'8S. 88. Cambridge, MA, USA: MIT Press, 1988, p. 305 313.
[77] S. Casas, A. Sadat, and R. Urtasun, "Mp3: A unified model to map, perceive, predict and plan," 2021. [Online]. Available: https://annivoorgga////0....0
14

---

[78] A. Sadat. S. Casas, M. Ren, X. Wu. P. Dhawan, and R. Urtasun, "Perceive, predict, and plan: Safe motion planning through interpretable semantic representations." in Computer Enion-ECCYY 2020 16th European Conference, Glaxgow: 2020, Proceedings. ser. Lecture Notes Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Germany: Springer Science and Business Media Deutschland GmbH, 2020, pp. 414-430, publisher Copyright: @ 2020, Springer Nature Switzerland AG.; 16th European Conference Computer Vision, ECCV 2020 Conference date: 23-08-2020 Through 28-08-2020.
[79] A. Cui. S. Casas, A. Sadat, R. Liao, and R. Urtasun, "Lookout: Diverse multi-future prediction and planning for self-driving." in 2021 IELE/CVF International Conference Computer Vision (ICCV). IEEE, Oct, 2021. [Online]. Available: http:/dx.doi.org 10.1 100/ICCC4499222221.01555
[80] S. Hu, I.. Chen, P. Wu, H. Li, J. Yan, and D. Tao, "St-p3: End-to-end vision-hased autonomous driving taansssspaaill-eemprall feature learning," " in Computer Vision ECCV 2022: /7th European Conference, Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII. Berlin, Heidelberg: Springer-Veree Verlag,
2022. p. 533-549. [Online] Available: https://doi.org/ org////oooror//. 10.
31
[81] Chekroun, Toromanoff. S. Hornauer, and Moutarde, "Gri: General reinforced
imitation and its application to vision-based autonomous driving." 2022. [Online]. Available:
[82] D. Chen, V. Koltun, and Krähenbühl, "Learning to drive from a world on rails," in 2021 IEBE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15557015 579.
[83] Prakash, K. Chitta, Geiger, "Multi-modal fusion transformer for end-to-end autonomous driving." 2021. [Online]. Available: hutps://aasw.orr////211110022220
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and last paragraphs accurately reflect our paper's s contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation is presented in appendix section "E Limitations".
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
15

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical assumptions and proof are presented in subsection "3.4 Dis- criminator" of section "Methods".
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The results of this paper can be reproduced with the code in supplement materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
16

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have attached code in supplement materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https Stw.iiddddeeoeSeemiiisiisieee
//nips.
ssinnoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Due to the page limitation, we only put several important parameters in the paper, in A.3 of AA Experiment Appendix", the whole settings are included in our code.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We put the related information in the section "Experiments" " and details are represented in our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources is identified in subsection A.3 Implementation Details" of "A Experiment Appendix".
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpe////aarrpppcc//p///lilatte
18

---

Answer:
[Yes]
Justification:
We have review the NeurIPS Code Ethics. Our paper meets this requirement.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in Appendix "F Social Impacts".
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
19

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our paper does not have such risks.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsswipppsseeeeeeeeeeeee thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will open-source all components of this paper and ensure that only assets with appropriate usage permissions are cited and utilized.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only use for checking grammar.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should be described.
21

---

A Experiment Appendix
A.1 More Ablation Study
A.1.1 Effectiveness of Projection
also train our module using different projection methods, as shown in Table In this experiment, we replace the original projection module, which consists of several MLP layers, with the Q-former from BLIP-2 to evaluate the effectiveness of our projection design. As shown in table, the MLP-based projection slightly outperforms the Q-former-based projection. To further analyze this phenomenon, we identify two possible reasons: (1) the Q-former contains significantly more parameters than the MLP, making it better suited for large-scale datasets; and (2) the Q-former compresses image features into 256 tokens, which may result in the loss of fine-grained visual information, such object positions, that is crucial for autonomous driving tasks.
Table 4: An ablation study is conducted to evaluate the effectiveness of the projection module on MuScenes-VLM dataset. The study compares MLP-based projection with Q-former-based projection. Notably, the proposed model (LLaVA-7b) is implemented using an MLP-based projection module.
Name
ADE +
Collision Rate (%)
Proposed with Qformer (LLaVA-7b)
1.47
1.60
Proposed (LLaVA-7b)
1.44
1.58
A.1.2 Effectiveness of Temporal Information
As shown in T we conduct experiments the nuScenes-Vii VL. dataset to assess the effectiveness of incorporating temporal information. As described the model section, our method filters previous answers, embeds them, and inputs them into the CoC VLM. In this experiment, temporal information aggregation process. As shown the first row of Table [ this modification results in slight performance drop in both ADE and collision rate.
remove the
Table 5: Ablation study on the effectiveness of temporal information. The experiment uses the nuScenes-VIAN dataset and the simulated dataset.
Name
ADE
Collision Rate (%) |
Proposed without Temporal Information (LLaVA-7b)
1.53
1.64
Proposed (LLaVA-7b)
1.44
1.58
A.1.3 Long-tail Performance
We conducted a more de-tepth analysis to demonstrate our model's ability of handle uncommon cases. Specifically, we split the challenging subset of the nuScenes dataset (comprising 122 scenes) as our test set, while the remaining, easier cases were used for training. Here are challenging scene examples: Scene-0026_55 (29: The ego vehicle is intercepted by construction worker give way to construction truck approaching from the left. Scene-004g 5688, Scene-0094 09044_9994444_ Scene-0131 0131_1153, Scene-0162_I1 1556: Pedestrian intrusions. Scene-0150. 1358: A construction worker blocks the road using traffic cones or water barriers in front of the ego vehicle. Scene-0201 _1998: obstructed by car attempting to park. We trained and evaluated our model using this challenging subset. The performance results are presented below, as shown in Table 6
-200-99011198: The ego vehicle
Table 6: An ablation study of handling uncommon cases.
Name
ADE
Collision Rate (%) J
Our Student VLM model baseline
2.08
2.16
Straight Regressor Baseline
2.89
3.61
Proposed
1.73
1.92
22

---

The results show that our proposed model achieved improvements of 0.35 and 0.24 compared to our student baseline in the new split dataset.
Furthermore, to analyze the performance on easier cases (e.g., straight-line driving), we designed a dummy regressor baseline that always predicts straight trajectory. This baseline helps quantify how much of the evaluation performance on the new nuScenes split can be attributed to genuine methodological improvements. The dummy regressor, termed the Straight Regressor Baseline, is based on our method. To generate straight trajectory, we ignore the y-axis and predict only the x-axis trajectory values (whereas existing methods predict both (x, y) coordinates). We trained this Straight Regressor Baseline, and the results are presented in Table [ This demonstrates that the new split of the nuScenes dataset, which includes challenging scenarios, effectively validates our strategy's s ability to transfer hhallenge-handlng capabilities to real-world models.
A.1.4 Closed-loop Evaluation
We employed NeuroNCAP [4 as the closed-loop simulator, as it supports the nuScenes dataset and provides pretrained rendering model checkpoints, making it well-suited for our method. We cloned the NeuroNCAP 44 and NeoRAD-Studio [45] repositories. replaced the example model (UniAD) with our own pretrained model from the main experiment, and integrated our checkpoints. Additionally, we downloaded the NeuroAD weights, modified the evaluation script accordingly, and conducted the closed-loop evaluation. Since NeuroNCAP offers standardized benchmark and evaluation metrics, and is commonly used by other methods, we followed its recommended configuration. We evaluated our method on the suggested scenarios (e.g., scene-099, scene-0103, etc.). As shown the table below, our proposed method outperforms UniAD, VAD, and our student baseline trained on the nuScenes dataset without adversarial transfer learning.
Table 7: An ablation study of closed-loop.
Model
NeuroNCAP Score avg
Collision Rate (%) avg
UniAD
1.84
68.70
VAD
2.75
50.70
Our Student VLM model baseline
3.07
48.83
Proposed (LLaVA-7b)
3.32
45.26
A.1.5 Datasets
During pre-training, the teacher and student VLMM trained separately using simulated and real- world datasets, respectively. The simulated dataset, referred to as CAMLA-VIM, is collected using the CARLA Leaderboard v2 simulator and comprises 61.6% normal scenes and 38.4 38.4% challenging scenarios (e.g., traffic jams, near-miss vehicle interactions, and pedestrian intrusions). The real-world dataset used prepared from the publicly available nuScenes dataset. To accelerate training, we initialize both models with checkpoints from LLaVA-v1.5 [40]. Each model is fine-tuned using image frames, temporal instructions, and end-to-end prompts, enabling the Chain-offCCussllee (CoC) VLMs to align instruction, visual, and temporal information effectively. Efficient training is further facilitated by sampling frames at fixed intervals and applying random temporal shifts as augmentation. We use a real-world dataset (nuScenes- (usScenes-VLM) and a simulated dataset (CARLA-VIL. to pre-train the student and teacher VLM models, respectively.
nuScenes- VLM dataset: The student VLM is pre-trained on the nuScenes- VLM dataset, which is derived from the publicly available nuScenes dataset and enriched with textual prompts. We follow the official train/validation split provided by nuScenes.
CARLA- LL- dataset: We use this dataset as our simulation dataset to pre-train teacher VLM model. This dataset is configured with settings similar to nuScenes and annotated with textual prompts consistent with those in the nuScenes-VLM dataset. To enhance dataset transparency, we provide the statistical distribution of scenarios within the CARLA-VIL dataset. A new supplementary table presents detailed counts for each scenario category. We believe these additions improve the clarity of our dataset description and enable more rigorous evaluation of the model's 's performance, particularly in challenging driving scenarios.
We have provided the statistical distribution of the simulated challenging scenarios to clearly charac- terize the dataset. A new supplementary table presents detailed counts for each scenario category. We
23

---

hope these additions enhance the transparency of our dataset and enable a more rigorous assessment of the model's performance in handling rare but critical driving events, shown in Table
Table 8: CARLA Scenarios Statistical Distribution.
Scenarios Category
Proportion(
Clips Count(total 600)
Normal (Straight, Left, Right Turn)
61.6
370
Pedestrian Intrusion
4.7
28
Fog
6.7
40
Rain
7.8
47
NearMiis Vehicle Interactions
2.7
16
Traffic Jam
4.5
27
Traffic Accident
1.8
11
Vehicle Cut In
2.2
13
Opposite Vchicle Intrusion
0.8
5
Vehicle U-Turning
0.5
Construction Obstacle
4.0
24
Bicycle Intrusion
0.5
3
Lane Merge
1.2
7
No Traffic Light Intersection
0.7
4
Turn Left and Merge In
0.3
2
A.1.6 N frames Fusion
We have conducted an experiment using two cached frames, with the results presented in Table The findings indicate that, compared using one cached frame, there is no significant improvement in ADE or Collision Rate. Therefore, we have chose to use single cached frame to reduce the number of input tokens.
Table 9: Ablation study on frame temporal information.
Model
ADE 4
Collision Rate (%)
With 1 Frame
1.44
1.58
With Frames
1.47
1.56
A.2 Implementation Details
We conduct experiments using nuScenes dataset and simulator dataset collected from CARLA. Our approach adopts the CoC VLM architecture as the baseline for both the teacher and student VLM models, which are pre-traine pre-trained on the simulation and real-world datasets, respectively. The LLM Brain component is initialized using the LLaVA pre-trained model [4]) Given the complexity of the training process and the associated convergence challenges, To prevent convergence to suboptimal local minima, we utilize a large batch size during training. Furthermore, different optimization algorithms are applied to the various modules improve training performance. All experiments are conducted on eight NVIDIA H100 GPUs. To reduce computational resource requirements. fine-tune our model using the LoRA method 446
adopt multi-step training strategy.
Before adversarial training, we load the pre-trained models for both the teacher and student VLM models. The adversarial training process consists of multiple steps, incorporating our unique back- propagation strategies.
Step 1:
Forward- Propaaatttooa We freeze both the teacher and student VLM models and train only the discriminator. Simulation and real-world data are fed into the two VLMs, respectively, and the

---

resulting features are passed to the discriminator to outputs the logits. Backward Propaardtroapaoioonn Using the discriminator's output logits, we compute the loss and perform prok-propagation. Since only the discriminator is being trained at this stage, we update only its parameters while keeping both VLM models frozen.
Step 2:
Norward-Propagatin In this process, only the student VLM and the discriminator are involved. Both real-world and simulation data are input into the student VLM, and the resulting features are passed to the discriminator to produce logits.
Backward- Propaaprtroa This step introduces our novel contribution. We first back-propagate the discriminator loss through the discriminator without updating its parameters. Then, the propagated gradients, along with the VLM-specific losses, are used to update the student VLM model. This strategy reduces convergence instability and accelerates the adversarial training process.
A.3 Adversarial Training Loss
Unlike existing methods, our approach incorporates additional distinct loss functions. During Step 1 of adversarial training, the discriminator loss is incorporated. This enables the discriminator to distinguish between the two data distributions, defined as follows:
stepp lg
(12)
In Step 2, both the discriminator loss and the VLM autonomous driving loss are utilized, as shown below:
Latep2 LVLM + la
(13)
However, as described in the Model Training section, our discriminator only passes the gradient parameters, and does not update their model parameters.
A.4 Evaluation Metrics
To evaluate the proposed method, we employ two categories of metrics: language evaluation metrics and planning evaluation metrics.
A.4.1 Language Evaluation Metrics
In our experiments, several standard metrics are used to assess language performance. SPICE. This is prevailing metric used in VQA and image captioning, to evaluate the structure similarity of predicted texts with ground truth while ignoring the semantic meanings [47]. In detail, it parses the text into a syntactic dependency tree using probabilistic context-free grammar, then maps the dependency tree into a scene graph in a rule-based manner. The scene graph describes the objects, attributes, and their relationship in the original text, and the SPICE score is computed as the F-score of the generated scene graphs from prediction and ground truth [47].
GPT Score. We employ GPT Score (48) to measure the semantic alignment of answers and comple- ment the SPICE metric. Specifically, the question, the ground truth answer, the predicted answer, and prompt asking for numerical score of the answer. GPT Score is a metric provided by ChatGPT. Traditional metrics mainly assess word-level performance and may not capture seman- tic nuances, potentially yielding unexpected evaluation outcomes. Leveraging ChatGPT's robust reasoning capabilities, we employ it to gauge prediction quality and derive a more rational score [48). BLEU. Bilingual Evaluation Understudy (BLEU) is used to measure the n-grams between prediction and ground truth, and is sensitive to the word order. The n ranges from I to in our experiment. With higher precision indicating a better match, The BLEU score is between 0 and where represents perfect match and 0 represents the opposite [49].
ROUGE L. Recall-Oriented Understudy for Gisting Evaluation- Iesiiuatiion-onngee Longest Common Subsequence (ROUGE L) calculates the precision and recall with the longest common sub-sequence, which utilizes
the B-grams policy similar to BLEU, but mainly based on recall [50].
CIDEr. Consensus--asee Image Description Evaluation (CIDEr) encodes the frequency of n-grams appearing in the text, calculates the weight of each n-gram through TF-IDF represents the sentence in vector form using n-grams, and then calculates the cosine distance of the TF-IDF vector between the two text to measure their similarity (51). Furthermore, we also use Accuracy and Match as suggested by to evaluate our method.
A.4.2 Planning Evaluation Metrics
ADE. Average Displacement Error (ADE) is used to measure the performance of the planning results, indicates the average 1.2 distance between the labeled ground truth trajectories and predicted trajectories.
25

---

Collision Rate.
This metric is used to compute the ratio of evaluation frames that collides with objects over all evaluation frames. Notably, these metrics follow the VAD 2 settings, it will consider the error/collision rate as an average over 0.5, 1.5, 2, 2.5, seconds, another word, it uses average over average strategy.
B
Data Generation
B.1 Data Collection
In our approach, the nuScenes dataset [52 is used the real-world dataset. This dataset is collected using multi-view cameras and LiDAR, with annotations provided for each key frame. A simulation dataset is also collected using the CARLA Leaderboard v2 simulator [1O]. The same configuration as nuScenes is adopted, including multi-view cameras, labels, and HD maps. The data are segmented into clips, consistent with the nuScenes format. In contrast, numerous challenging scenarios are simulated, such as accidents, traffic violations, and pedestrian trespassing. These scenarios are designed to enhance the generalization capabilities of the student VLM model.
B.2 CoC Answer Generation
VLM-based autonomous driving methods require textual prompts paired with corresponding answers. Our method takes data clips input, following the approach suggested by [42], with several enhancements. We augment the data with specific object deseriptions around the ego vehicle, including orientation (azimuth), pixel bounding box coordinates (2D coordinates of the object in the camera view), and dimensions (height and width of the bounding box, which serve as distance cues). We refine the end-to-end question/answer templates using two main strategies: employing classification-basee multiple-choice questions for improved stability, and structuring logical templates to coherently link tasks. These data are structured using saain-of-causaliiy reasoning extract end- to-end driving logic. Notably, both the real-world and simulation datasets are generated using the same prompt strategy and follow consistent CoC answer format. These datasets referred to as nuScenes- VLM and CARLA-VIM, respectively, in our paper.
C Motivations
Recently, LLM- autonomous driving methods have employed two distinct types of datasets. One type consists real-world datasets, such as nuScenes [52], while the other includes simulated datasets, such those generated using the CARLA simulator [1O. Methods trained real-world datasets suggest that models should applied real driving scenarios; hence, models trained on real data considered optimal. However, these methods face several challenges. First, collecting and labeling real-world data is expensive. Second, world data lacks rare and challenging cases, such pedestrian trespassing, traffic violations, and accidents. Finally, real-world data often lacks complex interactions and decision-makinn scenarios.
To address these limitations, other researchers have proposed models trained on simulated data. Simulations can be freely deployed with arbitrarily dynamic or static environments and can reproduce scenarios that are difficult or dangerous to capture in the real world, such reckless driving, traffic collisions, and extreme lighting conditions. Moreover, simulations facilitate the collection of large- scale labeled datasets. However, significant domain gap exists between simulations and real-world driving, limiting the applicability of simulation-traite trained models to real vehicles. Therefore, transfers strong handling capabilities from simulation to real-world deployment has become increasingly important.
Furthermore, incorporate temporal information from previous frames, existing methods typically input sequences images into LLM models (9. This strategy significantly increases token length and computational resouree consumption. Therefore, it is essential to develop simple and efficient strategy for temporal aggregation.
D Why CoC
These Chain-of- Sicin-of-Thoure Thought approaches often compromise the integrity of reasoning, resulting in frag- mented and incoherent decision-maie making chains. In contrast, our Chain-of Chain-of-Causalitt Causality (CoC) method outputs entire CoC answer (End2End answer) that preserves internal causal linkages across the entire process. Specifically, follows the sequence: T cached information + perception - prediction + planning - T+1 cached information. For example, if the ego vehicle was turning left in the previous frame, the model should detect an oncoming vehicle the opposite lane, predict trajectory, and generate appropriate plan accordingly.
26

---

E
More Related Works
E.1 Traditional Autonomous Driving
Traditional autonomous driving methods are typically composed of four subtasks: detection, online mapping, prediction, and planning. Researchers have extensively studied each of these components, contributing a wide range of solutions tailored to individual challenges in the driving pipeline.
Detection. CenterNet [53] introduces two specialized modules: Cascade Corner Pooling and Center Pooling, improve object detection performance. Differently, PointPillars [54] employs LiDAR point clouds to predict 3D bounding boxes. DETR3D [55], a DETR- -ETT--nspiiedd approach, utilizes 3D queries to extract image features and directly predict bounding boxes without requiring non- maximum suppression. Additionally, LSS [56] pioneers the use of depth prediction to construct a S-rd's-Eyee View (BEV) feature map for detection tasks. Similarly, PETR [57] enhances feature initialization by incorporating 3D positional encodings into image features, which are then processed via detection queries using a transformer-basee mechanism. BEVFormer [ further advances BEV- based detection through spatio-temporal transformers, significantly improving detection accuracy for autonomous driving.
Online Mapping. For constructing detailed road topology, researchers focus on accurately identifying lane locations and geometries. LaneNet [58] decomposes the lane detection task into edge proposal and line localization to mitigate confusion with visually similar objects. FIERY [59] performs dense segmentation on the BEV feature map to predict lane features. Furthermore, HDMapNet [60] transitions from dense segmentation to sparse map representations. Different from HDMapNet, VectorMapNet [6]] directly predicts polylines in the BEV space, removing the need for heuristic post-processing. MRpTR [62] introduces query-baser representations for constructing lane topology. Inspired by these methods, MapExpert [63] further refines this approach by distributing specialized experts to handle heterogeneous map elements with varying geometric characteristics.
Prediction. Conventional prediction models use historical trajectories to forecast future movements. Early methods, such as FaF and IntentNet [64] 65]. use neural networks for motion prediction. CoverNet and related works [66] [67] highlight the significance of dynamic behavior modeling. In addition, MultiPath (68) combines visual features extracted from cameras with convolutional neural networks to predict motion in BEV space. Different from methods above, VectorNet [69 introduces sparse representations for trajectory prediction. Some researchers also adopt sparse representations to forecast vectorized trajectories using transformer baaddd architectures [70] 71. In contrast, some models adopt dense representations predict occupancy and motion flow [59 72 Other methods, such as VIP3D and PIP (73 (74) integrate interactions with dynamic agents and static map elements to boost predictive performance. PnPNet [75] adds tracking strategy that derives trajectory estimates from detection results. Recently, unified frameworks such as UniAD, VAD, and SparseDrive 2 20 jointly perform perception and prediction within single model.
Planning. Planning represents the final stage in autonomous driving, where systems generate executable trajectories. ALVINN (76) is among the earliest ceural-network-basee planning models. More recent approaches integrate perception outputs to refine trajectory accuracy [77] 78], Others, such LookOut and ST-P3 learning--asee based methods introduce teacher VLM models to guide planning [8]] PlanT [3 utilizes
79 80), incorporate rule-based optimizations. Notably, reinforcement
standard transformers to extract object representations for planning, while VAD2 [3 models planning actions as probabilistic distributions, thereby enhancing local planning precision and achieving strong closed-loop performance on benchmarks.
F Limitations
As -LLa is chosen as the baseline, inference computational cost and speed present challenges for deployment in online systems, limiting potential applications. Furthermore, maintaining two baseline models incurs high training costs. The complexity of the training process also poses optimization challenges, particularly during the initial epochs. Since the objective of our method is to transfer simulation capabilities to real-world performance, evaluation is conducted on real-world datasets. Consequently, closed-loop evaluation metries are not assessed. 45
G
Social Impacts
The development of our Visual Language Model (VLM) methods holds significant potential to enhance transportation systems by improving road safety and mitigating traffic congestion. These methods exhibit robustness and strong generalization capabilities across diverse driving environments.
27

---

Moreover, their interpretability could facilitate the transition of autonomous driving systems from black-box white-box models, thereby accelerating deployment. However, because autonomous driving is closely tied to human safety, the current limitations in the safety and trustworthiness of our methods raise concerns about their widespread adoption. There is still a long way to go.
28


Reviews:
[{"review_id": "QDlLh5jYWo", "reviewer": "Reviewer_hDQ3", "summary": "This paper proposes a novel end-to-end autonomous driving framework that aims to transfer long-tail handling capabilities learned in simulation to real-world applications via adversarial domain transfer. The authors introduce an architecture comprising a teacher VLM, a student VLM, and a discriminator. A central component of this architecture is the proposed \"Chain-of-Causality Visual Language Model\" (CoC VLM), designed to integrate temporal information and facilitate causal reasoning for decision-making.", "questions": "Regarding the CoC VLM architecture described on lines 179-182, could you please clarify the following:\n\n1.  The 'filter' is described as a component to simplify LLM-generated responses and reduce token length. The paper later states that this filter is designed to \"summarize and cache the end-to-end answer, converting it into a concise instruction for the subsequent frame\". What is the specific mechanism or model used for this filter? Is it a rule-based system that extracts keywords, or another smaller language model?\n\n2.  The 'dedicated end-to-end answer generation module' is mentioned as a way to streamline output formatting. Does this refer to a separate trainable module, or does it describe a specific structured prompting and decoding process?", "limitations": "Please see the weakness section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n\nThe core idea of the paper is interesting and highly relevant. Leveraging simulation data to supplement the scarcity of rare and challenging cases in real-world datasets is a promising direction for developing more robust autonomous driving systems. The proposed adversarial transfer framework to bridge the sim-to-real gap for complex behaviors is reasonable.\n\n[Weaknesses]\n\nMy major concern involves the experimental part of the manuscript. The evaluation methodology does not sufficiently support the central claims of the paper.\n\n1.  The evaluation relies on macro-level metrics that may not validate the claimed improvement in handling long-tail scenarios. The paper reports the Average Displacement Error (ADE) and Collision Rate over the entire nuScenes-VLM test set. These overall metrics are likely dominated by a large number of simple, common scenarios (e.g., straight-line driving), which can dilute the performance results and mask the model's actual capabilities in critical, long-tail situations. To substantiate the claim of improved handling of uncommon cases, the authors must provide a more fine-grained analysis. For instance, they should isolate a subset of test clips containing specific long-tail events (e.g., 'pedestrian trespassing' or 'traffic violations' as mentioned in the simulation design ) and report performance metrics (ADE, Collision Rate) specifically on this challenging subset. Without this, the claimed improvements are not convincingly demonstrated.\n\n2.  A significant limitation is the lack of closed-loop evaluation. The paper's evaluation is conducted in an open-loop setting, which assesses the model's ability to predict a trajectory but does not account for how the ego-vehicle's actions influence the environment's subsequent states. Closed-loop evaluation, where the model's output is fed back into the simulation to create a continuous interaction loop, is a much stronger and more realistic test of an autonomous driving system's performance and safety. The authors acknowledge this in the appendix, stating that 'closed-loop evaluation metrics are not assessed', but this remains a major weakness that impacts the perceived real-world viability of the proposed method.\n\n3.  The paper lacks transparency regarding the statistical distribution of the simulated long-tail scenarios. While the supplementary material mentions that the simulation data is collected from CARLA Leaderboard v2's 39 challenging scenarios and aims for a 'balanced representation' to address the long-tail challenge, neither the main paper nor the supplement provides quantitative statistics on the final distribution of these cases. It is unclear what specific cases are included and at what frequency. This lack of transparency makes it difficult to assess the diversity of the training data and to fairly interpret the performance improvements. The effectiveness of the domain transfer heavily relies on the quality and distribution of these simulated events.", "was_revised": true, "final_justification": "Please see my comment below.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. Your insightful comments have significantly improved the quality of our research. Below, we address your concerns point by point.\n\nWeaknesses 1:\n\n\nResponse: Thank you for your thoughtful suggestion. We conducted a more in-depth analysis to demonstrate our model’s ability of handle uncommon cases. Specifically, we split the challenging subset of the nuScenes dataset (comprising 122 scenes) as our test set, while the remaining, easier cases were used for training. Here are challenging scene examples:\nScene-0026_379: The ego vehicle is intercepted by a construction worker to give way to a construction truck approaching from the left.\nScene-0046_568, Scene-0094_948, Scene-0131_1153, Scene-0162_1556: Pedestrian intrusions.\nScene-0150_1358: A construction worker blocks the road using traffic cones or water barriers in front of the ego vehicle.\nScene-0201_1978: The ego vehicle is obstructed by a car attempting to park.\nWe trained and evaluated our model using this challenging subset. The performance results are presented below.\n| Model               | ADE | Collision Rate        |\n| ----------- | -------- | ----- |\n| Our Student VLM model baseline | 2.08       | 2.16           |\n| Proposed (LLaVA-7b)  | 1.73         | 1.92 |\nThe results show that our proposed model achieved improvements of 0.35 and 0.24 compared to our student baseline in the new split dataset.\n\nFurthermore, to analyze the performance on easier cases (e.g., straight-line driving), we designed a dummy regressor baseline that always predicts a straight trajectory. This baseline helps quantify how much of the evaluation performance on the new nuScenes split can be attributed to genuine methodological improvements.\nThe dummy regressor, termed the Straight Regressor Baseline, is based on our method. To generate a straight trajectory, we ignore the y-axis and predict only the x-axis trajectory values (whereas existing methods predict both (x, y) coordinates). We trained this Straight Regressor Baseline, and the results are presented below.\n| Method            | ADE | Collision Rate        |\n| ------------------ | --------------- | --------------- |\n| Straight Regressor Baseline   | 2.89    | 3.61    | \nThis demonstrates that the new split of the nuScenes dataset, which includes challenging scenarios, effectively validates our strategy's ability to transfer challenge-handling capabilities to real-world models.\n\n\n\n\nWeaknesses 2:\n\n\nResponse: We sincerely thank you for your suggestion. We employed NeuroNCAP [1] as the closed-loop simulator, as it supports the nuScenes dataset and provides pretrained rendering model checkpoints, making it well-suited for our method. We cloned the NeuroNCAP [1] and NeoRAD-Studio [2] repositories, replaced the example model (UniAD) with our own pretrained model from the main experiment, and integrated our checkpoints. Additionally, we downloaded the NeuroAD weights, modified the evaluation script accordingly, and conducted the closed-loop evaluation.\nSince NeuroNCAP offers a standardized benchmark and evaluation metrics, and is commonly used by other methods, we followed its recommended configuration. We evaluated our method on the suggested scenarios (e.g., scene-099, scene-0103, etc.). As shown in the table below, our proposed method outperforms UniAD, VAD, and our student baseline trained on the nuScenes dataset without adversarial transfer learning.\n\n| Model               | NeuroNCAP Score avg | Collision Rate avg  |\n| ------------------ | --------------- | ------------------ |\n| UniAD         | 1.84         | 68.70          |\n| VAD | 2.75       | 50.70           |\n| Our Student VLM model baseline | 3.07       | 48.83           |\n| Proposed (LLaVA-7b)  | 3.32         | 45.26 |\n\nWeaknesses 3:\n\nResponse: We have provided the statistical distribution of the simulated challenging scenarios to clearly characterize the dataset. A new supplementary table presents detailed counts for each scenario category. We hope these additions enhance the transparency of our dataset and enable a more rigorous assessment of the model's performance in handling rare but critical driving events.\n| Scenarios Category | Proportion(\\%) |Clips Count(total 600)|\n|--|--|--|\n|Normal (Straight, Left, Right Turn) |61.6|370|\n|Pedestrian Intrusion | 4.7| 28|\n|Fog| 6.7| 40|\n|Rain|7.8| 47|\n|Near-Miss Vehicle Interactions|2.7| 16|\n|Traffic Jam |4.5|27|\n|Traffic Accident|1.8|11|\n|Vehicle Cut In|2.2|13|\n|Opposite Vehicle Intrusion|0.8| 5|\n|Vehicle U-Turning |0.5|3| \n|Construction Obstacle |4.0| 24|\n|Bicycle Intrusion |0.5| 3|\n|Lane Merge| 1.2|7|\n|No Traffic Light Intersection|0.7|4|\n|Turn Left and Merge In| 0.3| 2|\nTable Supp. CARLA Scenarios Statistical Distribution\n\nQ1:\n\nWe apologize for the unclear explanation. The filter referenced here is a rule-based system that removes the reasoning component of the End-to-End Answer and extracts only the final action instruction for the ego vehicle—for example, \"There is no safety threat, the vehicle maintains its current speed.\" as described on line 230. This filtering process reduces the input token size while preserving the most critical information from the previous frame.\n\nQ2:\n\n\nThe \"dedicated end-to-end answer generation module\" refers to our CoC VLM model. Unlike existing approaches that typically rely on multiple rounds of dialogue (they firstly generating a perception answer, then feeding it into the model to produce a prediction answer, and finally combining both to generate a planning answer), our model generates the end-to-end answer directly. This design simplifies the pipeline and streamlines output generation.\n\n[1] W. Ljungbergh, A. Tonderski, J. Johnander, H. Caesar, K. Åström, M. Felsberg, and C. Petersson, “Neuroncap: Photorealistic closed-loop safety testing for autonomous driving,” European Conference on Computer Vision (ECCV), 2024.\n[2] A. Tonderski, C. Lindström, G. Hess, W. Ljungbergh, L. Svensson, and C. Petersson, “NeuRAD: Neural rendering for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 895–14 904.598"}, {"review_id": "IiOvnreqwN", "reviewer": "Reviewer_orn8", "summary": "This paper proposes a Chain-of-Causality (CoC)-based transfer learning framework for autonomous driving, addressing the limitations of traditional methods that rely solely on either real-world or simulated data. By independently training two models and leveraging a teacher model (trained in complex simulated environments) to guide a real-world student model, the framework effectively utilizes simulated data. Additionally, the paper introduces Chain-of-Causality, which aggregates temporal information end-to-end via a text interface. Experimental results demonstrate that this method successfully achieves Sim2Real transfer in autonomous driving.", "questions": "The term \"End2End answer\" is quite ambiguous. While it seems to encompass textual descriptions of perception, prediction, and planning, is \"end-to-end\" the appropriate term? Would \"CoC answer\" be more accurate?\n\nHave experiments been conducted using the previous N frames' \"End2End answer\" as cached temporal information?\n\nThe paper does not clearly differentiate between CoT (Chain-of-Thought) and CoC (Chain-of-Causality). In Omnidrive, this process is referred to as CoT, whereas the authors label it CoC. Clarifying the boundaries between these definitions would be helpful.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper presents a novel perspective, and investigating the Sim2Real problem in autonomous driving holds significant research value.\n\nThe experiments are thorough, the analysis is well-reasoned, and the study offers strong analytical merit.\n\nWeaknesses:\n\nFigure 1 contains excessive lines, making the description overly complex and hindering readability.\n\nThe spacing between paragraphs appears inconsistent, and the excessive blank space before the \"Contribution\" section in the Introduction seems unnecessary.\n\nThe structure of Chapter 3 is somewhat unusual. Sections 3.2 and 3.3 could be condensed under \"Methods\" rather than being separate subsections, and \"Model Training\" might be better placed in Chapter 4 instead of Chapter 3.\n\nThe formatting of tables is inconsistent. For example, Table 1 lists each metric in separate columns, whereas Tables 2 and 3 do not.", "was_revised": true, "final_justification": "BA", "rebuttal": "Thank you for the great efforts and valuable comments. We have carefully addressed the main concerns in detail. We hope you might find the response satisfactory.\n\nWeakness 1\n\nResponse: Thank you for pointing this out. Specifically, we have revised Figure 1 by merging the flow lines to make the illustration more concise and clear. Additionally, we have moved the complex forward and backward propagation expression to the appendix (now shown as Figure 4 in our manuscript) and provided the corresponding explanations below:\nBefore adversarial training, we load the pre-trained models for both the teacher and student VLM models. The adversarial training process consists of multiple steps, incorporating our unique back-propagation strategies.\n\nStep 1:\n\nForward-Propagation:\nWe freeze both the teacher and student VLM models and train only the discriminator. Simulation and real-world data are fed into the two VLMs, respectively, and the resulting features are passed to the discriminator to outputs the logits.\nBackward-Propagation:\nUsing the discriminator's output logits, we compute the loss and perform back-propagation. Since only the discriminator is being trained at this stage, we update only its parameters while keeping both VLM models frozen.\n\nStep 2: \n\nForward-Propagation:\nIn this process, only the student VLM and the discriminator are involved. Both real-world and simulation data are input into the student VLM, and the resulting features are passed to the discriminator to produce logits.\n\nBackward-Propagation:\nThis step introduces our novel contribution. We first back-propagate the discriminator loss through the discriminator without updating its parameters. Then, the propagated gradients, along with the VLM-specific losses, are used to update the student VLM model. This strategy reduces convergence instability and accelerates the adversarial training process.\n\nWeakness 2\n\nResponse: We have removed the excessive blank space in the Introduction section, particularly before and after Figure 1.\n\nWeakness 3\n\nWe have condensed the Sections 3.2 and 3.3 into the first paragraph of Sections 3. The modified content is shown below:\n\nAs illustrated in Fig. 1, the proposed architecture consists of three components: a Teacher Visual Language Model (Teacher VLM), a Student Visual Language Model (Student VLM), and a Visual Language Model Discriminator. Both the Teacher and Student VLMs share a common base architecture, referred to as the Chain-of-Causality Visual Language Model (CoC VLM), which processes multi-view image pairs, end-to-end prompts, and historical instructions from previous frames to generate end-to-end outputs. The Teacher VLM is trained on simulated data to address diverse and rare scenarios, such as pedestrian trespassing, driving violations, and traffic accidents. In contrast, the Student VLM is trained on real-world data and serves as the final inference model, enabling the transfer of knowledge from simulated to real-world contexts.\n\nAdditionally, we have relocated Subsection 3.5, Model Training, to the beginning of Section 4 as its first subsection. Following this reorganization, the overall structure is significantly clearer and more readable.\n\nWeakness 4\n\nWe apologize for the oversight. We have revised the formatting inconsistencies in the tables by adding separate columns to Tables 2, 3, and 4, consistent with the format used in Table 1.\n\nQ1:\n\nThank you for your suggestion. We agree that \"CoC answer\" is more appropriate than \"End2End answer.\" Accordingly, we have revised the manuscript to replace all instances of “End2End answer” with “CoC answer.”\n\n\nQ2:\n\nWe have conducted an experiment using two cached frames, with the results presented in Table 6. The findings indicate that, compared to using one cached frame, there is no significant improvement in ADE or Collision Rate. Therefore, we have chose to use a single cached frame to reduce the number of input tokens.\nTable 6. Ablation study on N frame temporal information.\n| Model               | ADE | Collision Rate        |\n| ------------------ | --------------- | ------------------ |\n| With 1 Frame | 1.44       | 1.58    |\n| With 2 Frame  | 1.47         | 1.56 |\n\nQ3：\n\n\nExisting Chain-of-Thought (CoT) autonomous driving methods typically rely on multiple rounds of dialogue, with each round performing a single-step reasoning process. For example, OmniDrive employs a sequence of prompts such as:\n1. What traffic elements should I be aware of while driving in this area? \n2. If I decide to accelerate and make a left turn, what could be the consequences? \n3. What should be my next action given the current driving situation, and why?\n\nThese Chain-of-Thought approaches often compromise the integrity of reasoning, resulting in fragmented and incoherent decision-making chains. In contrast, our Chain-of-Causality (CoC) method outputs entire CoC answer (End2End answer) that preserves internal causal linkages across the entire process. Specifically, it follows the sequence: T cached information → perception → prediction → planning → T+1 cached information. For example, if the ego vehicle was turning left in the previous frame, the model should detect an oncoming vehicle in the opposite lane, predict its trajectory, and generate an appropriate plan accordingly."}, {"review_id": "YLZnuxdFsV", "reviewer": "Reviewer_a2Yp", "summary": "This paper introduces an end-to-end autonomous driving framework that uses a vision-language model (VLM) to bridge the domain gap between simulation and real-world scenarios. It employs a teacher VLM (trained on simulated “long-tail” driving events) and a student VLM (trained on real data), both using a shared Chain-of-Causality VLM backbone. A novel transformer-based discriminator is trained adversarially to align the student’s features with the teacher’s, effectively transferring rare-case handling skills from simulation to the real domain. The Chain-of-Causality architecture extends a pre-trained multimodal model (LLaVA-1.5) with a temporal text adapter that feeds the previous time-step’s summarized output into the language model, enabling chain-of-thought reasoning for explainable decision making. On the nuScenes-VLM benchmark, the proposed approach significantly outperforms prior LLM-based driving methods (e.g. DriveLM) on both language understanding and planning metrics, demonstrating effective sim-to-real transfer and improved interpretability.", "questions": "**1**, In line 187, what is the scheme for choosing LLM outputs to encode temporal consistency? \\\n**2**, How to understand the data domain denoted as $X^S$ and $X^R$? Are they sensor data and image data from simulation and real-world? Then what does $y$ represent here?\\ \n**3**, How is the distribution probability calculated in line 251? Do we explicitly know distribution $p^s$ and $p^r$?\\\n**4**, How does the convex function $\\delta$ in line 280 involves in the distance/f-divergence calculation in equation 7? Is $\\delta$ related with function $f$?\\", "limitations": "See above. The reviewer think this paper is interesting and novel with comprehensive experiments setting, but authors need to improve on the writing clarity: some of the technical details are missing but crucial for understanding. The reviewer will raise the score if concerns are properly addressed.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "##Strengths:\n\n**1**, The use of adversarial techniques for domain adaptation is well-established; however, combining this with VLMs to facilitate explainable decision-making in autonomous vehicles is a novel contribution.\\\n**2**, The proposed Chain-of-Causality VLM introduces a mechanism to model causal relationships in driving scenarios, which is a unique approach to improving interpretability in autonomous systems.\\\n**3**, Figure 1 is concise and straight-forward, which elaborates the structures and data flow.\\\n**4**, The experiments are solid and detailed, building upon a comprehensive set of metrics.\n \n\n##Weakness:\n\n**1**, Line 105, citation need to be fixed. \\\n**2**, From line 191 to 196, the example is straight forward, but still a bit vague to understand what kind of content are needed in temporal instruction.\\\n**3**, The clarity of section 3.4 needs to be improved.  A list of questions can be found in below Question part.\\\n**4**, In section 3.5.1, it would be better to state data used for pre-training of both student and teacher VLM.\\\n**5**, As a stand alone contribution stated in line 90-91, this back-propagation strategy have not been found in section 3 for elaboration.\\", "was_revised": true, "final_justification": "Authors have answered all questions raised and concerns are properly addressed with more intuitive explanations and experiments. The training method prosed by the authors are interesting and seemingly novel for the Autonomous Viechle community. As long as authors improved their writing particularly the method part in their final version, I recommend boarder-line accept.", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have carefully addressed your concerns in detail. We hope you might find the response satisfactory.\n\nWeakness 1:\n\nThank you for pointing out this issue. We have revised the manuscript and corrected the citation for FusionAD accordingly.\n\nWeakness 2:\n\nWe apologize for the unclear expression. The temporal instruction is primarily composed of three parts:\n1.A fixed prompt: “**In the previous frame,**”\n2.The main threat detected in the last frame, such as “**a white car ahead is moving away from the ego vehicle,**” or “**a car ahead is slowing down.**” If no threat is detected, the default is “**there is no safety threat.**”\n3.The action taken by the ego vehicle, such as “**the vehicle continues in its current driving state,**” “**the vehicle is turning left,**” “**the vehicle is turning right,**” or “**the vehicle is slowing down.**”\nThis temporal instruction clearly conveys the ego vehicle’s previous action, which is crucial for determining the next appropriate action, as prior actions often influence current decision-making.\n\nWeakness 3:\n\nWe have revised Section 3.4 of the manuscript to improve its readability. The specific concerns have been addressed in detail in the question part (please refer to  Q1, Q2, Q3, and Q4). We hope that our responses to Q1, Q2, Q3, and Q4 could address your concerns.\n\nWeakness 4:\n\nWe appreciate the reviewer’s insightful suggestion. In response, we have added a detailed explanation of the datasets used for pre-training both the student and teacher VLMs. Section 3.5.1 of the manuscript has been revised accordingly. The updated content is as follows:\n\nDuring pre-training, the teacher and student VLMs are trained separately using simulated and real-world datasets, respectively. The simulated dataset, referred to as CARLA-VLM, is collected using the CARLA Leaderboard v2 simulator and comprises 61.6\\% normal scenes and 38.4\\% challenging scenarios (e.g., traffic jams, near-miss vehicle interactions, and pedestrian intrusions). The real-world dataset used is prepared from the publicly available nuScenes dataset. To accelerate training, we initialize both models with checkpoints from LLaVA-v1.5 [40]. Each model is fine-tuned using image frames, temporal instructions, and end-to-end prompts, enabling the Chain-of-Causality (CoC) VLMs to align instruction, visual, and temporal information effectively. Efficient training is further facilitated by sampling frames at fixed intervals and applying random temporal shifts as augmentation.\n \nWe use a real-world dataset (nuScenes-VLM) and a simulated dataset (CARLA-VLM) to pre-train the student and teacher VLM models, respectively.\nnuScenes-VLM dataset:\nThe student VLM is pre-trained on the nuScenes-VLM dataset, which is derived from the publicly available nuScenes dataset and enriched with textual prompts (as illustrated in Appendix B.2). We follow the official train/validation split provided by nuScenes.\n\nCARLA-VLM dataset:\nWe use this dataset as our simulation dataset to pre-train teacher VLM model. This dataset is configured with settings similar to nuScenes and annotated with textual prompts consistent with those in the nuScenes-VLM dataset.\nTo enhance dataset transparency, we provide the statistical distribution of scenarios within the CARLA-VLM dataset. A new supplementary table presents detailed counts for each scenario category. We believe these additions improve the clarity of our dataset description and enable a more rigorous evaluation of the model’s performance, particularly in challenging driving scenarios.\n\n| Scenarios Category | Proportion(\\%) |Clips Count(total 600)|\n|--|--|--|\n|Normal (Straight, Left, Right Turn) |61.6|370|\n|Pedestrian Intrusion | 4.7| 28|\n|Fog| 6.7| 40|\n|Rain|7.8| 47|\n|Near-Miss Vehicle Interactions|2.7| 16|\n|Traffic Jam |4.5|27|\n|Traffic Accident|1.8|11|\n|Vehicle Cut In|2.2|13|\n|Opposite Vehicle Intrusion|0.8| 5|\n|Vehicle U-Turning |0.5|3| \n|Construction Obstacle |4.0| 24|\n|Bicycle Intrusion |0.5| 3|\n|Lane Merge| 1.2|7|\n|No Traffic Light Intersection|0.7|4|\n|Turn Left and Merge In| 0.3| 2|\nTable Supp. CARLA Scenarios Statistical Distribution\n\nActually, In all our experiments, we only used two datasets: the nuScenes-VLM dataset (real-world dataset) and the CARLA-VLM dataset (simulation dataset).  \n\nWeakness 5:\n\nThank you for your suggestion. We have revised the manuscript to enhance transparency. In our anonymous submission, our original intent was to explain the contribution of the back-propagation strategy in Section 3.5.2, with its corresponding data flow illustrated in Figure 1. We have now revised and clarified this content in Section 3.5.2 as follows:\n\n3.5.3 Adversarial Training with Multi-Step Back-Propagation\n\nAs illustrated in Fig.1, the adversarial training process involves multiple steps that incorporate our unique back-propagation strategy. Initially, the pre-trained teacher and student VLM models are loaded. The training procedure then proceeds in two steps:\n\nStep 1:\n\nForward-Propagation\n\nIn this forward propagation, we freeze the teacher VLM model and student VLM model, only train our discriminator. Simulation data and real-world data are fed separately into the teacher and student VLM models. The features extracted from both CoC VLMs are then passed into the discriminator.\nBack-Propagation\n\nAfter the discriminator outputs the final logits, we compute the discriminator loss and perform back-propagation. Since only the discriminator is trainable in this step, the gradients are back-propagated solely through the discriminator, and only its parameters are updated. The parameters of the teacher and student VLM models remain unchanged.\n\nStep 2: \n\nForward-Propagation\n\nIn this process, only the student VLM model and the discriminator are involved in the training pipeline. Both simulation and real-world data are fed into the student VLM model, and the resulting features are passed through the discriminator to produce logits.\nBack-Propagation \n\nThis back-propagate step introduces our key contribution in terms of back-propagation. The discriminator loss is back-propagated through the discriminator, but without updating its parameters. Then, the gradients from this discriminator, together with the VLM-specific losses, are used to update only the parameters of the student VLM model.\n\nQ1:\n\nIn line 187, our intention was to convey that temporal information from the previous frame is aggregated into the current frame to enhance temporal features. The term “temporal consistency” was used to indicate the continuity of information across the temporal dimension. To achieve this, we have employed a rule-based filter, this filter removes the reasoning part of End-to-end Answer, extract the final ego vehicle's action instruction such as \"There is no safety threat, the vehicle maintains its current speed.\", as described in lines 230. This filtering step reduces the input token length while preserving the most critical information from the previous frame.\n\nQ2:\n\nDear Reviewer, we have carefully addressed your concerns as follows.\n\n$X^{S}$ and $X^{R}$ are macro-level representations denoting the data distributions of the simulated and real-world autonomous driving domains, respectively. A domain consists of an infinite set of data. Human only sample a finite subset from these distributions to form the training and testing datasets. In our paper, the datasets represent small sample sets drawn from both the virtual and real-world domains. Additionally, we use  $y$ to represent the labels/ground truth..\n\n\nQ3:\n\nHere is our explanation, our labels/GT corresponding to samples data $x^{s}$ and $x^{r}$ are mainly generated by GPT-4 and subsequently verified manually. These GTs, such as \"There are two pedestrians ... In front of the ego vehicle...\" are distribution samples from GPT4 outputs. Since natural language allows for diverse expressions of the same underlying information, it can have plenty of expression to describe the GT, for example \"A pedestrian is on the left ...., another pedestrian is 10m away from ego...\".\nTo formalize this process, we use the statistical learning theory (also called VC-theory) to express general mathematical framework for estimating dependencies from samples. Different from most empirical methodologies (like deep learning) that often rely on intuitive, asymptotic, and/or biological arguments. \nWe use an expression that returns a label $y$ for every input $x$, according to a conditional distribution $p(y|x)$, this is determined by GPT4 and manual verification. The expectation is:\n$$\\bar y = \\int dp(y|x))$$\nIn our statistical field, we normally select enough samples to mitigate the bias. But in deep learning, like the existing LLM/VLM methods, they use few samples as the final expression. Thus, we explicitly highlight the distribution expression of language content generation.\n\nThe distribution $p^s$ and $p^r$ are determined by GPT4 and manual verification.\n\n\n\n\nQ4:\n\n\n\nIn our paper, the f-divergence is employed to measure the discrepancy between two domains. To enable optimization, the f-divergence function must be convex. Therefore, we adopt a convex function $\\delta$ as our f-divergence."}]

Author rebuttal:

Meta-review: The paper introduces a Chain-of-Causality (CoC) Visual Language Model and an adversarial transfer framework with teacher–student VLMs and a discriminator. The goal is to transfer rare-case driving behaviors from simulation to real-world data.

This paper proposes a novel integration of VLMs, adversarial transfer, and causal reasoning for autonomous driving and demonstrates strong sim-to-real transfer. Closed-loop evaluation (via NeuroNCAP) and additional analyses added during rebuttal strengthened claims and the revision addressed clarity, dataset transparency, and formatting issues.

The initial evaluation lacked fine-grained long-tail analysis and closed-loop tests (later added in rebuttal). Writing and structure in early versions were unclear; terminology (CoC vs. CoT, End2End answer) was confusing. Some theoretical explanations (e.g., convex f-divergence) were initially unclear but clarified after back-and-forth discussion.

Despite initial weaknesses in clarity and evaluation, the rebuttal substantially strengthened the paper.
Decision: Accept (poster)