Title: CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: VLA

VLM

LLM-AD

Domain Transfer

Autonomous Driving

End-to-end.
Abstract: Autonomous driving represents a prominent application of artificial intelligence. Recent approaches have shifted from focusing solely on common scenarios to addressing complex, long-tail situations such as subtle human behaviors, traffic accidents, and non-compliant driving patterns. Given the demonstrated capabilities of large language models (LLMs) in understanding visual and natural language inputs and following instructions, recent methods have integrated LLMs into autonomous driving systems to enhance reasoning, interpretability, and performance across diverse scenarios. However, existing methods typically rely either on real-world data, which is suitable for industrial deployment, or on simulation data tailored to rare or hard case scenarios. Few approaches effectively integrate the complementary advantages of both data sources. To address this limitation, we propose a novel VLM-guided, end-to-end adversarial transfer framework for autonomous driving that transfers long-tail handling capabilities from simulation to real-world deployment, named CoC-VLA. The framework comprises a teacher VLM model, a student VLM model, and a discriminator. Both the teacher and student VLM models utilize a shared base architecture, termed the Chain-of-Causality Visual–Language Model (CoC VLM), which integrates temporal information via an end-to-end text adapter. This architecture supports chain-of-thought reasoning to infer complex driving logic. The teacher and student VLM models are pre-trained separately on simulated and real-world datasets. The discriminator is trained adversarially to facilitate the transfer of long-tail handling capabilities from simulated to real-world environments by the student VLM model, using a novel backpropagation strategy. Experimental results show that our method effectively bridges the gap between simulation and real-world autonomous driving, indicating a promising direction for future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an end-to-end autonomous driving framework that uses a vision-language model (VLM) to bridge the domain gap between simulation and real-world scenarios. It employs a teacher VLM (trained on simulated “long-tail” driving events) and a student VLM (trained on real data), both using a shared Chain-of-Causality VLM backbone. A novel transformer-based discriminator is trained adversarially to align the student’s features with the teacher’s, effectively transferring rare-case handling skills from simulation to the real domain. The Chain-of-Causality architecture extends a pre-trained multimodal model (LLaVA-1.5) with a temporal text adapter that feeds the previous time-step’s summarized output into the language model, enabling chain-of-thought reasoning for explainable decision making. On the nuScenes-VLM benchmark, the proposed approach significantly outperforms prior LLM-based driving methods (e.g. DriveLM) on both language understanding and planning metrics, demonstrating effective sim-to-real transfer and improved interpretability.\n\nSTRENGTHS AND WEAKNESSES:\n##Strengths:\n\n**1**, The use of adversarial techniques for domain adaptation is well-established; however, combining this with VLMs to facilitate explainable decision-making in autonomous vehicles is a novel contribution.\\\n**2**, The proposed Chain-of-Causality VLM introduces a mechanism to model causal relationships in driving scenarios, which is a unique approach to improving interpretability in autonomous systems.\\\n**3**, Figure 1 is concise and straight-forward, which elaborates the structures and data flow.\\\n**4**, The experiments are solid and detailed, building upon a comprehensive set of metrics.\n \n\n##Weakness:\n\n**1**, Line 105, citation need to be fixed. \\\n**2**, From line 191 to 196, the example is straight forward, but still a bit vague to understand what kind of content are needed in temporal instruction.\\\n**3**, The clarity of section 3.4 needs to be improved.  A list of questions can be found in below Question part.\\\n**4**, In section 3.5.1, it would be better to state data used for pre-training of both student and teacher VLM.\\\n**5**, As a stand alone contribution stated in line 90-91, this back-propagation strategy have not been found in section 3 for elaboration.\\\n\nQUESTIONS:\n**1**, In line 187, what is the scheme for choosing LLM outputs to encode temporal consistency? \\\n**2**, How to understand the data domain denoted as $X^S$ and $X^R$? Are they sensor data and image data from simulation and real-world? Then what does $y$ represent here?\\ \n**3**, How is the distribution probability calculated in line 251? Do we explicitly know distribution $p^s$ and $p^r$?\\\n**4**, How does the convex function $\\delta$ in line 280 involves in the distance/f-divergence calculation in equation 7? Is $\\delta$ related with function $f$?\\\n\nLIMITATIONS:\nSee above. The reviewer think this paper is interesting and novel with comprehensive experiments setting, but authors need to improve on the writing clarity: some of the technical details are missing but crucial for understanding. The reviewer will raise the score if concerns are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Chain-of-Causality (CoC)-based transfer learning framework for autonomous driving, addressing the limitations of traditional methods that rely solely on either real-world or simulated data. By independently training two models and leveraging a teacher model (trained in complex simulated environments) to guide a real-world student model, the framework effectively utilizes simulated data. Additionally, the paper introduces Chain-of-Causality, which aggregates temporal information end-to-end via a text interface. Experimental results demonstrate that this method successfully achieves Sim2Real transfer in autonomous driving.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper presents a novel perspective, and investigating the Sim2Real problem in autonomous driving holds significant research value.\n\nThe experiments are thorough, the analysis is well-reasoned, and the study offers strong analytical merit.\n\nWeaknesses:\n\nFigure 1 contains excessive lines, making the description overly complex and hindering readability.\n\nThe spacing between paragraphs appears inconsistent, and the excessive blank space before the \"Contribution\" section in the Introduction seems unnecessary.\n\nThe structure of Chapter 3 is somewhat unusual. Sections 3.2 and 3.3 could be condensed under \"Methods\" rather than being separate subsections, and \"Model Training\" might be better placed in Chapter 4 instead of Chapter 3.\n\nThe formatting of tables is inconsistent. For example, Table 1 lists each metric in separate columns, whereas Tables 2 and 3 do not.\n\nQUESTIONS:\nThe term \"End2End answer\" is quite ambiguous. While it seems to encompass textual descriptions of perception, prediction, and planning, is \"end-to-end\" the appropriate term? Would \"CoC answer\" be more accurate?\n\nHave experiments been conducted using the previous N frames' \"End2End answer\" as cached temporal information?\n\nThe paper does not clearly differentiate between CoT (Chain-of-Thought) and CoC (Chain-of-Causality). In Omnidrive, this process is referred to as CoT, whereas the authors label it CoC. Clarifying the boundaries between these definitions would be helpful.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel end-to-end autonomous driving framework that aims to transfer long-tail handling capabilities learned in simulation to real-world applications via adversarial domain transfer. The authors introduce an architecture comprising a teacher VLM, a student VLM, and a discriminator. A central component of this architecture is the proposed \"Chain-of-Causality Visual Language Model\" (CoC VLM), designed to integrate temporal information and facilitate causal reasoning for decision-making.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\nThe core idea of the paper is interesting and highly relevant. Leveraging simulation data to supplement the scarcity of rare and challenging cases in real-world datasets is a promising direction for developing more robust autonomous driving systems. The proposed adversarial transfer framework to bridge the sim-to-real gap for complex behaviors is reasonable.\n\n[Weaknesses]\n\nMy major concern involves the experimental part of the manuscript. The evaluation methodology does not sufficiently support the central claims of the paper.\n\n1.  The evaluation relies on macro-level metrics that may not validate the claimed improvement in handling long-tail scenarios. The paper reports the Average Displacement Error (ADE) and Collision Rate over the entire nuScenes-VLM test set. These overall metrics are likely dominated by a large number of simple, common scenarios (e.g., straight-line driving), which can dilute the performance results and mask the model's actual capabilities in critical, long-tail situations. To substantiate the claim of improved handling of uncommon cases, the authors must provide a more fine-grained analysis. For instance, they should isolate a subset of test clips containing specific long-tail events (e.g., 'pedestrian trespassing' or 'traffic violations' as mentioned in the simulation design ) and report performance metrics (ADE, Collision Rate) specifically on this challenging subset. Without this, the claimed improvements are not convincingly demonstrated.\n\n2.  A significant limitation is the lack of closed-loop evaluation. The paper's evaluation is conducted in an open-loop setting, which assesses the model's ability to predict a trajectory but does not account for how the ego-vehicle's actions influence the environment's subsequent states. Closed-loop evaluation, where the model's output is fed back into the simulation to create a continuous interaction loop, is a much stronger and more realistic test of an autonomous driving system's performance and safety. The authors acknowledge this in the appendix, stating that 'closed-loop evaluation metrics are not assessed', but this remains a major weakness that impacts the perceived real-world viability of the proposed method.\n\n3.  The paper lacks transparency regarding the statistical distribution of the simulated long-tail scenarios. While the supplementary material mentions that the simulation data is collected from CARLA Leaderboard v2's 39 challenging scenarios and aims for a 'balanced representation' to address the long-tail challenge, neither the main paper nor the supplement provides quantitative statistics on the final distribution of these cases. It is unclear what specific cases are included and at what frequency. This lack of transparency makes it difficult to assess the diversity of the training data and to fairly interpret the performance improvements. The effectiveness of the domain transfer heavily relies on the quality and distribution of these simulated events.\n\nQUESTIONS:\nRegarding the CoC VLM architecture described on lines 179-182, could you please clarify the following:\n\n1.  The 'filter' is described as a component to simplify LLM-generated responses and reduce token length. The paper later states that this filter is designed to \"summarize and cache the end-to-end answer, converting it into a concise instruction for the subsequent frame\". What is the specific mechanism or model used for this filter? Is it a rule-based system that extracts keywords, or another smaller language model?\n\n2.  The 'dedicated end-to-end answer generation module' is mentioned as a way to streamline output formatting. Does this refer to a separate trainable module, or does it describe a specific structured prompting and decoding process?\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}