Title: FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous driving

Visual language action models

Chain of thought

Autoregressive image generation

Visual reasoning

Unified generation understanding
Abstract: Vision–Language–Action (VLA) models are increasingly used for end-to-end driving due to their world knowledge and reasoning ability. Most prior work, however, inserts textual chains-of-thought (CoT) as intermediate steps tailored to the current scene. Such symbolic compressions can blur spatio-temporal relations and discard fine visual cues, creating a cross-modal gap between perception and planning.
We propose FSDrive, a visual spatio-temporal CoT framework that enables VLAs to think in images. The model first acts as a world model to generate a unified future frame that overlays coarse but physically-plausible priors—future lane dividers and 3D boxes—on the predicted future image. This unified frame serves as the visual CoT, capturing both spatial structure and temporal evolution. The same VLA then functions as an inverse-dynamics model, planning trajectories from current observations and the visual CoT.
To equip VLAs with image generation while preserving understanding, we introduce a unified pre-training paradigm that expands the vocabulary to include visual tokens and jointly optimizes VQA (for semantics) and future-frame prediction (for dynamics). A progressive easy-to-hard scheme first predicts lane/box priors to enforce physical constraints, then completes full future frames for fine details.
On nuScenes and NAVSIM, FSDrive improves trajectory accuracy and reduces collisions under both ST-P3 and UniAD metrics, and attains competitive FID for future-frame generation despite using lightweight autoregression. It also advances scene understanding on DriveLM. Together, these results indicate that visual CoT narrows the cross-modal gap and yields safer, more anticipatory planning.
Code is available at https://github.com/MIV-XJTU/FSDrive.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FSDrive, a novel framework for autonomous driving that enables visual reasoning through spatio-temporal Chain-of-Thought (CoT), bridging the gap between perception and planning in a unified visual framework. The experiment part demonstrates that FSDrive achieves good results on the generation, understanding and planing tasks.\n\nSTRENGTHS AND WEAKNESSES:\nFSDrive presents an improved framework for driving policy prediction, guided by a novel Spatial-Temporal Chain-of-Thought (CoT) methodology. The authors conduct comprehensive experiments to validate the effectiveness of the proposed approach, covering generation, understanding, and policy prediction tasks. I recommend acceptance of this paper, primarily because extending the CoT paradigm into the physical world of autonomous driving is both novel and conceptually sound. The thorough experimental evaluation further supports its potential.\n\nHowever, there are several weaknesses that should be addressed:\n\n1) Limited architectural novelty: The proposed architecture for generation and understanding tasks largely mirrors existing designs in general vision-language domains. While the application to driving is appreciated, the model structure itself lacks sufficient novelty.\n2) Dataset limitation: The claim of using only 0.3% of training data relies solely on experiments conducted on nuScenes. This raises concerns about generalizability. The authors are encouraged to validate their method on additional datasets to strengthen the robustness of their claims.\n3) Real-time applicability: The auto-regressive formulation in Equation 6 for action prediction appears computationally expensive and may not satisfy the real-time constraints required in real-world driving systems. Further clarification or optimization is needed here.\n4) Insufficient qualitative evaluation: The paper relies on FID to evaluate generative quality, but comparing FID scores across different output resolutions can be misleading. The authors should provide more visual comparisons to convincingly demonstrate the quality of generated future frames.\n\nQUESTIONS:\n1) Limited architectural novelty: The proposed architecture for generation and understanding tasks largely mirrors existing designs in general vision-language domains. While the application to driving is appreciated, the model structure itself lacks sufficient novelty.\n2) Dataset limitation: The claim of using only 0.3% of training data relies solely on experiments conducted on nuScenes. This raises concerns about generalizability. The authors are encouraged to validate their method on additional datasets to strengthen the robustness of their claims.\n3) Real-time applicability: The auto-regressive formulation in Equation 6 for action prediction appears computationally expensive and may not satisfy the real-time constraints required in real-world driving systems. Further clarification or optimization is needed here.\n4) Insufficient qualitative evaluation: The paper relies on FID to evaluate generative quality, but comparing FID scores across different output resolutions can be misleading. The authors should provide more visual comparisons to convincingly demonstrate the quality of generated future frames.\n5) Format of Tab6: A more compact and structured format would help improve the overall clarity of the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FSDrive, an autonomous driving framework based on Spatio-Temporal Chain-of-Thought (CoT), which addresses the issues of ambiguous spatio-temporal relationships and fine-grained information loss caused by discrete textual CoT in existing Vision-Language Models (VLMs). The key contributions include: Visual Reasoning: By generating future frames (with lane dividers and 3D detection boxes) as intermediate reasoning steps, the model performs visual simulation akin to human cognition rather than symbolic logic. Unified Pretraining: Joint optimization of visual generation (future frame prediction) and understanding (VQA), coupled with progressive generation (coarse physical constraints first, then fine details). Experimental Validation: On nuScenes and DriveLM datasets, FSDrive outperforms SOTA methods (e.g., UniAD, Doe-1) in trajectory planning, future frame generation, and scene understanding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Clearly identifies limitations of textual CoT in autonomous driving (loss of physical laws due to over-abstraction) and proposes visual reasoning as a principled solution.\n2. The core innovation lies in elevating text CoT to visual CoT, directly using images as intermediate reasoning steps, which better suits the spatio-temporal continuity requirements of autonomous driving.\n3. Experimental validation is comprehensive and robust. \n\nWeaknesses\n1. Inference speed for generating future frames is unreported. Compliance with real-time requirements (e.g., >10Hz) remains unclear.\n2. Only front-view future frames are generated, while autonomous driving requires 360° environmental awareness. This may limit real-world deployment safety.\n\nQUESTIONS:\nHow does the model handle noisy annotations (e.g., occlusions) for lane dividers and 3D boxes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of current visual language models (VLMs) in autonomous driving, particularly the reliance on discrete, text-based Chain-of-Thought (CoT) reasoning. The authors argue that such symbolic, language-based intermediate steps can lead to ambiguity in representing spatio-temporal relationships and result in the loss of fine-grained visual information, which is critical for safe and effective autonomous driving.\n\nTo overcome these issues, the paper introduces a spatio-temporal CoT reasoning method that enables VLMs to \"think visually\" rather than abstracting visual information into text. In this approach, the VLM acts as a world model, generating unified image frames that predict future world states. These frames include both perception results (like lane dividers and 3D object detections) to capture spatial relationships, and ordinary future frames to represent temporal evolution. This visual reasoning process serves as an intermediate step, allowing the VLM to function as an inverse dynamics model for trajectory planning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The experimental evaluation is comprehensive, covering multiple relevant tasks (trajectory planning, future frame generation, scene understanding).\n2. The proposed spatio-temporal CoT method is clearly described.\n\nWeaknesses:\n1. Although the overall training process is intuitive and complete, the improvement in trajectory planning based on LLaVA-7B over other state-of-the-art methods is only marginal. Given the sophistication of the proposed methods, the performance gains do not seem commensurate. Furthermore, while the CoT process has been enhanced, the improvement over the DriveLM benchmark remains limited.\n2. As shown in Table 2, FID should be compared at the same resolution. Training with a lower resolution may yield a lower FID score, but this does not necessarily indicate better performance.\n\nQUESTIONS:\n- The authors should explain why the improvement is only marginal under fair comparison, as this is critical for assessing the significance of the proposed methods.\n- A fair comparison should be provided in Table 2 to validate the effectiveness of the proposed methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a spatio-temporal Chain-of-Thought (CoT) method to predict future trajectories in autonomous driving. In this framework, the Visual Language Model (VLM) functions as an inverse dynamics model, planning trajectories based on current observations and its own predicted future states. The training process is divided into pre-training and fine-tuning stages and is conducted without altering the existing MLLM architecture. During the pre-training phase, a \"progressive generation\" technique is employed to ensure the future predictions adhere to physical laws. The authors report that this method achieves state-of-the-art (SOTA) performance compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The method is cost-effective as it does not modify the existing MLLM architecture, only requiring an expansion of the vocabulary during the pre-training stage.\n2. The progressive generation technique effectively incorporates physical laws, ensuring the model predicts more realistic future scenarios.\n\nWeaknesses:\n1. There is a significant concern regarding real-time performance. The paper provides no metrics on inference latency, and it seems challenging to apply the prediction results immediately in a live driving context.\n2. The paper does not present failure cases or incorrect examples of the generated CoT, which makes it difficult to assess the model's limitations from the qualitative results.\n3. The generated CoT is an output of the model's inference, not the fundamental reason for the decision itself. Therefore, it is difficult to consider it a true basis for the model's actions.\n\nQUESTIONS:\n1. I am curious about the necessity of the VQA task. Could the authors provide the results of an ablation study using only the three generation-related tasks (Future frames, Future 3D detection, and Future lane divider) without the VQA task?\n2. Would it be challenging to produce results on other datasets? For instance, evaluating on Argoverse 2, which is often used alongside nuScenes, would significantly strengthen the claims of robustness.\n3. Considering that other SOTA papers like OmniDrive report results specifically for intersections, why was this critical scenario analysis omitted? Would it be possible to conduct and share additional experiments for such cases?\n\nLIMITATIONS:\n1. The method's ability to operate in real-time is a significant concern, making its practical application in live driving scenarios challenging.\n2. There is no dedicated module to handle cases where the CoT contains errors, which could compromise safety.\n3. The results are limited to a single dataset, raising concerns about how well the model will generalize to diverse, unseen environments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}