Title: Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving
Venue: NeurIPS 2024
Accepted: yes
Keywords: Autonomous Driving

Dual-process System

Knowledge-Driven

Vision Language Model
Abstract: Autonomous driving has advanced significantly due to sensors, machine learning, and artificial intelligence improvements. However, prevailing methods struggle with intricate scenarios and causal relationships, hindering adaptability and interpretability in varied environments. To address the above problems, we introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing. The Analytic Process leverages its logical reasoning to accumulate linguistic driving experience, which is then transferred to the Heuristic Process by supervised fine-tuning. Through reflection mechanisms and a growing memory bank, LeapAD continuously improves itself from past mistakes in a closed-loop environment. Closed-loop testing in CARLA shows that LeapAD outperforms all methods relying solely on camera input, requiring 1-2 orders of magnitude less labeled data. Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1.8B parameters can inherit the knowledge from a GPT-4 powered Analytic Process and achieve continuous performance improvement. Project page: https://pjlab-adg.github.io/LeapAD


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces **LeapAD**, an interesting paradigm for autonomous driving inspired by human cognitive processes, addressing the limitations of prevailing data-driven methods in complex scenarios. LeapAD incorporates a dual-process decision-making module consisting of an Analytic Process (System-II) for logical reasoning and experience accumulation, and a Heuristic Process (System-I) for quick, empirical decision-making based on the learned knowledge from System-II. By emulating human attention to focus on critical objects, LeapAD simplifies environmental interpretation and mitigates decision-making complexities. The system is tested in the CARLA simulator, demonstrating superior performance over camera-only methods with less labeled data. The Heuristic Process shows continuous improvement through a reflection mechanism and a growing memory bank, indicating the effectiveness of the dual-process approach.\n\nSTRENGTHS:\nThe paper presents several notable strengths across dimensions of originality, quality, clarity, and significance: \n\n  \n\n### Originality \n\n1. **Dual-Process Decision-Making**: The combination of an Analytic Process (System-II) and a Heuristic Process (System-I) emulates human cognitive functions, offering a biologically inspired framework for autonomous driving. \n\n  \n\n### Quality \n\nI think the quality is good. \n\n1. **Continuous Learning**: The reflection mechanism and growing memory bank enable continuous learning and improvement, showcasing the adaptability of the proposed system. \n\n  \n\n### Clarity \n\nThe paper is well-written, with clear and concise explanations of complex concepts. The dual-process framework and its components are described in detail, making the methodology accessible to a broad audience. \n\n  \n\n### Significance \n\n**Advancing the Field**: By introducing a dual-process decision-making framework, the paper opens avenues for research in autonomous driving and artificial intelligence, potentially influencing future developments in the field.\n\nWEAKNESSES:\nWhile the paper presents some interesting contributions, there are areas where improvements could be made: \n\n  \n\n### Methodological Concerns \n\n  \n\nWhile I appreciate the design of the Analytic Process and Heuristic Process, does the paper clearly distinguish between the two? My understanding is that the Analytic Process uses LLMs, while the Heuristic Process uses a lightweight language model. Why can it be called the Heuristic Process? It would be better to clearly state why can it be called the Heuristic Process and the Analytic Process. \n\n \n\n### Experimental Limitations \n\n  \n\n1. **Quantitative Metrics**: \nThe paper's experimental results are primarily based on the CARLA simulator, lacking real-world experiments. CARLA scenarios are still too simple. It would be better to report results that can comprehensively evaluate the performance of LeapAD, such as using the real-world dataset nuScenes. \n\n  \n\n### Clarity and Presentation \n\n  \n\n1. **Technical Details**: \nThis paper is based on Qwen VLM. It is not clear whether the performance improvement is due to this Qwen VLM or the two-system design. It would be better to include more ablation studies to explore the influence of VLMs, such as LLaVa. \n\n  \n\nBy addressing these weaknesses, the authors can provide a more thorough and robust evaluation of LeapAD.\n\nQUESTIONS:\nThere are areas where improvements could be made: \n\n \n\n### Methodological Concerns: \n\n \n\nThe paper should clearly distinguish between the Analytic Process and Heuristic Process? How are these processes defined and why is the Heuristic Process called such if it uses a lightweight language model? \n\n  \n\n### Experimental Limitations: \n\n \n\nCan results be reported to evaluate the performance of LeapAD using a real-world dataset like nuScenes? \n\n  \n\n### Clarity and Presentation:  \n\n \n\nIt is not clear whether the performance improvement is due to Qwen VLM or the two-system design. Can you report ablation studies to explore the influence of VLMs, for example, also try LLaVa? \n\n  \n\nBy addressing these questions, the authors can provide a more thorough and robust evaluation of LeapAD.\n\nLIMITATIONS:\nYes, the limitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

comments: SUMMARY:
This paper presents LeapAD, a dual-process closed-loop autonomous driving system.

LeapAD first uses a VLM to analyze the scene by selecting and locating critical objects in the scene, and then it uses a dual-process learning approach to learn driving behaviors.

The dual-process learning system contains an Analytical Process and a Heuristic Process. The Analytical Process is strong but expensive to run. It is used to summarize the driving experience into the Memory Bank. The Heuristic Process is more lightweight and is used to generate controls to control the vehicle. The Heuristic Process is trained with data in the Memory Bank.

The Analytical Process can also reflect from collision events in previous simulation runs. It will analyze the cause of the collisions and save the knowledge in the Memory Bank.

The authors evaluated the LeapAD method in closed-loop simulation with the CARLA simulator. They used the Qwen models as the VLMs and GPT-4 for the Analytical Process.

The evaluation result shows that LeapAD surpasses the performance of the other camera-only models on the CARLA Town05 benchmark.

STRENGTHS:
* The dual-process idea is neat and thought-provoking. It equips the autonomous driving system with the ability to learn from past experiences.

* The method achieves stronger performance than state-of-the-art methods in CARLA closed-loop simulation.

* This paper is well-written and provides sufficient details for reproducing their approach.

WEAKNESSES:
* The performance improvement is not very significant compared to the baseline.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-06

comments: SUMMARY:
This paper introduces a paradigm to design an annotation-efficient end-to-end autonomous driving system that harnesses the power and generalizability of open-source LLM models. It proves that critical frame/instance selection are critical to a decision-making module training. This method is evaluated by closed-loop testing in CARLA and achieves the SOTA performance among camera-based methods.

STRENGTHS:
1. The core idea is straightforward.
2. Achieves the SOTA result.
3. Complete adequate ablation studies to support its claim.

WEAKNESSES:
1. No quantitative benchmark on its VLM module on simulation and the real world. Only some samples are listed in the paper.
2. The paper only presents an overall benchmark on the system but no failure case analysis.
3. The result relies on the foundation model performance and the paper does not show a way to fill the gap between the simulation and the real world, which limits its impact.

QUESTIONS:
1. Why decouple into 2 separated modules, scene understanding, and decision making?
2. The scene understanding section mentions that the motion direction is one of the outputs. However, since the input sensor data is single-frame based, how does the model know the motion direction?
 3. It is not clear how the interaction with GPT4 completes in the reflection mechanism. It would be better to provide more details.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"LeapAD\" introduces a new approach to autonomous driving that addresses key challenges in adaptability and interpretability. It draws inspiration from human cognition to enhance decision-making processes in complex environments.\nThe system incorporates two complementary processes:\n- Analytic Process: Provides thorough analysis and reasoning, accumulating driving experience through logical reasoning.\n- Heuristic Process: Employs swift, empirical processing and learns from the Analytic Process through supervised fine-tuning. This dual-process setup enhances adaptability and performance.\n\nClosed loop testing in the CARLA simulator demonstrates that LeapAD outperforms methods relying solely on camera input. The Heuristic Process can inherit knowledge from an Analytic Process powered by GPT-4, leading to continuous performance improvements as the memory bank expands.\n\nSTRENGTHS:\n1.\tThe paper is generally well-written and but easy to follow. Good motivation for the model design in the introduction. \n2.\tI like the problem setup: how can we design AV systems that continually learn from its mistakes. \n3.\tThe experimental results seem to support the authors' claims.\n\nWEAKNESSES:\nI overall liked the idea of closed-loop autonomous driving approach that could emulate the critical attention mechanisms required for smooth driving environment in safety critical scenarios. The notion of heuristic and analytical processes for executing actions in robotics seems a novel approach.\nHowever, my primary concern lies in the setup of data and models for generating scene descriptions into text to identify critical objects. Operating within the text domain, which requires subsequent interpretation and tokenization by the analytical and heuristic modules, seems less efficient than using a direct vectorized representation. For instance, representing an object with parameters such as {v = 0.2 m/s, s = 3m, class = Car} is likely more efficient and robust than the text output \"The car is 3 m away from the ego vehicle and is moving at 0.2 m/s.\" This textual method could lead to inefficiencies, especially in scenarios with multiple dynamic actors.\n\nQUESTIONS:\n- The authors should detail the data generation process for complex driving scenarios like intersections, lane changes, and overtaking. Based on my understanding, the current model primarily focuses on simpler scenarios involving a single-lane and limited interaction with other actors. \n-  I recommend evaluations in more dynamic settings such as intersections and scenarios involving lane changes and overtaking, where multiple actors interact and cooperate for safety. \n- A comparison with traditional vectorized or feature-based planning systems, such as Wayformer or Precog, would be beneficial. These systems process scenes as images and convert data into vectors instead of text, which might offer insights into efficiency and performance. \n- I see DriveLM also has good performance in the case of VLM based driving. Is there any reason why that has not been put as a baseline, considering the similarity in dataset generation processes.\n\nI look forward to seeing how these suggestions might be incorporated to further enhance the robustness and applicability of the proposed approach in more complex driving scenarios.\n\nRef: \n- http://arxiv.org/abs/2207.05844 \n- https://openaccess.thecvf.com/content_ICCV_2019/papers/Rhinehart_PRECOG_PREdiction_Conditioned_on_Goals_in_Visual_Multi-Agent_Settings_ICCV_2019_paper.pdf\n- https://arxiv.org/abs/2312.14150\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}