SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models

Authors: Jiawei Zhang, Xuan Yang, Taiqi Wang, Yu Yao, Aleksandr Petiushko, Bo Li
Primary area: foundation or frontier models, including LLMs
Keywords: Autonomous Driving; Multimodal Large Language Models; Multimodal Retrieval-Augmented Generation; Probabilistic Graph Model

Abstract:
Traditional autonomous driving systems often struggle to harmonize high-level reasoning with low-level control, leading to suboptimal and even unsafe driving behaviors. The emergence of multimodal large language models (MLLMs), capable of processing visual and textual data, presents an opportunity to unify perception and reasoning tasks within a single framework. However, integrating precise safety knowledge into MLLMs for safe autonomous driving remains a significant challenge.
To address this, we propose SafeAuto, a novel framework that enhances MLLM-based autonomous driving systems by incorporating both unstructured and structured knowledge. In particular, we first propose the Place-Dependent Cross-Entropy (PDCE) loss function, which is specifically designed to enhance the accuracy of low-level control signal predictions when treating numerical values as text.
To explicitly integrate precise safety knowledge into the MLLM to enable safe autonomous driving, we build a reasoning component for SafeAuto, which first parses driving safety regulations into first-order logic rules (e.g., "red light $\implies$ stop") and then integrates these rules into a probabilistic graphical model, such as a Markov Logic Network (MLN). The environment attributes, identified by attribute recognition models (e.g., detecting a red light), are used to form the predicates in MLN.
In addition, the environmental attributes utilized for reasoning are also considered factors in retrieval to construct a Multimodal Retrieval-Augmented Generation (RAG) model, which aims to learn from past similar driving experiences more effectively.
Extensive experiments demonstrate that SafeAuto significantly outperforms baselines across multiple datasets. By bridging the gap between high-level reasoning and low-level control, SafeAuto paves the way for more accurate, reliable, and safer autonomous driving, facilitating systems that learn effectively from experience, adhere to traffic regulations, and execute precise control actions.

Decision: reject
Reviewer scores: 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730599359369

thread_type:
official_review

<thread_object>
creator: reviewer_1fdq

<content>
summary:
The paper presents "SafeAuto," a framework for enhancing autonomous driving through multimodal foundation models. It focuses on high-level action prediction and justification, as well as low-level motion prediction, leveraging datasets like BDD-X and DriveLM. The authors claim improvements in performance metrics such as BLEU, CIDEr, and METEOR. However, the paper largely reads like a technical report, lacking significant theoretical contributions or novel methodologies, which raises questions about its overall impact and originality.

strengths:
1. The paper provides a comprehensive set of experimental results demonstrating improvements in performance metrics, which is a solid contribution.
2. The experimental design and results are well-organized and clearly presented.
3. The plug-and-play nature of the framework allows it to be integrated with existing multimodal learning methods

weaknesses:
1. The paper appears to rely on established methodologies without introducing substantial theoretical advancements, limiting its originality and depth.
2. While the components such as the PDCE loss and post-safety verification are interesting, they do not significantly diverge from existing techniques.
3. There is little discussion on how different predicates are selected or their impact on the overall prediction performance
4. The evaluation does not seem to adequately address edge cases or scenarios where traditional models might fail. A deeper examination of how the framework handles uncommon but critical driving situations could provide insights into its robustness and safety.

questions:
1. Can the authors elaborate on the specific mechanisms used to extract and integrate environmental predicates into the retrieval process? How do these predicates influence decision-making during high-level action predictions?
2. The paper mentions a gradual increase in the value of σ during training for the PDCE loss. How was this approach determined, and what empirical evidence supports its effectiveness compared to alternative strategies?
3. How does the framework handle potential conflicts between high-level predictions and low-level control signals?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643981755

thread_type:
official_review

<thread_object>
creator: reviewer_uk4o

<content>
summary:
This paper proposes a framework to enhance multimodal large language based autonomous driving using structured and unstructured knowledge. The main contribution of the paper is proposal of a new loss function that the authors call place dependent cross entropy loss, that is supposed to enhance accuracy of text based control instructions. They also propose building a reasoning component for converting safety regulations into first order logic rules. These rules are then used in a probabilistic graphical model along with environmental features to contruct a RAG model.

strengths:
This paper proposes to use a lot of new methods for enhancing autonomous driving. It addresses multiple problems that are difficult to solve and poses a challenge for using large language models for autonomous driving related tasks. Understanding of numerical information from text is a common problem that this paper proposes to solve using a new loss function. Reasoning is also a challenging task for these systems, here the paper proposes a system of using safety regulations to logic within a graphical model. The RAG model is also meant to produce better driving experience by learning from past.

weaknesses:
This paper proposes a few different changes and the goal of the proposals are not always clear. If the goal is to produce a better driver, there should be comparison with other autonomous driving works in standard NuScenes like evaluation, both open and closed loop. But, driving comparison with standard models like UniAD and VAD is not performed (Table 3).

The loss PDCE formulation is unclear. Any new loss function should be defined clearly with notation. Here the description of the loss is unclear. The properties of the loss needs to be studied in a more varied settings. The authors just produce a graph for a specific setting to motivate this loss, but better experiments need to be planned to demonstrate the behavior of this loss.

The action generation step from the LLM is not grounded in any way, so it is not clear how the approach would deal with invalid responses or hallucination, where it might generate unsafe actions.

It is not discussed how this approach could be used in practice, do the authors see this running in realtime in a vehicle or is it just used for data labeling offline? We need some runtime information for questions like this. Also, not all the tasks they want to solve would make sense in either case.

questions:
1. Formulate the loss function in mathematical terms
2. Perform experiments on loss behavior with error bars and more varied situations
3. Provide runtime information
4. Discuss inference strategy
5. Produce more experiments with driving performance like NuScenes setup and UniAD / VAD.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684185571

thread_type:
official_review

<thread_object>
creator: reviewer_jpqm

<content>
summary:
The authors propose SafeAuto, a framework that can be incorporated into multimodal large language models and that consists of 3 components: (i) a new loss function called Place-Dependent Cross-Entropy, (ii) a reasoning module based on Markov Logic Networks, (iii) a multimodal Retrieval-Augmented Generation model.
The Place-Dependent Cross-Entropy loss aims to better capture numerical proximity between numeric strings generated by the large language models indicating low-level control numerical signals (e.g., speed).
The Markov Logic Network-based module serves as a post-processor that flags the large language model's outputs which do not satisfy pre-defined knowledge rules (written in first order logic and capturing relations between predicates representing actions of the ego-vehicle---such as stop or accelerate---and objects from the environment---such as a stop sign).
The multimodal Retrieval-Augmented Generation model aims at helping the decision process by retrieving driving experiences similar to the current scenario, utilising information gathered from the image embeddings, control signals and a vector containing the values of binary predicates (that capture whether certain objects, e.g. a stop sign, are present in the given image or not).
The authors conduct an experimental analysis (on two datasets), showing (i) how these three components impact the performance when compared to state-of-the-art baselines and (ii) how background knowledge can be used to enhance the predictions of multimodal large language models.

strengths:
Good referencing of prior works related to most of the involved components, including recent studies.

Additionally, the paper clarifies the relation to prior works and explains how the proposed approach differs from the previous ones.

The work is clearly motivated in the paper, the research context and goals are also clearly stated.

The method is described in a way that is easy to follow.

The experimental setting is well-explained.

weaknesses:
There is no discussion on how the proposed Place-Dependent Cross-Entropy Loss behaves when there is a difference in the order of magnitude between the input and the target values. From the description of the method, it appears that the Place-Dependent Cross-Entropy Loss does not take into account the decimal point position (and, for example, this means that the target probability distributions from Figure 2 for numeric string "2.69" will be the same as for numeric string "26.9").
I would appreciate it if the authors could clarify this point and the statement in lines 247-248: "Notice that when \sigma is set to 0, the loss reduces to the original definition of joint CE loss for the entire numeric string", which again is unclear as the cross-entropy loss takes into account the decimal point position.

The method offers no guarantee that reprompting the multimodal large langue models will generate a new high-level action that does not violate the knowledge rules, since such models are data-driven.

The paper is in the context of knowledge-enhanced autonomous driving and argues that the underlying models are limited as they do not account for any available knowledge when making their decision.
Specifically, the authors capture the knowledge rules in the form of logical formulae, yet the related work section does not discuss prior works that would be relevant here (e.g., representing and integrating background knowledge into neural network models).

Improvements in clarity and readability are needed. Please see below detailed comments.

Figure 2: contains pseudocode and could be moved to supplementary material, with its contents described more concisely in natural language in the main text.

Formatting: cluttered text, very little space between captions and main text (e.g., lines 73 and 237), all affecting readability.
All abbreviations should be introduced, e.g. "QA" is used without doing so.

Many typos:
- in the abstract, "the the Place-Dependent".
- lines 92, 93: "contextssuch", "signalsto"
- lines 105, 106: "scenariowhich", "informationwe"
- line 144: "knowledgespecifically", "rulesinto"
- and other such instances of words not being separated by a space when they should be.
- a space before each citation would improve the readability (line 108 "BDD-X(Kim et al., 2018) and DriveLM(Sima et al., 2023)")
- abbreviations are introduced multiple times: e.g., "cross-entropy (CE)" (lines 53, 127, 180); "Place-Dependent Cross-Entropy (PDCE)" (lines 95, 170, 397), similarly for "Markov Logic Networks", etc.
- line 277/278: "all formula"
- the title of section 6 is inconsistent with the naming convention of the rest (i.e., "Limitation.", rather than "Limitation").

Other comments:
- in Figure 2, the naming of "cumulative_probs" (line 231) could be confusing/misleading as it suggests traditional cumulative probabilities. A clearer term, such as "weights," would align with the main text (line 238).
- repetitions: e.g., lines 83-84: "regarding high-level action prediction, a significant limitation of current methods in high-level action prediction"
- line 193: "with temperature as 1.0." it is unclear what is meant here, as the temperature was not yet introduced (only later on line 374 it is introduced).

questions:
One of the main components of the proposed framework is a post-processor designed to identify predictions made by the large language model that do not adhere to the knowledge rules. It would be helpful to understand how often violations of the rules were observed in the conducted experiments.

How does the method deal with cases where the large language model keeps generating a new high-level action (after reprompting) that violates the knowledge rules?

As a suggestion (also mentioned earlier, along with detailed comments), the paper would need to be improved in terms of clarity and readability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732316958913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summarization

comment:
We would like to thank all the reviewers for their valuable feedback and suggestions. We are pleased that the reviewers found our paper well-written and recognized it as providing a novel and systematic approach to autonomous driving via MLLMs, featuring three innovative components: PDCE loss, post-safety verification via MLN, and the multimodal RAG for learning from similar driving experiments, all of which significantly outperform other baselines. Following the reviewers' suggestions, we have made the following main revisions to further enhance our work.

- Following **Reviewer jpQM**'s suggestion, we further clarified the re-prompting of the MLLM for mapping predicates to the corresponding high-level actions.
- We added related work that is relevant to representing and integrating background knowledge into neural network models, as suggested by **Reviewer jpQM**.
- We corrected all typos and addressed the comments pointed out by **Reviewer jpQM**.
- We further explained the intuition and formulation of our PDCE loss following guidance from **Reviewer uk4o**.
- We provided detailed running time information as suggested by **Reviewer uk4o**.
- At the suggestion of **Reviewer 1Fdq**, we conducted an additional ablation study exploring the impact of the number of selected predicates on final prediction performance.
- Following **Reviewer 1Fdq**'s feedback, we added additional empirical evidence supporting the effectiveness of a gradually increasing value $\sigma$ for better performance over other strategies.

We sincerely appreciate the reviewers for dedicating their time to review our paper and look forward to any further discussion and suggestions to help improve the quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper develops SafeAuto, a novel framework that enhances MLLM-based autonomous driving systems by incorporating both unstructured and structured knowledge. The paper has the strengths of clear motivation and some innovation in using LLM for autonomous driving. The paper has the weaknesses of lacking detailed discussion on the proposed loss function, unclear goal, and no discussion on the approach can be used in practice.

After discussion, the final scores from the reviewers are 3, 3, 6. Despite the discussions between the authors and the reviewers, the two negative reviewers remain negative on the submission and do not believe the comments have been well addressed.

The AC has checked the submission, the reviews, the rebuttal, and the discussion, and sided with the two negative reviewers and decided this work is not ready for publication in the current form. Thus a rejection is recommended.