SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models
Authors: Jiawei Zhang
Xuan Yang
Taiqi Wang
Yu Yao
Aleksandr Petiushko
Bo Li
Venue: ICML 2025 poster (2025)
Keywords: Autonomous Driving; Multimodal Large Language Models; Multimodal Retrieval-Augmented Generation; Probabilistic Graph Model; Markov Logic Network
Primary area: deep_learning->large_language_models
TL;DR: We propose SafeAuto that includes a specialized PDCE loss for low-level control to improve precision and safety, and enhances high-level action prediction by integrating past driving experiences and precise traffic rules into multimodal models.

Abstract: Traditional autonomous driving systems often struggle to connect high-level reasoning with low-level control, leading to suboptimal and sometimes unsafe behaviors. Recent advances in multimodal large language models (MLLMs), which process both visual and textual data, offer an opportunity to unify perception and reasoning. However, effectively embedding precise safety knowledge into MLLMs for autonomous driving remains a significant challenge.
To address this, we propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge. First, we introduce a Position-Dependent Cross-Entropy (PDCE) loss to improve low-level control signal predictions when values are represented as text. Second, to explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic (e.g., "red light => stop") and embeds them into a probabilistic graphical model (e.g., Markov Logic Network) to verify predicted actions using recognized environmental attributes.
Additionally, our Multimodal Retrieval-Augmented Generation (RAG) model leverages video, control signals, and environmental attributes to learn from past driving experiences. Integrating PDCE, MLN, and Multimodal RAG, SafeAuto outperforms existing baselines across multiple datasets, enabling more accurate, reliable, and safer autonomous driving. The code is available at https://github.com/AI-secure/SafeAuto.

SafeAuto: Knowledge-nnbaned Safe Autonomous Driving with Multimodal Foundation Models
Jiawei Zhang
Xuan Yang"
Taiqi Wang"
8 Yu Yao Aleksandr Petiushko
8 Bo Li
Abstract
1. Introduction
Autonomous Driving (AD) systems (Kim et al., 2018; Jin et al., 2023; Hu al., 2023; Jiang et al., 2023; Zhang et al., 2024b) have made significant strides in recent years, yet they often rely on separate modules for high-level decision- making (e.g., "the car should slow to a stop") and low- level control signal prediction (e.g., providing the specific speed or steering angle for the next few moments). How- ever, these two aspects are inherently correlated, as high- level actions directly guide low-level control signals. This modular design often overlooks this correlation, leading to inefficiencies and less cohesive driving behaviors. Re- cent advancements in Multimodal Large Language Models (MLLMs) (Liu et al., 2023b;a; Lin et al., 2023) offer a promising avenue to bridge high-level reasoning and low- level control autonomous driving (AD), which provide unified framework capable of processing and reasoning over multiple data modalities, such images, videos, and text. Some recent works (Wang al., 2023; Xu et al., 2024; Wang et al., 2024) have begun to leverage MLLMs to gener- ate both high-level action descriptions and low-level control signals an end-to-end manner. However, they are predom- inantly data-driven and often fail to perform at human levels due several limitations.
Traditional autonomous driving systems often struggle to connect high-level reasoning with low- level control, leading to suboptimal and some- times unsafe behaviors. Recent advances in mul- timodal large language models (MLLMs), which process both visual and textual data, offer an opportunity to unify perception and reasoning. However, effectively embedding precise safety knowledge into MLLMs for autonomous driving remains significant challenge. To address this, we propose Safe framework that enhances MLLM-based autonomous driving by incorporat- ing both unstructured and structured knowledge. First, we introduce a Position- Position-Dependdee Dependent Cross- Entropy (PDCE) loss to improve low-level control signal predictions when values are represented text. Second, explicitly integrate safety knowl- edge, develop reasoning component that translates traffic rules into first-order logic (e.g.. "red light - stop")) and embeds them into a probabilistic graphical model (c.g., Markov Logic Network) to verify predicted actions using rec- ognized environmental attributes. Additionally, our Multimodal Retrieval-Augmenteed Generation (RAG) model leverages video, control signals, and environmental attributes to learn from past driving experiences. Integrating PDCE, MLN, and Multimodal RAG, SafeAuto outperforms ex- isting baselines across multiple datasets, enabling more accurate, reliable, and safer autonomous driving. The code is available at https:// github. com/ Saamm/AA-seeureeaafaeeeee Safeuuu..
Firstly, for low-level action prediction, current approaches in adapting MLLMs generally follow two fashions. The first fashion treats the prediction of float numbers as text generation (Gruver et al., 2024: Xu et al., 2024), directly training the MLLM using cross-entropy (CE) loss for to- ken prediction. Some variations (Brohan et al., 2023; Sima et al., 2023) of this method involve tokenizing the prediction range into several bins and adding new tokens for each bin into the LLM's vocabulary, allowing the model to predict the corresponding bin token ID. However, these methods remain somewhat coarse compared to traditional regres- sion techniques (Hu et al., 2023) using Mean Squared Error (MSE) loss. Alternatively, another fashion (Jin et al., 2024b) employs a linear layer to decode the float number from the output hidden embeddings of the MLLM, enabling the use of MSE loss to train the model. While this approach may improve numerical accuracy, it compromises the au- toregressive capability of the LLM, as the model can then only be purely used for numerical prediction and cannot
Work done during an internship at Nuro, I Work done during an internship at UIUC. This work was done while the author was Nuro 'University of Chicago UUniversity of Illinois Urbana- Champaign 'Nuro Viirte AI. Correspondence to: Jiawei Zhang <jwz@ uchicago.cdu>, Bo Li <bol@uchicago.cdus.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SafeAuto: Knowledge-nnanneed Safe Autonomous Driving with Multimodal Foundation Models
Multimodal Input: video, text
Position- DeiionneDeeednnt Cross-Entropy Loss ($ 3.1)
Current video driving scenaring <videnn>
Control Siynal before the current frame Sequence:
Speed: [3.35, 3.26, 3.17, 3.08, 2.96, 2.87, 2.78]
Curvature: [0.0, 0.0, 0.0, 0.0,
0.0, 0.01
MLLM
Acceleration: [-0. 20.992-00 -0,9, -0.86, -0.85, -0.82, -0.8, -0.77)
Course: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0:
The Weighited Target Takeo Profability for float number speed "2.69"
SaíeAuto- SafeAuto-Reasoning: Knowledge- Enhanced Post-Safety Verification ($ 3.2)
Tasks of SafeAuto
Task High Level Action Query
Predicates
Weight
Furmula (Krowledge Rules]
MLLM Action Predicate:
10.02
aalddRedigghtt => -Accoferatets) A -ftofifssts A --Yilddo
[Current Context) + [Retrieved Context)
MLLMKespixisL, MLLMStophi)- = O,
0.03
StopSipott) =a Stapts) (ecoleeattri)) A -PullOOwrrii
Human: What the action of ego car?
Fnvironmental Predicates:
8.47
NotelTumSignfa) -Tumneefiss)
HLM:The-can moving forward
SalidRedl light))=1,, StopSign(()-O,... ...
1O.51
MHMMeeppo Keept))
LLM: The car is slowing to a stop
Observed Action Predicates
10.55
ML(MStop00 => Stop/0
Violate safely knowledge, Should he Stop
Keeptx), Accelerate(s), Stopíx),
Task 2: High Level Justification Query
SafeAuto-RAse RAG:: Learn from Similar Driving Experience ($ 3.3)
Human: Why does the ego car do this?
LLM: for the red light at the intersection ahead.
Video (8 image frames)
Rank Learning
MLLM
Control Signal (loat vector)
Task Low Level Action Query
Environnental Predicate (binary vector)
Human: Predict the control signal next frame.
Embedding
Unified
Embedding
Text
rank
TRetrieved Context]
LLM: Speed: 2.69 Course: 0.00
Figure I: Overview of our SafeAuto pipeline for end-to-end high-level and low-level prediction in autonomous driving, featuring: (1) the Position-Dependent Cross-Entropy Loss (Section 3.1) for improved low-level numerical predictions using soft, weighted digit probability distributions; (2) Knowledge- ledge-Fnhaneed Post-Safety Verification (Section 3.2) with Markov Logic Networks to verify high-level actions against traffic rules; and (3)+ Multimodal RAG (Section 3.3) training method that incorporates similar driving experiences via text- rankings for better context-aware decision- crtisinnmmaking....
perform any further QA-for example, handling high-level ireestion-nsweeete
ities. (2) Knowledge-Enhancee Post-Safety Verification: We employ Markov Logic Networks (MLNs) (Richardson Domingos, 2006) to explicitly encode domain knowledge and structured traffic rules into the recision-making process of the MILM. This Knowledge-enables reasoning allows us to verify and correct the high-level actions suggested by the MILM, ensuring they comply with traffic regula- tions and safcty constraints. (3) Multimodal RAG for Autonomous Driving: We introduce a method that utilizes video data, control signals, and the environmental predicates used in the MLN to retrieve similar driving experiences. By learning joint embedding across these modalities based on the ranking derived from text description of the current scenario which contain rich semantic information can effectively leverage past experiences to inform cur- rent decision-making. By integrating these components, SafeAuto provides a comprehensive solution to the chal- lenges faced by current MILMs in autonomous driving. We evaluate our approach X (Kim et al., 2018) and DriveLM (Sima et al., 2023), both featuring low-level control signals and high-level action de- scriptions. Our experimental results demonstrate significant improvements in both low-level control accuracy and high- level action prediction. First, for low-level prediction on the BDD-X dataset, it reduces the Root Mean Square Error (RMSE) for speed and course predictions by an additional 5.8% and 14.1% over the state-of-the-ar (SOTA) baselines, respectively. Furthermore, on the DriveLM dataset, it de- creases the Average Displacement Error (ADE) for motion prediction by 44.4%. Second, for high-level prediction on
Additionally, regarding high-level ac- tion prediction, a significant limitation of current methods is their inability to effectively utilize both structured and unstructured knowledge when making decisions. Specifi- cally, existing approaches often focus solely on data-driven techniques, inadequately incorporating structured knowl- edge such as traffic rules and safety constraints. Although some methods (Sima et al., 2023; Mao et al., 2023; Wang al., 2024) attempt to include traffic regulations by embed- ding them into the model's is context, this implicit approach is insufficient. Due to the inherent tendency of MLLMs to hallucinate, they may still generate unsafe or illegal ac- tions. Meanwhile, while RAG (Lewis et al., 2020) has been employed in language models (Semnani et al., 2023; Zhang et al., 2024a) to mitigate issues like hallucination by incorporating relevant information from external sources, Yuan et al. (2024) first propose combine the rich multi- modal data inherent in autonomous driving contexts- such as videos, images, and control signals- to learn from past driving experiences as unstructured knowledge.
two benchmark datasets: BDD-
To address these challenges, we propose a novel frame- work SafeAuto that enhances MI.LMs for autonomous driv- ing through three key contributions as shown in Figure 1: (1) Pesition-Deeendent Cross-Entren propose a PDCE loss that retains the autoregressive nature of the MLLM while behaving like an MSE loss during training. This loss function improves numerical prediction accuracy without compromising the model's 'ss language generation abil-
(PDCE) Loss: We
2

---

SafcAuto: Knowaeege-Enbaneed Safe Autonomous Driving with Multimodal Foundation Models
the BDD-X dataset, our method boosts high-level action performance beyond the SOTA by 28.0% under the CIDEr metric. Meanwhile, the DriveLM dataset, it improves the high-level behavior prediction accuracy by an additional 13.0% over the SOTA.
these methods remain purely data-driven and lack explicit safety verification for generated actions. Given the safety- critical nature of autonomous driving, ensuring that output actions are safe and compliant with traffic rules is essen- tial. To address this, we incorporate extracted knowledge- specifically structured traffic rules-into a probabilistic graphical model like a Markov Logic Network (MLN) for explicit post-safety verification, which has been widely used in previous work (Yang et al., 2022; Zhang et al., 2023b). Besides, RAGDriver (Yuan et al., 2024) further enhances reasoning by retrieving similar driving experiences through triplet loss-based metric learning. We extend this approach by developing a more flexible and efficient retrieval system, directly training a joint embedding based on multimodal in- puts to learn relative rankings from text similarity. Most im- portantly, we find that the incorporation of binary structured environmental predicates (e.g., the presence of stop sign) from the previous reasoning components, namely MINs, significantly improves retrieval performance.
2. Related Work
Advancements autonomous driving have produced com- prehensive frameworks like UniAD (Hu et al., 2023), which integrates modules for tracking, mapping, motion prediction, and occupancy estimation for low-level planning. However, UniAD lacks high-level action descriptions and textual jus- tifications. To address high-level explanations, Kim et al. (2018) proposed an attention- based video-to-text model gen- erating explanations of current driving actions. Similarly, ADAPT (Jin et al., 2023) employs video Swin Transformer (Liu et al., 2022) extract video tokens for separate high- level and low-level action predictions.
Autonomous Driving with MLLM. The emergence of MLLMs enables unified end-to-end generation of both high- level and low-level outputs. Most of these works often treat numerical control signals as text, training models us- ing token prediction with cross-entropy loss. For example, DriveGPT4 (Xu et al., 2024) just treats low-level control signals as text, fine-tuning an MLLM to sequentially predict high-level and low-level actions in a conversational manner using the BDD-X dataset. M-AeeL (Sima et al., 2023), influenced by RT-2 (Brohan et al., 2023), discretizes waypoints into bins, expanding the tokenizer vocabulary accordingly and fine-tuning the BLIP-2 (Li et al., 2023). While this facilitates end-to-end training, it remains coarse compared to UniAD (Hu et al., 2023), which uses MSE loss. Time-LLM (Jin et al., 2024b) decodes numerical predic- tions directly from output embeddings using a linear layer with MSE loss but diminishes the language model's 's autore- gressive capabilities, limiting high-level question-answering abilities. Additionally, Tan et al. (2024) suggest that em- ploying the LLM backbone in this way does not enhance regression performance. In contrast, we propose a novel PDCE loss that adapts the cross-entropy loss for numerical training to behave more like MSE loss while preserving the model's ' ability to perform high-level cesstion-answering.
3. SafeAuto
Motivation. Recent studies have begun to explore the in- tegration of MLLMs into autonomous driving systems to enhance both high-level reasoning and low-level control actions. As illustrated in Figure 1, the MLLM receives a sequence of current driving images or videos, accompanied by textual descriptions of historical control signals, includ- ing speed, curvature, acceleration, and course, as inputs. Then, during the conversation, the model expected to an- swer three types of queries: (I) High-Level Action Queries: These queries request a textual description of the action that the current ego vehicle is performing should per- form. For example, when asked "What is the action of the ego car?", the MLLM is expected to respond with an answer like "The car is slowing down to stop". (2) High
High-Level Level Justification Queries: These queries seek an ex- planation for the action provided by the MILLM. For in- stance, "Why does the ego car do this?" prompts the model to justify the action, such as "for the red light at the intersec- tion ahead." (3) Low-Level Action Queries: These queries request specific control signals or trajectories that the ve- hicle should execute in the future. For example, the query "Predict the control signals for the next frame" " would elicit a response like "Speed: 2.69, Course: 0.00", which can then be translated into actual control commands for the au- tonomous vehicle. Typically, low-level action queries follow high-level action and justification queries, ensuring that the generated control signals are conditioned on prior high-level actions for more accurate and coherent driving control.
Safety Guarantee. Providing safety guarantees (Li al., 2022: Zhang et al., 2023a) is always fundamental con- cem in machine learning, especially in safety-critical ap- plications like autonomous driving. Further advancements involve integrating perception and planning tools into the MILM context. Agent-Driver (Mao et al., 2023) incor- porates modules from UniAD into an MLLM framework, serving as a language agent for autonomous driving. Om- niDrive (Wang et al., 2024) introduces a framework com- bining 3D perception, reasoning, and planning. However,
Overview. In this section, we detail the three main components proposed within this framework, each elab- orated in subsequent sections: (1) Position--Dependent Cross- Loss function for improved low-level action
3

---

SafcAuto: Knowledge-Enhanee Safe Autonomous Driving with Multimodal Foundation Models
LAST busiinn of Speed
Dfdeestonal Speet
the CE loss to function more like MSE loss while main- taining textual predictions. Consider the previous exam- ple of predicting the float number "12.46." Originally, the MLLM is trained to maximize the probabilities p('1))p(22') ''''...('''' | 112.44) by minimizing the CE loss with one- hot labels. However, this does not ensure that predictions closer to the target value have a lower loss as each digit's probability is treated with equal importance in loss.
25
(2) Speed prediction distribution using original loss,
pont
(b) Speed prediction distribution using PIDCE loss.
To make the CE loss function behave more like MSE loss, we adhere to two key principles: ity, where digits closer to the target in the same position incur lower loss, and (2) Place-Level Importance, where more significant digit positions have greater influence on the loss. To implement these, we introduce two modifica- tions. First, Digit-Level Loss Adjustment replaces one-hot hard target labels with soft target distribution D(H, or) centered around the target digit μt, e.g., Gaussian distri- bution ((,, a)) to assign higher probabilities to numerically closer digits. The loss for each digit is then calculated as the Kullback-Leible (KL) divergence between D(u, or) and the predicted probability distribution P from the MILMM. Second, Place-Level Weighting assigns a weight wy each digit position, aiming for weights decrease with the posi- tion index i. Instead of using fixed weights, we adopt soft approach by utilizing the approximated joint probabilities as weights, allowing flexible adjustment via a. For exam- ple, in the number "12.46", the weight for digit '2' is the probability of '1' under D(1, o). and the weight for digit '4' is the probability of '1' multiplied by the probability of '2' under D(2, This ensures that later positions never have higher influence than earlier ones and also allows for dynamic control of the decreasing weight with o. Typically, as a approaches O, the weighting scheme converges to the original CE loss. As a result, the final loss can be summa- rized as the weighted sum of the KL divergence between the probabilities generated by the MLLM and the target digit-level soft probability distributions. Mathematically, it can be expressed as:
Figure 2: Sampled speed prediction distribution under different losses when the ground truth 12.46
(1) Digit-Level Proxim-
prediction (Section 3.1); (2) Knowledge-Enhanced Safety Verification using Markov Logic Network (MIN) for high-level action prediction (Section 3.2); (3) Multimodal Retrieval-Augne Augmmentdd Generation (RAG) for learning from similar driving experiences (Section 3.3). In summary, dur- ing training, we first fine-tune the MILM using the PDCE loss with the retrieved context to enhance the accuracy of lovelevel action predictions. During evaluation, we retrieve the top K similar driving experiences from the training database, generate high-level actions using the MLLM, and apply post-safety verification using the MLN to ensure that the actions comply with traffic rules and safety constraints.
Enhanced Post-
3.1.
CE loss
In existing approaches that utilize MLLMs for autonomous driving, the next-token prediction loss loss-specificallly the cross- entropy loss is commonly applied uniformly across all pre- diction tasks, including numerical value predictions. How- ever, for numerical regression tasks, it is standard practice to use the Mean Squared Error (MSE) loss, as it directly penalizes the squared difference between the predicted and true values. A fundamental difference between CE loss and MSE loss lies in how they handle proximity to the target: MSE loss decreases as the prediction gets numerically closer to the target value, whereas CE loss does not necessarily exhibit this property. As a result, an issue is empirically observed in the speed prediction distribution when using the original CE loss to fine-tune the MLLM on the BDD-X dataset, as shown in Figure 2 (a), which displays predictions over 200 samples given the same input driving context with temperature 1.0. As we can see, it reveals two distinct peaks, indicating that predictions closer to the ground truth value of " "1..66" do not necessarily occur with higher fre- quency or lower loss, contrary to the behavior expected from MSE loss. A potential solution is to append an MLP to the MLLM to decode the output embeddings into float values, using MSE loss for tuning. However, this will disrupt the MLLM's autoregressive token generation, transforming it into a pure transformer encoder (TTn et al., 2024) used only for regression tasks and losing its language generation capabilities necessary for high-level dnestion--nswerring
PDCE loss
Σ w: KL(P, || D(us, o)),
(I)
where is the number of digits used for representing the number, μi is the i-th digit, P. represents the probability distribution over the possible digits for the i-th digit posi- tion from MILM, wi = II D (Hz, o) [my]- The pseudo- code for implementing this loss during practice is provided in Appendix B. Besides, to balance the loss among various float numbers, we standardize their representation by using consistent digit lengths in text form. For example, in the BDD-X dataset, each number is formatted to five digits, such as representing 8. as 100" during training. The new prediction distribution using this loss with o = 0.35 is demonstrated in Figure 2 (b). As shown, the distribution exhibits predictions that are more centered and closer to the
PDCE loss. To overcome these challenges, we adapt
4

---

SafcAuto: Knowledge-nnanneed Safe Autonomous Driving with Multimodal Foundation Models
ground truth with a bell shape, which aligns with the desired outcome and verifies our intuition. Figure 7 demonstrates further case study showing the token probability distribution when using CE loss and PDCE loss during training.
assignment ay (which equals if ʃ is true under ay and O otherwise), and A is the set of all possible assignments to the arguments of formula f.
Autonomous Driving Context. In our approach, we cat- egorize predicates into unobserved predicates (L), repre- senting potential vehicle actions like Accelerate (x), Stop (x), and TurnLeft (x), and observed predicates (O), Observed predicates include (1) MLLM Action Pred- icates, such as MLLMAcellerte (x), MLLMStop (x), and MIIMTurnIeft (x), which reflect high-level actions suggested by the MLLM. We map these to their truth values, introducing formulas like MLLMAccelerate (x) + Accelerate (x) to align with MLLM's decisions. (2) Environmental Predicates, which describe conditions like StepSign (x) or Solid BolidRedLight (x), extracted from video data using object detectors. These predicates integrate with main action predicates to form logical for- mulas based on traffic rules from the California Driver Handbook I, e.g., StopSign (x) = Stop (x) V Decelerate (x) A -Pull0ver (x). Additionally, predicates like HCSTurnIeft (x) reflect historical ac- tions based on control signals, enhancing the vehicle's ac- tion mecision-making process. Details are deferred to Ap- pendix A.
3.2. SafeAuto-reasonee
Currently, most methods for autonomous driving that utilize MLLMs are still purely data-driven. While these data-driven approaches have led to significant advancements, they may not be entirely suitable for safety-critical scenarios like au- tonomous driving, where reliability and strict adherence to safety regulations are paramount. To address this concern, we propose incorporating Probabilistic Graphical Models (PGMs) to verify the safety of the high-level actions sug- gested by the underlying MLLM. Specifically, in this paper, we focus on demonstrating how to adopt Markov Logic Networks to integrate domain knowledge and traffic rules into the decision-making process, while other variants are also applicable. In this section, we begin by explaining what MLNs are and how they apply our AD context.
Definition. Essentially, an MLN consists of a set of first-order logic formulas, each associated with weight that reflects the strength or confidence of that formula. These weights allow us to model uncertainty and han- real-world knowledge. In our au- tonomous driving scenario, we use MLNs to model traf- fie rules and safety constraints. For example, traf- fie rule like "If there is stop sign, then the vehicle should stop or decelerate" can be represented as the logical formula: StopSign(x) (x) = Stop (x) V Decelerate (x), where x represents the current driv- ing scenario. Here, predicates such as StopSign (x), Stop (x), and Decelerate (x) are logical functions that return true or false, indicating whether the condition holds in scenario r. Formally, in MLNs, predicates are logical functions defined over a set of constants V = (11, u,,..., ) ), where each i represents an object or con- cept in the domain, such as "stop sign" or "red light." A predicate takes these constants as arguments and returns a truth value: k(.)= : V X ... X V + 0, While for- mulas are logical statements composed of predicates and logical connectives (e.g., = A, v), with each for- mula f associated with a weight wy indicating its impor- tance. Then, an MLN defines joint probability distribu- tion over all possible assignments of truth values to the ground predicates (predicates with specific constants as- signed). The probability of a particular world (an assign- ment of truth values to all ground predicates) is given by: P(X)= 1 exp (Erer Wy Eayes, bhy (uy)). where X is the set of all ground predicates, F is the set for all formulas f, 7 is the partition function ensuring the distribution sums to one, (y(aa) is the potential function for formula f with
dle exceptions
Inference. Our goal is to infer the most probable assign- ment of the unobserved main action predicates V given the observed predicates O. To determine the safest and most appropriate action, we perform inference by maximizing the conditional probability P(( Specifically, we seek the assignment to the main action predicates U/ that maxi- mizes this probability ut" = arg maxy P(4(O). Since the possible worlds for 1 (i.e., the possible assignments to the main action predicates) are inherently limited-a vehicle cannot simultaneously accelerate and decelerate or turn left and right-the the inference process thus computationally efficient. The detailed specifics of the possible worlds can be found in Appendix A.6.
Training, The training of the MLN is straightforward and involves learning the weights wy of the formulas to max- imize O() In our approach, we utilize a mix of real and simulated data for training. The real data serves as the ground training data, provided by datasets such as BDD-X, while the simulated data allows us model various driving conditions. This includes rare dangerous scenarios not present in the real data, by simulating different truth values for the predicates to perform inference. Details are deferred to Appendix A.4.
Safety Verification. Initially, we collect observed grounded environmental predicates and the MLLM action predicates
'nttps: // //www. dav ca. gov / poporraa/handbooooo cal i forni a derver-iiaare handbook/
5

---

SafcAuto: Knowledge-Enaaneed Safe Autonomous Driving with Multimodal Foundation Models
from high-level actions generated by the MILM, extracted through object detector and prompting with GPT4o. These predicates undergo inference within the trained MLN. If the MLN's final main action predicate output contradicts the MLLM's suggested action-Suggesting a potential safety violation or a breach of critical traffic rules, we overwrite the original high-level action query based on the MLN's output and re-prompt the MLLM generate a new high- level action, as depicted in Figure 1. Further details are available in Appendix A.5, In this way, the MLN serves cost-verification layer that can override unsafe suggestions from the MLLM, enhancing the overall reliability of the autonomous driving system.
may be attributed to the reduction of noise inherent in the raw video embeddings or control signals; the binary pred- icates provide a clearer and more robust representation of essential environmental information.
Unified Embedding Construction. The central question is: How can we train a unified embedding that effectively combines these different modalities for similarity compu- tation and retrieval? A key insight is that textual descrip- tions of the current driving scenario typically encompass all relevant semantic information, reflecting aspects of the video, control signals, and predicates. For instance, a text that concatenates action and justification suuiiiionnsuhhhhoe such as "The car is slowing to a stop for the red light at the intersec- tion ahead" as shown in Figure captures the essence of all three modalities. This comprehensive representation is particularly valuable for ranking the most similar driving scenarios. However, such ground text descriptions are often not available during evaluation. Building on this intuition, we propose learning a unified embedding that aligns these modalities in a shared space, akin to how text embeddings represent semantic information.
3.3. SafeAuto-Multinon RAG
In this section, we introduce a novel training method for con- structing a unified embedding that effectively integrates mul- tiple modalities-curret driving videos, historical control signals, and observed environmental predicate information from Section 3.2. Specifically, we aim train the joint em- bedding to mirror the similarity rankings derived from the embedding of the textual descriptions for the current driving scenarios, which encapsulate the semantic information of all modalities during training. This approach facilitates the retrieval of similar driving experiences, enabling the ego vehicle to make more informed and context-aware decisions in current driving situations.
Training Loss. We start by mapping each input vector- Ze. Ze. and Zp-to aligned embeddings Z Z/ and Z through individual projectors, each normalized to unit lz norm and sharing the same dimension. We then apply weighting factors We, We, and Wpp to adjust the contributions of each modality in the final unified embedding, calculated as 2, Pro Projertor (w, Z,, + w.Zz + 2,)))), which re- sides in RIIH Motivated by CLIP (Radford et al., 2021), our objective is to train these projectors so that Z,, effec- tively mirrors the relational properties of text embeddings Z, Ixx! derived from high-level scenario descriptions, including actions and justifications. To achieve this, we randomly sample a batch of cases, 2 E B and corre- sponding text embeddings Z E RBxI (assume each row has been normalized to a unit Eg norm). We compute the similarity matrices S² = 2,((2) (Z and St = Zi(Z;)T. T. The training loss then finally defined as the minimiza- tion of the divergence between the similarity matrix logits S" and the temperature-scal scalled target logits St/T,, where is a temperature parameter that sharpens the focus on the most similar examples. This minimization ensures that the unified embeddings preserve the relative rankings of the text embeddings, crucial for effective retrieval tasks without ground textual descriptions during inference.
Different Modality. (1) Image/ Video Embedding: for the image or video modality, we utilize the pre-trained Lan- guageBind encoder (Zhu et al., 2024). This encoder pro- cesses an input image in 2556 1024, while processing video into eight frames and generates a video embedding in 20488 1024, For simplicity and to reduce computational complexity, we apply global average pooling over the first dimension for both modalities here, resulting in com- pressed embedding Z, E RIx1024 for use in subsequent experiments. (2) Control Signal Vector: the control signals are numerical values representing various aspects of the ego vehicle's s historical state, such as speed, curvature, ac- celeration, and course. In datasets like BDD-X, each of these four types of control signals contains seven historical values (excluding the current frame), resulting in total of N 4 7 = 28 values. We concatenate these values into single vector 2, E IIN,, which serves as the initial control signal vector. (3) Environmental Predicate Vector: These environmental predicates introduced in Section 3.2 are bi- nary indicators of certain conditions or observations (e.g., presence of stop sign, status of traffic light). We encode these predicates into single binary vector Z, E (0. 1) I where M is the number of the whole environmental pred- icates. Empirically, we found that including this explicit binary representation significantly boosts retrieval perfor- mance, as demonstrated in Section 5. This enhancement
4. Experiments
In this section, we present our experimental results on two datasets: the BDD-X dataset (Kim et al., 2018) and the DriveLM dataset (Sima et al., 2023), both of which con- tain high-level action questions and low-level control ques- tions. Specifically, we find that: (1) when using the Position-
6

---

SafcAuto: Knowledge-nbanneed Safe Autonomous Driving with Multimodal Foundation Models
Table 1: High-level action and justification evaluation on BDD-X dataset. B4, C, and M represent BLEU4, CIDEr, and METEOR, respectively.
2023) with Vicuna 1.5 7B (Zheng et al., 2023) as the base LLM for fine-tuning. We fine-tune the model for 2 epochs with batch size of 128 on the BDD-X dataset and for 4 epochs with a batch size of 64 on the DriveLM dataset, using a learning rate of 5 x 10-2.
Action
Justification
Method
B4 f
Ct
M
B41
Ct
M
f
ADAPT
34.6
247.5
30.6
11.4
102.6
15.2
DriveGPT4
30.0
214.0
29.8
9.4
102.7
14.6
Experimental Details. (a) PDCE loss: During the fine- tuning of the MLLM, we initialize o in D(p, .)) at a small value of 0.01 and geometrically increase it after each op- timization step until it reaches the predefined value of = 0.35. This gradual increase helps stabilize the training process. (b) Post-safety verification via MLN: we fine-tune YOLOv8 (Jocher et al., 2023) using LISA dataset (Jensen et al., 2016) as the object detector for both traffic lights and signs. For the BDD-X dataset, we define 16 action predi- cates, 20 environmental predicates, and 35 formulas based traffic rules. Similarly, for the DriveLM dataset, we de- fine 7 action predicates, 29 environmental predicates, and 29 formulas. Further details are provided in Appendix A. (c) Multimodal RAG: we consistently employ four-layer mul- tilayer perceptrons (MLPs) as projectors to obtain aligned embeddings for each modality and to generate the final uni- fied embedding, and we use sentence 2022) as our text encoder. The weighting factors wi, and w. are both set to 0.4, while the weight for the predicate embed- ding up is set to 0.2. We consistently set the learning rate to 0.001 and the temperature parameter T to 0.5 for training. On the BDD-X dataset, the projectors are trained for 100 epochs with a batch size of 2,048; while for the DriveLM dataset, the projectors are also trained for 100 epochs but with batch size of 512. Finally, we retrieve the Top K 2 examples on BDD-X dataset, and Top K = 1 example for DriveLM dataset on finetuning MLLM and inference.
RAGDriver
34.3
260.8
30.7
11.1
109.1
14.8
SafeAuto
38.6
337.4
35.5
9.4
96.0
14.0
Table 2:
High-level behavior and low-level motion prediction evaluation on DriveLM dataset.
High-Level Behavior
Motion
Method
Acc f Speed f Steer f
ADE
UniAD-Single
1.80
UniAD-Full
0.80
BLIP-RT-2
2.63
Drivel M--geet
61.60
65.40
81.61
1.51
SafeAuto
74.60
81.61
81.90
0.84
Dependent Cross-Entropy loss, the numerical prediction of float numbers is significantly improved; (2) with the post- safety knowledge-enhanced verification via MIN, many dangerous high-level actions have been corrected; (3) the incorporation of Multimodal RAG, specifically integrating environmental predicate information from the MLN com- ponent, leads to significant improvements in the MI.LM's high-level prediction performance. Notably, our framework is plug-an slug-and-plar and-play and can be directly applied to any new methods based on MLLMs. All experiments are conducted on eight NVIDIA A6000 GPUs.
c5-x1 (Ni et al.,
Datasets and Tasks, (a) BDD-X: In this work, we adopt the processed version from RAGDriver (Yuan et al., 2024), where the task involves using an input video along with control signals from the past seven frames context for conversation that focuses high-level action queries, (ii) high-level justification queries, and (iii) low-level action predictions for speed and course in the next frame. This processed dataset contains 16,390 training video QA conversations and 2.123 test conversa- tions. (b) DriveLM: The DriveLM dataset is built upon the nuScenes dataset (Caesar al., 2020). In this work, we primarily focus on tasks that involve using six multi-view images from the current frame, and control signals includ- ing trajectory positions from the past three seconds as input context. The conversation concentrates on: (i) planning for possible high-level safe actions, (ii) high-level behavior involving predicting speed and steering actions, which serve as multiple-choice questions, and (iii) low-level motion, pre- dicting 2D trajectories for the next three seconds, similar to UniAD (Hu et al., 2023). We filter instances to include only those with prediction horizon of at least 3 seconds, resulting in a final dataset of 3,447 training conversations and 685 test conversations.
Baselines. (a) On the BDD-X dataset, we compare our method with several baselines: (1) ADAPT (Jin et al., 2023), state-of-the-ard video transformer-based method that pro- vides high-level and low-level answers using two separate branches; (2) TimeLLM (Jin et al., 2024a), which repurposes frozen large language models for general time series fore- casting by reprogramming numerical inputs into text-based patches; (3) DriveGPT4 (Xu et al., 2024), the first work to provide both high-level action descriptions and low-level vehicle control signals in an end-to-end fashion using an MLLM; and (4) RAGDriver (Yuan et al., 2024), a state-of- the-art method that leverages triplet loss train multimodal retrieval models for autonomous driving. (b) For the Driv- eLM dataset, we use: (1) Drivel the current state- of-the-art method that employs graph-based visual question answering to improve high-level responses and uses mo- tion tokenization for low-level prediction; (2) UniAD (Hu et al., 2023), the state-of-the-an method on the nuScenes dataset used here for comparing low-level predictions- consider two versions: UniAD (Full), which utilizes the entire historical video input, and UniAD (Single), a variant
three types of questions: (i)
Model. We use the pretrained Video-LLLaV (Lin et al.,
7

---

SafcAuto: Knowledge-nbanneed Safe Autonomous Driving with Multimodal Foundation Models
Table 3: Low-level control signal prediction evaluation on BDD-X dataset.
Speed
Course
Method
RMSE 4
Ao.1 f
Ao.5 f
A1.0 f
As.o t
A20.0 t
RMSE
Ao.1 t
Ao.5 t
A1.0 ?
As.o f
A10.0 t
ADAPT
2.68
11.77
31.79
47.48
92.75
95.87
5.87
54.49
86.39
91.06
97.36
98.20
TimeLLM
1.17
21.34
53.13
74.14
99.67
99.86
4.10
65.70
83,47
89.59
97.60
98.59
DriveGPT4
1.09
56,93
77.77
87.97
99.00
99.57
4.57
69.22
79.14
84.47
95.72
96.74
RAGDriver 0.69
51.12
85,54
94.49
99.81
99.91
4.48
74.32
88.69
93.12
98.30
99.10
SafeAuto 0.65
55.49
88.84
95.34
99.81
99.91
3.85
76.26
89.68
94.11
98.30
99.25
Table 4: Ablation study of the contribution from each module in SafeAuto focusing high-level action and justification assess- ment the BDD-X dataset. "Acc" denotes the high-level action predicates accuracy.
Table 5: Ablation study of the contribution from each module in SafeAuto both high-level and low-level predictions using the DriveL. dataset.
High-Level Behavior
Motion
Action
Justification
Method
Method
Ace
Speed f
Steer
ADE $
B4 f
C
M
Acc
B41
Ct
M
Base
30.8
221.5
29.2
61,75
7.8
85.4
13.2
Base
60.58
64.67
80.29
0.86
PDCE
31.4
231.4
29.3
61.94
7.9
84.2
13.2
PDCE
63.21
67.88
79.27
0.85
PDCE + MLN
31.5
232.2
29.4
62.97
7.9
84,5
13.2
PDCE + MLN
66.86
71.39
80.29
0.85
PDCE + RAG
38.2
334.8
35.3
91.00
9.4
95.5
13.9
PDCE + RAG
74.01
79.27
81.61
0.84
PDCE + MLN
+ RAGG
38.6
337.4
35.5
92.18
9.4
96.0
14.0
PDCE MLN RAG
74.60
79.85
81.90
0.84
modified to use only the current frame's input for fair com- parison; and (3) BLIP-RT-2, which fine-tunes BLIP-2 (Li et al., 2023) the DriveLM data and utilizes trajectory tokenization proposed in RT-2 (Brohan et al., 2023).
tion, SafeAuto achieves further reduction of 5.8% in RMSE for speed prediction and 14.1% in RMSE for course predic- tion. The contributions of each component to the overall performance are detailed in Section 5. (b) DriveLM Dataset: The final results are demonstrated in Table 2. For high- level behavior prediction, SafeAuto improves accuracy by 13.00% compared to the SOTA baseline MrAeeLM- For low-level motion prediction, it achieves a further reduc- tion of 44.4% in ADE over the DriveLM-Agent, Notably, the ADE of SafeAuto is even comparable to UniAD (Full) which is trained purely for low-level prediction. Additional case study is provided in Figure 6.
Metries. (a) For the BDD-X dataset, we adopt widely used metrics for high-level prediction, including 4-gram BLEU (B4) (Papineni et al., 2002). METEOR (M) (Banerjee & Lavie, 2005), and CIDEr (C) (Vedantam et al., 2015). For low-level prediction, we use the Root Mean Square Error (RMSE) for both steering angle (in degrees) and speed (in meters per second). We also present "tolerant accuracy" metrics, As, representing the accuracy of predictions when binarized as being within a tolerance threshold â of the ground truth. (b) For the DriveLM dataset, the high-level behavior questions are multiple-choice problems concerning speed and steering. We report the overall accuracy, as well as individual accuracies for speed and steering predictions. For low-level trajectory prediction, we use the Average Displacement Error (ADE), as in UniAD, which indicates the average &g distance between the predicted trajectory and the ground truth trajectory and is calculated as the average of the errors at the 1st, 2nd, and 3rd seconds.
5. Ablation Study
In this section, we conduct various ablation studies on SafeAuto to assess the impact of each module and different hyperparameters, as outlined in Section 3. For elarity, we refer to the base model-trrined without any enhancements described in our paper-as 'Base".
Contribution from Each Module. The contributions of dif- ferent modules to both high-level and low-level performance on the DriveLM dataset are shown in Table 5, while the high- level results for the BDD-X dataset are presented in Table 4. The contributions of each module in SafeAuto to low-level control signal prediction on the BDD-X dataset are deferred to Appendix C. To accurately measure improvements in action performance, we introduce a new metric called high-level action predicate accuracy for the BDD-X dataset, which converts high-level action descriptions to one of 16 predefined actions using GPT4o strictly and measures accu- racy. Our results reveal that: (1) PDCE loss markedly en- hances low-level prediction while preserving high-level pre- diction performance; (2) post-safety verification via MLN
The final results for high-level prediction, including both action and justification, pre- sented in Table 1, while the low-level predictions for speed and course are shown in Table 3. For high-level action pre- diction, SafeAuto improves performance by 1 11.6%, 29.4%, and 15.6% for the BLEU4, CIDEr, and METEOR met- rics, respectively. Although the justification performance is slightly lower than the ofte-offthhear method, it still sig- nificantly outperforms the vanilla ine-tuned Video-LLaVa model, demonstrated in Section 5. Additional case study is provided in Figure 5. For level control signal predic-
Results. (a) BDD-X Dataset:
8

---

SafeAuto: Knowledge-nhanned Safe Autonomous Driving with Multimodal Foundation Models
0.80
-420
Table 6: The impact of incorporating Environmental Predicates (EP) in- formation for retrieval, along with the number of retrieved examples K in Multimodal RAG, on high-level action and justification performance the BDD-X dataset.
Prediction RMSE
4.10
BRRN uogo SRumO
0.70
4.00
Action
Justification
Method
B4
C
M t
Acc
B4 f
Ct
M f
0.65
Base
30.8
221.5
29.2
61.75
7.8
85.4
13.2
RAG w/o EP
29.4
219.2
28.5
59.06
7.3
74.8
12.6
0.60
3,80
RAG w/o EP
29.7
218.6
28.7
59.91
7.3
73.7
12.5
0.15
0.20
0.25
0.30
0.35
0.40
0.45
the for POCE loss
RAG EP
38.1
334.8
35.4
91.47
8.8
89.2
13.5
Figure 3: RMSE variation of low-level speed and course predictions with different PDCE loss a values on the BDD- X dataset. The dashed line represents the result of using the original CE loss.
RAG w/ EP
38.2
334.8
35.3
91.00
9,4
95.5
13.9
still corrects certain unsafe actions, even though the base model is conservative; the impact of each module on the rates of rule violation is detailed in Appendix C.3. (3) the use of Multimodal RAG significantly enhances performance, increasing high-level action predicate accuracy by 30% on BBD-X, and 14% on DriveLM.
dataset, the critical traffic rule Sol SolidRedLighht (x) = -Accelerate (x) A -LeftPass (x) A -Yield and in the DriveLM dataset, the key rule is RedYield serrieldiign (x)
-Fast (x). These rules carry the highest weights in their respective MLNs. A case of MLN correcting aggressive driving behavior is illustrated in Figure 8
PDCE Loss with Different a Values. We investigate the effect of varying a values on low-level predictions in the BDD-X dataset, demonstrated in Figure 3. Our findings show that PDCE loss consistently achieves lower RMSEs for speed and course predictions than the CE loss, with minimal sensitivity to a changes, indicating strong stability.
Limitation
There are some limitations in SafeAuto that could be ad- dressed in future work. These include: (1) using better designed distribution D(u, a) for PDCE loss to enhance performance, (2) improving the effectiveness of safety veri- fication by refining predicate extraction, especially in scenar- ios with limited predicates, and (3) adding further filtering or reranking processes after retrieval in the Multimodal RAG within the MLLM context to enhance accuracy.
Influence of Environmental Predicates on Retrieval. Our approach incorporates explicit Environmental Predicate (EP) information extracted from video and control signals. As in- dicated in Table 6, omitting environmental predicates yields performance akin to the base model, while including them markedly improves high-level prediction performance. This underscores the potential of using explicit binary environ- mental predicates to refine retrieval, eliminating the noise in the original data sources.
A Acknowledgment
This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, NSF AI Institute ACTION No. IIS 2229876, DARPA TIAMAT No. 80321, the National Aeronauties and Space Adminis- tration (NASA) under grant No. SONSSC20M0229, ARL Grant W91 1991INF-23 22-2--112--7 Alfred P. Sloan Fellowship, the research grant from eBay, AI Safety Fund, Virtue AI, and Schmidt Science
Multimodal RAG with Different K. We explore the im- pact of varying top K selections for BDD-X dataset in Ta- ble 6. As we can see, significant improvements in high-level action prediction are achieved even with K 1, and the performance is already comparable to the K = 2 scenario. Furthermore, selecting a larger K value enhances perfor- mance in high-level justification prediction.
Impact Statement
This paper presents work aimed at enhancing the safety and reliability of autonomous driving systems through the inte- gration of multimodal foundation models, a newly designed loss function, structured safety knowledge based traffic rules, and unstructured data from previous driving experi- ences. By improving the ability of autonomous vehicles to reason about complex driving scenarios, provide more accurate control signals, and adhere to traffic regulations, our framework, SafeAuto, has the potential to significantly reduce traffic accidents and enhance overall road safety with autonomous vehicles.
Impact of Predicate Selection. Predicates are crucial for retrieval and vost-verification, shown in an ablation study in Appendix C.2. This study highlights the importance of MLLM action and environmental predicates, demonstrating that certain environmental predicates significantly improve accuracy in driving scenarios when aligned with relevant traffic regulations for predicting lawful high-level actions.
Case Study on Post-Safety Verification. In the BDD-X

---

SafcAuto: Knowledge-nnanneed Safe Autonomous Driving with Multimodal Foundation Models
References
guage models. In The Twelfth International Conference on Learning Representations, 2024b.
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 63-72, 2005.
Jocher, G., Qiu, J., and Chaurasia, A. Ultralytics YOLO, January 2023. URL /ttttpp://github.... ui ultralytics/ caaiice//ltttaan wuttraltitts..s....moooo
Kim, J., Rohrbach, A., Darrell, T., Canny, J., and Akata, Z. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV), pp. 563-578, 2018
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: lisionlaaguuage-attion models transfer web knowledge to robotic control. arXiv preprint 1XXiv23377.18888 2023.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t,, Rocktäschel, T., et al. aetrieval-auggeente generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-94477 2020.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Li, L., Zhang, J., Xie, T., and Li, B. Double sampling randomized smoothing. In International Conference Machine Learning, pp. 13163-13208. PMLR, 2022.
Hu, Y., Yang. J.. Chen, L., Li, K., Sima, C.,, Zhu, X., Chai, cernine- S., Du, S., Lin, T., Wang. W., al. Planning oriented
autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17853-17862, 2023.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representa- tion by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Jensen, M. B., Philipsen, M. P., Møgelmose, A., Moes- lund, T. B., and Trivedi, M. M. Vision for looking at traffic lights: Issues, survey, and perspectives. IEEE transactions on intelligent transportation systems, 17(7): 1800-1815, 2016.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a.
Liu, H. Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023b.
Jiang B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q. Liu, W., Huang, C., and Wang. X. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8350 2023.
Liu, Z., Ning. J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. Video swin transformer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2200-3211, 2022.
Jin, B., Liu, X., Zheng, Y., Li, P., Zhao, H., Zhang, T., Zheng, Y., Zhou, G., and Liu, J. Adapt: Action-awan driving caption transformer. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 7554-7561. IEEE, 2023.
Mao, J., Ye, J., Qian, Y., Pavone, M., and Wang, Y. A language agent for autonomous driving. arXiv preprint arXiv:2311.108/1 2023.
Ni, J., Abrego, G. H., Constant, N., Ma, J., Hall, K., Cer, D., and Yang, Y. Sentence-t5: Scalable sentence encoders from tre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874, 2022.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang. J., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-f., Pan, S., et al. Time-llm: Time se- ries forecasting by reprogramming large language models. In International Conference on Learning Representations, 2024a.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.- Liang, Y., Li, Y.-F., Pan, S., et al. Time- Im:: Time series forecasting by reprogramming large lan-
10

---

SafcAuto: Knowledge-nhannee Safe Autonomous Driving with Multimodal Foundation Models
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Zhang, J., Chen, Z., Zhang. H., Xiao, C., and Li, B. (DiffSmooth): Certifiably robust learning via diffusion models and local smoothing. In 32nd USENIX Secu- rity Symposium (USENIX Security 23), pp. 4787-4804, 2023a.
Richardson, M. and Domingos. P. Markov logic networks. Machine learning, 62:107-136, 2006.
Zhang, J., Li, I.,, Zhang, C., and Li, B. Care: Certifiably robust learning with reasoning via variational inference. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 554-574. IEEE, 2023b.
Semnani, S., Yao, V., Zhang, H., and Lam, M. Wikichat: Stopping the hallucination of large language model chat- bots by few-shot grounding on wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 202.3, pp. 2387-2413, 2023.
Zhang, J., Xu, C., Gai, Y., Lecue, F., Song, D., and Li, B. Knowhalu: Hallucination detection via multi- form knowledge based factual checking. arXiv preprint urXiv:2404.084 2024a.
Sima, C., Renz, K., Chitta, K., Chen, I.,, Zhang, H., Xie, C., Luo, P., Geiger, A., and Li, H. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.44156 2023.
Zhang, J., Xu, C., and Li, B. Chatscene: Knowledge- enabled safety-criticee scenario generation for tonomous vehicles. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15469, 2024b.
Tan, M. Merrill, M. A., Gupta, V., Althoff, T., and Hartvigsen, T. Are language models actually useful for time series forecasting? arXiv preprint arXiv:2406. 16964, 2024.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 2023.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4575, 2015.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., HongFa, W. Pang, Y.. Jiang, W., Zhang, J., Li, Z., et al. Languagebind: Extending lideo-language pretraining to n-modality by language- based semantic aligmment. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., Kautz, J., Li, Y., and Alvarez, J. M. Omnidrive: A holis- tic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01333 2024.
Wang, W., Xie, J., Hu, C., Zou, H., Fan, J., Tong, W., Wen, Y., Wu, S., Deng, H., Li, Z., al. Drivemlm: Align- ing multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.092455 2023.
Xu, Z., Zhang. Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.- Y. K., Li, Z., and Zhao, H. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
Yang, Z.,, Zhao, Z.,, Wang, B., Zhang, J., Li, L.., Pei, H., Karlaš, B., Liu, J., Guo, H., Zhang, C., et al. Improving certified robustness via statistical learning with logical reasoning. Advances in Neural Information Processing Systems, 05:33459-348777 2022.
Yuan, J., Sun, S., Omeiza, D., Zhao, B., Newman, P., Kunze, L., and Gadd, M. Rag-driver: Generalisable driving explanations with retrieval- augmented in-context learning in multi-modal large language model. arXiv preprint arXiv:2402. 10828, 2024.

---

SafcAuto: Knowledge-nbanned Safe Autonomous Driving with Multimodal Foundation Models
A. Details on SafeAuto-Reasoningg
A.1. Traffic Rule Mapping
This section outlines the methodology for extracting first-order logic formulas from the California Driver Handbook 2. Initially, all traffic rules are transformed into a structured format using GPT4o, based on the template: 'When [conditions], you should/should not [action] (unless [conditions])." Subsequently, GPT is utilized again translate the structured traffic rules into first-order logic formulas. The complete set of prompts is provided in Table 7 and Table 8.
egent for autononous driving, your task so rast inet rules fron the provided ay filiwsing out isselevant infornal lun, In speci please extrast rules Eron the Lext relating Lo apecific driving naneuvers listed as follows: keep, accelerate, dacalezace, acos, nake left tuzna, nake right curcs, reverse, nazce, change lanes, park, nake O turns, ovwrtake, yiu'd, follow differens traffic signs. Disregard unselated actions for actoronous driving like "looking around/ checking nistora"
conserning autonoouu driwing, whilh
sinilar non qaaniiiiable action.
Use the atructured format: "Ween (conditiona you should/should not [act (unless io Utilize 'OR' or 'AND" to connect iiliiple conditions that nay Lrigger the sane action. Optionally, Include "unless 'condil conditiona] where except ions apply. Bach rule should direct and apotinable, ensuring
the precise safe execution of self-driving nanauvers. If the does provide relevart advice for the aut ions liated, resgand with 'None'.
Here is examples
ITiclod; bouble Bolid Yellow Lines
Paasaged: bo not pass over double solid yellow lines, Stay to the right of these " iiee unl you are:
In high cocupancy vehicle (BOV) carpoel lane that hax designat and rance un the left.
Instructed constsuction or other aigns drive
the other aide of the road because your aide closed or blocked.
Turning left across a sing'e set of double ye'lou low liee to enter or exit a drivessy or private roed or nake U-turn.
Tho suta of sulid double yellow linex spaced Lwo us nore feel apari are pans sderidered a barsiee, bo not drive on or over Uhis barrimt, make left turn, make U-turn
it, except at designated openings. driving near double solid low lines, you high uccupancy vehicls (BGV) daspuel lane Chat has a designated wil canow on the leftz (il) You are inst ructed by contisuction or other signs to drive on the other side of the road because your side is closed or blooked; 1151) You are turring lett across a single set et double yow Ties to enter or exit drivessy or private rowd, or make J-turn.
Bulesk:
stay the right of
lines unless:
You
then Leo sets of sulid duuble yellow lines apaced
or Seet apazt possent, you should drive or Lhis barsims, make a left torn, or make a U-turn acroes it, unless there is a designated opening for such maneuvers.
New, extract the rules for tollowing passage:
iTitlet; Ititlel
Paasaged: (passage)
#Extracted Rulest:
Table 7: Prompt for converting traffic rules to structured format
Your goal is transform natural languege driving rules into first order logicel rules for autoronous driving systens, start by lcentitying the relevant actions and conditions trem the text, Actione must choose iron presetined precicates like Keep, Accelerate, Decelerate, Stop, MakelleftTorn, MokeRightTurr, Reverse, Merge, Toliefetaee ChangeToRiig gatter, Park, NakelTorn, leftPass, Pasht and Yield.
Pirst, analyse the natural driving rules to idertify clear obl igstions Ireguired act tonst and bisionisions (benned actione), ixplicttly ignoring any aud lons described ibed as condit lonal pernissions ("may*). Rath sule will wither distate requited antions ander specific cenditions explicitly ban sestain actions in detined scenerios. For esch rules
antions
Icontity Required Actions (obligationsl: : It a rele specidies an action that must be taken under cartain conditions, tormlltt this into a logical statemert using the ternat "Condition Action." " Tais represence an ogaigay action,
Isentity Frohibized actions (Bensi: It rule bans cestain actions in specitic circumetances, express this logical statement using the cormat "Cordition Action." This capturee actions that are explicitly torbideen,
Hero exemle::
Natural Ralost:
then driving near double solid yellow lines, you should stay to the right of these lines unless: (1) You are in high-occuparey vehiele (BOV) carporl lane taat has a designated eatrance on the lett; (11) You are instructed by construction oz other
signs drive on the other side of the road becsuse your side is closed blocked; (15t) You turring lett across single set deuble yellow 1iaes to eater or exit a driversy or private road, or nake à J-turn.
whon two sets ot solid douale yellow lines spaced two or nore rest apart ars present, you chould not drive on or ever this barrier, nake a left turn, or make a U-turn across it, urless there is a desigrated opening for such maneuvers. construction ChanesTolettiane LottPess
tooical Ralost: (11 LertSingloRettoiiiis acgccotstaggeseetooo..eo (2) beteeeeeieeeeeeooollllleeee bettboessssettpeeieeellloo
MiooraaroolWithhctttttttt Enterorsxitrivveeyy Ente:orsxittesvaaceooo MaketettTern
perignateddeen:rgieetttt MekeleftTorn cestomatsdopenniiirreen MakelTurn
Now, extract she first-ardar logical rules for the follosing natural ruiss, and label each locical zuic clearly with tlciccll Rulest and include an insox that zorsesponds to the incox of the original ruie as shem in the exampin. Berides when there are only conditioani permiesions i"may"I and no cleat abligations or progibitions, you zan simply cutput None. Natural Ralost: (rules)
Table 8: Prompt for further converting traffic rules to first-order logic formulas
Phttps: htttpp:://www.. wttp:/ddw..cc..o//prrtaallll htwp://ortttt/aaaooooooooo /cal /aa1 1i aoomaaaddiver--aaddaai
12

---

SafcAuto: Knowiedge-Enbanced Safe Autonomous Driving with Multimodal Foundation Models
A.2. YOLOv8 Fine-tuning
We fine-tuned the YOLOv8 model using the LISA dataset (Jensen et al., 2016), which contains annotations for both traffic signs and traffic signals. The dataset comprises four daytime sequences and two nighttime sequences, primarily designated for testing purposes, with a total duration of 23 minutes and 25 seconds of driving footage recorded in Pacific Beach and La Jolla, San Diego. It contains 43, 007 frames, annotated with 113, 888 traffic lights and 7, 855 traffic signs across 6, 610 frames. The YOLOv8m model was fine-tuned over 500 epochs, with an input image resolution of 640 x 640 pixels.
A.3. Predicate Extraction
For environmental predicates, we use the fine-tuned YOLOv8 for detection, as described in Appendix A.2. To ensure consistency with RagDriver (Yuan et al., 2024), we uniformly divide video segments into 8 frames and select the final frame as input. Additionally, in DriveLM, images from three perspectivesthe front camera, left front camera, and right front cameraare utilized. We leveraged the nuScenes map expansion to extract lane line information for both sides of the lane in which the ego vehicle is positioned. Historical control signals in BDD-X and DriveLM were obtained by querying GPT and mapping the results corresponding environmental predicates (e.g., HCSKeep (x)) ). Specific details of the prompts used in this extraction process are provided in Table 9 and Table 10.
Giver the curzent speed, curvsture, acceleration, and course of the car, use one velsaity predicate and ore cirectional. predicate to best deseribe the behavior of the car. The velecity predicates
Keep, Acceicrate, Decelerate, Stop, Reverse.
The cirectional predicates are: Straight, Sett, Right, Octput the predicstes directly without any additional information. aere are come examples:
répeed:: (2,18, 5.76, 4,45, 3,30, 2.24, 1,20, 0.361
tcurvaturets [1.32, 0,38, 0.5e, 1.85, 2.76, 1.61, 0.64]
tcccelerationt: [-1.22, -1.85, -2.39, -2.22, -2.01, -1,46, -0.07]
sCoursef: [0.00, -13.03, -8.33, -3.23, -0.97, -0.32, -0.c8)
iFrecicate:: #CSstep, acsieft
rspeedt: (12.31, 9.51, 7,24, 5.30, 3.61, 2.76, 3.001
ICurvacured: [-0.00, 0.co, 0.00, -0.05, -0.18, -0.57, -0.791
naccierationt: [-1,85, -2,79, -2.73, -2.23, -1.67, -0.07, 0.71)
rCourset: 10.00, 0,30, 0,00, 0.00, -20,26, -60.10, 7.17)
iEsodicator: HCSDocelarato, #CsRight
rSpecdt: (1.27, 4.19, 6,83, 8,87, 19.48, 12.22, 14.05]
(eurvacured: 10,20, 0.00, 0.00, -0.30, -0.01, -0.00, -0.c0)
Acccsessttonf:: [2.27, 2.15, 1.81, 1.35, 1.28, 1.36, 1.45]
#Couzsef: [0.00, -0.09, 0.00, 0.00, 0.20, 0.00, 0.00]
Frodicater; HCtkecetarate, Bosstraight
ESpocdi: (spead]
cCurvsturo:: (curvature)
Mocolesaaionna [scceleration]
#coursef: (course)
Table 9: Prompt for Extracting High-level Control Signal Environmental Predicates from the BDD-X Dataset
Giver the curzent speed and of the cas, velocity prodicato and disectienal prodicate best describe behavior she car, The vel ool predi area Nornel, Fast, Slox, Stop.
The circational predicates are: Straight, left, Right. Output the predicates directly without additional intormation. are exempless
rSpeedt: (64.54, 0.01, 15.36, 0.01, 15.57, 0.01, (5.7, 0.01, 16.46, 0.01, (6.63, 0.003
#Coursef: (11,0, 0,31, 11.0, 0.0), C.0,, 0.01, 11,0, 0.0), (1.0, 0.03, (1.0, 0.011 Fredicator:: aCSFest, acsstraight
repecdt: ((10.01, 0.0), (9.88, 0.01, (9.52, 0.01, 19.39, 0.0), 19.25, 0.0), (8.91, 0.011
iCourset: 110.84, 0.03, (0.84, 0.0, 10.06, 0.01, 10.89, 0.0), 10.93, 0.00, (0.95, 0.011
IErodicater: #CsFest, acssight répeest: (12.51, 0.31, 12.49, 0.01, 12.45, 0.01, 12,43, 0,0), (2.43, 0.31, 12,37, 0.011
eCoursef; 110.85, 0.0), 10.85, 0.01, 10.06, 0.01, 10.85, 0.01, 10.82, 0.0), 10, 0.011
|Fresieater: SCsslexly, aesteft rspectt:: ((1.65, 0.21, 11.37 0.01, 10,ra 0.03, 10.09, t,0,, 10.0, 0.05, 16,0, 0.01, 10.0, 0.00, (0.0, 0.0:1
icoursef: [10.86, 0.0), (o.85, 0.0), (0.87, 0.01, 10.86, 0.c), 10.86, 0.0), (0.86, 0.0), (0.85, 0.01, 10.84, 0.0)1
rEsodicater: #Csstep, acsatraight
rSpeedt:: (speed)
scoursef: (courso)
rErodicator:
Table 10: Prompt for Extracting High-level Control Signal Environmental Predicates from the DriveLM Dataset
With respect to MLLM action predicates, since the output of MLLM consists of high-level action descriptions such as "The car is slowing down to stop, we map these to predicates represented as (MLLMDece lerate (x), MLLMSt (x))
13

---

SafcAuto: Knowledge-nnhaned Safe Autonomous Driving with Multimodal Foundation Models
Giver the cuzzent behavior of tha car, please use predicates belox to best describe the behavior of the caz. The predicates aze: Neep, Accelarate, bepelerate, Seop, Reverse, Turnleft, Turnkiggh,, UTarn, Merge, leftPass, Right Pans, ChengeTolefttang ChaiggeoRRight Lane, Pack, PullOves.
Haze are sone exanples:
eCurtent Behavior4; The car is trevelling down the road.
Eredicalest:: Keep
#Current Rehaviort: The ear is nsking left turn.
Predicalass: Turnileft
Currentt Behavior4; The cas is sloving down and Chen conea Lo stop.
#recicateett
Stop
The eat is acoelerating and then Lurns right. iF Peedicatest: Accelerace, PurnRight Correent Rehaviorr: The car is neeing left turn and accel brateer Peddioalest: Turslefl, Actulerate
CCrroent Behavlort:
Furrenn Behaviort: The car decelerates and stops. o Predicatess: becelerate, Stop
cusrent sehavior car described, provide predicates thst best describe the benevior cas.
Courswal Behavior4; (action)
#recicatest::
Table 11: Prompt for Extracting Environmental Predicates from the BDD-X Dataset
In the BDD-X dataset, due to the increased number and complexity of high-level action descriptions for MLLM action predicates, we employ GPT4o with specifically designed prompts to extract these predicates, with the detailed prompts provided in Table 11. In DriveLM, given that the question-ande and-answwrr format comprises multiple-choie questions with fixed option descriptions, we predefine mapping rules to translate high-level action descriptions into predicates, as described in Table 12.
Table 12: Mapping of High-level Action Descriptions to MLLM Action Predicates.
High-level Action Description
MLLM Action Predicate
Going straight
Slightly steering to the left Slightly steering the right
MaiMstraight (x)
Driving fast
MLLMF (x)
Driving very fast
Driving slowly
MLLMSloW (x)
Driving with normal speed
MLLMNormal (x)
Not moving
MTLMStop (x)
Steering to the left
MLLMLeft (x)
Steering to the right
MLLMRight (x)
A.4. Training Details
The learning rate for the Markov Logic Network (MLN) is set at 1 x 10-5, To mitigate the risk of overfitting and to avoid excessive reliance on frequently occurring scenarios, such as straight movements, regularization is incorporated into the training process, also at 1 10-5, The models are trained for a total of 300 epochs, unless interrupted by predefined early stopping criterion: specifically, if the model's accuracy fails to improve by more than x 10 over 10 consecutive epochs, training will be terminated.
A.5. Post-verification Details
As outlined in Section 3.2, during safety verification, we initiate the process by extracting observed grounded environmental predicates and MILM action predicates using the object detector and GPT4o. If the final main action predicate output of the Markov Logic Network (MLN) conflicts with the suggested action from MLLM, we modify the high-level action query based on the output of the MLN. In the BDD-X dataset, we replace the original high-level action queries with new actions inferred from the MLN. For example, if the MLN predicts the possible world represented as "Stop (x) = 1" with the highest probability, we append the suggestion "The ego vehicle should stop" to the high-level action query. This approach
14

---

SafcAuto: Knowledge-nbbnnee Safe Autonomous Driving with Multimodal Foundation Models
str_num: a string representing float number (excluding the decimal point '.') with N digits
logits: the logits distribution output from MLLM for each digit in str_num, with shape of N 10
sigma: the standard deviation of the Gaussian distribution
Precompute the digit-level probabil lity distributions from scipy.stats import norm
distribution_dict - {
for num in rang
prob_distribution np. array([normm(num sigma) cdf + 0.5)
norm(num, sigma) cdf(i 0.5) for i in range(10)])
rrob_distribution /- prob_di sttodiiitributtoo....u
distrrbuttion_dcc((ssrrmmm))) = grob_distribution
calculate weights for each digit position
tgt_probs []
weight 1.0
for digit in str_num:
The place-level weighting
digit_probs = distributioon_dice(ciitt weight
weight == digit__orobs(iniiiiiittt igg_proos.apenncciiit..rroo
tgt_probs np.arrayyttt__pobss
Compute the KL loss, constants are ignored
loss (tgt_probs 1og_softtmax(loiits, axis=1)) summ))
Figure 4: The numpy-style pseudocode on PDCE loss.
facilitates the mapping back to the corresponding high-level action description and ensures the flow of conversation for subsequent queries.
In DriveLM,, since high-level action queries are presented a multiple-choice format, the final main action predicate output from the Markov Logic Network (MLN) may not always align directly with one the options. In such cases, we filter the available options by the probability possible worlds. Given that MLLM action predicates may map multiple high-level action descriptions, it is feasible for multiple valid options to arise simultaneously. We then overwrite the high-level action queries by removing incorrect options and prompt the MLLM to regenerate an option.
A.6. Predicates and Traffic Rules
This section provides detailed overview of the specific aspects of the MLN construction for both the BDD-X and DriveLM datasets. Table 16, Table 17, Table 18, present the predicate of BDD-X, all possible worlds, and first-order predicate logic, respectively; while Table 19, Table 20, and Table 21 show those of DriveLM.
B. Pseudo-Code of PDCE loss
The pseudo-code for calculating the PDCE loss provided in Figure 4.
C. Experiements on Ablation Study
C.1. Contribution from different module
Table 13 presents an ablation study evaluating the contribution of each module in SafeAuto to the low-level control signal prediction on the BDD-X dataset. Interestingly, we find that the MLN reasoning and RAG modules have only a minimal impact on the low-level prediction accuracy, with the primary improvement stemming from the PDCE loss, as expected. Additionally, we observe that incorporating RAG slightly increases the RMSE for speed prediction but decreases the RMSE for course prediction.
15

---

SafcAuto: Knowledge-nbanneed Safe Autonomous Driving with Multimodal Foundation Models
Table 13: Ablation study of the contribution from each module in SafeAuto focusing on low-level control signal assessment on the
DDD-X dataset.
Speed
Course
Method
RMSE
Ag.1 T
Ap.s 1
A1.0
As.o 1
A10.0 1
RMSE 4
Ao.1
Ao.5 f
An.o 1
A5.o 1
A10.0 1
Base
0.76
53.65
87.38
95.10
99.76
99.81
4.18
76.31
89.87
94.49
98.21
99.15
PDCE
0.63
55.63
88.04
95.24
99.86
99.91
3.89
76.64
89.97
94.35
98.21
99.20
PDCE4MLN
55.58
87.99
95.24
99.81
99.91
3.89
76.68
90.01
94.35
98.21
99.20
PDCE+RAG
0.65
$5.49
88.79
95.34
99.81
99.91
3.85
76.31
89.68
94.07
98.30
99.25
PDCE+MIN++RAG
0.65
55.49
88.84
95.34
99.81
99.91
3.85
76.26
89.68
94.11
98.30
99.25
C.2. Predicate Selection
Table 14 indicates that incorporating MLLM action predicates significantly enhances SafeAuto's effectiveness. Subsequently, we ranked all environmental predicates based on their total occurrence frequency across all scenarios. For each experiment, we selected the top n (n 5, 10, 15, 20) most frequently occurring environmental predicates, retrained the Markov Logic Network, and evaluated its performance. Notably, selecting only the top five environmental predicates achieved relatively high accuracy, suggesting that the majority erroncous scenarios are associated with these predicates, such as Sol id
Table 14: Ablation study the impact of different predicate selections SafeAuto performance.
BDDX
DriveLM
Method
1
Speed f
Steer
Avg.
Acc f
Speed 1
Steer
Avg. f
MILM Action Predicates All Environmental Predicates
92.18
79.85
81,90
84.65
74.60
79.85
81.90
78.12
All Environmental Predicates
49.88
49.49
46.28
48.55
38.83
49.49
46.28
44.20
MLLM Action Predicates & Top 5 Envirommental Predicates
87.75
79.27
81.75
82.26
74.16
79.27
81.75
78.39
MILMM Action Predicates & Top 10 Environmental Predicates
92.10
79.56
81.75
84.47
74.30
79.56
81.75
78.54
MLLM Action Predicates Top Environmental Predicates
92.18
79.42
81.90
84.50
74.31
79.42
81.90
78.54
C.3. Rule Violation
Table 15 presents the impact of PDCE, RAG, and MINN on the violation of traffic rules SafeAuto's action prediction. On the BDD-X dataset, PDCE and RAG significantly reduce the violation rate in the underlying MLLM's decision-making. The MLN post-verification further decreases the violation rate of SafeAuto. However, on the DriveLM dataset, PDCE and RAG do not reduce the violation rate of MLLM's prediction, DriveLM contains large number of simple driving scenes with only straight-line movement, making it challenging for MILMM to effectively learn different driving patterns. Nevertheless, using MLN to correct MLLM's errors reduces the violation rate.
Table 15: Ablation study on the impact of each module on the traffic rule violation rate of MLLM-predicter actions.
Method
BDDX
DriveLM
Base
11.64%
1.03%
PDCE
8.44%
1.66%
RAG+PDCE
5.90%
1.03%
RAG+PDCE-MLN
4.50%
0.75%
D. Case Study
D.I. High-Level Action Query
The two examples in Figure use the base model and SafeAuto, respectively, to predict high-level actions. The actions predicted by SafeAuto are closer the ground truth.
16

---

SafcAuto: Knowledge-nbanneed Safe Autonomous Driving with Multimodal Foundation Models
Multimodal Input: Video, text
High-Level Action Query: What is the action of ego car?
Video: <video>
VideoLLaVA-78 (Base)
SafeAuto
Speed: (0.56,0.73,.. ..
Curvoture: [0.0, 0.0, ..
Video
RAG
MLN
Acceleration: [0.19, ...]
Control Signel
Course: (0.0, 0.0, .J
Environment
Pedestrion Crossing; Yield Sign
Predicates
Ground Truth: The car is moving slowly
The car accelerates
The car slows down
Multimodal Input: Video, text
Video: <video>
RAG
Speed: (4.45,4.47, -]
Curvature: [0.0, -0.28,.. ..]
Signal
Acceleration: (8.78, ...
Course: [0.0, 4.22, ..
Environment Predicates
tuin Crossing; Solid Green Light
The car is turning left
The car is turning right
Ground Truth: The car is steering to the right
Figure Case Study on High-Level Action Queries.
D.2. Low-Level Action Query
Figure shows that the low-level control signals obtained through SafeAuto for low-level action queries numerically closer to the ground truth. Moreover, the predicted control signals highly correlated with the previous high-level queries.
D.3. Token Probability Distribution in Control Signal Prediction
Figure illustrates the probability distribution of numerical tokens when predicting low-level control signals using the base model and SafeAuto.
D.4. MINN Post- Vost-Verification
Figure illustrates how the MLN rejects dangerous and illegal actions predicted by the MLLM and enhances safety by recommending high-level actions for the MILM to re-output.
17

---

SafcAuto:
Knowledge-Enbaneed Safe Autonomous Driving with Multimodal Foundation Models
Muitimodal Input:
Video, text
Multimodal Input: Video, text
Video: cvidear-
Speed: 23,6.56, 5.63, 6.78, 7.1, 6.6,, 10.24] Curvoture: 0.02, -0.01, -0.07, 0.0) Acroleration: [0.27, 0029, 0. 0,13, 0,65, 4,14,, 0,22,157]
Speed: (11.82,700 10.25, 8 78, 7.34, 6.62, 6.57, 7.92] Corvature: [0.0, -0.26, +0.01, -0.02, -0.18, 0.23, 0.0]
Arzelevation: 1.28, 0.94, 1.91]
Course: (0.0, 0.56, 0.45, 0.45, 0.22, 0.18)
Course: (0.0, 0.18, 0.32, 0.28, 2.09, 1.33, 0.0j
High-Level Action Query: What is the action of ego car?
High-Level Action Query: What is the action of ego car?
VideoLLoVA-75 (Base)
SafeAuto
VideoLLoVA-78 (Base)
SafeAuto
The car moves forward
The car is driving forward
The driving forward
The car is merging into the right lane
then comes to stop
Ground Truth: The car is travelling down the road
Ground Truth: The car slowly angling right on the highway
High-Level Justification Query: Why does the car do this?
High-Level Justification Query: Why does the ego do this
Because the light is red
Because the road clear
Because are
Because it is free traffic
nearby cars
Ground Truth: No obstructions ahead
Ground Truth: The trying get different
exit needs let pass first
Low-Level Action Query: Predict the control signal for next frame
Speed: 1.69 Course: 0.0
Speed: 12.330 00.000
Speed: Coure: 0.0
Speed: 09.610 Course: 00.000
Ground Truth:
Ground Truth:
Speed: 12.8 0.0
Numerical closer
Speed: 10.750 Course: 0.05 Numerical closer
Figure 6: Case Study on Low-Level Action Queries.
Multimodal Input: Video,
Multimodal Input: Video,
Video: evideo>
Video: <video>>
Speed: [15 80, 25, 29,29, Curvature: (00.1,, 0.00,
Speed: [11.87, 12.22, ..] Eurvature: [-0.00
Acceleration: (-1.16, 0.28..] (0.0) O.0.
Asceleration: (-0.12, 0.28... [0.0,
①
Highh-Lavel Action Query
High-Level Query High Level Justification
High-LLval Justification Quary
Low-Level Action Query: Predict the control signal for next frame
VideoLLaVA--2 (Bose)
Speed: 17.05 Course:8 0.
Videol inVA- 78 (Rose)
Speed: Course:8.0
The Meighted Target Tokou Arahmilltt for fLort nuther speed "37.8" 87"
The Mrighted Tonget Taboo Frobability for Floot number Spood "10.21"
SofeAuto Speed:15.66
SafeAuto
Speed:10. Course:0.
The Weightee Tukeu Probabilityy for flout number
"15.63"
1P6 wiightad ranget token probobiL Cty for Float suebon speed "Id.22"
Ground Truth: Speed:
Ground Truth: Speed: & 10 Course: 0.0
Figure 7: Case Study Token Probability Distribution.
18

---

SafeAuto: Knowledge-nhanned Safe Autonomous Driving with Multimodal Foundation Models
1. High-Level Action Queries
What is the action of car?
The accelerates
Post-Safety Verification Predicate Extraction
extract
Environmental Predicates:
MLLM Action Predtcate:
extroct
Sol Soliddddighhtt))111,,, StapSign(x)-B,
MLLNKccellenstexx-11,, MLLMMeep(xx==)
YOLOv8
GPT40
MLN
Weight
ForsulaUknowwedge Rules)
18.82
SolidRedLight acceleratee-lasttass---lll
8.83
Stop5ign(x) Stop(x) Decelerate(x) -Pulll0ve(x)
8.47
NoLeftTurnšignte => -TurnLef((x)
8.77
MLLMAccelerate(x) Accelerate(x)
Inference
Observed Action Predicates Accelerate(x) Stop(x)
Probability
0.02 Suggestion ego should stop
Re- Re-prompt
The ego vehicle should stop.
The car slows to stop What is the action of ego car?
overwrite
8
2. High-Level Justification Queries
op
Why does the car doing this?
Because the light is
3. Low-Level Action Queries
Predict the control signal for frame.
Speed: 02.510 Course: 00.000
Figure 8: An example of rejecting and correcting aggressive behavior through MLN
Predicates
Unobserved Predicates:
Keep (x), Accelerate (x), Decelerate (x), Step (x), Reverse (x), TurnLeft (x), TurnRight (x),
UTurn (x), Merge (x), LeftPass (x),
RightPass (x), vieid(x), Change (x),
ChangeToRi gat (x), Park (x), Pul 10ver (x)
Observed Predicates:
MLLM Action Predicates:
MI.LMKeep (x), ALiMAAccelerateo (x), Mi1MDecelerate( MLLMStop (x), MLLMReverse (x),
MIL.MTornliet: (x), MiJMTurnRight (x), MLLMITurn (x), MLL.Merge (x), MLLMMefftPss (x),
ght.iniggtPssss (x), MLLMYieTd MiJlCChangeVVolattiians (x), MulMChangeeToRRighhtlaaaa
Mi.IMPark (x), MI.LMPu11Qver (x)
Environmental Predicates:
Sol ioliiRedliigh ght (x), Sol SolidVellowi.ihtt (x), Ye! VellowileftArrowiighh (x), RedleeftArrowlightt (x),
MerginggTrafficiigg (x), NoleftTornSi (x), NoRight (x), PedCrossingSigg (x),
St.osSign (x),
RedYieldSige (x), SlowSign (x), SolidGreenlight (x), HCSKeep (x),
#CSAccelerate (x), #CSDecelerate (x), HCSStop (x), HCSReverse (x), #CSStraight (x).
#CSLeft (x), HCSRight (x)
Table 16: The predicate set of BDD-X.
19

---

SafcAuto: Knobaeege-Ebaaneed Safe Autonomous Driving with Multimodal Foundation Models
Traffic Rules
idRedlightt (x) = ~Accelerate (x) A AlleftPass (-Yield(x)
SelidVeLiowtight (x) = Turnieft (x) TurnRight (x) Keep (x) Stop (x) V Deceierate
Acceeiraate (x)
VeLLcwleftArrowwighh (x) = Stop (x) V Decelerate (x)
RedleftAtrrowwigght (x)
-
(Turnieft (x) V UTurn (x))
Merginggrrafficsign (x)
= Decelerate (x)
NoleffTurnSign (x) = -TurnLeft (x)
NoRightTurnSigh (x)
= -TurnRight (x)
RedYieidSign (x)
Decelerate (x)
SlowSign (x)
= AAceelerate
StopSign (x) = Stop (x) V Decelerate (x) A -PullOver (x)
HCSKeep (x) = Keep (x) V Accelerate
ECSAcceierate (x) + Keep (x) V Accelerate (x)
ECSDecelerate (x)
Decelerate (x) V stop (x)
HCSStop = becelerate (x) Stop (x)
ACSReverse (x) = Reverse (x)
HCSLeft(x) (x) = Turnieft (x) V Change)lolefftiiamee.. (x)
HCSRight (x) = TuznRight (x) V ChangeToRightiane (x)
ECSLeft (x) A MGJNChaangTTRigghtaana (x) ChangeToieftIane (x)
HCSRight (x) MiiMChanngToLeetLann (x) = ChangeToRighttann (x)
MLLMMeep (x) + Keep (x)
MLLMAcceierate = Accelerate (x)
MLLMDDccieratee (x)
Decolcrate (x)
MLLMStcp + Stop (x)
MLLMReverse (x) = Reverse (x)
MLLMMurnieft (x)
= Turnieft (x)
MLIMMurnRight (x)
TurnRight (x)
MILMUTurN (x)
UTurn (x)
ML.LMerrg (x)
Merge (x)
M.i.MeftPass (x)
Past (x)
M.L.MiightPas (x) RightPass (x)
MLLMYieid (x)
SLLLChanngeToeetttaaee
ChangeToleftiane
MiLMChanngeooniigtttaaeee (x) = ChangeToRightien (x)
MLIMPark (x) = Park (x)
ALLMPullOver (x) = PullOver (x)
Table 17: First-order logic formulas of BDD-X.
Possible Worlds
(Keep), (Accelerate), (Decelerate). (Stop). (Turnefft). (TurRRight) (UTurn), (PullOver). (Reverse). (Park), (Merge), (LefiPass), (RightPass), (ChangTToLefiLane), (ChangeToRighLLane) (Yield), (ChangeToRighaLane, Merge), (Accelerate. ChangeToRighaLanee) (Decelerate. Stop). (Keep. Stop), (Accelerate, Keep), (Merge. Stop), (Accelerate, LeftPuss). (ChangeToLefiLane, Merge), (Stop. Yield), (Accelerate, TurnRight), (Decelerate, Keep), (Decelerate, PullOver), (ChangeTooLefiLann, PullOver), ( (Keep, TurnRight), (PullOver, Stop), (Park, Stop), (Decelerate, TurnRight), (Keep. LeftPass), (Accelerate, ShanneTTLLefi.coo (Accel- erate, TurnLeft). (Accelerate, Stop), (Keep, TurnLeft), (Accelerate, Merge), (Decelerate, TurnLeft), (Park, PullOver), (Keep, Merge), (Keep, Park), (TurnLeft. TurnRight), (TurnLeft, Reverse), (TurnRight, Stop), (ChangeTTLefitaeee Decelerate), ChaangTToiggaLaane Decelerate), (TurnLeft. Stop), (TurnRight, Park), (ChangeTool ToaeeteTTeeLLenee ChangeToRiihal (Keep. RightPass), (ChngeTToLefiLan Stop), (Keep, PullOver), (LeftPass, RightPass), (ChangeToRighis ToRightLane, Keep), (TurnRight, PullOver), (ChangeToLegti Keep), (TurnRight, Reverse), (PullOver, Reverse), (ChangeToRight Yield). CChaneTToRighLLane PullOver), (TurnLefi. PullOver), (Decelerate, TurnLefi, Stop), (Decelerate, Merge, Stop), (Decelerate, PullOver, Stop), "ChangTToRighuLane Decelerate. Stop), (ChangeToLeftLane, Decelerate, Stop), (Decelerate, TurnRight, Stop), (Accelerate, Change Tol eftl ane, ToRightl ane), ChangeTToigght aiee, ane, Decelerate, Merge), (ChangeToRighht ane, Decelerate, Merge, Stop)
Stop),
TurnLeft), (Accelerate, Decelerate), (TurnRight, Yield), (Decelerate,
Table 18: The possible worlds of BDD-X.
20

---

SafcAuto:
Knobneege-Ebbaneed Safe Autonomous Driving with Multimodal Foundation Models
Predicates
Unobserved Predicates:
Normai (x), Fast (x), Slow (x), stop (x), Left (x), Right (x), Straight (x)
Observed Predicates:
MILM Action Predicates:
MLLMNormal (x), MLLMFast (x),
MLLMSIOW (x), MLLMStop (x), MLLMLeft (x),
MLLMRight (x),
MLLMStraight (x)
Environmental Predicates:
So1 ColidReLLight (x),
SolidYelloxtigh (x),
YelLovlettArrowiight (x),
RedleeftArrowwighht (x), Mergingtrafficc (x), NoiieftuunSign (x),
NoRightTurnSignn (x),
(ecdcrossiingiiinnnx),, (x), StopSign (x), RedYieldSign (x), SlowSign (x),
SolidGreenLight (x),
hoobbeellshhcchhhttcc...hieetett (x), (x),
hrg.apaeeecaaaaaaaaaaaaaaaaaaaar htt..aaaesaallllliteeeeeeeeeeeee (x))
(x),
(x),
(x),
shtg:jaee.eggggiiii.ccciioongoga (x), http::/www..aaaaa.o...........aa
(x),
Singleefolickeelloo..nnoiggg.gg (x), HCSNormal (x), HCSFast (x), HCSSlow (x),
HCSStop (x),
#CSLeft (x), Hht (x), #Csstraight (x)
Table 19; The predicate set of DriveLM.
Traffic Rules
dRReLLight (x) = Fast
Sel itt (x) = -Fast (x)
VellowIeftArrowlighht (x) = Stop (x) V Slow (x)
RedleftArroowiight (x) = - Left (x)
Merginggrrafffcsiim (x)
Fast (x)
NoIcftTurnSign (x)
-Left (x)
NoRight (x) = -Right (x)
RedrieidSign (x) = -astt (x)
81 SlowSign = -ast (x)
(x)
=
(x)
Singjessii.@hhhtee.iiieeeihhhh (x)
= -Right (x)
Doubleesci.dlhhiteeiineeeeett (x) = Left (x)
DoubleesciiddMiittellleeeetittie (x)
= RRight (x)
Siaggleiiigaaphhhhttaaalhheeeeee (x)
SSop
siagleeiiggagghh.ccc..eeeeee (x)
= -stp (x)
BCSNormai (x) = Normal (x)
#CSFast (x) = Fast (x)
RCSSIcw (x) = Slow (x)
RCSStop (x)
Stop (x)
RCSLeft (x) = Left (x)
SCSRight (x) + Right (x)
raigttaight (x)
> Straight (x)
MILMNormai (x)
Normai (x)
ML.LMPast (x) = Past (x)
MLLMS1OW ow (x) = Slow (x)
oP (x) + Stop (x)
MLLMLeft (x) = Left (x)
MLLMRight (x)
Right (x)
MLIMstraight (x) Straight (x)
Table 20: First-order logic formulas of DriveLM.
Possible Worlds
(Normal, Left), (Normal, Right), (Normal, Straight), (Fast, Left), (Fast, Right), (Fast, Straight), (Slow, Left), (Slow, Right), (Slow, Straight), (Stop, Left), (Stop, Right), (Stop, Straight),
Table 21: The possible worlds of DriveLM.
21


Reviews:
[{"review_id": "Evr5wJj6wp", "reviewer": "Reviewer_X7JM", "summary": "SafeAuto proposes a unified framework to enhance autonomous driving systems by leveraging multimodal foundation models. It integrates three core components:\n\nPosition-Dependent Cross-Entropy (PDCE) Loss: An adaptation of the standard cross-entropy loss that incorporates digit-level proximity and place-level weighting, making it behave more like MSE for low-level numerical predictions.\n\nKnowledge-Enhanced Post-Safety Verification: Utilizes a Markov Logic Network (MLN) to explicitly encode traffic rules and safety constraints, serving as a verification layer that can override unsafe high-level action predictions from the multimodal language model.\n\nMultimodal Retrieval-Augmented Generation (RAG): Combines video data, control signals, and environmental predicates into a unified embedding space to retrieve similar driving experiences and inform both high-level and low-level decision making.\n\nThe proposed methods are evaluated on datasets BDD-X and DriveLM, which shows improvements in both low-level control signal accuracy (e.g., reductions in RMSE for speed and course) and high-level behavior prediction (except justification).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to the reviewer for their insightful and thorough feedback, and we appreciate the recognition of our work's contribution! The suggestions and comments made for our work have significantly helped to improve its quality.\n\n> **Q1: Could the authors provide more details on the inference efficiency of the proposed (M)LLM-based method?**\n\nThanks for raising this insightful question! We employ KV-caching to accelerate the MLLM’s inference during multi-turn conversations, enabling faster and more efficient responses. Using a single NVIDIA A6000 GPU in a standard academic setting, we recorded an average inference time of approximately 2.33 seconds per case with the full pipeline (i.e., including RAG, covering both high-level and low-level predictions with Safe-Reasoning) on the BDD-X dataset, and 3.50 seconds per case on the DriveLM dataset. In our practical industry deployments, the inference speed can be improved by 2–3 times. We will incorporate more detailed results on the inference efficiency into our final version.\n\n> **Q2: Why are the justification scores lower compared to baselines?**\n\nThanks for your insightful observation. While our justification score appears lower relative to some baselines, it's substantially improved when compared to the vanilla base model without any components (see Table 11 on Page 15). While for other baselines, ADAPT is not an LLM-based method and independently predicts justifications separate from low-level actions. Compared to DriveGPT, our justification scores are comparable, but we significantly outperform it in high-level action prediction accuracy. Regarding RAGDriver, it uses both train and test data for training RAG, which we believe introduces evaluation bias; while our experiments adhere strictly to using only the training data. Notably, when adopting the setting of RAGDriver, actually we will have 50% further improvement in justification scores, but we don’t think it is a correct setting for evaluation.\n\n> **Q3: Could the authors provide more details on how the RAG dataset was constructed and clarify its similarity to the test data, given its significant impact on performance?**\n\nThanks for your thoughtful question! Our RAG dataset is built exclusively from training data without incorporating any information from the test set. Upon manual inspection, we observed they are not extremely similar, suggesting the model generalizes effectively rather than relying on direct memorization. The superior performance of SafeAuto over RAGDriver also highlights the efficiency and generalizability of our RAG design.\n\n> **Q4: Essential References Not Discussed**\n\nThanks for the helpful references! We have now cited these works and expanded the related discussion in our new version accordingly.\n\n> **Q5: Could the authors provide insights into the sensitivity of performance to hyperparameters in the PDCE loss and multimodal RAG component?**\n\nThank you for highlighting this important point, and we apologize for the confusion. We indeed conducted sensitivity analyses for key hyperparameters, but the corresponding results are deferred to Appendix C. Specifically, Figure 4 on Page 15 shows our ablation study on the weighting factor $\\sigma$. The results indicate that our PDCE loss could still consistently outperform the CE loss across a wide range of different $\\sigma$ values, demonstrating robustness to hyperparameter selection.\n\n> **Q6: Could the authors discuss potential vulnerabilities arising from reliance on external modules such as pretrained detectors (YOLOv8) and text encoders?**\n\nThanks for your insightful observation! While we acknowledge that external modules like YOLOv8 and pretrained text encoders can introduce errors—a common limitation shared by related works (e.g., AgentDriver). However, as we incorporate post-hoc safety verification through our SafeAuto-Reasoning component, it can correct certain errors originating from these modules and thus enhance driving stability. The substantial overall improvements obtained by leveraging these external modules also indicate that they provide more benefits than drawbacks.\n\n> **Q7: Limited Domain Evaluation.**\n\nThank you for the valuable suggestion! We fully agree that evaluating our framework in more diverse driving scenarios would further demonstrate its generalizability and effectiveness. Currently, however, publicly available multimodal autonomous driving datasets are limited and BDD-X and DriveLM are the main public multimodal benchmarks providing both high-level and low-level: as for other datasets, nuScenes lacks high-level action or reasoning annotations, while Waymo does not provide detailed image/ video data. Although richer private datasets exist within industry, these typically cannot be publicly shared due to policy constraints. We sincerely hope future availability of diverse multimodal datasets will enable broader evaluation of methods like ours.", "claims_and_evidence": "The paper claims that by integrating a modified loss function, explicit safety verification via MLN, and a multimodal retrieval mechanism, SafeAuto significantly improves both low-level control precision and high-level decision-making in autonomous driving.\n\nThe work is generally well-motivated and the PDCE loss is especially targeted to improve the numeric prediction ability for autonomous driving tasks. The evaluations also reflect the effectiveness of this design - the prediction accuracy increases (especially the low-level motion/control signal prediction). The MLN is intended to improve the safety and the RAG is meant to enhance the reasoning. The ablation study also shows how different modules contribute to the final improvement. However, in the experiments, it would be great if the authors can provide more detailed results on safety-critical scenarios (besides the rule-following). Besides, the justifications scores on the BDD-X data is behind all baselines and it is worth explaining and further investigation. How the RAG datasets are built and how feasible the RAG is are still unclear, especially given the significant improvement brought by the RAG module.", "methods_and_evaluation_criteria": "The authors provide extensive results on two major multi-modal driving dataset BDD-x and DriveLM and they also presents the results on both high-level and low-level prediction/decision-making, which support most of their claimed contribution. \n\nGenerally one concern of the (m)LLM-based is how efficient the inference process is - although the paper mentions the inference is computationally efficiently. It would be great if the author can provide more details on that.", "theoretical_claims": "The theoretical contribution mainly lie on the design of PDCE loss, which makes sense to me. I believe it helps mitigate the LLM' intrinsic issue of poor numeric reliability. \n\nThe theoretical framework for using MLNs to encode traffic rules is solid, but its practical performance depends on accurate predicate extraction and the proper setting of rule weights. The paper could delve deeper into how uncertainties in rule extraction might affect inference in the MLN.", "experimental_designs_or_analyses": "As mentioned above, the paper provide extensive results/ablation study on both high-level and low level motion prediction/decision-making, which demonstrates the effectiveness of the proposed methods. \n\nHowever, there are still some concerns regarding the evaluation: \n\n1) the justification score falls behind all baselines and I didn't find convincing explanation or ablation for that - the explainability is one of the key motivation why we use the (M)LLM for such safety-critical tasks. If we only cares about the motion prediction/action decision-making, LLM-based methods may not be necessary or efficient.  \n\n2) If we look into the detailed ablation study, we find for most metrics the RAG brings most improvement, especially for the high-level predictions. It is unclear how the RAG dataset is built and how similar they are to the actual test data? This can be critical to understand the feasibility and generalizability of the proposed methods.", "relation_to_broader_scientific_literature": "The paper is based on the recent advance of (M)LLMs and a series of pioneering works on utilizing LLMs to autonomous driving tasks. They focus on improving loss design and enhancing the safety of such syetems.", "essential_references_not_discussed": "The paper includes the recent advances in MLLMs for autonomous driving and the safety guarantees for such systems. However, I also encourage the authors to include the works on enhancing safety for general LLM-based autonomous systems because the system-level design of MLLM-based and general LLM-based AV systems are similar. For instance, the work 'DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models' (ICLR 24) utilize a similar memory module to enhance the performance and reasoning ability. The works 'Empowering Autonomous Driving with Large Language Models: A Safety Perspective' (ICLRW 2024) discussed a couple methods with verification modules and in-context learning to enhance the safety.", "other_strengths_and_weaknesses": "* Strength:\n\nThe PDCE loss is a creative solution to the problem of numerical prediction with autoregressive models, retaining language generation abilities while improving regression performance.\n\nThe explicit incorporation of traffic rules using MLNs directly addresses the safety-critical requirements of autonomous driving.\n\nThe framework is evaluated on two datasets with both qualitative and quantitative results, and ablation studies clarify the contributions of each component.\n\n* Weakness:\n\nHyperparameter Sensitivity: The performance of PDCE loss and the multimodal RAG component depends on carefully chosen parameters (e.g., σ, weighting factors). A deeper exploration of this sensitivity would be beneficial.\n\nReliance on External Modules: The approach relies on pretrained object detectors (YOLOv8) and text encoders, making it vulnerable to errors from these components.\n\nLimited Domain Evaluation: Although evaluated on two datasets, additional tests in diverse driving scenarios or simulated environments could further demonstrate the generalizability and robustness of SafeAuto. LLM's strength for AD tasks is its general common sense and generalizability.", "other_comments_or_suggestions": "Please kindly refer to the previous comments.", "questions_for_authors": "Please kindly refer to previous comments/questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yzaZ8DOIOw", "reviewer": "Reviewer_EhoY", "summary": "This paper proposes SafeAuto, a novel framework that enhances MLLM-based autonomous driving systems by incorporating both unstructured and structured knowledge. The model can predict high-level and low-level action prediction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to the reviewer for their thorough and insightful feedback. Their expertise and dedicated time have significantly contributed to improving the quality of this work!\n\n\n> **Q1: The experiments are only conducted on the BDD-X/DriveLM datasets and may only be compared with some narrow works.**\n\nThank you for the valuable suggestion. We fully recognize the need to test our framework on a broader range of datasets to better demonstrate its generalizability and effectiveness. Unfortunately, the availability of comprehensive multimodal autonomous driving datasets is currently limited. For example, nuScenes, while extensive, lacks high-level action or reasoning annotations (and DriveLM is just developed based on nuScenes which provides the high-level annotation). Similarly, the Waymo dataset does not provide essential modality data like images or videos. Given these constraints, BDD-X and DriveLM represent the most relevant and widely used datasets for evaluating multimodal autonomous vehicle systems involving LLMs. We hope that the future development of diverse multimodal datasets will allow for more comprehensive evaluations of frameworks like ours.\n\n> **Q2: The proposed method uses Weighted Target Token Probability to predict the float number of speed/course, somehow not a very accurate predicting approach for these fine-grained signals.**\n\nThank you for your insightful question! As detailed in our paper, our PDCE loss yields notably higher accuracy on these fine-grained signals. For example, on the DriveLM dataset—where the goal is to predict the next six waypoints—our method achieves an ADE of 0.84, substantially lower than the 1.51 reported for DriveLM-agent and even approaching the performance of the full regression model UniAD (full) with ADE as 0.80. Additionally, on the BDD-X dataset, and following reviewer KjQv’s suggestion, our approach still outperforms TimeLLM, which employs an MSE loss on LLM-extracted hidden features. The performance comparison for speed prediction is summarized below:\n\nSpeed:\n\n| Method       | RMSE     | A0.1      | A0.5      | A1.0      | A5.0      | A10.0     |\n| ------------ | -------- | --------- | --------- | --------- | --------- | --------- |\n| TimeLLM      | 1.17     | 21.34     | 53.13     | 74.14     | 99.67     | 99.86     |\n| **SafeAuto** | **0.65** | **55.49** | **88.84** | **95.34** | **99.81** | **99.91** |\n\nCourse:\n\n| Method       | RMSE     | A0.1      | A0.5      | A1.0      | A5.0      | A10.0     |\n| ------------ | -------- | --------- | --------- | --------- | --------- | --------- |\n| TimeLLM      | 4.10     | 65.70     | 83.47     | 89.59     | 97.60     | 98.59     |\n| **SafeAuto** | **3.85** | **76.26** | **89.68** | **94.11** | **98.30** | **99.25** |\n\nSo it is indeed accurate and the weights here are also not arbitrarily chosen but determined by a predefined soft target distribution. Moreover, the ablation study on the hyperparameter $\\sigma$, as shown in Fig. 4 on page 15, further validates the robustness and effectiveness of this weighting strategy.\n\n> **Q3: I have some doubts about whether the proposed method is very effective for low-level actions. The BDD-X dataset is not a universally recognized best dataset for evaluating low-level actions. Maybe it needs to be compared with some end-to-end autonomous driving works.**\n\nThank you for your insightful question! In our experiments, we also used the DriveLM dataset, which is derived from the standard nuScenes dataset and specifically designed for end-to-end prediction. The goal is to forecast 3 seconds into the future (i.e., 6 future waypoints), thereby providing a standard framework for low-level action evaluation. As demonstrated in Table 2, our method still achieves substantially improved low-level action prediction, with an ADE of 0.84 compared to the 1.51 ADE reported for the original DriveLM. This performance is also on par with the full regression method UniAD (full), which records an ADE of 0.80. Currently, there are few multimodal autonomous driving datasets that include high-level annotations. Nevertheless, we are still excited to evaluate our method on any new datasets offering comprehensive multimodal annotations for both high-level and low-level actions as they become available!\n\n> **Q4: The writing is not very friendly for those readers in the autonomous driving field. I recommend adding relevant background information about the methods.**\n\nThank you for your valuable suggestion! We acknowledge that the current version may not be fully accessible to readers in the autonomous driving field. We will add more background details, such as for RAG, in the related works section to better support these readers.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. However, the proposed method uses Weighted Target Token Probability to predict the float number of speed/course, somehow not a very accurate predicting approach for these fine-grained signals.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "The experiments are convincing and show the effectiveness of each component. However, the experiments are only conducted on the BDD-X/DriveLM datasets and can only be compared with some narrow works.", "relation_to_broader_scientific_literature": "This paper provides a Knowledge-Enhanced large model for autonomous driving to predict high-level and low-level actions.", "essential_references_not_discussed": "No other essential references are not discussed.", "other_strengths_and_weaknesses": "Strengths:\n- This paper describes the details of the method very clearly. Although it takes some effort to follow, the proposed method is very systematic.\n\nWeaknesses:\n- The experiments are only conducted on the BDD-X/DriveLM datasets and may only be compared with some narrow works.\n- The proposed method uses Weighted Target Token Probability to predict the float number of speed/course, somehow not a very accurate predicting approach for these fine-grained signals.\n- I have some doubts about whether the proposed method is very effective for low-level actions. The BDD-X dataset is not a universally recognized best dataset for evaluating low-level actions. Maybe it needs to be compared with some end-to-end autonomous driving works.", "other_comments_or_suggestions": "The writing is not very friendly for those readers in the autonomous driving field. I recommend adding relevant background information about the methods.", "questions_for_authors": "Please refer to the weakness mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hk1lfWur19", "reviewer": "Reviewer_v68f", "summary": "This paper proposes SafeAuto, a novel framework for autonomous driving using multimodal foundation models. It addresses the challenges of integrating high-level reasoning and low-level control. \nThe main algorithm ideas include three key components. First, the Position-Dependent Cross-Entropy (PDCE) loss function improves low-level numerical prediction accuracy while maintaining the autoregressive nature of the Multimodal Large Language Model (MLLM). Second, Knowledge-Enhanced Post-Safety Verification uses Markov Logic Networks (MLNs) to integrate traffic rules into the MLLM's decision-making process, verifying and correcting high-level actions. Third, Multimodal Retrieval Augmented Generation (RAG) learns from similar driving experiences by integrating video data, control signals, and environmental predicates.\nThe paper's main findings show that SafeAuto outperforms existing baselines. On the BDD-X and DriveLM datasets, it reduces the Root Mean Square Error (RMSE) for speed and course predictions and improves high-level action prediction performance. On the BDD-X dataset, it reduces the RMSE for speed and course predictions by 5.8% and 14.1% respectively, and boosts high-level action performance by 28.0% under the CIDEr metric.\n\n## Update after rebuttal\nThanks for authors to provide answers that address all my concerns. I've also read the comments from the other reviewers. In overall consideration, I'd like to raise my assessment score to be positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback, and for recognizing the value of our work! The insightful suggestions and detailed comments provided have substantially contributed to enhancing the quality of our work!\n\n> **Q1: The PDCE is the weighted sum of the KL divergence, what is new in this loss?**\n\nThanks for the insightful question! Our motivation for proposing the PDCE loss for LLM digit prediction is twofold. (1) When fine-tuning an LLM for digit prediction, one natural idea is to use a combination of CE loss for the word part and MSE loss for the digit part (which we have also tried internally before). However, this approach introduces some challenges: it is challenging to balance the magnitudes of the CE and MSE losses, the self-attention mechanism injects noise into the final output features for forecasting digits, and applying MSE loss on hidden features would also impair the LLM's autoregressive capability (i.e., it cannot do the next-token generation). (2) Additionally, using CE loss as existing work directly on digit prediction treats each digit as equally important, which introduces bias as highlighted in Figure 2.\n\nTo address these challenges, we introduce the PDCE loss, which employs a careful weighted sum of the KL divergence. However, note that as shown in Fig 3, both the weights and the target distribution in the KL divergence are not randomly chosen, but determined formally by a predefined soft target distribution, such as a Gaussian distribution, which aims to approximate the behavior of the MSE loss on digits in string form. In this way, it solves the above challenges and eliminates the need to balance disparate losses for finetuning LLM, preserves the autoregressive property of the LLM, and also emulates the behavior of MSE loss during training on float numbers in string, as demonstrated in Figure 2.\n\nWe also present a performance comparison with TimeLLM, which directly applies MSE loss on the extracted hidden features from LLM for regression, as suggested by Reviewer KjQv, below, and our method still performs much better. Due to the word limit, we kindly encourage you to review our rebuttal in Q1 for Reviewer KjQv.\n\n> **Q2: What is the difference between this work and the Markov Logic Networks (MLNs) of Richardson & Domingos, 2006?**\n\nThank you for your insightful question! The key distinction in our work lies in how we integrate them into a data-driven vision-language model framework for autonomous driving.\n\nSpecifically:\n\n1. **Integration with Data-Driven Paradigms:** Our approach is the first to combine MLNs with Multimodal LLM in the context of autonomous driving. This integration allows us to inject explicit safety knowledge—such as traffic rules—into the prediction process. Directly training a model on data alone does not guarantee adherence to these safety constraints.\n\n2. **Different Source of Predicates:** In our work, the grounding of predicates comes from multiple external modules. We incorporate outputs from our self-trained YOLOv8 detector and the original predictions from the MLLM to form the predicates.\n\n3. **Joint Simulated and Real-Data Training:** Another novel aspect of our work is the joint training strategy for the MLN weights using both simulated and real-world data instead of purely relying on real-world data which may be hard to collect in practice.\n\n> **Q3: What are the special challenges addressed in the M-RAG?**\n\nThank you for the insightful question! In contrast to relying solely on the video modality (as in RAGDriver), our Multimodal RAG must integrate three key modalities—(i) video or image, (ii) control signals, and (iii) environmental predicates—into a single unified embedding for ranking and retrieval. One challenge here is the lack of a clear objective for aligning these diverse modalities in autonomous driving retrieval tasks.\n\nTo address this gap, we exploit textual scenario descriptions, since they naturally encapsulate information about all modalities. During training, our approach guides the unified embedding to replicate the relative ranking derived from these textual embeddings. Concretely, we use a contrastive learning-based procedure where each batch’s embedding distribution is pushed to match the ranking distribution from the text embedding for that scenario. This approach is both efficient (as it avoids iterating over external databases mid-training) and effective at aligning multiple modalities.\n\nBy contrast, methods such as RAGDriver use a triplet-loss formulation, which only focuses on top-k similarities but ignores fine-grained ranking signals and requires pre-fetching positive/negative examples before training. The improved performance results also highlight the effectiveness of our proposed multimodal RAG, and as demonstrated in Table 10 on page 15, we find incorporating environmental predicates significantly boosts retrieval accuracy by mitigating some of the noise present in raw video embeddings.", "claims_and_evidence": "The paper claims that SafeAuto, with its novel components like PDCE loss, MLN-based safety verification, and Multimodal RAG, outperforms existing baselines. This is supported by experiments on BDD-X and DriveLM datasets. For example, it shows significant improvements in low-level control accuracy, such as reducing RMSE for speed and course predictions, and high-level action prediction performance, like boosting CIDEr scores. Ablation studies are provided to help in understanding the contribution of each component.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are suited for autonomous driving. The PDCE loss, MLN-based safety verification, and Multimodal RAG directly address the challenges of integrating high-level reasoning and low-level control. \nBenchmark datasets like BDD-X and DriveLM, along with relevant metrics such as RMSE and CIDEr, comprehensively assess system performance in both high-level action prediction and low-level control accuracy, making them appropriate for this application.", "theoretical_claims": "The paper focuses more on empirical validation through experiments instead of theoretical proofs.", "experimental_designs_or_analyses": "For the PDCE loss experiments, using the BDD-X dataset to compare its performance against the original CE loss is valid. \nMeasuring RMSE for speed and course predictions directly assesses the improvement in numerical prediction accuracy. \nThe MLN-based safety verification experiments are well-designed in terms of using real and simulated data for training. \nThe Multimodal RAG experiments use appropriate datasets and metrics.", "relation_to_broader_scientific_literature": "Existing methods using MLLMs struggle with low-level control and safety. SafeAuto's PDCE loss improves low-level prediction accuracy compared to traditional cross-entropy loss. Its use of MLNs for safety verification addresses the lack of explicit safety checks in previous approaches. Multimodal RAG enhances decision-making by leveraging past experiences.", "essential_references_not_discussed": "There seem no missing essential related works.", "other_strengths_and_weaknesses": "Strengths: The paper introduces concepts like PDCE loss, use of MLNs and Multimodal RAG. Experimental results shows that by integrating the designs, SafeAuto outperforms existing baselines across multiple datasets.\n\nWeaknesses: \n- Despite of the integration to form the SafeAuto pipeline, the novelty for each component seems marginal, which is my major concern. \n- The writing is lengthy, and it seems some parts are overlength such as the abstract section.\n- The arrangement of space between texts and figures need to be checked.", "other_comments_or_suggestions": "See questions for authors.", "questions_for_authors": "Despite of the integration to form the SafeAuto pipeline, the novelty for each component seems not clear. \n(1) The PDCE is weighted sum of the KLdivergence, what is new in this loss design?\n(2) What is the difference between this work and the Markov Logic Networks (MLNs) of Richardson& Domingos, 2006?\n(3) What are the special challenges addressed in the multimodal RAG method design?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RIWOdaWMci", "reviewer": "Reviewer_KjQv", "summary": "This paper presents SafeAuto, an MLLM-based autonomous driving system. SafeAuto has three major innovations. First, it uses a new Position-Dependent CE loss (PDCE) loss, which supervises the predicted number tokens based on their numerical difference from the ground-truth number. Second, it has a knowledge-enhanced post-safety verification module using Markov Logic Networks (MLNs). It can explicitly encode domain knowledge and structured traffic rules into the decision-making process of the MLLM, which can be used to verify and correct the predicted high-level actions. Third, it uses a novel training method for constructing a unified embedding that effectively integrates all modalities.\n\nThe authors evaluated the SafeAuto method on the BDD-X dataset and the DriveLM dataset. They showed that SafeAuto has better driving performance than the state-of-the-art baselines.\n\n\n## update after rebuttal\n\nNo updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude to the reviewer for their meticulous and constructive feedback. Their insightful observations and valuable recommendations have greatly contributed to improving the rigor and clarity of our work!\n\n> **Q1: The motivation for the PDCE loss is that using the MSE loss on regression outputs will \"disrupt the MLLM’s autoregressive token generation, transforming it into a pure transformer encoder (Tan et al., 2024) used only for regression tasks and losing its language generation capabilities necessary for high-level question-answering\". I hope the authors can compare their method against this MSE-loss baseline and show how the high-level question-answer capabilities are impacted.**\n\nThank you for the valuable suggestion! In fact, once we apply an MSE loss at the MLLM’s hidden layer to directly regress the low-level control signals, the model stops generating tokens—only numeric outputs are produced via an MLP head, rather than text tokens, and thus we cannot do a further next-token generation. Consequently, it loses its autoregressive language generation capability, making further high-level QA impossible. As a result, the TimeLLM can only handle low-level predictions but cannot generate further textual answers or explanations.\n\nMoreover, even when focusing exclusively on low-level tasks, our proposed SafeAuto still outperforms TimeLLM by a wide margin on BDD-X dataset, as illustrated in the tables below:\n\nSpeed:\n\n| Method       | RMSE     | A0.1      | A0.5      | A1.0      | A5.0      | A10.0     |\n| ------------ | -------- | --------- | --------- | --------- | --------- | --------- |\n| TimeLLM      | 1.17     | 21.34     | 53.13     | 74.14     | 99.67     | 99.86     |\n| **SafeAuto** | **0.65** | **55.49** | **88.84** | **95.34** | **99.81** | **99.91** |\n\nCourse:\n\n| Method       | RMSE     | A0.1      | A0.5      | A1.0      | A5.0      | A10.0     |\n| ------------ | -------- | --------- | --------- | --------- | --------- | --------- |\n| TimeLLM      | 4.10     | 65.70     | 83.47     | 89.59     | 97.60     | 98.59     |\n| **SafeAuto** | **3.85** | **76.26** | **89.68** | **94.11** | **98.30** | **99.25** |\n\n\n\n> **Q2: Figure 2. Having two peaks is not necessarily worse than having just one peak. It could be that there are two different valid maneuvers in this scenario. It will be useful to have more analysis on why the traditional CE loss leads to the observed behavior.**\n\nThank you for the insightful question! In that case, the historical speed values are [6.23, 6.56, 5.62, 6.27, 7.09, 6.67, 10.24] with the ground truth being 12.46. This means that the peak around 13 is actually more valid in this scenario. But we agree that in some cases, two peaks might indicate different valid maneuvers. However, since we are predicting for the very next frame—a short time interval—the two peaks should not be widely separated as shown in Fig 2. In such cases, we would expect the distribution to resemble a Gaussian with one dominant peak more.\n\nThe issue with CE loss lies in how it computes the joint probability. The calculation follows p('1') × p('2'|'1') × p('4'|'12') × p('6'|'12.4'), treating every digit as equally important during training. Thus, when the training data is not large enough, the model might learn to assign higher values to the last three digits instead of the first digit to make the overall probability high. As a result, even if the latter probabilities are high, a lower value for p('1') can cause the model to generate a different starting digit (for instance, '3' instead of '1'), which then leads to the subsequent digits being generated incorrectly.\n\nWe also provide two case studies in Figure 6 on Page 16 to illustrate these failure cases with CE loss more clearly. We will also include more additional analysis in our final version to further explain this behavior!\n\n> **Q3: It is not clear how SafeAuto leverages past experiences to inform current decision-making and how Multimodal RAG helps.**\n\nSorry for the confusion, to clarify, once we retrieve similar past experiences, we incorporate the corresponding historical video along with both high-level and low-level predictions into the current context—that is, we append them before the current video and question in prompt. This process is similar to few-shot learning because it provides the MLLM with additional context from similar driving scenarios, thereby informing its current predictions. We will provide a more detailed explanation in our final version.\n\nIf you have any further questions or suggestions, please feel free to let us know. Your feedback is greatly appreciated and will certainly help us improve our work a lot!", "claims_and_evidence": "* The proposed PDCE loss and MLN-based reasoning module are well-designed\n* The overall SafeAuto method shows strong performance against state-of-the-art baselines on popular benchmarks.", "methods_and_evaluation_criteria": "* The datasets used for evaluation make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "* The experimental setup is valid. However, I think all the results are from open-loop simulation. I would like to see some results in closed-loop simulation as well.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "* I don't have anything to add.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* The motivation for the PDCE loss is that using the MSE loss on regression outputs will \"disrupt the MLLM’s autoregressive token generation, transforming it into a pure transformer encoder (Tan et al., 2024) used only for regression tasks and losing its language generation capabilities necessary for high-level question-answering\". I hope the authors can compare their method against this MSE-loss baseline and show how the high-evel question-answer capabilities are impacted.\n\n* Figure 2. Having two peaks is not necessarily worse than having just one peak. It could be that there are two different valid maneuvers in this scenario. It will be useful to have more analysis on why the traditional CE loss leads to the observed behavior.\n\n* It is not clear how SafeAuto leverages past experiences to inform current decision-making and how Multimodal RAG helps.", "questions_for_authors": "See \"Other Comments or Suggestions\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes and evaluates SafeAuto, a novel framework for autonomous driving that utilizes multimodal LLMs to address the challenges of integrating high-level prediction and low-level control.

The reviewers identify several strengths of the paper: (1) The SafeAuto framework defines a novel integration of three key components - use of the position-dependent-cross-entropy (PDCE) loss function to improve low-level prediction accuracy, use of Markov Logic Networks (MLN) to factor in explicit safety knowledge, and a contrastive multimodal retrieval automated generation (RAG) for properly aligning video, control signals, and state predicates; (2) The overall SafeAuto framework is shown to produce strong performance against state-of-the-art baselines on two popular benchmark data sets (BDD-X and DriveLM); and (3) the presentation describes the details of the SafeAuto method very clearly (even though it takes some effort to follow).

The weaknesses of the paper that were identified include the following: (1) The experiments are restricted to BDD-X/XDriveLM data sets (for reasonable reasons), which limits the approach's empirical comparison to a narrow number of other approaches and impacts the significance and generality of the results accordingly; (2) despite their integration to form an effective prediction pipeline, the novelty of each component seems marginal, which limits the magnitude of the paper's contribution somewhat; (3) All results appear to have been generated using open-loop simulation, which de-emphasizes the ability of the framework to keep pace with execution; and (4) the presentation is a little verbose in some spots and could also benefit from refinement to make it more relatable to readers working in the autonomous driving field.

Overall, there is consensus that the paper makes an interesting contribution and should be accepted. Please address all of the reviewers comments and suggestions in preparing your final version.
Decision: Accept (poster)