HCRMP: An LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving
Authors: Zhiwen Chen
Hanming Deng
Zhuoren Li
Huanxi Wen
Guizhe Jin
Ran Yu
Bo Leng
Venue: NeurIPS 2025 poster ()
Keywords: Autonomous Driving，Motion Planning
Primary area: applications
TL;DR: This paper's HCRMP architecture implements an "LLM-Hinted" paradigm, using LLM hints for RL-based motion planning with crucial relative LLM-RL independence to improve driving safety and reliability.

Abstract: Integrating the understanding and reasoning capabilities of Large Language Models (LLM) with the self-learning capabilities of Reinforcement Learning (RL) enables more reliable driving performance under complex driving conditions. There has been a lot of work exploring LLM-Dominated RL methods in the field of autonomous driving motion planning. These methods, which utilize LLM to directly generate policies or provide decisive instructions during policy learning of RL agent, are centrally characterized by an over-reliance on LLM outputs. However, LLM outputs are susceptible to hallucinations. Evaluations show that state-of-the-art LLM indicates a non-hallucination rate of only approximately 57.95\% when assessed on essential driving-related tasks. Thus, in these methods, hallucinations from the LLM can directly jeopardize the performance of driving policies. This paper argues that maintaining relative independence between the LLM and the RL is vital for solving the hallucinations problem. Consequently, this paper is devoted to propose a novel LLM-Hinted RL paradigm. The LLM is used to generate semantic hints for state augmentation and policy optimization to assist RL agent in motion planning, while the RL agent counteracts potential erroneous semantic indications through policy learning to achieve excellent driving performance. Based on this paradigm, we propose the HCRMP (LLM-Hinted Contextual Reinforcement Learning Motion Planner) architecture, which is designed that includes  ①Augmented Semantic Representation Module to extend state space. ②Contextual Stability Anchor Module enhances the reliability of multi-critic weight hints by utilizing information from the knowledge base. ③Semantic Cache Module is employed to seamlessly integrate LLM low-frequency guidance with RL high-frequency control. Extensive experiments in CARLA validate HCRMP's strong overall driving performance. HCRMP achieves a task success rate of up to 80.3\% under diverse driving conditions with different traffic densities. Under safety-critical driving conditions, HCRMP significantly reduces the collision rate by 11.4\%, which effectively improves the driving performance in complex scenarios.

HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving
Zhiwen Chen Tongji University thw.enccheen250163.coo com
Hanming Deng Sense Time Research
Zhuoren Li" Tongji University 1911055@tongj1 edu. cn
danghaamiinglseesstiinne com
Huanxi Wen
Guizhe Jin Tongji University jgz135730018922163..
Ran Yu
Tongji University 231 9511385toongj..co edu. on
Tongji University ranyußtongji. edu, edu cn
Bo Leng" Tongji University
lengbo@tongj i edu. cn
Abstract
Integrating the understanding and reasoning capabilities of Large Language Models (LLM) with the self-learning capabilities of Reinforcement Learning (RL) enables more reliable driving performance under complex driving conditions. There has been lot of work exploring LLM-DDoinated RL methods in the field of au- tonomous driving motion planning. These methods, which utilize LLM to directly generate policies or provide decisive instructions during policy learning of RL agent, are centrally characterized by an over-reliance on LLM outputs. However, LLM outputs are susceptible to hallucinations. Evaluations show that state-of-the- art LLM indicates non hallucination rate of only approximately 57.95% when assessed on essential driving-related tasks. Thus, in these methods, hallucinations from the L.M can directly jeopardize the performance of driving policies. This paper argues that maintaining relative independence between the LLM and the RL. is vital for solving the hallucinations problem. Consequently, this paper is devoted to propose a novel LLM-Hinted RL paradigm. The LLM is used to generate semantic hints for state augmentation and policy optimization to assist RL agent in motion planning, while the RL agent counteracts potential erroneous semantic indi- cations through policy learning to achieve excellent driving performance. Based on this paradigm, we propose the HCRMP ment Learning Motion Planner) architecture, which is designed that includes ! Augmented Semantic Representation Module to extend state space. @ Contextual Stability Anchor Module enhances the reliability of multi-critic weight hints by utilizing information from the knowledge base. 3 Semantic Cache Module employed to seamlessly integrate LLM low-free now-frequency guidance with RL high- frequency control. Extensive experiments in CARLA validate its strong overall driving performance. HCRMP achieves task success rate of up to 80.3% under diverse driving conditions with different traffic densities. Under safety-critical driv- ing conditions, HCRMP significantly reduces the collision rate by 1. 1.4%, which effectively improves the driving performance in complex scenarios.
Hinted Contextual Reinforce-
"Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Reinforcement learning (RL) is a method for learning optimal policies by maximizing expected returns through interactions with the environment. It has proven to be effective for solving complex decision-makee problems and has attracted significant attention in various fields [ 2 For the motion planning task in autonomous driving (AD). the RL agent can dynamically generate trajectories or control commands that follow the learned driving policy, based on the fused multimodal traffic features 4 [ 6 However, RL has limited understanding and reasoning in complex driving conditions and often fails to identify critical traffic features
g which can result in unreliable driving actions. In contrast, large language models (LLM) poss strong semantic understanding and common- sense reasoning abilities 9 [0], and have achieved significant advances in multitasking in recent years II [22 [3]. AD systems require the extensive common sense and high-level decision-making capabilities, which are exactly what large language models can provide [4, [6] [8]. Therefore, integrating LLM and RL within unified AD system leverages the strengths of both approaches [T]]
This integration enhances policy understanding and reasoning during self-learning (20). leading to
more reliable and safer driving actions in complex driving conditions 21].
Despite the significant potential benefits of integrating LLM and RL for AD, effective integration remains critical and unresolved challenge. Current methods primarily use RI. agent to assist in the policy optimization of L.LM 22 23
24 or employ L.LM to directly instruct RI. agent policy 32). In the former approach, the L.LM outputs directly generate the driving policy, while in the latter, the LLM strongly provides decisive instructions during the policy learning process of the RL agent. Because both methods demonstrate strong reliance on the LLM, we refer to such methods as LLM-Dominated RL Methods
generation [25] 26 27
28 29 30
31
However, LLM outputs are known to be susceptible to hallucinations can distort
33
34
35
36
37
which
and compromise policy stability. The Gemini-2.5 2.5-Pro model, a state-
of-the-art (SOTA) LLM [38], is evaluated on five key dimensions related to two essential driving capabilities: scenario understanding and action response, as illustrated in Figure (a). The result shows a non-hallucination rate of only 57. 37.95% implying that over 40% of its outputs are prone to hallucinations. Further details are provided in the Appendix A.
J-M-Damintted RL Methed Agent Distaile Q-Miin Eviluttion
Unfeasible Actionl
Hozard Identification
LLW Desiger Hollcciaatinn
Reward Purchooy
78. .7%
Lateral Driving Decisions
Action Enttrcctin
Ineffiiinn briesg Pulicy
29. 68%
Road Type Comprehen-
Extro Scurcas
RL Mathod
Feasible Actionl
39, .3%
Hints
RL Agent
22%
Fallay-
Strlin O-Vikun
Longitudinal Driving Decisions
Road Sign Comprehension
Optenbiing
Masi
Shofs-
Augusering Mirte
protly
Everding Erre
(0)
Figure 1: LLM performance evaluation and hallucination impact on LLM-RL methods. Figure (a) shows the SOTA LLM's non non-hallucinatio rates across five driving tasks. Figure (b) illustrates the LLM-Dominated RL methods. LLM hallucinations directly distort the RL agent's Q-value estimation and degrade policy efficiency, leading to unfeasible driving actions. Figure (c) presents the LLM-Hinted RL method. LLM provides semantic hints to the RL agent instead of directly dictating decisions. The RL agent, through its own policy learning process, effectively buffers the negative impact of these hallucinations, thereby preventing unfeasible actions.
To clarify how these capabilities interface with RL, LLMs are typically integrated in two primary modes. L.M-Instructed RL Policy Generation, the LLM primarily acts as an instructor, leveraging its scenario understanding capability. It interprets complex traffic environments (e.g.. Hazard Identification, Road sign comprehension) and translates this high-level understanding into dynamically shaped reward functions for the RL agent. Conversely, in RL-Assisted LLM Policy

---

Optimization, the LLM is the core decision-matee leveraging its action response capability to directly generate high-level driving policies (e.g.. Longitudinal Driving Decisions Lateral Driving Decisions), which the RI. agent then assists in optimizing through environmental feedback. In both modes, the L.LM's 's output tightly coupled with the final driving decision. As shown in Figure (b), when LLM hallucinations occur, these incorrect signals are directly propagated to the downstream decision-makee process. This distorts the Q-value evaluation and compromises the safety and stability of the policy. The direct result is significant degradation in overall driving performance. It thus imperative for AD systems combining LLM and RL to mitigate the policy instability caused by LLM ballucinations.
(b),
This paper argues that maintaining relative independence between the LLM and the RL is an essential way to solve the hallucinations problem. The fundamental reason is that such separation preserves the RL agent's s autonomy in decision-making and adaptation, while allowing the L.LM to provide semantic hints as auxiliary inputs and intrinsic modulation. Accordingly, We propose an L.MM-Hinte RL motion planning paradigm, as illustrated in Figure (c). Even if the LLM outputs are unstable, the RL agent is able to counteract potential erroneous semantic indications through policy learning, avoiding the direct generation of unreasonable actions. At the same time, this separated structure preserves the utilization of LLM strengths in driving conditions comprehension and common-sense reasoning, enabling context-aware guidance for the RL agent in a way that maintains its fundamental self-optimization capabilities.
Based on the LLM-Hinted RL paradigm, we develop an architecture called HCRMP (LLM- Linted Contextual Reinforcement Learning Motion Planner). The architecture comprises three key com- ponents: @ the Augmented Semantic Representation Module, which utilizes semantic hints from the LLM to extend the state space; @ the Contextual Stability Anchor Module, which leverages information from the structured knowledge base to improve the reliability of the weight hints that the LLM generates for each critic network; and @ the Semantic Cache Module, which enables efficient and stable training through exeuuncry hierarchical outputs and the historical context cache matching strategy. The LLM provides state-augmenting and policy-optimizing semantic hints as auxiliary inputs and intrinsic modulation to the RL agent, rather than directly controlling policy generation. Mcanwhile, the RL agent autonomously executes motion planning, and the LLM and RL modules collaborate asynchronously at different temporal scales, ensuring training stability. In addition, extensive experiments in the CARLA simulator validate the effectiveness of our proposed HCRMP framework, highlighting its superior overall performance, particularly in demanding driving conditions. HCRMP achieves a task success rate of up to 80.3% across diverse conditions with varied traffic densities. Critically, in safety-critical driving conditions, HCRMP achieves a significant 11.4 % reduction in the collision rate. The contributions of this study can be summarized as follows:
We classify existing LLM- Dominnteee RL methods, clarify their strong reliance on LLM outputs, and highlight the problem that hallucinations from the LLM can degrade driving performance. To address these challenges, we propose the LiM--ddted RL paradigm.
We propose a novel motion planning architecture named HCRMP. By combining the semantic hints for state augmentation and policy optimization provided by LLM with the self-learning capabilities of RL, it significantly improves driving performance in diverse driving conditions.
Extensive experiments in CARLA validate HCRMP's strong overall driving performance. HCRMP achieves a task success rate of up to 80.3% under diverse driving conditions and, critically, reduces the collision rate by 11.4% % in safety-criticll driving conditions.
2 Related Works
LLM--Domnntted RL methods for AD motion planning fall into two main categories: RL-assisted LLM policy optimization and I.L.Minnttructed RI. policy generation. Additional related work is provided in Appendix B.
2.1 RL-Assisted L.MM Policy Optimization
This type of methods typically convert conditions information into linguistic inputs to generate action instructions or probabilities distributions [39, 40
RI. agent enables the fine-tuning of LLM parameters using reward signals from the environment, optimizing its policy to maximize
3

---

Recently, the integrated paradigm of LLM and RL has demonstrated considerable potential for motion planning in AD. Existing studies primarily use LLM generate trajectories or control commands, with RI submodules integrated for action suggestion generation or policy optimization. LLM [22 drives an L.M agent using meta-actions output by pre-trained RI. model, combining the current state and similar trajectories to generate specific actions. AlphaDrive [23], in contrast, leverages GRPO-based RL reward function to enhance the driving policy of its vision-language models (VLM). This enhancement enables the VLM to better adapt dynamic driving conditions. While these methods can leverage the LLM strengths complex scenario understanding and
cumulative rewards [45, 46].
their risk is that any erroneous instructions generated by LLM can directly mapped to unreasonable driving actions, which fundamentally threaten the driving safety.
2.2 I.M-IIsttrcted RL Policy Generation
For the other type of method, RL agent is used to generate control commands. LLM serves as sub-module to support the policy optimization. Specifically, LLM is utilized provide intrinsic rewards, which is proven to improve the learning efficiency of RL 47
48 49
50, [5]]. For AD motion
planning. LearningFlow [28] and Autoreward [29] utilize a closed-loop framework which the LLM automatically generates reward signals to guide RL agent training. Similarly, Clip-RL -id-iee [30],
LORD [31] and REvolve 32 directly rely on the LLM to generate reward values for policy learning. However, these methods are highly sensitive to the quality of LLM outputs and are therefore quite susceptible the adverse effects of hallucinations.
contrast, our proposed method utilizes a weakly coupled integration of the LLM and RL. This significantly diminishes the system's vulnerability to fluctuations in the LLM outputs, preserving the LLM inherent semantic and reasoning ad advantages while ensuring RL maintains its fundamental self-optimization capabilities.
Augmented Semantic Representation
Levei-Level Hints
State Space
Actor
raw
S
Road
Vehicle
n < 8c)
Exprasswer High- derelit
LLM AMM-Augmented Representations
S lm
N (ot $c)
Global scenario-level
Fine grained object- level Analysis
Abstraction
L(O) dtip
LLM
Specific Prompts
Semantic Cache Module
Multi Critics
similarity
History...
X1
Environment
Contextual Stability Anchor
Driving Attributes
Knowledge Prompt
Task Prompt
Advantage Function
Aint (s, a)
Dataset
Data chunking
Embedding
Safety
N
Efficiency
(8,a)
Looa Requlation
Deleliy showkord
Comfort
The framework of our proposed HCRMP. LLM acts in the Augmented Semantic Represen- tation module to fetch information at the scenario level and object level, extending the state space. Meanwhile, LLM acts in the Contextual Stability Anchor module to generate reliable weights between multi critics, utilizing the knowledge base to mitigate the output fluctuations. When LLM fails to provide timely guidance, the Semantic Cache module replaces the missing weights by retrieving most similar historical driving conditions. Hints from LLM are ultimately input to the RL agent's actor and mulli-critie networks for optimal policy learning.
Figure 2:

---

3
Methodology
This section outlines the proposed HCRMP framework. As illustrated in Figure| 2 HCRMP comprises three key components: the Augmented Semantic Representation Module (Section 3.2), the Contextual Stability Anchor (Section 3.3), and the Semantic Cache Module (Section 3.4).
The Augmented Semantic Representation (ASR) module employs LLM for hierarchical scenario reasoning, from global abstraction to object-level analysis. The resulting multi-level semantic hints are encoded into compact vectors and integrated into the RL agent's 8 state space to enhance driving condition awareness. The Contextual Stability Anchor (CSA) Module utilizes external knowledge sources, including traffic regulations and funddamental priors, to generate more reliable semantic hints. These hints then dynamically influence Q-value evaluation by modulating the impact of various driving attributes, thereby guiding the RL agent toward more effective pocision-making Furthermore, the Semantic Cache Module(SCM) addresses the temporal mismatch between the low-frequency semantic guidance from the LLM and the high-free teghfrreuency control commands of the RL.
3.1 Problem Formulation
We formulate the AD task Markov Decision Process (MDP), represented by the tuple (S, P. R) 7). Here, the action space A consists of two continuous control variables: throttle/brake control commands and steering angle, each constrained within the normalized range [-1, The state space 5 is composed augmented semantic state characteristies !!lm The transition function P(s' s, defines the probability distribution over next states, the reward function R provides feedback signals reflecting the quality of actions, and 7 E [0, 1] is discount factor that trades off between immediate and future rewards. The agent employs a deep neural network (DNN) policy, denoted by Te. where 0 represents the learnable parameters. At each time step i, given the current state 8 E S, formed by concatenating the raw sensory inputs sam and the LLM-enhanced semantic embeddings simm the policy samples a bounded continuous action a TT(- 8t) from Beta distribution, ensuring constrained control commands. Then, the agent executes action a E A. receives an immediate reward E R, and transitions to the next state 81+1 according to the probability P. The agent's objective is to discover an optimal policy x² through interaction with the environment that maximizes the expected discounted cumulative return in E
two components: the raw state characteristics st and the L.I.--
R r (s az)
π arg max 2
I
(I)
3.2 Augmented Semantic Representation
The LLMM-based ASR module designed to provide the RL agent with multi-level semantic encodings, which in turn augments its state space. The LLM performs hierarchical situational reasoning, transitioning from a global scenario-level abstraction to a fine-grained object-level analysis. Based on the scenario's s topological configuration and traffic dynamics, the LLM executes semantic parsing to categorize road situations. Then, this categorized understanding is encoded into a 4-dimensional scenario-level vector. Concurrently, by analyzing the semantic characteristics of surrounding traffic participants, the LLM pinpoints critical agents. Following this identification, it quantifies their relative spatial configurations and maps them to discrete directional categories. These are then encoded as 9-dimensional object-level vector. To ensure consistent representation, particularly in low-density traffic conditions, semantic compensation mechanism can be implemented preserve integrity.
At each time step f, the state space of the agent S; constructed by combing heterogeneous feature sets, including raw sensory inputs s!aw and LLM-enhanced semantic embeddings s!!m! where s!w (f/ f2 ..... fí'), f.." view RGB image features 1/, road topology features 77. ego-vehicle dynamics EY, and navigation path embeddings N². The RL agent processes incoming multimodal inputs via a lightweight yet effective visual backbone (ShuffleNetV2 (52) for spatial encoding and Gated Recurrent Units (GRU) for temporal semantic modeling. In this pipeline, the semantic representation s!!n functions as an auxiliary information source, serving semantic guidance to enhance situational awareness.
and each feature frame f/ (I..T.. E/ NV?) comprises multi-
5

---

3.3 Contextual Stability Anchor
To satisfy the demands for adaptive trade-offs among driving attributes (e.g.. safety, comfort, effi- ciency), we design multi-critic framework based on Proximal Policy Optimization (PPO) [53]. The framework decouples the Q-value evaluation of multiple driving attributes and enable independent representation learning for distinct objectives. By formulating prompts, the LLM performs contextual analysis of dynamic traffic conditions and dynamically generates a set of adaptive weights {Ac where N is the number of attributes, A [0, 1], and Σi=1 X = 1, which are then used as part of an integrated advantage function Am estimated via Generalized Advantage Estimation (GAE) in Eq
N (m a) Σ A,
(2)
The clipping objective is defined in Eq where the clip(-) function constrains the update range to stabilize training. ratio,()) denotes the probability ratio between the current policy Th and the previous policy Tex..
Letip (0) EE min (O) (0) +,,1+))Amm
To further enhance the stability of the L.M output dynamic weights, we propose semantic anchoring module that utilizes Retrieval-Augmenter Generation (RAG [54]) for stability optimization. The proposed module utilizes a semantic reference corpus. This corpus is compiled from authoritative sources such national standards, local regulations, industry guidelines, and technical specifications [55] [6). This module performs a text-to-text alignment: using a pretrained embedding model (text- eeeetdembbdding-- a00-00)), we embed both the real-time driving conditions query (Query) and the reference corpus (Knowledge) into high-dimensional vector representations. Semantic relevance between the query and corpus vectors is subsequently assessed using FAISS- basSe similarity search (57). A Top-3 selection strategy is employed to retrieve the most semantically aligned textual fragments. The retrieved passages are aggregated into semantic context set, which is then provided as auxiliary input to the I.MM. By anchoring weight generation to a knowledge base of traffic regulations and safety norms through RAG, this module mitigates weight drift, cutting LLM weight variance by 54.67% in complex driving (e.g., road construction) as detailed in Appendix C, and enhancing multi-attribute coordination. Importantly, the LLM-generated attribute weighting mechanism is implemented in a weakly coupled manner. This design leverages the LLM's strength in context-awas aware weighting while mitigating the potential adverse effects hallucinated outputs on Q-value evaluation, thereby suppressing policy misguidance caused by hallucinations.
3.4 Semantic Cache Module
The inference latency of LLMs inherently limits the control frequency of existing LLM- L.M--Domittted RL methods. To address this problem, we propose cooperative asynchronous training framework that coordinates low-frequency semantic planning by the LLM with the high-frequency control execution handled by the RL. However, this decoupling introduces a new challenge: the LLM can fail to deliver timely semantic outputs since unexpected system delays or interruptions.
To mitigate this, SCM compensates for such missing outputs by retrieving pertinent historical semantic representations. The SCM maintains dedicated memory bank. This bank stores structured representations of past driving conditions, which encompass both scenario-level and object-level information. To manage this memory bank, the SCM employs Least Recently Used (LRU) eviction policy retain data from rare, critical scenarios. Crucially, it also includes the corresponding multi- critic weight vectors that are generated in previously similar conditions. When the LLM fails to return a valid semantic signal within a predefined time window, the SCM performs rapid nearest-neighbor search over the memory bank using semantic embedding vectors. The module then identifies the historical entry most semantically aligned with the current driving context. It extracts the associated weight vector from this entry to serve temporary guidance signal for the LLM.

---

Experiments and Results
4.1 Experiment Setting
4.1.1 Driving Conditions
All experiments are conducted in Town of the CARLA simulator (58) Driving conditions include conventional conditions, such as overtaking and merging, as well safety-critical conditions, such as trilemma and occluded pedestrian. To further assess the performance of the AD system, the traffic flow densities across three levels are established: low, medium, and high, as detailed in Appendix D.
4.1.2 Evaluation Metrics
We evaluate the driving policy using quantitative metrics for safety, efficiency, and comfort.
Safety is evaluated using two primary metrics: Success Rate (SR)-the percentage of episodes completed without major violations and Collision Rate (CR)-the the proportion of episodes involving collisions.
Efficiency is evaluated by Average Speed (AS), Total Distance (TD) and Time Steps (TS), reflecting travel speed, distance covered, and task completion time, respectively.
Confort is evaluated via Speed Variance (SV) and Acceleration Variance (AV), which reflects the smoothness and stability of driving behavior.
4.1.3 Baselines
We systematically compare the proposed method with the following methods:
Vanilla PPO 53
Directly train the policy using PPO in tasks as a basic RL bascline.
E2ECLA 59:
Combine curriculum learning and RL, which learns end-to-end AD policies in CARLA by gradually increasing task difficulty without prior knowledge.
AutoReward 29
An RL method that iteratively refines LLM-generated rewards post-training.
VLM-RL 25
A method that integrates pre-trained VLM with RL. It generates semantic rewards through language objective comparison, replacing manually designed reward function.
4.2
Main Results
Table I: Performance Comparison in Conventional Conditions under Different Traffic Densities
Density
Medium Density
High Density
Methods
Category
Condition
SR (%)
CR (%)
SR (%)
CR (%)
SR (%)
CR (%)
Vanilla PPO
RL
Overtaking
80.0
20.0
69.0
31.0
62.0
38.0
Merging
89.0
11.0
75.0
25.0
68.0
32.0
E2ECLA
RL
Overtaking
60.0
40.0
56.0
44.0
38.0
62.0
Merging
56.0
44.0
54.0
46.0
42.0
58.0
AutoReward (iter=0) LLM--Dominttd RL
Overtaking Merging
73.0
60.0
40.0
52.0
82.0
18.0
71.0
29.0
51.0
49.0
AutoReward (iter=5) LLM--ominated RL
Overtaking Merging
85.0
15.0
76.0
24.0
70.0
30.0
94.0
85.0
15.0
71.0
29.0
VLM-RI.
I.MM--Dmintted RI.
Overtaking Merging
54.0
45.0
52.0
48.0
50.0
50.0
56.0
44.0
53.0
47.0
50.0
50.0
HCRMP
LLM--iined RL
Overtaking Merging
99.0
93.0
7.0
87.0
13.0
97.0
3.0
92.0
8.0
86.0
14.0

---

As illustrated in Table [ SR and CR of various methods are compared in conventional conditions under different traffic densities. Results indicate that HCRMP matches or outperforms other baselines in different driving conditions, particularly in medium- and high-density conditions, where it achieves an average SR of 89.5%. By optimizing the multi-critic coordination strategy, the agent able to adopt safer actions, significantly reducing the collision risk.
Additionally, HCRMP shows relatively minor advantage over the baselines in low-density driving conditions, which is due to limited vehicle interactions. It reduces the difficulty of the driving task and restrict the full exploitation of the CSA's dynamic adjustment capabilities. We further conduct systematic evaluation of HCRMP and three Muselines-E2ECLA VLM-RL, and Autoreward underr safety-critical scenarios, with the latter two being representative LLM- L.M-Dominated RL methods. The corresponding results are presented in T
under
Table 2: Performance Comparison in Safety-Critice Driving Conditions
Traffic Density
SR
CR
AS
TD
TS
SV
AV
Methods
Condition
(%)
(%)
(m/s)
(m)
(8)
(m/s)
(m/s²)
Low
36.0
64.0
8.22
36,99
47.04
4.29
1.49
Occluded Pedestrian
Medium
37.0
63.0
7.19
31.53
39.72
3.02
2.62
E2ECLA
High
31.0
69.0
7.87
33.86
37.04
3.25
1.52
Low
36.0
64.0
6.99
52.82
48.38
3.01
2.89
Trilemma
Medium
34.0
66.0
6.99
40.64
50.28
2.18
2.70
High
38.0
62.0
8.68
48.96
46.44
4.59
2.89
Low
70.0
30.0
6.30
28.07
35.05
1.40
2.96
Occluded Pedestrian
Medium
31.0
69.0
7.79
27.42
55.78
1.34
3.16
Autoreward(iter=S)
High
24.0
76.0
7.37
31.26
53.65
2.15
3.21
Low
60.0
40.0
5.60
24.91
34.3
2.45
2.68
Trilemma
Medium
58.0
42.0
6.16
31.22
33.9
3.05
3.33
High
32.0
68.0
7.05
25.84
55.27
1.90
2.95
Low
65.0
35.0
5.02
139.20
45.04
8.43
1.72
Occluded Pedestrian
Medium
53.0
47.0
6.34
170.45
42.30
7.69
1.78
VLM-RL
High
41.0
59.0
5.98
172.87
40.23
7.06
1.61
Low
67.0
33.0
8.89
283.13
50.01
9.87
1.75
Trilemma Medium
58.0
42.0
7.76
278.82
48.13
7.75
1.69
High
54.0
46.0
7.79
176.06
46.98
7.72
2.70
Low
73.0
27.0
9.98
86.06
50.07
10.04
1.69
Occluded Pedestrian
Medium
67.0
33.0
9.96
82.09
51.17
9.97
1.74
HCRMP
High
61.0
39.0
8.97
77.95
48.59
9.69
1.72
Low
75.0
28.0
10.24
88.20
51.58
9.57
1.44
Trilemma
Medium
69.0
31.0
10.08
79.91
49.72
10.14
1.23
High
64.0
36.0
77.34
47.13
9.96
1.72
E2ECLA exhibits an AV exceeding 2.5 m/s² under certain conditions, which can cause discomfort to the passenger [60]. This phenomenon can primarily attributed to E2ECLA's failure to ade- quately consider vehicle acceleration. Consequently, the system prioritizes rapid maneuvers over comfort. AutoReward utilizes LLM to construct the reward function based on the analysis of the scenario. Similarly, unreasonable values of acceleration changes occur due to the LLM's emphasis on maneuverability at the expense of comfort considerations in safety-critical scenarios. VLM-RL, leveraging efficient navigation approaches, achieves notably high TD; however, its average CR in safety-critical conditions reaches 43.7%, indicating a significant safety concern.
The proposed HCRMP demonstrates well-balanced performance across all metrics, particularly in high-density conditions, where it achieves SR of 62.5%, a CR of 37.5%, and an AV of 1.72 m/s?, This comprchensive performance is mainly due to the integration of ASR and CSA. The former

---

pursues efficiency and comfort in low-density traffic flows, while prioritizing safety in medium- and high-density traffic conditions. The latter enhances the system's 's ability to understand complex environments by extending the state space.
HCRMP prioritizes ensuring safety in immediate driving tasks. This focus, however, means its global adaptation to the map's inherent static path features may be less developed, which consequently results in a lower traveled distance compared to VLM-RL. Through the synergistic effect of CSA and ASR, HCRMP achieves an effective balance of safety, efficiency, and comfort in safety-critical conditions, with particularly outstanding performance in medium- and high-density conditions.
4.3 Ablation Study
Table 3 presents the results of the ablation study in the medium-density trilemma, evaluating the performance of the HCRMP with the removal of different modules: HCRMP without ASR, HCRMP without CSA, and HCRMP with ASR. In the HCRMP w/o CSA configuration, the multi-critic weights are set to fixed equal values to create a clear baseline against the dynamic weighting provided by CSA, thus isolating and highlighting the CSA module's contribution.
Table 3: Ablation Study Results
Model
SR(%)
CR(%)
AS (m/s)
TD (m)
SV (m/s)
AV (m/s²)
Collision per Mile
HCRMP w/o ASR
40
60
7.52
44.34
47.74
6.94
2.95
21.65
HCRMP w/o CSA
48
52
6.01
29.82
54.44
2.89
2.63
27.90
HCRMP w/ ASR
54
7.27
29.04
34.64
5.89
2.26
25.34
The SR of HCRMP without ASR drops to mere 40.0%, indicating that the absence of ASR significantly diminishes the system's is ability to comprehend complex environments, resulting in information deficits that increase collision risks. contrast, HCRMP with ASR, achieves an SR of 54.0%, underscoring the critical role of ASR in enhancing situational awareness. By leveraging the LLM to interpret the current driving conditions, ASR expands the state space, strengthening the system's 's awareness of surrounding driving risks and thereby improving the safety of its decision- making tasks.
Furthermore, HCRMP without CSA exhibits an SR of 48.0%, which is even lower than that of HCRMP with ASR. Specifically, CSA enhances contextual stability through a knowledge base by dynamically constraining the priority weights within the multi-critic framework, effectively mitigating excessive fluctuations during the training process. Without CSA, the multi-critic system experiences pronounced instability in weight adjustments, hindering convergence toward an optimal policy. This instability directly undermines the system's decision-making consistency and safety in the medium-density trilemma, consequently leading to substantial decline in SR.
Figure illustrates the performance differences between HCRMP variants through dynamic reward trends. The reward curve for HCRMP without CSA exhibits pronounced fluctuations, indicating that, in the absence of CSA constraints, the system struggles stabilize reward values during evaluating. reflecting inherent instability in policy optimization. In contrast, the HCRMP variant equipped with CSA demonstrates a smoother upward trend in its reward curve.
Conclusion
Current LLM--ominated RL approaches for AD motion planning heavily depend on LLM outputs, making them vulnerable to hallucinations that can compromise policy reliability and lead to unsafe behavior. We propose LLM--Hintd RL motion planning paradigm and the corresponding HCRMP framework, aiming to preserve the relative independence between LLM and RL. The framework mitigates the impact LLM hallucinations, while still preserving the strengths of LLM in semantic understanding and high-level decision-making while ensuring RL maintains its fundamental self- optimization capabilities. The HCRMP architecture comprises three key components. First, the ① Augmented Semantic Representation Module refines the state space via semantic guidance.

---

Reward Comparison Across HCRMP Variants
75 Held
HCAMP ESA HCAMP w/o CSA
18
0.6
0.4
HCRИ w/o CSA
HCRMP w/ CSA
(a)
(b)
Figure 3: HCRMP variants rewards: dynamic trends and statistical distributions. Figure (a) visualizes that the dynamic reward curve for HCRMP without CSA exhibits significant fluctuations, indicative of performance instability, while the curve for HCRMP with CSA shows smaller fluctuations and more stable performance. Figure (b) illustrates that the overall reward distribution for the variant without CSA is wider and more pronounced in lower reward regions, whereas the CSA variant's rewards are more concentrated in higher value ranges with a more prominent peak.
Second, the @ Contextual Stability Anchor Module enhances the reliability LLM-provided multi-critic weight hints through retrieval-augmenned semantic anchoring based on information from the structured knowledge base. Finally, the @ Semantic Cache Module primarily improves training efficiency by asynchronous decoupling low-frequency LLM reasoning from high-frequency RL execution. To handle delayed LLM outputs, it employs historical semantic cache matching strategy as fallback. Extensive experiments in CARLA validate HCRMP's strong overall driving performance. HCRMP has a high task success rate 80.3% under diverse driving conditions with range of
different traffic densities. Especially, it achieves a I 11.% reduction in collision rate complex driving conditions. HCRMP provides promising framework for RL motion planning for AD with integrated LLM.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 52522219 and 52232015, and in part by the Fundamental Research Funds for the Central Universities under Grant 22120230311.
References
[1] Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. Reinforcement learning algorithms: A brief survey. Expert Systems with Applications, 231:1 2023.
[2] Xu Wang, Sen Wang, Xingxing Liang. Dawei Zhao, Jincai Huang, Xin Xu. Bin Dai, and Qiguang Miao. Deep reinforcement learning: A survey, IEEE Transactions on Neural Networks and Learning Systems, 35(4):5064 5078, 2022.
[3] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint 1XXi:22055.10300 2022.
[4] Long Chen, Yuchen Li, Chao Huang, Bai Li, Yang Xing. Daxin Tian, Li Li, Zhongxu Hu, Xiaoxiang Na, Zixuan Li, et al. Milestones autonomous driving and intelligent vehicles: Survey of surveys. IEEE Transactions Intelligent Vehicles, 102):1446-1056 2022.
[5] Fengying Dang, Dong Chen, Jun Chen, and Zhaojian Li. Event-triggered model predictive control with deep reinforcement learning for autonomous driving. IEEE transactions on intelligent vehicles, 9(1):459 2023.
10

---

[6] Sina Alighanbari and Nasser L Azad. Deep reinforcement learning with nmpe assistance nash switching for urban autonomous driving. IEEE Transactions on Intelligent Vehicles, ((3):260--6115, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1 13:7197119 2020.
[8] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13397/ 2023.
[9] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv: 02361, 2019.
[10] Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H Gilpin. Can large language models explain themselves? study of llm-generated self-explanations. arXiv preprint arXiv:2310. 12207, 2023.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178.. 2024.
[12] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, The Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0222 1.29544 2024.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12945: 2025.
[14] Daocheng Fu, Wenjie Lei, Licheng Wen, Pinlong Cai, Song Mao, Min Dou, Botian Shi, and Yu Qiao. Limsim++: A closed-loop platform for deploying multimodal Iims in autonomous driving. In 2024 IEEE Intelligent Vehicles Symposium (IV), pages 1084-009, IEEE, 2024.
[15] Yunsheng Ma. Can Cui, Xu Cao. Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15141-15151, 2024.
[16] Xuemeng Yang. Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving. arXiv preprint arXiv:2408. 004155 2024.
[17] Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with 1lms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093 14100. IEEE, 2024.
[18] Zhenjie Yang. Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey large language models for autonomous driving, arXiv preprint arXiv:2311.01043. 2023.
[19] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[20] Sheila Schoepp, Masoud Jafaripour, Yingyue Cao, Tianpei Yang, Fatemeh Abdollahi, Shadan Golestan, Zahin Sufiyan, Osmar R Zaiane, and Matthew Taylor. The evolving landscape of Ilm-and vim-integrated reinforcement learning. arXiv preprint
2025.
11

---

[21] Sihao Wu, Jiaxu Liu, Xiangyu Yin, Guangliang Cheng, Xingyu Zhao, Meng Fang, Xinping Yi, and Xiaowei Huang. Robust rl with llm-driven data synthesis and policy adaptation for autonomous driving. arXiv preprint arXiv:2410. 12568, 2024.
[22] Mustafa Yildirim, Barkin Dagda, and Saber Fallah. Highwayllm: Decision-making and naviga- tion in highway driving with rl-informed language model. arXiv preprint 1rXiv:2405.13547 2024.
[23] Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vims in autonomous driving via reinforcement learning and reasoning. arXiv preprint 2rX.002503.07608, 2025.
[24] Yuan Sun, Navid Salami Pargoo, Peter Jin, and Jorge Ortiz. Optimizing autonomous driving for safety: A human-centrr approach with Ilm-enhanced rlhf. In Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 76-80, 2024.
[25] Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, and Sikai Chen. Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driving. arXiv preprint arXiv:2412.1 15544, 2024.
[26] Zihao Sheng. Zilin Huang, Yansong Qu, Yue Leng, Sruthi Bhavanam, and Sikai Chen. Curricu- vim: Towards safe autonomous driving via personalized safety-critical curriculum learning with vision-language models. arXiv preprint arXiv:2502.15119 2025.
[27] Hao Pang. Zhenpo Wang, and Guoqiang Li. Large language model guided deep reinforcement learning for decision making in autonomous driving. arXiv preprint arXiv:2412 1851 2,024.
[28] Zengqi Peng, Yubin Wang, Xu Han, Lei Zheng, and Jun Ma. Learningflow: Automated policy learning workflow for urban driving with large language models. arXiv preprint arXiv:2501.05057, 2025.
[29] Xu Han, Qiannan Yang. Xianda Chen, Zhenghan Cai, Xiaowen Chu, and Meixin Zhu. Autore- ward: Closed-loop reward design with large language models for autonomous driving. IEEE Trunsactions on Intelligent Vehicles, 2024.
[30] Erfan Doroudian and Hamid Taghavifar. Clip-rldrive: Human-aligned autonomous driving via clip-based reward shaping in reinforcement learning. arXiv preprint arXiv:2412.1620) 2024.
[31] Xin Ye, Feng Tao, Abhirup Mallik, Burhaneddin Yaman, and Liu Ren. Lord: Large models based opposite reward design for autonomous driving. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5072-5081. IEEE, 2025.
[32] Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models for autonomous driving. arXiv e-prints, pages 2406, 2024.
[33] Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. Lims know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707, 2024.
[34] Guangzhi Sun, Potsawee Manakul, Adian Liusie, Kunat Pipatanakul, Chao Zhang, Phil Wood- land, and Mark Gales. Crosscheckgpt: Universal hallucination ranking for multimodal founda- tion models. arXiv preprint arXiv:240S.13684, 2024.
[35] Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, and Zibin Zheng. Llm hallucina- tions in practical code generation: Phenomena, mechanism, and mitigation. arXiv preprint arXiv:2409.200 20550, 2024.
[36] Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint 00X1v:2404.00971, 2024.
[37] Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. LIm-check: Investigating detection of hallucinations large language models. Advances in Neural Information Processing Systems, 01:3418--34216 2024.
12

---

[38] Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti- bench: Are milms ready for precise opatial-temporal world understanding? arXiv preprint arXiv:2503.23765 2025.
[39] Wanpeng Zhang and Zongqing Lu. Rladapter: Bridging large language models to reinforcement learning in open worlds. 2023.
[40] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31 199-31212, 2022.
[41] Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S Du, and Huazhe Xu. Unleashing the power of pre- trained language models for offline reinforcement learning. arXiv preprint arXiv:2310.20587, 2023.
[42] Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, and Jun Wang. Ask more, know better: Reinforce-leareed prompt questions for decision making with large language models. arXiv preprint arXiv:2310.1812) 2023.
[43] Xue Yan, Yan Song, Xidong Feng, Mengyue Yang. Haifeng Zhang, Haitham Bou Ammar, and Jun Wang. Efficient reinforcement learning with large language model priors. arXiv preprint arXiv:2410.07927, 2024.
[44] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676-3713 PMLR, 2023.
[45] Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:110935.
1 110971, 2024.
[46] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 244444 2024.
[47] Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: Lim-empowered credit assignment in episodic reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20095-20103, 2025.
[48] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprint arXiv:2309.1 1489, 2023.
[49] Kristy Choi, Chris Cundy. Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task spec-fpe priors. arXiv preprint arXiv:2210.1255 2022.
[50] Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pages 8657-8677. PMLR, 2023.
[51] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[52] Ningning Ma, Xiangyu Zhang, Hai-Tao Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018.
[53] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 4rX1v:1707.06347, 2017.
13

---

[54] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Retrieval-auge aaugeeeedd generation for inowledge-intensiie nlp tasks. Advances in neural information processing systems, 33:9459-9477 2020.
[55] Tianhui Cai, Yifan Liu, Zewei Zhou, Haoxuan Ma. Seth Zhao, Zhiwen Wu, and Jiaqi Ma. Driving with regulation: Interpretable decision-making for autonomous vehicles with retrieval- augmented reasoning via Ilm. arXiv preprint XXX::2410.04459, 2024.
[56] Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model. arXiv preprini arXiv:2402. 10828, 2024.
[57] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. JEEE Transactions on Big Data, 7(3):535 547, 2019.
[58] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[59] Luca Anzalone, Paola Barra, Silvio Barra, Aniello Castiglione, and Michele Nappi. An end-to- end curriculum learning approach for autonomous driving scenarios. IEEE Transactions on Intelligent Transportation Systems, 23(10):19817-1998 2022.
[60] Ksander N De Winkel, Tugrul Irmak, Riender Happee, and Barys Shyrokau. Standards for pas- senger comfort automated vehicles: Acceleration and jerk. Applied Ergonomics, 106: 103881, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly state the problem of LLM hallucinations in autonomous driving, the proposed HCRMP framework with its LLM--iited RL paradigm, and its key contributions in improving driving performance and safety, These claims are substantiated by the experimental results presented Section 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper's appendix discusses "Limitations and Broader Impacts," where po- tential issues such as inter-module information transfer loss and the gap between simulation and real-world complexity are discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper primarily focuses empirical framework and experimental validation of the HCRMP architecture for autonomous driving. does not introduce new theoretical results requiring formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper details the experimental setup in Section 4.1, including the CARLA simulator environment (Town 2), driving conditions, traffic densities (with Appendix D for details), evaluation metries, and baselines. Key architectural components and their interactions described in Section 3. While specific hyperparameters for baselines might rely on their original papers, the HCRMP methodology is laid out.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release the code for HCRMP and relevant experimental configura- tions upon publication to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
publicc/uuides/CoeSSuuuuu wttsssammissioooooiiiooooooooeee onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. ctajps.cc/puulic/uuiiae gwwp///ieee//eee/ Codeeubmii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 4.1 outlines the experimental setup, including the CARLA environ- ment, specific driving conditions, traffic density levels, and evaluation metrics. Section describes the HCRMP architecture, including the PPO-based multi-critic framework and LLM integration. Appendix C mentions prompt formulation and weight distribution, and Appendix D details traffic densities.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

Justification: Figure 3 in the paper presents information regarding the variability and distribution of rewards for different HCRMP variants. Specifically, Figure 3(a) shows reward curves over training steps with shaded regions indicating the variability (e.g., standard deviation or range over multiple runs) around the mean reward. Figure 3(b) uses violin plots combined with box plots to illustrate the distribution of rewards, showing the mean, median, interquartile range (25 (25%)75% ) and overall spread for HCRMP w/ CSA and HCRMP w/o CSA. This provides insights into the stability and performance consistency of the models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Details regarding the computational resources used for the experiments, in- cluding hardware specifications and approximate training times for the HCRMP framework and baselines, are provided in Appendix D.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research aims improve autonomous driving safety and reliability, adhering to ethical considerations in Al development. No human subjects were directly involved in experiments, and the work focuses simulated environments.
Guidelines:
18

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 6, "Limitations and Broader Impacts," discusses the positive potential of improving AD systems and intelligent transportation. While not extensively detailing negative impacts, it acknowledges the gap with real-world complexity, which implicitly relates to safety if deployed prematurely.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper proposes new framework (HCRMP) and conducts experiments in a simulator. While it utilizes LLMs, it does not release new pre- pre-trained LLMs or large-scale datasets scraped from the internet that would pose direct high risk for misuse in the sense outlined. The focus is on the interaction methodology.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
19

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper cites CARLA [58], PPO [52], ShuffleNetV2 [53], RAG [54], FAISS [57], and the Gemini-2.5- 2.5-i model [38] which is SOTA LLM. It is assumed that the use of these assets conforms to their respective licenses and terms of use, common in academic research.
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pappsswiteeeateeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces a new framework, HCRMP, which is described in detail in Section 3. No new datasets standalone software packages released primary contributions that would require separate documentation beyond the paper itself at this stage.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: The research presented in this paper does not involve crowdsourcing or direct experiments with human subjects. All experiments are conducted using the CARLA simulator.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The research does not involve human subjects, as all experiments are simulation- Therefore, IRB approval was not required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMS if is an important, original, or non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The use of LLM is a core and original component of the proposed HCRMP framework. Section details how LL.M-generated semantic hints used for policy optimization (Contextual Stability Anchor Module) and state augmentation (Augmented Semantic Representation module), and how the Contextual Stability Anchor module uses LLM with RAG.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Appendix
Table of Contents
LLM Evaluation: Driving Scenario Understanding and Response
A.1
Evaluation Setup
A.1.1
Driving Conditions Specification
A.1.2
Tasks Definition
A.1.3
Evaluation Dimensions
A.2
Test Question Examples
A.3
Comparison Methods
A.4
Conclusions
Additional Related Work
B.I
Taxonomy of LLM-RL Integration Paradigms Motion Planning in Autonomous Driving
B.2
Methodology Details
C.I
Augmented Semantic Representation Prompts Contextual Stability Anchor
C.2
C.2.1
Prompts
C.2.2
Overview of Weight Distributions
C.2.3
Comparison with Learnable Alternatives
C.3
System Flexibility and Prompt Robustness
C.3.1
Experimental Setup
C33.2
Results and Analysis
C.3.3
Prompt Versions
D
Experiments and Results
D.1
Computational Resources
D.2
Experiments setup
D.3
HCRMP with Varied LLM
D.4
Impact of LLM Hallucinations Q-value Estimation
4
E
Limitations and Broader Impacts
16

---

A
LLM Evaluation: Driving Scenario Understanding and Response
A.1
Evaluation Setup
A.1.1
Driving Conditions Specification
Driving conditions encompass both conventional scenarios such as overtaking and merging- and safety-critical situations [ illustrated in Figure
Ego Vehicle
(a) Trilemma
(b) Occluded Pedestrian
Figure Safety-Critical Driving Conditions: Trilemma and Occluded Pedestrian
The Trilemma driving condition refers to situation where the ego vehicle (indicated by the red box) surrounded by other vehicles on all sides. In this case, the ego vehicle must balance among three competing objectives: speed optimization, safe distance maintenance, and the need to change lanes. The Occluded Pedestrian driving condition is defined blind spot and intersecting the path of the ego vehicle.
pedestrian suddenly stepping out from
The driving conditions used in this part of the experiment consistent with those used in the
other experiments this paper.
A.1.2 Task Definition
Each task systematically evaluates different capabilities of LLM in the context of autonomous driving(AD). We categorize these tasks into two main groups: Scenario Understanding and Action Response. The former focuses the recognition and interpretation of driving conditions, while the latter assesses the model's 's ability to respond and make decisions various driving situations.
Scenario Understanding:
Hazard Identification: the ability to accurately identify potential hazards in diverse driving conditions.
Road Type Comprehension: types.
the ability to correctly interpret and respond to different road
Road Sign Comprehension: the ability correctly interpret and respond traffic signs.
Action Response:
Longitudinal Driving Decisions: The ability to make appropriate decisions related to ego-vehicle speed.

---

Lateral Driving Decisions: The ability to make appropriate decisions related to lane changes and steering.
A.1.3 Evaluation Dimensions
The primary metric used is the non-hallucination rate, which quantifies the accuracy and factual correctness of the LLM outputs.
Non-Hallucination Rate
For each dimension, the LLM output evaluated for the presence of hallucinations. hallucination is defined as any generated content that is factually incorrect, inconsistent with the driving condition, or irrelevant to the query. The non-hat non-hallucination is calculated as:
Number Hon-Hallueinated Outputs
Non-Hallucination Rate
Total Number of Outputs x 100%
A.2 Test Question Examples
We design a set of test questions to evaluate different LLM across the five defined tasks. Each test problem consists of a textual description of driving condition, along with specific query or instruction. The example problems are shown in Fig. and aims comprehensively evaluate the various capabilities of LLM in driving scenario understanding and response generation.
Hazard Identification Example
You're self- helf-driving car, What do you think is the biggest influence on in
neighborhood right now?
Road Type Comprehension Example:
You're self-driving car, where do you think you're traveling right now?
Road Sign Comprehension Example:
You're self-driving car, what do you think are the traffic signs ahead?
Longitudinal Driving Decisions Example:
You're self-driving car, What should your current longitudinal driving behavior look like?
Lateral Driving Decisions Example:
You're self-driving What should your current lateral driving behavior look like?
Figure 2: Sample test question for each task
A.3 Comparison Methods
Gemini-2.5-Pro []: On March 21, 2025, Google DeepMind officially launched Gemini- 2.5-Pro, its latest flagship model. Hailed as Google's 'most intelligent Al model' to date, it marks revolutionary breakthroughs in reasoning, context understanding, and multimodal processing.

---

Gpt-4o 3 A language model released for ChatGPT, offers real- sal-time reasoning across audio, visual, and text inputs. It supports 50 different languages with improved speed and quality.
Deepseek-rl [4]: An Al model developed by Chinese artificial intelligence startup DeepSeek. Post-trained with reinforcement learning, it's designed to enhance reasoning capabilities, particularly excelling at complex tasks such as mathematics, code, and natural language reasoning.
Alibaba has launched a Qwen model on its Alibaba Cloud Bailian platform, featuring significantly increased context length from 128k tokens to 1M tokens. This is equivalent to approximately one million English words or one and a half million Chinese characters.
Qwen-Turbo [5]
1ettaa3...-70I-nssttee (6]: Developed by Meta, it is the LLM with 70 billion parameters.
It's specifically designed for multilingual dialogue scenarios and has been optimized through Supervised Fine- Fine-Tuning Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), enabling it to excel natural language processing tasks such as text generation.
A.4
Conclusions
Hazard Identification
Hazard Identification
78.92%
81.94%
Lateral Driving Decisions
Road Type Compre-
61.35%
Road Type Compre- hension
Lateral Driving Decisions
hension
6.90%
14.59%
63.75%
3.51%
Road Sign comprehension
Road Sign comprehension
80.54%
80.28%
Longitudinal Driving Decisions
Longitudinal Driving Decisions
55.56%
(a) GPT-4o
(b) 1aaima--.3-70-mmme
Hazard Identification
Hazard Identification
71.95%
62.82%
Road Type Compre- hension
Lateral Driving Decisions
Lateral Driving
Road Type Compre- hension
Decisions
17.29%
5.98%
19.66%
79.37%
21.95%
Road Sign comprehension
70.94%
90.24% Longitudinal Driving Decisions
Longitudinal Driving Decisions
Road Sign comprehension
85.04%
cc) Deepseck-rl
(d) Qwen-Turnt
Figure Hon-alllucintiion Rates of Different LLM Across Driving Tasks

---

Regarding the performance evaluation of LLM in autonomous driving tasks, results indicate that their performance is relatively strong Longitudinal Driving Decisions and Hazard Identification, exhibit- ing higher non-hallucination rates. However, for tasks involving complex and precise operations such as Lateral Driving Decisions, the non-halllcination rates across all evaluated models are generally low, suggesting significant hallucination rate in this area. Furthermore, apart from Deepseek-rl which demonstrates prominent performance in Road Sign Comprchension, other models generally show deficiencies in this dimension. This highlights the inherent limitations of LLMs in accurate physical world perception and dynamic decision-making which remain crucial challenges for their application in autonomous driving.
B
Addition Related Work
B.1
Taxonomy of LLM-RL Integration Paradigms for Autonomous Driving
To ensure the focus of our discussion in the main paper. we concentrate on studying the methods most relevant to our work based on the core mechanism of "how an LLM's ss output influences and integrates into the RI. decision loop." This section provides a comprehensive taxonomy L.MM-RI. integration paradigms to address the broader landscape of existing methods.
B.1.1 Overview of LLM-RI. Integration Paradigms
Table presents a taxonomy of mainstream paradigms for L.LM-instructed RI. in autonomous driving, categorized by the L.M'' primary role and key characteristics.
Table I: Taxonomy of LLM-RL Integration Paradigms for Autonomous Driving
Papers Venues
Policy Optimization
Common-Sense Reasoning High-terel Policy Seeaario Generation Humon-In-Loow
I.M ar Reward Designer
I FearningFoww (arXiv 2025)
LORD (WACV 2025)
AutoReward (used
Decision
TeLL-Drive (arXiv 2025)
LaViPlan (arXiv 2025)
LLM az Language Translator
DriveGPT4 (ICLR 2124)
Human Centric AD (arXiv 2025)
2024)
CRITICAL (arXiv 2024)
L.M az Hinter
HCRMP (Ours)
B.1.2 Rationale for Inclusion and Exclusion in Main Paper Discussion
We clarify our reasoning for including or excluding these paradigms from our core discussion in the main paper:
B.1.2.1 LLM as a Reward Designer
This paradigm fully leverages the LLM's powerful common-sense reasoning capabilities for col- laborative policy optimization. The LLM analyzes driving scenarios and generates reward signals to guide RL agent training. Methods such LearninFllow, LORD, and AutoReward fall into this category. As this direction directly aligns with our research focus on how LLM outputs influence RL policy learning, it is key focus of our discussion in Section 2 (Related Work) of the main paper.
Key Characteristic: L.M outputs (reward functions or reward values) have a decisive influence on the RL agent's optimization objective, making these methods highly sensitive to LLM hallucinations.
5

---

B.1.2.2 LLM as Decision Provider
This paradigm positions the LLM as high-level deeision-maker maker that outputs semantic policies (e.g., "turn left," "slow down"). which are then translated into specific vehicle control signals by an RL. module or rule-based planner. Examples include TelvL-Drive and ViPiln.. This approach is fundamentally different from our end-to-end RL decision mechanism, where the RI. agent directly generates low-level control commands (throttle, steering). Therefore, this paradigm is not a core topic of our main paper discussion.
Key Difference: The RL component acts as an executor rather than a learner, which represents different technical direction from our work.
B.1.2.3 LLM Scenario Generator
This paradigm uses the LLM to enhance the training environment by generating diverse or safety- critical scenarios (e.g., ChatScene, CRITICAL). The LLM does not participate policy optimization during training or inference. As this is focused on data augmentation rather than policy learning, it represents a different technical direction and outside the scope of our related work.
Key Difference: LLM contributes to training data diversity rather than policy optimization guidance.
B.1.2.4 LLM Language Translator
This paradigm focuses on parsing human language commands (e.g., "drive faster," "turn right at the next intersection") and translating them into actionable instructions for the AD system, offen in human-in-the-loop settings (e.g., DriveGPT4). Since our work does not process such natural language commands and focuses on autonomous policy learning without human intervention, we consider its relevance to be limited.
Key Difference: This paradigm addresses human-vehicle interaction rather than autonomous policy optimization.
B.1.3 Positioning of Our LLM-Hinted RL Paradigm
Our proposed LLM-Hinted RL paradigm introduces novel integration mechanism that differs from the LLM as Reward Designer category in a fundamental way:
LLM as Reward Designer (LLM-Dominated); The LLM's outputs (reward functions or reward values) dominate the RL optimization objective. Any hallucination directly corrupts the training signal, leading to unsafe policies.
The LLM provides semantic hints (state augmentation and multi-critic weight modulation) that assist but do not dominate the RL agent's 's policy learning. The RL agent retains its self-optimization capability and can buffer LLM hallucinations through policy learning and environmental feedback.
LLM as Hinter (LLM-Hinted, Ours):
This weakly coupled integration mechanism is the contribution of our work, distinguishing it from existing LLM--DDminttdd RL methods.
B.2 Motion Planning Autonomous Driving
The core task of motion planning in autonomous driving is to rapidly generate safe and robust local trajectories or motion commands that guide the vehicle to effectively avoid obstacles and operate smoothly in complex dynamic environments (7). As a key component for achieving safe and efficient autonomous mobility, it has long been a highly active and closely studied area of research.
Current motion planning methods are mainly divided into two categories: pipeline planning and end-to-end planning. The traditional pipeline planning method, also known as the rule-based planning method, is composed of multiple interrelated modules. These modules succh such as perception, local- ization, planning, and control are designed and developed independently. Graph search planning algorithms determine path between a start point and goal point by performing iterative searches based on the environmental map and obstacle information. Common graph search algorithms include Dijkstra's is algorithm, the A* algorithm, and the Hybrid A algorithm. While Dijkstra's algorithm [ can find the shortest path between two points, it lacks goal-directed efficiency and becomes computa- tionally expensive in long-range searches. To address this limitation, Stanford University developed

---

the A algorithm [9] in 1968, which significantly improves efficiency through the use of well-designed heuristic functions. However, A is mainly suited for static environments and does not adequately account for the motion constraints of moving vehicles. In 2008, Stanford University introduced the Hybrid A algorithm [TO), which incorporates kinematic constraints, thereby enhancing the practicality and applicability of path planning in real-world driving scenarios. Esposto et al. [TT) combined the Hybrid A and classical A* algorithms to propose a path planning method based on Reeds-Shepp curves, which not only accommodates the kinematic characteristics of vehicles but also improves planning speed.
However, end-to-end planning methods have become focal point of current research due to their superior adaptability and efficiency, leveraging artificial intelligence to directly map raw perception data to control commands. Representative approaches include: Behavior Cloning, which uses supervised learning to mimic expert trajectories and quickly generate reliable driving policies. Reinforcement Learning, which optimizes dynamic decision-making through interaction with the environment, enabling adaptation complex scenarios.
Behavioral Cloning (BC) is primary imitation learning approach in autonomous driving, where the agent learns replicate expert behavior by training a classifier regressor on demonstration trajectories. As a passive method, it assumes that state-action pairs in the demonstrations are independent and learns the target policy purely through observation of complete expert executions. Early BC applications in driving [12] used end-to-end neural networks to map camera inputs directly to control commands. To improve performance in complex urban settings, later work introduced enhancements such as multi-sensor inputs 16) auxiliary learning tasks [17.88_ 18 and more sophisticated expert demonstrations [19].
To reduce reliance on labeled data, some researchers have turned to reinforcement learning (RL) for
autonomous crrisiio-making. Unlike imitation learning, RL agents learn policies by interacting with the environment and maximizing cumulative rewards through trial and error. Over time, the agent refines its policy to achieve optimal performance based feedback from the environment. RL has demonstrated success in learning lane following on real vehicle in low-traffic conditions (20). Saxena et al. [2I) use the Proximal Policy Optimization (PPO) algorithm learn control policy in continuous motion planning. Their model implicitly accounts for interactions with surrounding vehicles to prevent collisions and improve ride comfort.
C Methodology Details
C.1 Augmented Semantic Representation Prompts
ASR Module Prompt: Global Scenario-Level Abstraction
You are an advanced Al assistant for an autonomous driving system. Your task is to analyze the provided driving condition data and extract key scenario-level semantic information.
Given Scene Data:
Road topology features: (road_topoll
* Current traffic dynamics (overall flow, presence of traffic signals/signs):
(traffic {traffic_dynamiccs cetss_desrriiioo__
Ego-vehicle state (position, speed): fego_vehicle_statee
Instructions: Based on the provided driving condition data, please provide a concise answer for each of the following aspects. This information will be used to generate 4-dieensional semantic vector representing the global driving context.
Road Category:
Classify the current road type. (Examples: Highway, Rural Lane, Urban
Road)
2. Traffic Density: Describe the prevailing traffic density. (Examples: Low, Medium, High)
For each numbered point above, provide short descriptive phrase. Example: 1. Road Category: Urban Road 2. Traffic Density: Medium
Output Format:
7

---

ASR Module Prompt: Fine-Grained Object-Leevel Analysis
You are an advanced AI assistant for an autonomous driving system. Your task to analyze the provided driving condition data, focusing surrounding traffic participants and obstacles, to extract critical object-level semantic information.
Given Scene Data:
Detected surrounding traffic participants (vehicles, pedestrians, cyclists):
{list_of {1ist_of_detectedd l___edeeted_paaae walllpprrttiaanteae cipants_wiieieeeeeeeees porpe_position__ellloo
Detected static obstacles (roadblocks, debris): (detected_static_oo swtsssttttsseeeeeeeeeee seatticobstacless Ego-vehicle state (position, speed, current lane, heading): {ego_ wwwicleeeeeee.eeee state}
Instructions: Based on the provided scene data, identify up to 3 of the most critical (highest risk most influential ego-vehicle's decisions) traffic participants obstacles. For each identified critical entity, provide the following:
Entity Type: (e.g., Car, Truck, Bus, Motorcycle, Pedestrian, Cyclist)
Relative Direction: Its primary direction relative to the ego vehicle. Choose from: Front, Front-Lefft, Front-Right, Left, Right, Rear-Left, Rear, Rear-Right.
This information will be used to generate 9-dimensional semantic vector representing the object-level context. Focus conciseness and relevance for immediate driving decisions. Output Format: List each critical entity as a separate item. Example: Entity I: Type: Car, Direction: Front-Left Entity 2: Type: Pedestrian, Direction: Right Entity 3: Type: Truck,
Direction: Front
C.2 Contextual Stability Anchor
C.2.1 Prompts
Prompt for Autonomous Driving Attribute Weights Generation
Background: need to use three-element vector to represent the weighting charac- teristics safety, comfort, and efficiency in autonomous driving. Please analyze current driving scene based on the data read from the JSON file. The ego vehicle's coordinates are: (eggo_positionxx
pago_position_/)) The ego vehicle's velocity
is: (Cego_velocity x), aago_veloiity_io ) Surrounding environment information is: (surrounding
sacene_flag!
The question is: {question). Based my requirements and
please generate a three-clement vector for me. The three elements have the following requirements: The three elements represent: Safety, Comfort, Efficiency.
The of the three elements is 1.
All three elements should be retained to 2 decimal places.
Please determine the specific values based on my requirements and only output this vector without any other redundant content.
C.2.2 Overview of Weight Distributions
By anchoring the weight generation process to a corm-constrained semantic source, this module significantly mitigates the risk of weight drift. To empirically validate this, we conducted experiments within the same complex driving condition, specifically road construction environment, comparing -.nMe-gee crities weights with and without the integration of the Contextual Stability Anchor (CSA) module. As illustrated in Figure this module successfully reduces the variance of L.ne-agee critics weights by approximately 54.67% % within the same driving condition, thereby enhancing the consistency of multi-attribut coordination.

---

15%
2556-7595 within 1.50RR
Mollas
0.75
0.65
0.70
0.00
0.6
Weight
11:55
Weight
0.00
0.55
0.45
0.50
1..01
1.35
Step
Step
(a) Weight Distributions without CSA
(b) Weight Distributions with CSA
Figure 4: Comparison of I.MM-Generaeed Weight Distributions With and Without CSA
Building upon the previous findings that demonstrated CSA's ability stabilize LLM--generated weights within single driving condition, we further investigated its impact across diverse driving conditions. Figure 5 presents the Kernel Density Estimation (KDE) of the LLM's output safety weights across various scenarios. In the left panel, representing the system without the CSA module, boundary between safety weights for safe and dangerous scenarios indistinct, indicating that the LLM struggles to consistently differentiate risk levels. Conversely, the right panel, which incorporates the CSA module, clearly illustrates a distinct separation safety-critic weight distributions between safe and dangerous driving conditions. This evident demarcation highlights CSA's critical role in guiding the LLM to generate more context-aware and robust safety weight assignments, thereby enabling clearer distinction between varying levels of risk and enhancing the reliability of safety- critical decision-making.
Kernel Density Estimation
Kerne Density Estimation
10
erggan
20
istegran
8
15
Density
Density
10
4
0.4
0.5
0.6
0.7
0.8
0.9
0.60
Value
Value
(a) KDE without CSA
(b) KDE without CSA
Figure 5: Kernel Density Estimation of LLM--Gnnrated Safety-Critic Weights Across Diverse
Driving Scenarios
C.2.3 Comparison with Learnable Alternatives
To justify the design choice of using the LLMM-based CSA module over purely learnable alternative, we conducted a supplementary experiment comparing our approach with an end-to-end learnable network for critic weight generation.

---

C.2.3.1 Experimental Setup
We replaced the CSA module with an end-to-end network that employs a 4-head self-attention mechanism followed by multi-layer perceptron (MLP) to generate critic weights. This attention- based network is trained jointly with the PPO algorithm to learn the weight generation directly from driving data. The comparison was conducted in the challenging medium-density "Trilemma" scenario, which requires balancing speed optimization, safe distance maintenance, and lane change decisions.
C.2.3.2 Results
the performance comparison between the two approaches.
Table 2: Performance Comparison: CSA Attention Prasttio-BBsed Weight Generation
Model
SR (%)
CR (%)
HCRMP with Attention Network HCRMP with CSA (Ours)
57.0
43.0
69.0
31.0
C.2.3.3 Analysis
The experimental results indicate that the attention mechanism alone struggles to learn robust weight dynamics in the complex feature space of autonomous driving, which requires understanding multi- source inputs and dynamically adapting to diverse scenarios. We attribute this performance gap the following fundamental differences:
Nature of the Task: Determining appropriate safety weights is fundamentally reasoning problem involving traffic regulations, risk assessment, and causal understanding, rather than a simple pattern recognition or correlation task. For instance, deciding that safety should be prioritized in construction zone requires understanding traffic rules and consequences, not just recognizing visual patterns.
Knowledge Grounding: Our CSA module leverages the LLM's pre-trained common-sense reasoning capabilities and explicitly anchors weight generation to a structured knowledge base consistent with safe driving
(via RAG). This grounding ensures that the generated weights regulations and established safety principles. In contrast, the attention- based network must learn these complex reasoning patterns from scratch using only driving trajectories, which is substantially more challenging and data-inefficient.
Stability and Interpretability: The inwwlede-annchortd approach provides inherent stability (54.67% variance reduction shown in Section C.2.2) and interpretability the weights are grounded in retrievable knowledge sources. The end-to-end learned weights, while adaptive, lack this explicit grounding and exhibit higher variance across similar scenarios, leading to less consistent safety prioritization.
This comparison validates our design choice: for the safety-critin critical task of weight generation in autonomous driving, leveraging LLM reasoning with knowledge anchoring (the "hinting" mechanism) more effective than relying solely on end-to-end learning from driving data.
C.3 System Flexibility and Prompt Robustness
critical concern for LLM-based systems their flexibility and adaptability to scenarios not explicitly covered by predefined prompts. To address this, conducted a comprehensive evaluation HCRMP's robustness to prompt variations, testing whether the system maintains stable performance across different prompt formulations.
C.3.1 Experimental Setup
We selected three representative driving scenarios under medium traffic density: "Overtaking," "Merging," " and "Trilemma." For each scenario, we designed three prompt sets with different levels of detail and wording:
Prompt (Detailed Version): An expanded version of the original prompt with more elaborate instructions and contextual descriptions.
10

---

Prompt B (Concise Version): A simplified version with condensed wording and minimal instructions.
Prompt C (Original Version): The baseline prompt used in the main experiments (as shown in Sections C. and C.2.1).
The full text of all three prompt versions is provided in Section C.3.3 below. We evaluated performance across three key metrics: Success Rate (SR), Collision Rate (CR), and the frequency of Semantic Cache Module (SCM) calls, which indicates when the LLM fails to provide timely guidance.
C.3.2 Results and Analysis
Table presents the experimental results across all three scenarios and prompt variations.
Table 3: System Performance Across Different Prompt Formulations (Medium Traffic Density)
Driving Conditions
Prompt Sets
SR (%)
CR (%)
SCM Calls (%)
Prompt A (Detailed)
91.0
9.0
19.0
Overtaking
Prompt B (Concise)
88.0
12.0
4.0
Prompt C (Original)
93.0
7.0
4.0
Prompt A (Detailed)
92.0
8.0
21.0
Merging
Prompt B (Concise)
90.0
10.0
4.0
Prompt C (Original)
92.0
8.0
6.0
Prompt (Detailed)
62.0
38.0
27.0
Trilemma
Prompt B (Concise)
61.0
39.0
5.0
Prompt C (Original)
69.0
31.0
7.0
Overall Robustness: The experimental results indicate that the HCRMP framework exhibits degree of robustness to prompts with different styles. While performance varied across prompts-moon notably in the challenging "Trilemma" scenario-th framework maintained reasonable safety perfor- mance across all tests, with SR ranging from 61% to 93% and CR from 7% 39%.
Notably, the detailed prompt (Prompt A), despite providing more information, failed to improve and in some cases even degraded performance compared to the original prompt (Prompt C). For instance, in the Trilemma scenario, Prompt A achieved only 62% SR compared 69% for Prompt C. Simultaneously. dramatically increased the frequency of Semantic Cache Module (SCM) ralls-reaching 27% in Trilemma compared to just 7% for Prompt This indicates that overly complex prompts may introduce counterproductive noise or constraints, leading to clear trade-off where both safety performance and timeliness are negatively impacted.
Counterintuitive Effect of Detailed Prompts:
The concise prompt (Prompt B) exhibited moderate performance, generally falling between the detailed and original versions. While it reduced SCM call frequency compared to Prompt A, it did not achieve the optimal balance of safery and efficiency provided by Prompt C.
Balance Between Conciseness and Detail:
Implications for Prompt Design: These findings suggest that prompt engineering for safety-critical systems requires careful consideration. Excessive detail overload the LLM's reasoning process and increase inference latency (reflected higher SCM calls), while overly concise prompts may lack sufficient context for optimal decision-make making. The original prompt (Prompt C) achieves the best overall effectiveness by ensuring lower inference latency without sacrificing core safety metrics, making it the optimal choice within the HCRMP framework.
System Adaptability: Importantly, even with suboptimal prompts, the system maintained functional performance due to the M----Hitted paradigm's 's inherent safety mechanisms: (1) the RL agent retains self-optimizatiin capability to buffer poor L.LM guidance, and (2) the Semantie Cache Module provides fallback when LLM outputs are delayed, ensuring continuous operation.
11

---

C.3.3 Prompt Versions
Below we present the three prompt versions used in the robustness evaluation. These prompts are designed for the Contextual Stability Anchor (CSA) module to generate multi-critic weights.
Prompt A: Detailed Version
You are an expert autonomous driving system analyst. Your task is to carefully analyze the current driving scenario and generate three-element weight vector that represents the relative importance of three critical driving attributes: Safety, Comfort, and Efficiency.
Objective:
Input Information:
Ego vehicle position: (Lego_position.s) __ (ego_position_y))
Ego vehicle velocity: (fego_veloct) www.weelolyyaa..e _x,
Surrounding environment: (surrounding.infe)
Scenario type: {scene_flee __ffag)
Reference knowledge: (rag_output)
Detailed Instructions:
Please perform comprehensive analysis of the driving scenario considering traffic density, road conditions, nearby obstacles, and potential risks. Based on this analysis and the provided reference knowledge from traffic regulations, determine how to balance the three driving attributes.
The three elements of the output vector represent:
1.
Safety: Prioritize collision avoidance, maintaining safe distances, and responding to hazards.
Comfort: Ensure smooth acceleration/decelrrttoo and gentle steering to enhance passen- ger experience.
3. Efficiency: Optimize travel time and maintain appropriate speed while respecting traffic flow.
Requirements:
The sum of the three elements must equal 1.0
Each element must be a positive decimal value between and
Round each value to 2 decimal places
In high-risk scenarios (e.g., dense traffic, pedestrians nearby), prioritize Safety
In low-risk scenarios, balance all three attributes more evenly
Output Format: Provide only the three-element vector in the format: [Safety, Comfort, Efficiency]
Example: [0.65, 0.20, 0.15]
Prompt B: Concise Version
Generate weight vector for autonomous driving attributes.
Input:
Ego position: (fego_position x), (ego_position__))
Ego velocity: (fego_velocity.xx) {ego_velocity_y})
Environment: Ysurrounding_inff
Output:
Three-element vector [Safety, Comfort, Efficiency] where elements sum to 1.0.
Requirements: 2 decimal places, prioritize safety in high-risk scenarios.
12

---

For safety-critical trilemma and occluded pedestrian scenarios, the definitions for low, medium, and high traffic flow densities are unified. These are determined by counting the number of surrounding vehicles in the "borrowed lane" (i.e.. the adjacent/oncoming lane that the autonomous vehicle temporarily occupies or interacts with for specific maneuvers). Specifically, low, medium, and Prompt C: Original Version (Baseline)
Background: need to use a three-clement vector to represent the weighting characteristics of safety, comfort, and efficiency in autonomous driving.
Please analyze the current driving scene based on the data read from the JSON file. The ego vehicle's coordinates are: (fego carto_posiiioo__1 The ego vehicle's velocity is: ('egovvllocity) Surrounding environment information is: {surrounding_ info), {scene_flag'
dego_position_1) bego__elocity_y))
The question is: {question). Based on my requirements and stag_outputh) please
generate three-element vector for me. The three elements have the following requirements:
The three elements represent: Safety, Comfort, Efficiency.
The sum of the three elements 1.
All three elements should be retained to 2 decimal places.
Please determine the specific values based on my requirements and only output this vector without any other redundant content.
Experiments and Results
D.1 Computational Resources
The model training experiments in this study are accomplished relying the following computational resources: total of four NVIDIA A A100-PCIE GPUs (40GB graphics memory on a single card) are used. Each GPU is configured with 10 virtual CPU cores (Intel Xeon Gold 6248R) and 72GB of system memory. For each model, 200 epochs are performed, with each epoch consisting episodes.
D.2 Experiments setup
Town02 provides a variety of challenging driving conditions for evaluating basic driving ability and complex decision-making actions, as encompasses circular road networks, regular neighborhoods, and multi-level interchanges. We introduce varying traffic density settings as follows: in the on-ramp merging scenario, low, medium, and high densities correspond to 4, and 8 surrounding vehicles, respectively, located on the two lanes adjacent to the merging lane on the main road. In the multi-lane overtaking scenario, these levels represent 1, 2, or 3 surrounding vehicles ahead of the ego vehicle.
high densities correspond to the presence of 2, 3, and 4 surrounding vehicles, respectively, in the aforementioned "borrowed lane".
These traffic density levels are designed to create diverse traffic environments and interaction sit- uations, thereby inereasing task complexity and enabling more comprehensive evaluation of the algorithm's ' decision-making ability and adaptability under different traffic conditions.
D.3 HCRMP with Varied LLM
Our study primarily conducts experimental validation based on the Gemini-2.5-Pro model. To comprehensively assess the generalization capability and performance of the HCRMP architecture across different LLMs, we further evaluated its effectiveness using four mainstream LLMs: GPT-4o, Llama-3.3-70B 3.strut-----ttttttettt,t Deepseek-rl, and Qwen-Turbo in trilemma's 's high-density driving condition.
Experimental results show that, as shown figure 3 and table 4 the overall performance of HCRMP remains consistent across different L.M.. Although these LLM exhibit some variation in hallucination- free rate evaluations, their differences have minimal impact when serving as semantie prompt sources for HCRMP. This demonstrates that HCRMP,
LLM-Hinted RL paradigm, effectively
13

---

leverage the semantic prompting capabilitiess of various LLM. Through the self-learning mechanism of the RL agent, it compensates for disparities in LLM performance, thereby achieving similarly strong driving performance regardless of the underlying L.M backend.
Table 4: HCRMP with Varied LLM
SR(%)
CR(%)
AS(m/s)
(D(m)
TS(s)
SV(m/s)
AV(m/s*)
MCRMP-GGeini-22
64
36
9.94
77.34
47.13
9.96
1.72
HCRMP-GPT-4o
61
39
6.85
53.09
52.88
2,40
1.84
SERMP-Deepseeke
87.35
51.44
8.46
1.97
HCRMP SeeiIl--wee--Mnee Turbo
65.04
56.84
2.17
1.79
63
37
6.84
47.68
51.70
2.26
1.79
Success Rate(SR) Comparison of Different Methods
AutoReward
E2ECLA
VLM-RL
54.0
HCRMP-Qwen-Turbo
59.
HCRMP-GPT-40
HCRMP-Deepseekv11
MCRMP-Liama-333-7022
63.0
HCRMP-GGemin--25--r0
010
20
60
Success Rate(%)
Figure Success Rate (SR) Comparison of Different Methods
D.4 Impact of LLM Hallucinations on Q-value Estimation
critical challenge for LLM- L.M--intgrrted RL systems is understanding how incorrect LLM outputs lead to incorrect Q-value estimates or rewards. To address this, we conducted controlled experiment to establish a quantitative link from an erroneous LLM prompt, to a distorted Q-value estimation, and ultimately degraded driving performance.
D.4.1
Experimental Setup
D.4.1.1 Scenario
The experiment was conducted in high-density overtaking scenario (as described in Section 4.1 of the main paper), which requires careful judgment of when it is safe accelerate and overtake surrounding vehicles.
D.4.1.2 Control and Experimental Groups
Instances where HCRMP receives correct LLM outputs. For example, the LLM correctly identifies construction site ahead and outputs appropriate semantic hints (e.g., elevated safety weight, recognition of hazard in state representation).
Control Group:
Experimental Group: Instances specifically selected where the LLM produces hallucinated outputs. For example, the LLM fails recognize a construction site ahead, providing incorrect semantic hints that do not reflect the true risk level.
14

---

D.4.1.3 Measurement
We focus on the Q-value estimation for the critical action "accelerate to overtake" (Anventake) and measure:
Q-value Estimation Error: The relative difference between Q-values estimated under correct vs. incorrect LLM outputs, calculated as:
Qincorseet 8, Govertake)
Qoorrcer 8, Govertake
Q-value Error
Q correaa (8, Ggvertake
where Qoorrect is the Q-value estimated with correct LLM output, and Qincorrett is the Q-value with hallucinated LLM output.
Task Success Rate: The percentage of successful task completions (collision-free overtaking) in each group.
D.4.2 Results
Table S presents the quantitative impact of LLM hallucinations on Q-value estimation and driving performance.
Table 5: Impact of LLM Hallucinations on Q-value Estimation and Task Performance
Method
Estimated Q-value Error (%)
Success Rate (%)
HCRMP (Correct LLM Output)
85.0
HCRMP (Incorrect LLM Output)
23.0
67.0
Performance Degradation
+23.0% error
-18.0%
D.4.3
Analysis and Implications
D.4.3.1 Quantitative Evidence of Hallucination Impact
The experimental results provide concrete empirical numbers demonstrating how LLM hallucinations propagate through the system:
When the L.M produces hallucinated outputs (e.g., failing to recognize construction hazard), the estimated Q-value for the "accelerate overtake" action exhibits 23% error compared to the correct estimation. This distortion occurs because:
Q-value Distortion:
The incorrect semantic state representation from ASR module fails to capture the true risk level.
The CSA module, misled by the incorrect scenario understanding, assigns inappropriate critic weights (c.g., underweighting safety).
-The resulting integrated advantage function (s, a) (Eq. 2 in main paper) becomes biased, leading overoptimistic Q-value estimates for dangerous actions.
Performance Degradation: The 23% Q-value estimation error directly translates to an 18 percentage point drop in task success rate (from 85% 67%). This demonstrates the causal chain: hallucinated prompt - distorted Q-value - dangerous behavior (collision).
Why HCRMP Still Maintains 67% Success:
Importantly, with hallucinated LLM outputs, HCRMP maintains a 67% success rate rather than complete failure. This robustness is attributed the LLM--inned paradigm's design:
The RL agent retains self-optimization capability based on environmental feedback, allowing it to partially correct for LLM errors through policy learning.
The Semantic Cache Module (SCM) can replace extremely poor LLM outputs with more reliable cached weights from similar historical scenarios.
The weakly coupled design prevents L.L. hallucinations from completely dominating the decision decision-making process.
15

---

E Limitations and Broader Impacts
While HCRMP demonstrates promising motion planning capabilities, its current limitations primarily involve potential inter-module information loss and the sim-to-real gap inherent in simulator-based evaluations, defining clear avenues for future work. Future work will enhance data fusion and module integration reduce information loss, and validate HCRMP in real-world tests. This will strengthen HCRMP, an LLM-hinted RL paradigm that promises to significantly advance intelligent transportation.
References
[1] W Sing, C Xu, Lin, B Li, et al. A survey on safety-critical scenario generation for autonomous driving-a methodological perspective. arXiv preprint wxXi:2222..02255. 2022.
[2] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055: 03500330, 2024.
[3] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.22277 2024.
[4] DeepSeek-An AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, 7. B. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, and et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[5] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, and et al. The Ilama 3 herd of models, 2024.
[7] Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xuanyuan, Fenghua Zhu, al. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions on Intelligent Vehicles, 866)3369,)7 2023.
[8] Edsger W Dijkstra. A note on two problems in connexion with graphs. In Edsger Wybe Dijkstra: his life, work, and legacy, pages 287-290. 2022.
[9] Peter Hart, Nils Nilsson, and Bertram Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE transactions Systems Science and Cybernetics, 402):100-107, 1968.
[10] Dmitri Dolgov, Sebastian Thrun, Michael Montemerlo, and James Diebel. Practical search techniques in path planning for autonomous driving. ann arbor, (001(48105):18-8) 2008
[11] Francesco Esposto, Jorrit Goos, Arjan Teerhuis, and Mohsen Alirezaei. Hybrid path planning for non-holonomic autonomous vehicles: An experimental evaluation. In 2017 5th IEEE international conference on models and technologies for intelligent transportation systems (MT-ITS), pages 25-30. IEEE, 2017.
[12] Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network, Advances in neural information processing systems, 1, 1988.
16

---

[13] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End to end learning for self-driving cars. arXiv preprint arXiv:/604 1604.073166 2016.
[14] Lecun, E Cosatto, Ben, U Muller, and B Flepp. Dave: Autonomous off-road vehicle control using end-to-end learning. DARPA-IPTO Final Report, 36, 2004.
[15] Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17222-17231, 2022.
[16] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving- In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 7077-7087, 2021.
[17] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9329-9338, 2019.
[18] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-base based sensor fusion for autonomous driving. IEEE transactions pattern analysis and machine intelligence,
2022.
[19] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In Proceedings of the IEEE/CVF international conference computer vision, pages 15222-15232, 2021.
[20] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), pages 8248 2254..2..4 IEEE, 2019.
[21] Dhruv Mauria Saxena, Sangjae Bae, Alireza Nakhaei, Kikuo Fujimura, and Maxim Likhachev. Driving dense traffic with model-free reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5385-5392. IEEE, 2020.
17


Reviews:
[{"review_id": "53vRDMuocb", "reviewer": "Reviewer_it5r", "summary": "This paper argues that maintaining relative independence between the LLM and the RL is vital for solving the hallucinations problem.\n\nReinforcement learning (RL) is <the: you may want to use “a”; there is real-time dynamic programming, and Model-predictive control that also does this; RL is special in that it is sample-based> method for learning optimal policies by maximizing expected returns through interactions with the environment. \n\nFrom line 52, it is a little confusing. How the LLMs are used with RL for driving? According to the text, “scenario understanding and action response, as illustrated in Figure 1 (a).” How are LLMs used in these scenarios?\n\nWithout explaining that clearly, continuing arguing the hallucinations is even more confusing. \n\nRL-Assisted LLM Policy Optimization: I thought it’s the other way around: LLM-Assisted RL Policy Optimization, but this way is also interesting. The review in this section covers a few papers, but it feels still insufficient. \n\nSection 2.2: I see. Here is the other way. So this direction LLM is used to generate intrinsic rewards for RL to improve policy learning. Is this the only way that LLM is used to instruct RL agent? I doubt it. \n\nFigure 2: this is main illustration of the key/framework of the paper. This does not seem to convey easily. It’s better to use some simple example. “On the Expressway”: these texts are helpful but too small and not obvious that they are from LLM. \n\nFine-grained object-level Analysis: this is not clear what it means. There are no texts. Where and how is the LLM used?\n\nWhat do you mean by LLM-augmented representations? Texts?\n\nWhat “prompts” do you mean in the figure?\n\nWhat is A_int? Value function?Then don’t use A for that. Anyhow, it’s not explained anywhere in the figure or caption. \n\nThis is a confusing figure. You don’t want your paper’s idea blurred by a figure that takes an important position and tells not much but confusion. \n\nSection 3.2:\nIt’s not clear where and how s_t^{llm} is generated? Is it from s_t^{raw}?\n\nOkay A_int is defined here. It’s advantage. \n\nClipping is a tricky (introduces a few hyperparameters). But Okay. You are following PPO’s way to deal with off-policy sampling ratio. There is a paper that soft-clips the objective without hard clipping. \n\n\nExperiments are performed on Carla, on Town 2. \n\n\nReferences:\n\nDeep Reinforcement Learning for Autonomous Vehicle Intersection Navigation\nDistributional Reinforcement Learning for Efficient Exploration: they used Carla too for experiments. \n\nTable 2 and Figure 3 is the main result of experiments. \nPut shorthand names like SR in the table caption so it’s easy to find them.", "questions": "see above.", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "LLM-Hinted RL: this has very good result for collision rate. I think you need to do some careful studies into why this is so good. It seems there is no discussion of this in the text.", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Further clarification is requested on how LLMs specifically assist Reinforcement Learning in the aspects of scenario understanding and action response.\n> \n\n**A1:** Thank you for highlighting this narrative shortcoming in our introduction, which we will address in the final version. To clarify, we will detail how, within these methods, an LLM's scenario understanding and action response affect the RL agent, and how hallucinations propagate.\n\n1. **\"RL-Assisted LLM Policy Optimization\":** The LLM is the core decision-maker. It primarily leverages its **action response** capability to directly generate high-level driving policies, such as \"Longitudinal Driving Decisions\" (e.g., accelerate/decelerate) and \"Lateral Driving Decisions\" (e.g., change lanes). The RL agent then assists in optimizing this LLM-dominated policy through environmental feedback.\n    - If the LLM has a hallucination in its **action response** (e.g., its \"Lateral Driving Decisions\" capability fails and it outputs a \"change lane\" command in an unsafe situation), this incorrect command is executed directly, leading to danger.\n2. **\"LLM-Instructed RL Policy Generation\":** The LLM primarily acts as an \"instructor\" leveraging its core **scenario understanding** capability. It is responsible for interpreting and reasoning about complex traffic environments, performing tasks such as \"Hazard Identification,\" \"Road Type Comprehension,\" and \"Road sign comprehension.\" It then translates this high-level understanding into decisive instructions for the RL agent, most commonly by dynamically shaping the reward function.\n    - If the LLM has a hallucination in its **scenario understanding** (e.g., its \"Road sign comprehension\" capability fails, misinterpreting a clear \"Construction Ahead\" sign as \"road clear\"), it will generate a fundamentally flawed reward signal (e.g., continuing to reward \"efficient driving\" instead of \"slowing down\"). When optimizing for this contaminated reward, the RL agent is inevitably misled and will learn a dangerous policy, such as driving at high speed into the construction zone.\n\nIn both of these modes, the LLM's **scenario understanding** and **action response** are strongly coupled with the final driving decision. If the LLM hallucinates at any stage, the negative impact can propagate directly to vehicle control without a buffer, posing a severe safety hazard.\n\n> Q2: The reviewer is confused by the term \"RL-Assisted LLM Policy Optimization\" and finds the literature review in that section insufficient. The reviewer also questions whether \"LLM-Instructed RL\" is limited to only generating intrinsic rewards, suggesting that other methods of instruction likely exist.\n> \n\n**A2:** To ensure the focus of our discussion, we concentrate on studying the methods **most relevant to our work** based on the core mechanism of \"how an LLM's output influences and integrates into the RL decision loop,\" and have not covered all existing interaction paradigms.\n\nWe now outline several mainstream paradigms for LLM-instructed RL for autonomous driving.\n\n| Papers & Conferences/Years | Integration Framework (LLM's role) | Collaborative Policy Optimization | LLM Common-Sense Reasoning | LLM makes high-level policy advice | Scenarios Generation | Human-in-the-Loop |\n| --- | --- | --- | --- | --- | --- | --- |\n| Learningflow (arXiv 2025) | **Reward Designer** | **√** | **√** |  |  |  |\n| Lord (WACV 2025) | **Reward Designer** | **√** | **√** |  |  |  |\n| TeLL-Drive (arXiv 2025) | **Decision Provider** | **√** | **√** | **√** |  |  |\n| LaViPlan (arXiv 2025) | **Decision Provider** | **√** | **√** | **√** |  |  |\n| DriveGPT4 (ICLR 2024) | Language Translator | √ |  |  |  | √ |\n| Towards Human-Centric Autonomous Driving (arXiv 2025) | Language Translator | √ |  |  |  | √ |\n| ChatScene  (CVPR 2024) | Generator |  | √ |  | √ |  |\n| CRITICAL (arXiv 2024) | Generator |  | √ |  | √ |  |\n| HCRMP (ours) | Hinter | **√** | **√** |  |  |  |\n\nWe will now clarify our reasoning for including or excluding these paradigms from our core discussion:\n\n- **LLM as a Reward Designer:** This approach fully leverages the LLM's powerful common-sense reasoning capabilities for collaborative policy optimization. As this direction aligns with our research, it is a key focus of our discussion.\n- **LLM as a Decision Provider:** This approach positions the LLM as a decision-maker that outputs high-level semantic policies, which are then translated into specific vehicle control signals by an RL module. This is fundamentally different from our RL decision mechanism and is therefore not a core topic of our discussion.\n- **LLM as a Generator:** This approach uses the LLM to enhance the training environment (e.g., by generating long-tail scenarios) rather than for policy optimization. As this is a different technical direction, it is outside the scope of our related work.\n- **LLM as a Language Translator:** This paradigm focuses on parsing human language commands (e.g., \"drive faster\"). Since our work does not process such commands, we consider its relevance to be limited.\n\n> Q3: The reviewer found Figure 2 to be confusing and raised the following questions and suggestions for revision :\n> \n> 1. Terms like \"Fine-grained object-level Analysis\" and \"LLM-augmented representations\" are undefined.\n> 2. The meaning of \"prompts\" in the figure is unclear.\n> 3. The term Aint is not explained in the figure or its caption.\n\n**A3:** We sincerely apologize for the lack of clarity in Figure 2 and thank you for the detailed feedback.\n\nOur goal for this figure is to intuitively illustrate the HCRMP framework's technical workflow, especially the interaction between its three core modules (ASR, CSA, and SCM) and their influence on the RL agent. Detailed definitions for terms used in the figure, such as \"Fine-grained object-level Analysis,\" are provided in the subsequent subsections of Sec. 3.\n\nTo resolve this immediately, we will now clarify the key concepts from the figure:\n\n1. \"LLM-augmented representations\": This is the $s_{t}^{llm}$ vector, which is composed of both the \"scenario-level\" (4-dimensional) and \"object-level\" (9-dimensional) analyses. As a 13-dimensional semantic hint, it significantly enriches the RL agent's state space.\n2. \"prompts\": The \"prompts\" refer to the structured query texts we input to the LLM. We designed two types of prompts for state augmentation and policy optimization, respectively (detailed prompt design is available in Appendix C).\n3. $A_{int}$: We apologize for omitting the definition of $A_{int}$ in the caption. $A_{int}$ is our proposed integrated advantage function. As shown in Equation 2, it is the weighted sum of the advantage functions from multiple critics, using the weights ($\\lambda_{i}$) generated by the LLM.\n\nFor the final version, we will overhaul Figure 2 and its caption for improved clarity. The figure's layout will be redesigned for a more intuitive workflow. Furthermore, the caption will be substantially enriched to define key terms directly and to guide readers step-by-step through our framework's architecture and logic.\n\n> Q4: The paper is unclear on how the $s_{t}^{llm}$ vector is generated and whether it is derived from the $s_{t}^{raw}$ state.\n> \n\n**A4:** Thank you for your question. In Sec. 3.2 (lines 167-174), we described the composition of $s_{t}^{llm}$ conceptually. The semantic information within $s_{t}^{llm}$ is generated via carefully designed prompts fed to the LLM. These detailed prompts, which precisely guide the LLM's \"scenario-level abstraction\" and \"object-level analysis,\" are fully provided in our Appendix C.1.\n\n$s_{t}^{llm}$ is not generated directly from $s_{t}^{raw}$. They are parallel information streams: $s_{t}^{raw}$ is the agent's direct perceptual input, while $s_{t}^{llm}$ is a semantic supplement from the LLM's reasoning on broader environmental information. The two are then concatenated to form the final state, $\\mathcal{S}_{t}$.\n\nWe recognize the importance of articulating this key mechanism clearly in the main text. In the final version of our paper, we will **add a more detailed description to Sec. 3.2** to clarify the complete generation process of the $s_{t}^{llm}$ vector from comprehensive environmental information. \n\n> W: The paper presents a significantly improved collision rate for the \"LLM-Hinted RL\" method but lacks an in-depth analysis to explain the underlying reasons for this strong performance.\n> \n\n**A:**  We appreciate you acknowledging our method's strong performance in reducing the collision rate. \n\nIn **Sec. 4.2** of our paper (specifically, lines 245-248 and 263-267), we already provide a preliminary attribution for this phenomenon. \n\nWe state that the superior performance is largely due to the CSA module, which reduces collision risk by \"dynamically enhances the emphasis on safety\" and optimizing the multi-critic coordination strategy. We further attribute this success to the synergy between the ASR and CSA modules, where CSA improves the system's ability to understand complex environments by \"extending the state space,\" ensuring comprehensive driving performance.\n\nTo provide stronger empirical evidence as per your suggestion, we will add **Appendix D.4, \"Efficacy Analysis of HCRMP,\"** to the final paper. This new section will detail further experiments:\n\n- **Critic Weight Trajectory Analysis:** We will plot the critic weights ($\\lambda_i$) over time in safety-critical scenarios to show how the CSA module dynamically increases the safety weight ($\\lambda_{safety}$) in response to risk.\n- **Counterfactual Intervention Experiment:** We will introduce a \"hallucinated\" prompt to the LLM during a successful risk-avoidance maneuver. This experiment is designed to prove the robustness of our \"loosely-coupled\" framework by showing the RL agent can still perform safely.\n\nThese additions will rigorously demonstrate the mechanisms behind HCRMP's effectiveness and provide robust support for our central claims."}, {"review_id": "sfKYhshzFp", "reviewer": "Reviewer_L4iX", "summary": "- In contrast to LLM-dominated RL approaches this work tries to loosely couple ingestion of LLM’s information as auxiliary inputs for RL decision-making process to mitigate the problems due to LLM hallucinations. \n- The paper introduces three modules to augment information from LLMs to learn driving policies. The proposed paradigm LLM-Hinted RL and framework HCRMP (LLM-Hinted Contextual Reinforcement Learning Motion Planner) generates semantic ‘hints’ for state augmentation and RL policy optimization (through the semantic information in the reward design). \n  - Augmented Semantic Representation (ASR) module - for semantic guidance by augmenting information in the state space,\n  - Context Stability Anchor (CSA) module - adaptive weights for the multi-critic function based on some knowledge base, and\n  - Semantic Cache Module - to compensate for any missing output from LLMs due to mismatch in LLM inference and policy execution frequency. \n- Proposes to use the semantic understanding of the LLMs but still trying to reduce the bad policy behavior and instability due to LLM hallucinations. \n- The work makes the RL agent negate or mitigate the impact of the hallucinated outputs during the policy optimization process.", "questions": "- How do we define “hints”? What is the difference between (mathematical, if any) “hint” and LLM-dictating output?\n  - How much percentage of these “hints” are “absorbed” by the policy, Can we regulate that?\n- How is the embedding model trained? How are those embedding aligned to the same representation space as the scene in order to fetch the most similar embeddings?\n- How do we take care of the eviction policy of the semantic cache module? \n  - For how do we maintain the old history states?\n- Can we have a learned multi-head attention and pooling to learn the adaptive trade-offs among the driving attributes for the critic function instead of having the CSA module to manage the weights? How is it different?\n- Section 3.3: Line 195: Sorry, but what do we mean by “norm-constrained semantic source”? Is there a formula/equation/reference to support that statement?\n- Section 4.3 - Line 286: For the experiment without CSA, where are the dynamic weights coming from if not from the LLM?", "limitations": "- The authors talk about sim-to-real gap and inter-module information loss as some of the limitations of the HCRMP framework.\n- How much weightage is given to textual inputs as compared to the other modality (refer to [this](https://arxiv.org/pdf/2503.02199)) (image) inputs while generating the semantic scene information is important in order to get proper semantic understanding to be augmented to the state input is not clear.\n- The framework relies on online interaction with the environment to correct the erroneous actions making this framework difficult to perform well on offline RL settings.\n- For now, the LLM seems to only output hierarchical semantic information about the scene. How would it be to incorporate both observation level and expected behaviour level information?\n- The driving trajectories might still not be optimized for human-like driving behavior. There can be some work to incorporate that (may be in the CSA module to balance the human-like critic). Can we include expert driving/trajectory behavior in the CSA knowledge base as well?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "** Strengths:**\nCoined the following terms and tried to have distinctive categories of different LLM-RL works: LLM-dominated (RL-Assisted LLM Policy Optimization and LLM-instructed RL policy generation), LLM-Assisted RL methods. It gives a clear picture of the current research directions and which category those fall into.\nStudied the effect of incorrect signals on Q-value estimation that might lead to degradation in overall policy performance.\nThe CSA module tries to add more context based knowledge and seems to reduce the weight drifts in the critic network, thereby reducing the variance in complex driving  scenes and the stable reward estimates.\nAnalysed the non-hallucination rates of the different LLMs on driving tasks related QA. It gives a better picture of where these VLMs can be improved or which ones can be preferred. \n\n\n**Weakness:**\n- Any empirical numbers on how the incorrect outputs lead to incorrect Q-value estimates or rewards? \n- There have been some efforts like [DriveVLM](https://arxiv.org/pdf/2402.12289) that provide scene understanding information from VLMs to learn the driving policy. \n- In Table 2, HCRPM makes relatively less progress as compared to VLM-RL, can that be attributed to the less collision rates in HCRMP as well? It would be nice to have other metrics like collision per mile or miles per collision to have a closer comparison.\n\n\nThe paper does the decent job in bringing together the different works of LLM+RL integration. Although, the idea of using LLM information or scene understanding to train an RL policy doesnt seem to be novel.", "was_revised": true, "final_justification": "The authors tried to give better clarity as to why this work is not just leveraging the power of LLMs but is a complementary approach to mitigate the negative effects of LLM hallucinations. Based on their clarification and other experimental results this can be considered a valuable contribution.\nThere are still some future directions that were agreed upon by the authors in the rebuttal and those would be great if worked upon or at least included in the future directions in the final version of the paper.\nI did raise the score from 3 to 4 during the rebuttal and that would keep it that way.", "rebuttal": "> Q: Novelty and Contributions\n> \n\n**A:** We believe your concern regarding novelty may stem from a slight misunderstanding of our work's core contribution. We would like to take this opportunity to clarify that our core contribution is not proposing a method to *use* LLM information, but rather to solve a fundamental challenge within that methodology: **how to build an autonomous driving decision system that is robust to the inherent risk of hallucinations in LLMs**.\n\nYou mentioned, \"The idea of using LLM information or scene understanding to train an RL policy doesn't seem to be novel.\" If our work were merely categorized as \"using LLM information to train RL,\" then it would indeed fall into the same category.\n\nHowever, the innovation of our work lies in **how to leverage the benefits of an LLM's powerful reasoning capabilities while effectively mitigating its inherent risks.** In the safety-critical domain of autonomous driving, LLM hallucinations (such as misinterpreting a scene) can lead directly to catastrophic consequences.\n\nWe propose the novel LLM-Hinted RL paradigm. We wish to emphasize that **this is not intended to conflict with excellent LLM-Dominated methods like DriveVLM, but rather to offer a complementary research perspective.** If the focus of works like DriveVLM is on exploring how to maximize the powerful capabilities of LLMs, then our work focuses on an equally important problem: how to effectively mitigate their intrinsic negative impacts. The core innovation of our framework is the **loose coupling mechanism** between the LLM and the RL agent. We believe that the LLM-Dominated paradigm, which explores the potential of LLMs, and the LLM-Hinted paradigm, which explores how to control their risks, are complementary.\n\n> Q1: The paper needs to clearly define \"hints\" and explain the difference between \"hints\" and an LLM's \"dictating output.”\n> \n\n**A:** As discussed regarding the novelty, the core principle of our \"Hinted\" paradigm is to avoid using LLM outputs as direct commands or interventions. Instead, our approach aims to preserve the LLM's strengths while mitigating its negative impacts.\n\n- Hints: Auxiliary information generated by the LLM helps guide the RL Agent's decision-making process in two ways without directly determining the final action.\n    - State Augmentation: The LLM's scene understanding vector is concatenated with the raw state vector, jointly serving as the input to the RL policy network.\n    - Policy Optimization: The weights ($\\lambda_{i}$) provided by the LLM for the multi-critic framework are used to compute a weighted sum of different advantage functions.\n- LLM-Dictating Output: The LLM's output directly supplants a core function of the RL Agent, for instance, by directly generating the reward function or the action command.\n\n> Q2-3: The paper is missing key implementation details. For instance, it fails to explain the training and alignment of the CSA module's embedding model, or the maintenance strategy including the eviction policy for the Semantic Cache Module.\n> \n\n**A:**  Thank you for your question regarding implementation details. For the CSA module, we use the pre-trained text-embedding-ada-002 model. The alignment is text-to-text: a real-time scene description (Query) is matched against our pre-encoded knowledge base of regulations (Knowledge) using FAISS for Top-K retrieval. The Semantic Cache uses a Least Recently Used (LRU) eviction policy to retain data from rare, critical scenarios.\n\nWhile these engineering choices are crucial for system stability, they are not our core innovation. Given the page limits, we focused our discussion on our main contributions: the LLM-Hinted paradigm and its core modules. We will **add these specific details to the appendix**.\n\n> Q4: The paper does not justify its choice of the CSA module over a learnable alternative, like a multi-head attention network, nor does it discuss the differences between these two approaches.\n> \n\n**A:** We conducted a supplementary experiment that replaces the CSA module with an end-to-end network that uses a 4-head self-attention mechanism and an MLP to generate critic weights, trained jointly with our PPO algorithm. We compared the two approaches in the challenging medium-density \"Trilemma\" scenario. \n\n| Model | SR(%) | CR(%) |\n| --- | --- | --- |\n| HCRMP-Attention | 57.0 | 43.0 |\n| HCRMP | 69.0 | 31.0  |\n\nThe experimental results indicate that an attention mechanism alone may struggle to learn robust weight dynamics in the complex feature space of driving, which requires understanding multi-source inputs and dynamic scenarios.\n\n\"Safety,\" for instance, is a reasoning problem involving traffic rules and risks, not just a simple correlation. This motivates our use of an LLM for \"hinting,\" leveraging its superior reasoning for this critical task. The core advantage of our CSA module is its use of a knowledge base (via RAG) to \"anchor\" this reasoning. This ensures the generated weights are stable, interpretable, and consistent with safe driving regulations.\n\n> Q5: The term \"norm-constrained semantic source\" is not clearly defined.\n> \n\n**A:**  This term refers to our knowledge base of driving regulations (\"norms\") used to constrain the LLM's weight generation via a Retrieval-Augmented Generation (RAG) mechanism. We agree the phrasing is unclear and **will revise it throughout the paper** to be more explicit, e.g., \"Weight generation is constrained by a semantic source of traffic norms, implemented via RAG.\"\n\n> Q6: For the ablation study experiment \"without CSA,\" the paper does not specify the source of the dynamic weights for the multi-critic function.\n> \n\n**A:**  In the \"HCRMP w/o CSA\" ablation, we used fixed weights to create a clear baseline against our dynamic weighting, thus highlighting the CSA module's effectiveness. **We will clarify this in Section 4.3.**\n\n> W1: The paper does not provide empirical numbers to quantify how incorrect LLM outputs lead to incorrect Q-value estimates or rewards.\n> \n\n**A:** The experimental scenario is the high-density overtaking environment mentioned in the paper. The control group consists of data from normal decision-making instances where the model received correct LLM prompts. In contrast, the experimental group comprises instances specifically selected where the LLM produces the dangerous \"hallucinated\" prompts (e.g., failing to recognize a construction site ahead). The experiment focuses on the difference in Q-value judgments for the key action \"accelerate to overtake\" and records the final success rate. This approach allows us to establish a quantitative link from an erroneous prompt, to a distorted Q-value, and ultimately to dangerous behavior.\n\n| Method | Estimated error in Q (%) | Success Rate (%) |\n| --- | --- | --- |\n| HCRMP (LLM output incorrect) | 23 | 85.0 → 67.0 |\n\nTable Metrics Explained:\n\nQ-value Estimation Error: Calculated as $\\frac{|Q_{\\text{incorrect}} - Q_{\\text{correct}}|}{|Q_{\\text{correct}}|}$.\n\nTask Success Rate: Presented in the format: Control Group (correct LLM output) → Experimental Group (incorrect LLM output).\n\n> W2: There have been some efforts like DriveVLM that provide scene understanding information from VLMs to learn the driving policy.\n> \n\n**A**: Please see Q1, Thank you!\n\n> W3: The paper's analysis should Include normalized metrics like 'miles per collision'.\n> \n\n**A:** We have re-processed our data and will update Table 1,2 and 3 in the revision to include 'Collisions per mile' to better reflect the true risk profile. Below is an excerpt from the updated Table 3, which serves as an example of the changes made.\n\n| Model | SR(%) | ··· | Collision per Mile |\n| --- | --- | --- | --- |\n| HCRMP w/o ASR | 40 | ··· | 21.65 |\n| HCRMP w/o CSA | 48 | ··· | 27.90 |\n| HCRMP w/ ASR | 54 | ··· | 25.34 |\n\n> L1: The authors talk about sim-to-real gap and inter-module information loss as some of the limitations of the HCRMP framework.\n> \n\n**A:** Experimental data can be found in our response to Question 3 from Reviewer HcAE.\n\nWhile some inter-module information loss is inevitable, our successful real-world tests demonstrate that our LLM-Hinted paradigm is resilient enough to handle this uncertainty. \n\n> L2: It is unclear how the system weighs textual inputs against visual (image) inputs when generating the semantic information that is used to augment the agent's state.\n> \n\n**A:** Our framework intentionally avoids manual weighting. The RL agent learns to **implicitly weigh the concatenated visual features and LLM hints** during training. The used weighting is therefore dynamic and task-driven, not a fixed hyperparameter.\n\n> L3: The framework relies on online interaction with the environment to correct the erroneous actions making this framework difficult to perform well on offline RL settings.\n> \n\n**A:** HCRMP is intentionally designed to be online, as real-time feedback is core to our 'hint-and-verify' paradigm for safety. We prefer to think of Offline RL as a powerful complement rather than an antithesis. Your question highlights the promising \"Offline-to-Online\" trend, where offline pre-training provides a strong initialization to accelerate online learning and improve sample efficiency. We thank you for this valuable suggestion and will explore integrating this paradigm into our future work.\n\n> L4: The LLM's role is currently limited to providing semantic scene information.\n> \n\n> L5: The paper does not address how to optimize driving trajectories for human-like behavior, nor does it discuss potential methods for doing so.\n> \n\n**A:** Your suggestions perfectly demonstrate the flexibility of our framework. We are grateful for these ideas, which clarify an exciting direction for our future work. We will detail these potential extensions in the conclusion of our revised manuscript.\n\nThank you again for your valuable feedback. We will revise the manuscript carefully based on all your comments."}, {"review_id": "g5iG1kpNci", "reviewer": "Reviewer_HcAE", "summary": "The paper introduces HCRMP, an architecture designed for autonomous driving that integrates Reinforcement Learning (RL) with semantic hints from Large Language Models (LLMs). The authors highlight issues with current methods, specifically the susceptibility to errors (hallucinations) from LLM outputs. Their approach involves modules such as Augmented Semantic Representation, Contextual Stability Anchor, and Semantic Cache, aiming to mitigate these issues. Experiments conducted in the CARLA simulator demonstrate improved driving performance compared to baseline methods", "questions": "- Can the authors clarify the contradiction in their approach? They criticize reliance on LLM outputs but continue to use LLM extensively.\n- How flexible is the system in adapting to scenarios not predefined in the semantic hints?\n- What are the potential practical limitations when transitioning this method from simulation to real-world driving?", "limitations": "- The claims on LLM suffering from hallucination and use LLM as hints to train RL is contradicting.\n- Over-reliance on predefined semantic hint generation may hurt generalizability.\n- Lack of real-world testing hinders practical applicability.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- Clear identification of issues with LLM hallucinations in autonomous driving.\n- Structured modular approach that is clearly presented and easy to follow.\n- Comprehensive experimental validation in simulation.\n\n**Weaknesses:**\n\n- Fundamental contradiction in rationale: authors criticize reliance on LLM outputs but still significantly rely on LLM outputs as hints. The hints may also suffer from hallucination and require heavy prompt engineering and predefined concepts.\n- Predefined prompt engineering heavily constrains the flexibility and general applicability of the solution.\n- No real-world testing. Results only depend on the simulator environment and the results may overfit in the simulator.", "was_revised": true, "final_justification": "Thank you for the detailed response and additional experimental results. The additional experiments strongly enhanced the technical quality of this work. I have updated the score accordingly. In addition to the experiments, i still believe the explanation on \"Strong Reliance\" and \"Hint-based Reliance\" have some space to improve. I hope the authors can highlight this key inspiration of their method in the revision.", "rebuttal": "We sincerely thank you for your detailed evaluation of our paper. Before we address the specific questions, however, we must first state our firm belief that there has been a significant misunderstanding of our paper's core premise and contribution.\n\nOur reasoning is based on your summary in the \"Strengths And Weaknesses, Limitations\" section, where you state that the *\"authors criticize reliance on LLM outputs\"*—**a conclusion that does not align with the actual arguments and terminology in our paper**. In our abstract and introduction, we explicitly state that we analyze and critique the problems of \"over-reliance\" (line 7) and \"strong reliance\" (line 50) in existing LLM-Dominated methods. In these approaches, the LLM's output is used as a direct command or a decisive signal, meaning its hallucinations can directly compromise the driving policy.\n\n**Our argument is not to negate \"reliance\" on LLMs; rather, we aim to explore how to utilize their powerful capabilities more safely and effectively.** This is the core motivation for our LLM-Hinted paradigm. As we state in our paper, the key to our approach is a separated structure:\n\n*“At the same time, this separated structure preserves the utilization of LLM strengths in driving conditions comprehension and common-sense reasoning, enabling context-aware guidance for the RL agent in a way that maintains its fundamental self-optimization capabilities.\"* (line 68)\n\nThis passage clearly shows that our method preserves and utilizes the LLM's strengths in scenario comprehension and reasoning, while mitigating its negative impacts.\n\nTherefore, the premise of our work is not to outright negate the value of LLMs, but rather to acknowledge the limitations of existing LLM-Dominated approaches and propose what we believe is a superior integration method: LLM-hinted RL.\n\nWe have invested significant time and effort into this research. We sincerely ask for a fair and accurate evaluation based on the true merit of our work—a principle we believe is central to the academic spirit of NeurIPS. **We thus hope you will reconsider the genuine motivation and contributions of our work.**\n\n> Question 1: The paper appears to present a contradiction by criticizing heavy reliance on LLM outputs while the proposed framework itself continues to use LLMs extensively.\n> \n\n**Answer 1:** Based on the clarifications above, we can respond to this question more accurately. We argue that no contradiction exists; the key is to distinguish **how one relies on the LLM's output.**\n\n- **\"Strong Reliance\" in LLM-Dominated Methods:** In the LLM-Dominated methods we critique, the LLM's output is decisive. For instance, the LLM directly outputs an action command (\"turn left\") or defines the reward function. In this mode, if the LLM hallucinates (e.g., outputting \"turn left\" at an intersection where a left turn is not allowed), the RL agent will execute it blindly, leading directly to dangerous behavior. This is what we refer to as \"strong reliance\" or \"over-reliance.\"\n- **\"Hint-based Reliance\" in HCRMP:** In our proposed LLM-Hinted paradigm, the LLM's output is auxiliary and advisory.\n    - **As part of the state:** The $s_{t}^{llm}$ vector only enhances the state representation. The final decision, $a_{t}$, is still made by the RL policy network after synthesizing all available information:\n    $$\n    \\pi_{\\theta}(a_t | s_t^{raw}, s_t^{llm})\n    $$\n    Through true environmental feedback, the RL agent learns how to correctly interpret and utilize $s_{t}^{llm}$. If $s_{t}^{llm}$ consistently provides misleading information, the agent can learn to downweigh its importance in the decision-making process.\n    - **As critic weights:** The $\\lambda_{i}$ weights provided by the LLM only influence the relative importance of different driving objectives, rather than directly dictating whether an action is good or bad. What ultimately drives policy learning is the integrated advantage function, $A_{int}$, which is calculated based on real environmental interactions.\n\n> Question 2: The system's flexibility and adaptability to scenarios not explicitly covered by the predefined semantic hints are unclear.\n> \n\n**Answer 2:** We selected \"Overtaking,\" \"Merging,\" and \"Trilemma\" scenarios under medium traffic density and designed three prompts with different wording to test the system's flexibility and adaptability.\n\nThe three prompt sets are: a Detailed Version (Prompt A), which expanded upon the original; a Concise Version (Prompt B), which was a simplified version; and the Original Version (Prompt C), which served as the baseline. The full versions of the prompts and experimental details will be added to Appendix C.3.\n\nThe test results are as follows:\n\n| Driving Conditions | Prompt Sets | Success Rate (%) | Collision Rate (%) | Probability of calling SCM (%) |\n| --- | --- | --- | --- | --- |\n| Overtaking | **Prompt A** | 91.0 | 9.0 | 19.0 |\n|  | **Prompt B** | 88.0 | 12.0 | 4.0 |\n|  | **Prompt C** | 93.0 | 7.0 | 4.0 |\n| Merging | **Prompt A** | 92.0 | 8.0 | 21.0 |\n|  | **Prompt B** | 90.0 | 10.0 | 4.0 |\n|  | **Prompt C** | 92.0 | 8.0 | 6.0 |\n| Trilemma | **Prompt A** | 62.0 | 38.0 | 27.0 |\n|  | **Prompt B** | 61.0 | 39.0 | 5.0 |\n|  | **Prompt C** | 69.0 | 31.0 | 7.0 |\n\nThe experimental results indicate that the HCRMP framework exhibits a degree of robustness to prompts with different styles. While performance varied across prompts—most notably in the challenging \"Trilemma\" scenario—the framework maintained reasonable safety performance across all tests.\n\nNotably, the detailed prompt (Prompt A), despite providing more information, failed to improve and in some cases even degraded performance compared to the original prompt (Prompt C). Simultaneously, it dramatically increased the frequency of Semantic Cache Module (SCM) calls. This indicates that overly complex prompts may introduce counterproductive noise or constraints, leading to a clear trade-off where both safety performance and timeliness are negatively impacted.\n\nTherefore, considering both its superior performance and better real-time capability, the original prompt (Prompt C) achieves the best overall effectiveness by ensuring lower inference latency without sacrificing core safety metrics, making it the optimal choice within the HCRMP framework.\n\n> Question 3: The potential practical limitations of transitioning this method from simulation to real-world driving are not discussed.\n> \n\n**Answer 3:** We thank the reviewer for raising the question about practical applications and fully agree that on-vehicle testing is the ultimate standard for verifying an algorithm's feasibility. To directly address your concern about the \"lack of real-world testing\" and to demonstrate the effectiveness of our method, **we have conducted a preliminary yet crucial validation experiment on a real vehicle.**\n\nWe deployed our HCRMP framework on a vehicle platform. The primary hardware components of the intelligent driving system are as follows:\n\n| Device | Model |\n| --- | --- |\n| Industrial Control Computer | Nuvo-8108GC |\n| GPU | RTX 2080 Ti |\n| Integrated Positioning System | Huace CGI-610 |\n| LiDAR | 128-beam RS-Ruby, 4 RS-Lidar-16 units |\n| Stereo Vision System | Basler acA 1300-60gc front camera |\n\nThe test scenarios involve challenging overtaking maneuvers. We use Lead Vehicle Speed, Average Speed, Longitudinal Distance to Lead Vehicle in Target Lane (m), and Task Success as key evaluation metrics. The results are as follows:\n\n| **Trial No.** | **Lead Vehicle Speed (m/s)** | **Average Speed (m/s)** | **Longitudinal Distance to Lead Vehicle in Target Lane (m)** | **Task Success** |\n| --- | --- | --- | --- | --- |\n| 1 | 2.00 | 4.60 | 16.48 | Yes |\n| 2 | 2.43 | 4.17 | 28.84 | Yes |\n| 3 | 2.33 | 5.35 | 34.22 | Yes |\n| 4 | 2.54 | 5.21 | 33.08 | Yes |\n| 5 | 2.31 | 5.32 | 33.73 | Yes |\n\nThe experimental results demonstrate that the HCRMP architecture is more than just a theoretical concept. It has not only been **successfully transferred from simulation to the real world** but also exhibits strong performance in actual traffic scenarios, proving the framework to be robust, reliable, and viable."}]

Author rebuttal:

Meta-review: This paper explores the integration of large language models (LLMs) with reinforcement learning (RL) to improve motion planning for self-driving cars. While prior approaches often rely heavily on LLM outputs, which can be unreliable due to hallucinations, the authors propose a framework where LLMs provide auxiliary “hints” while the RL agent learns independently, thereby limiting the impact of errors. The system enriches scene understanding, stabilizes decision-making, and bridges the gap between slower LLM guidance and fast driving control. Experiments in CARLA Town2 show higher success rates and fewer collisions than existing LLM–RL baselines.

Reviewers highlighted several strengths. They appreciated the paper’s clear problem motivation around LLM hallucinations, as well as its effort to categorize existing LLM–RL approaches, situating the work in the broader literature. The modular design, comprising semantic augmentation, contextual stabilization, and caching, was seen as well-structured, with the CSA module in particular offering a practical means of stabilizing learning under noisy semantic signals. The analysis of non-hallucination rates of LLMs on driving-related tasks also provides useful insights into current model limitations.

At the same time, reviewers raised important concerns. A recurring issue was a conceptual contradiction: while the paper criticizes over-reliance on LLMs, it still depends heavily on LLM-provided hints, which may themselves suffer from hallucinations and require substantial prompt engineering. Some reviewers also found the methodological presentation confusing, particularly regarding how LLM outputs are generated and incorporated (e.g., figures, definitions, and descriptions of modules). From an evaluation perspective, the lack of key metrics (such as collisions per mile) and the absence of real-world experiments limit confidence in the findings, raising concerns about overfitting to simulation. The degree of novelty was also debated, as similar ideas of using LLM or VLM guidance for RL already exist, and the reported improvements over baselines were seen as modest.

In their rebuttal, the authors clarified that their central contribution lies in introducing an LLM-Hinted paradigm, distinct from LLM-dominated methods, and supplemented their submission with new results. These included a quantitative analysis showing the robustness of hint-based versus strong reliance, a real-world vehicle deployment demonstrating feasibility beyond simulation, and additional ablations on prompt sensitivity and comparisons of the CSA module with alternative designs.

The post-rebuttal discussion was generally positive. One reviewer noted that the clarifications and new experiments effectively addressed earlier concerns, particularly the distinction between hint-based and strong reliance, and expressed satisfaction with the paper. Another reviewer remained unconvinced by this distinction but acknowledged the technical soundness, practical potential and expressed no objection to acceptance.
Decision: Accept (poster)