AD-H: Autonomous Driving with Hierarchical Agents

## Abstract

Abstract Due to the impressive capabilities of multimodal large language models (MLLMs), recent works have focused on employing MLLM-based agents for autonomous driving in large-scale and dynamic environments.
However, prevalent approaches often directly translate high-level instructions into low-level vehicle control signals, which deviates from the inherent language generation paradigm of MLLMs and fails to fully harness their emergent powers. As a result, the generalizability of these methods is highly restricted by autonomous driving datasets used during fine-tuning.
To tackle this challenge, we propose to connect high-level instructions and low-level control signals with mid-level language-driven commands, which are more fine-grained than high-level instructions but more universal and explainable than control signals, and thus can effectively bridge the gap in between.
We implement this idea through a hierarchical multi-agent driving system named AD-H, including a MLLM planner for high-level reasoning and a lightweight controller for low-level execution.
The hierarchical design liberates the MLLM from low-level control signal decoding and therefore fully releases their emergent capability in high-level perception, reasoning, and planning.
To effectively train AD-H, we build a new autonomous driving dataset with action hierarchy annotations encompassing multiple levels of instructions and driving commands. Comprehensive closed-loop evaluations demonstrate several key advantages of our proposed AD-H system.
First, AD-H can notably outperform state-of-the-art methods in achieving exceptional driving performance, even exhibiting self-correction capabilities during vehicle operation, a scenario not encountered in the training dataset. Second, AD-H demonstrates superior generalization under long-horizon instructions and novel environmental conditions, significantly surpassing current state-of-the-art methods.
We will make our data and code publicly accessible at https://github.com/zhangzaibin/AD-H

## 1 Introduction

Autonomous driving systems represent a major advancement in contemporary transportation, which requires vehicles to automatically operate in large-scale and dynamic environments.
With the rapid advancement of Multimodal Large Language Models (MLLMs) Liu et al. [], Dai et al. [], Li et al. [], Yin et al. [], Zhang et al. [], Zhu et al. [] and MLLM-based agents Driess et al. [], Brohan et al. [, ], Belkhale et al. [], Wang et al. [, ], Lifshitz et al. [], Qin et al. [], Zhou et al. [], recent attempts Sima et al. [], Wang et al. [], Shao et al. [], Chen et al. [], Liu et al. [], Sha et al. [], Wen et al. [], Tian et al. [] have been made to explore MLLMs as the central agent of autonomous driving systems for better perception, reasoning, and interactions, which have achieved remarkable progress.
A predominant paradigm adopted by these methods is to translate high-level contextual instructions into low-level control signals using MLLMs. As MLLMs are pre-trained to generate natural languages, their ability to decode low-level control signals is highly reliant on the autonomous driving datasets used during fine-tuning, causing significant overfitting to specific scenarios and instructions.
As an example, Figure 1 (a) depicts an oversteering scenario that is absent in the training dataset. Most existing methods struggle to adapt to this case and often maintain straight motion even after excessive turning, leading to dangerous situations.
These limitations motivate us to delve into an intriguing and pivotal question:
Is it possible to develop an autonomous driving system that can fully unleash the emergent capabilities of pre-trained MLLM for more intelligent reasoning and stronger scalability towards unseen scenarios and instructions?

Figure: Figure 1: Comparison between the previous method and AD-H in an oversteering scenario not encountered during training. The previous method keeps moving straight, deviating from the intended route. Conversely, the planner of AD-H can provide corrective mid-level commands to the controller, facilitating the vehicle’s re-alignment.
Refer to caption: x1.png

To answer the above question, we explore the idea of hierarchical policy Belkhale et al. [], Chen et al. []. Rather than directly predicting the final control signals, we propose to fill the gap between high-level instructions and low-level control signals with intermediate mid-level commands. On the one hand, compared to high-level contextual instructions, mid-level commands offer a finer granularity and lie closer to low-level control signals, permitting more precise reflection on real-time environmental feedback. On the other hand, different from low-level control signals, mid-level commands are natural language-driven and are therefore better aligned with the pre-training target of MLLMs to leverage their world knowledge. In addition, breaking down high-level instructions into mid-level commands further enables more flexible human interaction and effective shared policy structure learning across similar tasks Belkhale et al. [], giving rise to stronger generalization abilities to novel instruction and scenarios.

In light of the above motivation, we design a Hierarchical Multi-Agent System for Autonomous Driving (AD-H), which comprises two agents: a MLLM-based planner and a lightweight controller. As shown in Figure 1 (b), the planner aims to perform planning and decision-making based on the input contextual high-level instruction and predicts a mid-level command at each decision frame. The mid-level command is then decoded into the low-level control signals by the controller given the current visual input and the contextual instruction. The high-level planner and low-level controller together form a hierarchical policy system, which effectively frees the MLLM from low-level decoding and unlocks its potential for high-level perception, reasoning, and planning. The last issue remaining is the lack of annotated data for training the hierarchical systems, as existing autonomous datasets do not contain mid-level commands. To this end, derived from LMDrive dataset Shao et al. [], we further build a new training dataset including 1,753K frames with hierarchical annotations encompassing multi-level instructions and commands.

Through intensive evaluations under the closed-loop environment, we show that our AD-H enjoys the following two advantages.
First, AD-H can better generalize to novel scenarios. Since the high-level reasoning and low-level execution are decoupled in our hierarchical multi-agent system, the planner solely focusing on high-level reasoning can more effectively leverage the emergent capability of pre-trained MLLMs, yielding stronger generalization power and reasoning ability under unseen driving scenarios and even challenging corner cases. For example, in cases of oversteering, the planner issues corrective instructions to guide the vehicle back on the right track (Figure 1 (b)). In contrast, previous methods tend to severely overfit to control signal patterns within the training set, resulting in a tendency to persistently move straight (Figure 1 (a)). As a result, AD-H achieves a notable improvement in driving performance compared to state-of-the-art methods.
Second, AD-H can better generalize to novel long-horizon instructions.
Our long-horizon experiments reveal that AD-H can comprehensively understand novel long-horizon instructions, perform effective planning, and generate precise driving commands at appropriate decision frames. This has led to a significant improvement in performance for long-horizon tasks. In contrast, existing methods show poor generalization to long-horizon instructions, often resulting in erroneous routes.

The contribution of this paper can be summarized as follows:

- •
We propose AD-H, a hierarchical multi-agent system for autonomous driving, which can significantly unleash the power of MLLMs to achieve higher control precision and generalization.
- •
We construct an autonomous driving dataset with 1,753k multi-level driving command annotations, which can effectively facilitate hierarchical policy learning.
- •
We perform intensive experiments and demonstrate that our approach can considerably outperform state-of-the-art methods and exhibits stronger generalization to novel scenarios and long-horizon instructions.

## 2 Related Works

### 2.1 End-End methods in Autonomous driving

In autonomous driving, precise perception Li et al. [], Yang et al. [], Liu et al. [], Philion and Fidler [], Liang et al. [], Qin et al. [], Li et al. [], Jiao et al. [], Yoo et al. [], Li et al. [], Bai et al. [], Chen et al. [], Huang et al. [], Li et al. [], Park et al. [], Li et al. [], Zhou et al. [], Wang et al. [, , ], Zhang et al. [], Ge et al. [], Li et al. [] and planning are critical. To tackle the prevalent issue of long-tail distribution in autonomous driving scenarios, several generative network-based World Models have been developed Wang et al. [], Jia et al. [], Zhao et al. [], Wen et al. []. These networks can generate a vast array of realistic urban street scenes. However, in order to control the vehicle, a separate planning model needs to be designed to utilize the perception results. To solve this problem, many end-to-end autonomous driving models have been proposed, including reinforcement learning based Prakash et al. [], Wu et al. [], Chitta et al. [], Codevilla et al. [], Cui et al. [] and imitation learning based methods Xiao et al. [], Hanselmann et al. []. Besides these, UniAD Hu et al. [] addresses the problem of end-to-end autonomous driving by utilizing multiple modules in BEV space.

Since the emergence of multimodal large models, the field of autonomous driving has been continuously exploring the possibility of using such large models in an end-to-end manner to solve this problem. LLM-Driver Chen et al. [] uses Vector-former to characterize the perception of the environment by autonomous driving in vector space. Drivegpt4 Xu et al. [] proposes a novel two-stage training multimodal autonomous driving paradigm, which directly regresses control signals and text responses through multi-frame image input and text instructions. DOLPHINS Ma et al. [] innovatively introduces in-context learning into the autonomous driving framework, which can better mimic human higher-order control abilities. Unlike the methods mentioned above that are trained and tested on static datasets, LMDrive Shao et al. [] first conducts closed-loop autonomous driving training and testing on the Carla simulator, demonstrating strong closed-loop control capabilities and scene generalization. As well as several other notable contributions in this area Li et al. [], Zhou et al. [], Ding et al. [], Wang et al. [], Ye et al. [], Peng et al. [], Paul et al. [], Wang et al. []. Besides, there have been some exploratory endeavors to leverage agent-based approaches in the domain of autonomous vehicular navigation Yang et al. [], Mao et al. [].

### 2.2 Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have garnered considerable attention for their remarkable abilities in multimodal perception. Several studies Liu et al. [], Dai et al. [], Zhang et al. [], Zhu et al. [], Lai et al. [], Peng et al. [] focus on integrating visual content into language models, specifically designed to comprehend and reason about images. Among these, LLaVA Liu et al. [] employs a two-stage instruction-tuning pipeline for comprehensive visual and language understanding. InstructBLIP Dai et al. [] combines the language model with an instruction-aware Q-Former to extract visual content highly pertinent to the provided instruction. Additionally, research Deshmukh et al. [], Li et al. [], Zhang et al. [], Guo et al. [], Hong et al. [] is expanding MLLMs to include audio, video, and point clouds, enhancing their ability to handle complex multimodal tasks. This integration allows MLLMs to process spatial, auditory, and visual data simultaneously, significantly improving performance in applications like autonomous navigation and multimedia analysis.

### 2.3 LLMs in Task Planning

In various fields, LLMs have demonstrated their potential in task decomposition for advanced planning. LLMs can incorporate additional visual modules, such as caption descriptions, to perceive environments and influence planning outcomes. SayCan Ahn et al. [] integrates LLMs with robotic capabilities, allowing robots to follow complex, long-term natural language instructions. Here, the LLM provides a high-level understanding of the instructions and identifies skills that can offer corresponding low-level controls. To avoid error accumulation due to model stacking, recent research has explored using MLLMs for planning. ViLa Hu et al. [] leverages the world knowledge inherent in MLLMs, including spatial layouts and object attributes, to make more rational task planning for manipulative tasks. RT-H Belkhale et al. [] improves task execution accuracy and learning efficiency by decomposing complex tasks into simple language instructions that are then converted into robotic actions. Nevertheless, it has mainly been investigated under small-scale and static scenarios. It is unknown whether this philosophy can also generalize to large-scale and dynamic autonomous driving environments. More importantly, it lacks suitable training datasets for learning such systems. Our work has filled the above gaps.

## 3 Method

In this section, we will first delineate the technical details of our proposed AD-H autonomous driving system, and then present the new dataset for training hierarchical multi-agent systems.

Figure: Figure 2: (a) Pipeline of AD-H. The planner breaks down a high-level instruction into mid-level driving commands and the controller decodes low-level waypoints from the mid-level commands. (b) Examples of a high-level instruction, a mid-level command, and low-level waypoints.
Refer to caption: x2.png

### 3.1 Method Overview

The AD-H system consists of two MLLM-based agents, namely a planner and a controller, as illustrated in Figure 2 (a).
At each decision frame, the planner consumes the current visual input and a high-level contextual instruction (*e.g.*, “turn left at the next intersection“), performs reasoning & planning, and makes a decision for the current frame by predicting a mid-level driving command (*e.g.*, “slow down to ensure safety“). The controller then receives the predicted command and converts it into future waypoints to control the vehicle. The planner and controller, together with the input high-level instruction, the predicted mid-level commands, and low-level waypoints form a hierarchical structure of action policy for autonomous driving.
The overall pipeline can be mathematically expressed as

$$ $\mathbf{y}_{t}=g(f(\mathbf{x}_{t},\mathbf{i}),\mathbf{x}_{t},\mathbf{i}),$ (1) $$

where $\mathbf{i}$ denotes the contextual driving instruction, $\mathbf{x}_{t}$ and $\mathbf{y}_{t}$ denotes the visual input and the predicted control signals (*i.e.*, waypoints) for the $t$-th frame, respectively, and $f$ and $g$ represent the high-level planner and low-level controller, respectively.

### 3.2 High-level Planner

In the AD-H system, the planner focuses solely on high-level decision-making without getting involved in the generation of low-level control signals and therefore becomes more specialized. To do so, the planner needs to perform not only visual perception to understand the surrounding environment as well as its ego status but also effective reasoning and planning to break down the contextual instruction into mid-level driving commands. To this end, we adopt a MLLM as the high-level planner to leverage their strong emergent capabilities (We mainly explore LLaVA-7B Liu et al. [] and Mipha-3B Zhu et al. [] in our experiments). Figure 2 (a) illustrates an overview of the MLLM-based planner. At each decision frame, 4 surround-view images are concatenated and fed into a pre-trained vision encoder Radford et al. []. The encoded visual features are further transformed into the textual token space through a projector. Finally, the visual feature together with the tokenized high-level instruction are sequentially fed into the MLLM to predict the mid-level command in an auto-regressive manner.

Through internet-scale pre-training and massive instruction tuning, MLLMs have acquired powerful reasoning ability, along with a wealth of world knowledge, which allows MLLMs to generalize better across various tasks and application scenarios. We then proceed with downstream fine-tuning on our collected autonomous driving dataset (Section 4) to teach MLLMs how to generate precise mid-level commands through the next token prediction given the contextual information. Since the driving commands are also natural languages, this downstream task is essentially consistent with the pre-training objectives of MLLMs. As such, the emergent capabilities of the pre-trained MLLMs can be fully unleashed. Our experiments show that the MLLM-based planner can better generalize to novel driving scenarios, long-horizon instructions, as well as unseen environments, and even exhibits self-correction abilities.

### 3.3 Lightweight Controller

The role of the controller is to translate the intermediate driving commands generated by the planner into executable control signals, which is much easier than directly predicting the control signals from the high-level instructions. Therefore, instead of using the 7B LLaMA model Liu et al. [] as in LMDrive Shao et al. [], we adopt the more lightweight OPT-350M Zhang et al. [] for this purpose. Since OPT-350M is a pure language model, we empower it with visual perception ability by adding an additional vision encoder He et al. [] and a Q-Former Li et al. [].
As shown in Figure 2, the pipeline of the controller is similar to that of the planner. The input images are also encoded by the vision encoder and then projected into the textual feature space through the pre-trained Q-Former. OPT-350M then takes as input the visual embeddings as well as the textual tokens of the high-level instruction and mid-level commands. The hidden state of its output layer serves as the action embedding and is finally decoded into 5 future waypoints through 2-layer MLP. These waypoints can be input into downstream control algorithms (*e.g.*, PID) to produce numerical information for vehicle control, such as speed, throttle, and steering angle.
The above pipeline for the controller can be mathematically expressed as

$$ $\mathbf{h}_{t}=g_{l}(\mathbf{x}_{t},\mathbf{i},\mathbf{c}_{t}),$ (2) $$

$$ $\mathbf{y}_{t}=g_{w}(\mathbf{h}_{t}),$ (3) $$

where $\mathbf{c}_{t}$ represents the mid-level command generated by the planner,
$g_{l}$ and $g_{w}$ denote the OPT-350M model and the MLP for waypoint regression, respectively, and $\mathbf{h}_{t}$ indicates the hidden state output of OPT-350M. During training, we feed the ground-truth mid-level command into the controller and minimize the $L_{1}$ loss between the predicted and ground-truth waypoints.

## 4 Training Dataset Construction

The mid-level driving commands play a pivotal role in training a proficient planner and controller.
Recent works Sima et al. [], Shao et al. [], Zhou et al. [] have collected a significant amount of image and instruction data in real-world scenarios and closed-loop simulators like CARLA Dosovitskiy et al. [].
However, these datasets lack consideration for mid-level driving commands, rendering training impractical. To address this issue, we create a novel action hierarchy dataset LMDrive-H derived from LMDrive dataset Shao et al. [].
Our dataset comprises annotations across three distinct hierarchical levels: high-level instructions, mid-level driving commands, and low-level vehicle control signals.
Initially, we extract about 160k video-instruction pairs from LMDrive dataset Shao et al. [], alongside low-level vehicle control signals for each frame. Subsequently, leveraging the detailed measurement record provided by CARLA Dosovitskiy et al. [] for each frame, including throttle, speed, steering angle, etc., we employ a rule-based methodology (See Supplementary Materials) to retrospectively deduce the mid-level driving commands for each frame.

Specifically, we first develop a comprehensive set of driving commands. Autonomous driving, unlike robotic grasping scenes, takes place in a dynamic and complex environment, necessitating a more fine-grained command construction than merely selecting actions like acceleration, braking, or turning left. Our fine-grained driving command encompasses both perceptual information and motion details, including crucial data about pedestrians, vehicles, and road signs. This approach aligns with the structure of LLMs and reflects the chain of thought ideology Wei et al. [], Sima et al. []. After resampling, we obtain 1,753K hierarchal annotations.
More details about our dataset are presented in Supplementary Materials A.2.

## 5 Experiments

### 5.1 Experimental Settings

#### 5.1.1 Implementation Details.

Our main experiments are achieved by using the pre-trained LLaVA-7B-V1.5 Liu et al. [] with ViT Dosovitskiy et al. [] vision encoder and the OPT-350M Zhang et al. [] with a ResNet50 vision encoder as the high-level planner and low-level controller, respectively.
We also explore other MLLM architectures Liu et al. [], Zhu et al. [] in Section 5.2.4.
Unless otherwise stated, the AD-H is fine-tuned on our LMDrive-H dataset with only vision encoders fixed. For the high-level planner, the initial learning rate is set to 2e-5, and a few steps of warm-up are incorporated into the training process. For the low-level controller, the learning rate is set to 1e-5. Training is conducted using the Adam optimizer with a batch size of 32 on 4 NVIDIA A800 GPUs. Please see Supplementary Materials A.1 for more implementation details.

#### 5.1.2 Evaluation Benchmarks and Metrics

We conduct standard closed-loop evaluations using CARLA simulator Dosovitskiy et al. [] on the LangAuto Benchmark Shao et al. []. On top of LangAuto, we further build two additional benchmarks termed LangAuto-Long-Horizon and LangAuto-Novel-Environment, which contain long-horizon instructions and novel environments, respectively. We present their details as follows.

##### LangAuto Benchmark.

The LangAuto benchmark encompasses a variety of test routes spanning eight towns, diverse weather conditions, and misleading interference. Throughout the testing procedure, algorithms navigate vehicles within the environment, utilizing solely language commands and visual input. The LangAuto benchmark is further divided into three sub-tracks: LangAuto, with routes longer than 500 meters; LangAuto-Short, with routes between 150 and 500 meters; and LangAuto-Tiny, with routes shorter than 150 meters. We follow the prior method Shao et al. [] and perform evaluations separately on these three sub-tracks.

##### LangAuto-Long-Horizon Benchmark.

Planning and decision-making over long-time horizons is a central concern in embodied AI Pirk et al. [], Huang et al. [], Zeng et al. [], Ahn et al. [], Huang et al. [], which typically necessitate a series of sub-instructions to fulfill a primary goal. To ascertain the effectiveness of AD-H in such scenarios, we have built LangAuto-Long-Horizon based on the LangAuto-Tiny Benchmark by combining multiple instructions to form long-horizon instructions. For instance, the instruction series "Alright, you can start driving", "Keep on rolling straight till you get to the next junction," and "Continue in a straight line along your current path" are condensed into a streamlined directive: "Go straight ahead, turn left at the end of the road, then continue straight." Additionally, given that neither our approach nor the baseline model incorporates historical frame information, we include environmental cues in long-horizon instructions to avoid ambiguity (such as uncertainty about whether to turn at a particular intersection). For instance, "Go straight until you see a turning point with palm trees ahead, then turn right and follow the road." Details of long-horizon instructions are presented in Supplementary Materials A.4.2. Since all the long-horizon instructions are absent from the LMDrive-H training set, the LangAuto-Long-Horizon benchmark can also verify the generalization ability of autonomous driving systems to novel instructions.

##### LangAuto-Novel-Environment Benchmark.

To evaluate the generalization ability of autonomous driving systems under new environments, we have built LangAuto-Novel-Environment based on the LangAuto-Tiny Benchmark by only retaining driving routes from 7 out of 8 Towns (Town02-07, 10). To ensure non-overlap between training and testing, we have further removed training data belonging to the above 7 Towns from the LMDrive-H training set.

##### Evaluation Metrics.

We employ three widely used evaluation metrics from the CARLA LeaderBoard Dosovitskiy et al. [], including route completion (RC), infraction score (IS), and driving score (DS). Among them, RC measures the percentage of the planned route that is successfully completed, with a specific focus on the distance covered along designated segments. Any significant deviation from the intended route leads to the episode being marked as a failure. The IS metric keeps track of violations such as collisions or traffic infractions, which decrease the score with each offense. The DS metric combines both the RC and IS scores to provide a comprehensive assessment of progress and safety, serving as the primary evaluation metric.

### 5.2 Results and Analysis

In this section, we mainly investigate the performance of the autonomous driving models from four perspectives: (1) standard evaluation in a closed-loop manner, (2) generalization to novel long-horizon instructions, (3) generalization to novel environments, and (4) performance achieved by using different MLLMs as planners.
As the LangAuto is a new benchmark, only the result of LMDrive Shao et al. [] is available. Therefore, we adopt LMDrive as our main competitor. It should be noted that
LMDrive is one of the pioneering works in language-guided closed-loop driving and can serve as a strong baseline of our method without using hierarchical agents.

#### 5.2.1 Closed-loop Driving Performance

Table 1 reports the
quantitative comparisons on the LangAuto benchmark. It shows that our AD-H significantly outperforms LMDrive for the three different sub-tracks, especially in terms of the main score DS, indicating that the mid-level driving commands generated by our planner enable the controller to act more accurately within large-scale and complex environments.
Through extensive analysis, we further observe that AD-H exhibits frequent self-correction behaviors.
As shown in Figure 3, LMDrive fails to recognize the road conditions after a left turn, causing the vehicle to continue maintaining the steering wheel straight as the previously received high-level navigation instruction. Consequently, the vehicle crosses the lane boundary and deviates from the correct path. In contrast, our AD-H utilizes its planner to dynamically generate mid-level commands, enabling the controller to adjust its posture accordingly, which effectively reduces the risk of traffic jams and accidents. More visualizations are provided in the Supplementary Materials A.4.2.

**Table 1: Comparison on LangAuto benchmark.**
| Method | LangAuto | LangAuto-Short | LangAuto-Tiny |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | DS($\uparrow$) | RC($\uparrow$) | IS($\uparrow$) | DS($\uparrow$) | RC($\uparrow$) | IS($\uparrow$) | DS($\uparrow$) | RC($\uparrow$) | IS($\uparrow$) |
| LMDrive (LLaVA-7B) Shao et al. [] | 36.2 | 46.5 | 0.81 | 50.6 | 60.0 | 0.84 | 66.5 | 77.9 | 0.85 |
| AD-H (Mipha-3B + OPT350m) | 41.1 | 48.5 | 0.86 | 54.3 | 61.8 | 0.86 | 68.0 | 74.4 | 0.87 |
| AD-H (LLaVA-7B + OPT350m) | 44.0 | 53.2 | 0.83 | 56.1 | 68.0 | 0.78 | 77.5 | 85.1 | 0.91 |

Figure: Figure 3: Results of self-correction scenario. (a) High-level instruction; (b) Visualization results of LMDrive; (c) Visualization results of AD-H; (d) Mid-level driving commands predicted by the planner of AD-H. The visual results show that LMDrive maintains a straight trajectory after oversteering, deviating from the intended path. However, AD-H is able to issue precise commands to guide the vehicle back on track.
Refer to caption: x3.png

Table: Table 2: Comparison on LangAuto-Long-Horizon benchmark.

Figure: Figure 4: Results with long-horizon instructions (a). (b) LMDrive persists in following the initial instructions, continuing forward; (c) AD-H can adeptly assess environmental cues to determine the appropriate timing for turning; (d) Mid-level commands produced by AD-H.
Refer to caption: x4.png

#### 5.2.2 Generalization to Long-Horizon Instruction

Table 3 presents the results on the LangAuto-Long-Horizon benchmark, where the high-level navigation instructions are long-horizon and are provided only at the beginning of the driving task.
Considering that both LMDrive and AD-H are trained under short-horizon instructions during the driving process, these unseen long-horizon instruction settings pose a significant challenge in terms of their generalization ability.
Nevertheless, our AD-H still delivers strong performance, surpassing the LMDrive method by a considerable margin.
As illustrated in Figure 4, LMDrive, which directly predicts control signals, struggles to properly understand the global instructions and road conditions provided in the long-horizon instruction. Consequently, it continues straight instead of making a right turn when necessary.
In comparison, the planner of our AD-H continuously analyzes the instructions and the visual environment during the driving process, providing accurate and fine-grained commands to the controller based on the current driving conditions.
These results indicate the promising generalization capability of AD-H for unseen navigation instructions.

#### 5.2.3 Generalization to Novel Environments

Table 3 compares AD-H and LMDrive on the LangAuto-Novel-Environment benchmark to assess their zero-shot adaptation capabilities to the unseen environment.
Our AD-H consistently outperforms LMDrive across all the metrics, which verifies its strong generalization ability to novel environments.

#### 5.2.4 Experiment on Different Planner Architectures.

Table 4 reports the results using different MLLM for the planner.
Following Shao et al. [], We also test additional metrics including Vehicle Collisions (VC), Pedestrian Collisions (PC), Layout Collision (LC), Red Light Violations (RV), and Offroad Infractions (OI).
The results suggest that using a smaller 3B MLLM for the planner leads to a slight performance decrease. However, it remains competitive compared to LMDrive, which is based on the larger LLaVA-7B model.

**Table 4: Experiments of Different Planners.**
| Method |  | DS($\uparrow$) | RC($\uparrow$) | IS($\uparrow$) | VC($\downarrow$) | PC($\downarrow$) | LC($\downarrow$) | RV($\downarrow$) | OI($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LMDrive | (LLaVA-7B) | 66.5 | 77.9 | 0.81 | - | - | - | - | - |
| AD-H | (LLaVA-7B) | 77.5 | 85.1 | 0.908 | 0.000 | 0.000 | 0.000 | 0.082 | 1.620 |
| AD-H | (Mipha-3B) | 66.4 | 79.4 | 0.793 | 1.828 | 0.000 | 2.370 | 0.082 | 1.765 |

## 6 Conclusion and Limitation

##### Conclusion

In conclusion, our proposed hierarchical multi-agent driving system, AD-H, bridges high-level instructions and low-level control signals with mid-level language-driven commands.
By liberating the multimodal large language models from the burden of decoding low-level control signals, AD-H fully leverages their emergent capabilities in high-level perception, reasoning, and planning. This hierarchical design not only enhances the efficiency and reliability of autonomous driving systems but also enables them to achieve remarkable driving performance even in scenarios not encountered during training.
Through comprehensive evaluation, AD-H outperforms the state-of-the-art method, demonstrating remarkable driving performance and adaptability to novel scenarios and instructions. The proposed AD-H harnesses the emergent powers of multimodal large language models, enhancing the efficiency and reliability of autonomous driving systems.

##### Limitation

Given that AD-H operates as a hierarchical agent system, both its size and computational needs are significant. Achieving a lighter version for deployment on actual vehicles will require notable advancements.
Moreover, since our experimental data mainly comes from simulations, it’s crucial to gather more real-world data to improve domain transfer effectively.
Additionally, because virtual scenarios offer limited data diversity, it’s urgent to have richer datasets. These datasets are essential for refining instruction tuning and enhancing the capabilities of MLLMs.

## Appendix A Supplemental Material

### A.1 Implementation Details

#### A.1.1 High-level Planner

##### Visual Input.

The motion planner receives visual input from four directional cameras, each capturing an RGB image. To maintain consistency with the pre-trained VLM, we concatenate these four images vertically and feed them into the visual encoder together. This approach offers two advantages: firstly, it aligns with the input format of the pre-trained VLM, preventing confusion that might arise from separate inputs; secondly, it reduces computational complexity by minimizing the token count. Preliminary experiments indicate that combining the four images adequately meets the requirements for input.

##### Textual Input.

The planner in AD-H is pivotal as it breaks down high-level navigation instructions into mid-level driving commands. In detail, the textual input of the planner is “What motion should the car currently take to accomplish the instruction <High-level Instruction>?”.

##### Training.

In our experiments, we employ two scales of MLLMs: LLaVA-7B-V1.5 Liu et al. [] and Mipha-3B Zhu et al. []. We fine-tune their pre-trained versions on the LMDrive-H dataset for one epoch using 4 $\times$ A800s, with the visual encoder kept frozen. During the independent training of the planner, we assess its performance by measuring accuracy on the validation set, as the AD-H system only supports combined testing. The batch size is set to 32, and 3% of the total steps are allocated for warm-up. We utilize the Adam optimizer with an initial learning rate of 2e-5.

#### A.1.2 Low-level Controller

##### Model.

Similar to the planner, the controller uses the ResNet50 He et al. [] model to extract features from images captured from four different angles. textual input of the planner is “What action should the car do to <High-level Instruction> with the perception and motion <Mid-level Driving Command>?”. These features are then projected into the controller’s input space for the LLM by an adapter made up of MLP, Which are concatenat with motion embeddings, which are processed from driving commands provided by the high-level planner through a tokenizer.We ultimately chose OPT-350m for its optimal balance of performance and speed. The final layer’s hidden features from this model are fed into an MLP-based waypoints predictor, which generates the vehicle’s position for the upcoming five-time steps. These position details are then translated into direct control signals like steering and throttle through a PID algorithm to interact with the vehicle.

##### Training.

Specifically, our experiments are conducted on four A800 GPUs with a batch size of 32. The visual encoder, ResNet50, underwent the same pre-training as used in the LMDrive Shao et al. [] . As with the controller, we set the learning rate at 1e-5, with a weight decay of 0.06. Since the controller directly generates waypoints. We train
controller with L1 loss and use it as evaluation metrics.

### A.2 Dataset Details

#### A.2.1 Overview

The AD-H Dataset is an innovative action hierarchy dataset specifically designed for autonomous driving. It focuses on mid-level driving commands, making training more practical. Specifically, the AD-H Dataset includes 1.7 million entries, each containing RGB images from four directions, high-level instructions, mid-level driving commands, and low-level vehicle control signals. The process of dataset generation is illustrated in Figure 5.

#### A.2.2 Driving Command

In our study, we analyze 26 distinct types of driving sub-commands within the AD-H dataset. These sub-commands cover nearly all the key perceptual objects in various driving scenarios and encompass all necessary driving actions. By combining these sub-commands, we generate over 160 different variations of driving commands. The complete list of driving sub-commands is provided in Table 10.
For instance, when encountering a red light, the appropriate driving command would be, “There is a red light ahead. Apply brakes safely.”

Figure: Figure 5: Dataset Generation Pipeline.
Refer to caption: x5.png

#### A.2.3 Annotation

The mid-level driving command is determined by the detailed driving data provided by CARLA for each frame, which includes information such as throttle, speed, and steering angle. We employed a rule-based methodology to retrospectively deduce the mid-level driving commands for each frame. For example, when encountering a red light, the appropriate driving command would be, ”There is a red light ahead. Apply brakes safely”. The data from CARLA provides information about the red light in the scene and whether the vehicle is braking. Details are presented in Table 10.

#### A.2.4 Resampling

Initial annotation reveals a significant long-tail distribution issue within the dataset: some motion instructions occur hundreds of times more frequently than others. Common driving scenarios, such as maintaining a steady speed or stopping, predominate, while actions like turning and decelerating are relatively rare. This imbalance can severely impact the model’s performance. To address this, we resample the dataset based on the frequency of motion instructions, reducing its size from 3 million to 1.7 million entries, thereby enhancing the dataset’s quality.

### A.3 LangAuto-Long-horizon Benchmark

We present the long-horizon instructions in Table 9.

### A.4 More Results

#### A.4.1 Visualization of an Example

We present a complete example of AD-H, from high-level instructions and sensor input to mid-level driving commands and finally to waypoints, as shown in Table 5.

#### A.4.2 More Visualization

We present more visualization results in Figure 5, Figure 6, Figure 7 and Figure 8.

**Table 5: An example of how our AD-H predicts future waypoints. Our planner provided accurate motion instructions, and the controller accurately execute navigation and motion instructions.**
| Challenging examples of novel and complex environments. |  |
| --- | --- |
| Sensor Input |  |
| High-level Instruction | What motion should the car currently take to accomplish the instruction ”Continue in a straight line along your current path until you reach the upcoming intersection.”? |
| High-level Planner | Slightly below target speed, gently increase acceleration. Make a slight left turn. |
| Low-level Controller | Waypoint: [-0.1512451171875, -2.8828125], [-0.439697265625, -5.984375], [-0.71630859375, -9.1796875], [-1.0048828125, -12.4296875], [-1.201171875, -15.828125] |
|  | Visualization: |
|  |  |

**Table 6: AD-H performs well in complex nighttime turning environments, whereas LMDrive may result in the vehicle stopping in the middle of the road. The green dots in the figure represent waypoints. When a waypoint coincides with the vehicle’s position, it indicates that the vehicle has come to a stop. Navigation Instruction: Upon covering [x] meters, a right turn at the traffic signal is mandatory.**
| Time | AD-H | LMDrive |  |
| --- | --- | --- | --- |
| Driving command | Veritcal View | Vertical View |  |
| $T_{0}$ | Watch out for the car ahead, there’s a vehicle in front. Apply brakes safely. |  |  |
| $T_{1}$ | Slow down to ensure safety. Make a slight right turn. |  |  |
| $T_{2}$ | Slightly below target speed, gently increase acceleration. Keep the steering wheel straight. |  |  |

**Table 7: AD-H performs well in complex turning environments, whereas LMDrive may result in the vehicle stopping in the middle of the road. The green dots in the figure represent waypoints. When a waypoint coincides with the vehicle’s position, it indicates that the vehicle has come to a stop. High-level instruction: Upon covering [x] meters, a right turn at the traffic signal is mandatory.**
| Time | AD-H | LMDrive |  |
| --- | --- | --- | --- |
| Driving command | Vertical View | Vertical View |  |
| $T_{0}$ | Approaching a junction, prepare to follow traffic rules. Slow down to ensure safety. Make a slight right turn. |  |  |
| $T_{1}$ | Approaching a junction, prepare to follow traffic rules. Slow down to ensure safety. Apply brakes safely. |  |  |
| $T_{2}$ | Approaching a junction, prepare to follow traffic rules. Slow down to ensure safety. Make a slight right turn. |  |  |

**Table 8: Our method has stronger instruction following performance.**
| High-level Instruction | Upon completing 10 meters,a left turn at the intersection is compulsory. |  |
| --- | --- | --- |
| Method | LMDrive | AD-H |
| Vertical View |  |  |
| Mid-level Driving Command | None | Slow down to ensure safety. Make a slight left turn. |

**Table 9: Full list of long-horizon instructions in LangAuto-Long-horizon benchmark.**
| ID | Driving command |
| --- | --- |
| 0 | Go straight ahead, turn left at the end of the road, then continue straight. |
| 10 | Go straight until the intersection ahead, then turn right, and continue along the road. |
| 12 | Go straight to the first intersection ahead and turn left, then continue straight. |
| 20 | Turn right ahead and then go straight. |
| 26 | Turn right ahead, go straight, then turn right again. |
| 34 | Go straight to the T-junction ahead, then turn left and follow the route. |
| 44 | Go straight to a crossroads, then turn left, then continue straight. |
| 46 | Go straight to the T-junction, turn right, and continue straight. |
| 48 | Follow the route, and continue straight when you reach the crossroads. |
| 57 | Go straight to the intersection where, on the left front side, there is an open space with some parked vehicles, and turn left. |
| 68 | Keep going along this road. |
| 70 | Turn left at the T-junction ahead, then follow the road. |
| 74 | Turn left ahead when you reach the cornfield, then turn left again when you encounter an open area. |
| 81 | Slightly turn left along the road ahead, then turn right, turn left at the T-junction, and then go straight. |
| 84 | Go straight until you see a turning point with palm trees ahead, then turn right and follow the road. |
| 88 | Turn right at the T-junction, go straight, then turn right at the T-junction where there are grid lines on the ground. Then continue straight. |

**Table 10: Full list of the 26 different types of driving sub-commands in AD-H dataset. Combining sub-commands can result in over 170 variations of driving commands.**
| Type | Driving command |
| --- | --- |
| Perception | Approaching a junction, prepare to follow traffic rules. |
| A vehicle is present at the junction. Be cautious. |  |
| Multiple vehicles are present at the junction. Be cautious. |  |
| Watch out for the car ahead, there’s a vehicle in front. |  |
| Watch out for the cars ahead, there are multiple vehicles in front. |  |
| A vehicle is present in the lane. Be cautious. |  |
| Multiple vehicles are present in the lane. Be cautious. |  |
| There is a bike ahead. Be cautious. |  |
| Multiple bikes are ahead. Be cautious. |  |
| There is a pedestrian ahead. Be cautious. |  |
| Multiple pedestrians are ahead. Be cautious. |  |
| There is a red light ahead. |  |
| There is a stop sign ahead. |  |
| Speed | Slow down to ensure safety. |
| Start accelerating gradually towards the target speed. |  |
| Remain stopped due to brake application. |  |
| Significantly below target speed, accelerate if safe. |  |
| Slightly below target speed, gently increase acceleration. |  |
| Above target speed, decelerate. |  |
| Maintain current speed to match the target speed. |  |
| Steer | Steer right sharply. |
| Make a slight right turn. |  |
| Steer left sharply. |  |
| Make a slight left turn. |  |
| Keep the steering wheel straight. |  |
| Break | Apply brakes safely. |

### A.5 Societal Impacts

The proposed approach of utilizing mid-level language-driven commands in autonomous driving systems presents several potential positive societal impacts. By bridging the gap between high-level instructions and low-level control signals, AD-H could lead to safer and more efficient autonomous driving in diverse and dynamic environments. This could ultimately reduce traffic accidents and fatalities, alleviate congestion, and improve accessibility for individuals with mobility limitations. Moreover, the enhanced generalizability of AD-H may foster wider adoption of autonomous vehicles, potentially leading to reduced greenhouse gas emissions and enhanced urban planning.

However, there are also potential negative societal impacts to consider. Dependence on advanced autonomous driving systems like AD-H may exacerbate existing societal issues such as job displacement in transportation sectors and exacerbate privacy concerns related to the collection and utilization of vast amounts of personal data. Additionally, the deployment of such sophisticated systems could widen the digital divide, as access to and understanding of these technologies may not be equitable across all socioeconomic groups. It’s crucial to address these challenges through thoughtful regulation, education, and inclusive design practices to ensure that the benefits of autonomous driving technologies are equitably distributed across society.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Please fix the `??` in line 312. 
- Does the model intake video sequence of just single frame multi camera images?
- Does the model consume vehicle state information (such as historical speed)?
soundness: 3
strengths: There are existing works that applies MLLMs to autonomous driving and the hierarchical MLLM agent paradigm introduced by this paper is similar to the chain-of-thoughts technique in the LLM area. This paper is original in that it brings this chain-of-thoughts technique to MLLM based autonomous driving and shows advantage in generalization ability and driving behavior.

Overall the paper is easy to follow and clear written.
summary: This paper introduces a new paradigm of applying Multi Modality Large Language Model (MLLM) in autonomous driving motion planning. Compared to existing MLLM planning methods that tries to let the MLLM produce a plan directly from sensor inputs, this work divides the process into a hierarchy where a larger MLLM first produce a high level plan, followed by a smaller MLLM to produce a fine grained waypoint  embedding based on that high level plan. The author trains each part separately based on LMDrive dataset, with the larger model against driving intentions expressed in natural language and the smaller model against waypoints. The authors perform experiments against the baseline method LMDrive that directly produce waypoints from sensor inputs and observed improvements in driving quality.
weaknesses: - The author did not demonstrate failure cases. On the LangAuto benchmark, the best route complete rate is 53.2%, what are the cases where it did not complete?

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: From the comments above, here are the main problems I have for the authors:
1. What theory supports your assertion that incorporating a hierarchical model that decouples perception, planning, and control can help a system built on LLM to achieve better reasoning and generalizability?
2. For your empirical experiment, can you compare other methods for training and evaluating your proposed dataset?
3. What are your method's average extra time and memory requirements compared to the counterpart that does not decouple the planner and controller? Is this trade-off truly worth it in real-life deployment?
soundness: 2
strengths: 1. From their statements, we can agree that the authors have made tremendous efforts to build a large autonomous driving dataset with multi-level driving command annotations to evaluate the model.
2. The ablation study of the experiment has been carefully designed to investigate influences from the training datasets and the different downstream controllers.
summary: The paper addresses the limitations of existing works in autonomous driving that implement large language models (LLMs) for directly translating high-level instruction into low-level control signals. The authors suggest that such a translation procedure, on the one hand, deviates from the original problem paradigm of language generation. On the other hand, it fails to leverage some of the promising features of the LLMs. Therefore, the authors proposed a hierarchical structure where LLMs act only as high-level perception and planning modules, generating natural language commands for controllers to execute. To make their work self-consistent, the authors built a driving dataset for evaluation.
weaknesses: In short, the paper, overall, makes the most minuscule contribution to society. Here are some key problems in the fundamental logic and their experiment design:
1. The authors suggest that by decoupling the perception-planning and controlling stage, one can better leverage LLMs' *emergent capabilities*. However, their justifications are rather feeble - the hierarchy helps free the LLM from generating low-level control signal decoding and focus on perceptive reasoning and planning. From a methodology perspective, the decoupling of planning and control can introduce additional bias during message passing from upstream to downstream. For example, if the LLM upstream yields an arbitrary command such as "slow down to ensure safety," how can the authors make sure that the target speed generated by the downstream controller can guarantee safety? Meanwhile, during fine-tuning, how can you determine whether the controller should be further fine-tuned to improve control quality or the upstream LLM should be fine-tuned to improve command generation?
2. On the other hand, the authors attribute the limitations of previous methods to a cause that **they deviate from the inherent language generation paradigm**. However, existing works have proposed several methods as well as benchmark datasets to exploit and evaluate the reasoning capabilities of the LLMs in autonomous driving systems. For example, in the paper "LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs" from CVPR 2024, the authors propose and provide a collection of atomic command building blocks with high-level objectives and use LLM to fill in blanks in these commands to achieve control. The authors should investigate and compare their work to this paper and other works that similarly showcase their method's superiority.
3. This paper's experiment design is rather poor. On the one hand, the authors built a dataset for self-consistently evaluating their proposed method but **only** compared their results to one of the existing methods. This is far from sufficient to justify their claimed benefits. Meanwhile, it is hard to convince the audience that the dataset is not built with a bias in favor of the proposed method over the competitors. Lastly, the primary concern from decoupling planning and controlling arises from extra time and memory requirements, which the authors didn't investigate further in their experiments.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section.
soundness: 3
strengths: 1. **Conceptual Simplicity and Design Insight**: The hierarchical agent framework, separating the MLLM planner and the controller, is a conceptually straightforward and interesting approach. The design highlights MLLMs’ strengths in high-level reasoning while minimizing their drawbacks in directly outputting low-level action. 
2. **Dataset Contribution**: The new dataset with hierarchical action annotations is a valuable resource for advancing autonomous driving research, especially in tasks requiring a separation between high-level instructions and mid-level driving commands.
3. **Comprehensive Experiments**: The closed-loop evaluations cover a wide range of scenarios, highlighting AD-H’s strengths in generalization, and performance under long-horizon instructions and unseen conditions. The results showcase AD-H’s good performance over SOTA methods and emphasize the effectiveness of hierarchical planning.
4. **Clear and Engaging Presentation**: The writing and presentation are well-structured, with clear explanations of the hierarchical design and insights into MLLM limitations. The quantitative results and qualitative illustrations provide a convincing demonstration of the framework’s effectiveness.
summary: This paper presents a hierarchical framework, AD-H, that leverages multimodal large language models (MLLMs) for autonomous driving in complex, dynamic environments. The framework introduces a dual-agent approach, with an MLLM planner responsible for perceiving environmental information and high-level instructions to generate mid-level commands, and a controller that executes these commands. This separation enables the MLLM planner to focus on high-level reasoning and planning, thus avoiding the limitations of directly mapping language to low-level control signals. Additionally, the authors contribute a new autonomous driving dataset annotated with hierarchical action levels, which supports the training of the AD-H system. Experimental results demonstrate AD-H’s stronger performance compared to state-of-the-art driving baseline (LMDrive), and robust generalization under long-horizon instructions and novel environmental conditions.
weaknesses: 1. **Evaluation of Consistency between Language and Action**: While the hierarchical framework is well-conceived, further evaluation of how consistent the language command and executed actions are, would be beneficial. In other words, do the final control actions comply with the mid-level commands? This is essential for fully validating the model’s real-world safety and applicability.

2. **Labeling of Mid-Level Instructions**: While the authors clearly introduced the hierarchical design, the methodology for labelling mid-level instructions lacks sufficient detail. For example, how do the authors define the high-level and mid-level instructions, and what is the principal difference between them? What is the design principle for mid-level instructions? While code snippets for generating mid-level commands are attached in the supplementary, descriptions of these design principles would better help readers grasp the gist of the design.

3. **Inclusion of Point Cloud in the Controller Only**: As shown in Fig 2, the choice to include point cloud data only within the MLLM controller, but not in the MLLM planner, is interesting but unexplained. More context regarding this decision would clarify whether it impacts the controller’s performance or if it’s just to balance computational resources. 

4. **Ablation Study on the Motion Resampling**: In Section 6.1 and Table 5, the ablation study findings appear counterintuitive, as resampling is expected to improve performance but seems to lower certain metrics in LMDrive. An explanation regarding the discrepancy, and an additional experiment with AD-H trained without resampling, would possibly strengthen the study. It's fine that the resampling increases the performance, since that is also part of the contribution in this paper. And clarifying the performance contributions of each component would improve the message to the community regarding AD-H’s core strengths.

5. **Minor Presentation Issues**: A few minor presentation improvements are recommended, including correcting the typo of reference failure on Line 313 and marking the second-highest metrics in tables with an underscore to improve clarity.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: 1. The paper is clearly presented, easy to understand.
2. AD-H fully leverages the MLLM’s capabilities in perception, reasoning, and planning by focusing on generating mid-level commands rather than direct control signals, which maximizes the utility of the MLLM in autonomous driving.
3. The proposed system demonstrates impressive performance, including self-correction during operation, and outperforms current state-of-the-art methods in generalization, especially in new or complex driving scenarios.
summary: This paper introduces AD-H, a hierarchical framework for autonomous driving using multimodal large language models (MLLMs). Unlike conventional methods that translate high-level instructions directly into control signals, AD-H separates high-level planning from low-level control. The MLLM planner generates mid-level commands, which the controller then executes. This paper also provides an autonomous driving dataset with multi-level instructions and driving commands. Closed-loop evaluations show that AD-H outperforms existing methods in driving performance, exhibits self-correction capabilities, and demonstrates excellent generalization under long-horizon instructions and new environments.
weaknesses: 1. The innovation of hierarchical planning is limited: DriveMLM has investigated the LLM-based mid-level lateral and longitudinal decision commands by setting decision states in Carla.
2. This paper uses LLaVA-7B and OPT-350M as the high-level planner and low-level controller, respectively. Can the 7B planner provide accurate mid-level instructions? Can the 350M controller output a well-planned trajectory based on ground truth instructions? 
3. Separate evaluation experiments for each module should be added to better assess the performance of each module and the effectiveness of mid-level instructions in enhancing the controller.
4. Figure 3 demonstrates the self-correction capability of the MLLM controller, but what is the source of this capability? More examples should be provided to analyze the self-correction ability. Additionally, when training the MLLM controller, we aim for strong instruction-following capability. Can the current model balance instruction-following with self-correction effectively?
5. In Table 5, I understand that the ADH dataset used to train LM-Drive does not include mid-level instructions and only uses the resample method. The superior performance of ADH compared to LM-Drive may stem from the mid-level instructions in the dataset. An experiment should be added where mid-level instructions are input along with instructions to train LM-Drive for comparison, analyzing the advantages of using a low-level controller versus having LLaVA-7B generate trajectories directly.
6. Regarding model-related questions: After training the MLLM planner and controller separately, was joint training conducted? Are the decision frequency of the planner and the planning frequency of the controller consistent?
7. The hyperlinks for tables and figures are set incorrectly, such as in Tables 5 and 6, and Figures 4, 5, 6, and 7. There is also a hyperlink error in lines 312–313.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why not compare with other recent approaches like CarLLaVA or other hierarchical systems?
2. Have you analyzed how different levels of granularity in mid-level commands affect performance?
3. What is the computational overhead of the hierarchical system compared to end-to-end approaches?
4. How does the system handle conflicts between high-level instructions and mid-level commands?
soundness: 3
strengths: - The hierarchical decomposition of driving tasks into high-level planning and low-level control is well-motivated and effectively leverages the strengths of MLLMs
- The mid-level driving commands provide an elegant interface between language understanding and vehicle control
- Clear and well-organized presentation
- Creation of a new dataset with hierarchical annotations
summary: This paper proposes AD-H, a hierarchical multi-agent system for autonomous driving that decomposes the driving task into two levels handled by separate agents: (1) a high-level MLLM-based planner that translates natural language driving instructions into mid-level driving commands, and (2) a lightweight controller that converts these commands into low-level vehicle control signals. The key insight is separating the high-level reasoning from low-level control, allowing the MLLM to focus on its strengths in language understanding and planning rather than directly generating control signals. The authors also contribute a new dataset called LMDrive-H with 1.7M frames annotated with hierarchical actions at multiple levels. Through extensive experiments in the CARLA simulator, AD-H demonstrates superior performance compared to state-of-the-art methods, particularly in generalizing to novel scenarios and long-horizon instructions.
weaknesses: ### Limited baseline comparisons：
- Should include more recent works like CarLLaVA and other hierarchical approaches
### Insufficient ablation studies:
- No analysis of impact of different granularities of mid-level commands

### Limited novelty
 The core idea of hierarchical decomposition is relatively straightforward and similar approaches have been explored in diverse domains, including autonomous driving and robotics control.

### The experimental validation could be more comprehensive:
- Only one main baseline (LMDrive)
- Limited analysis of failure cases


### Limited Real-World Validation:

- All experiments are conducted in simulation (CARLA)
- No discussion of potential sim-to-real transfer challenges


### Computational Requirements:


- The system requires two large models (7B parameter MLLM + 350M parameter controller). Runtime analysis and latency considerations are not discussed


### Dataset Construction:
- The rule-based annotation process for mid-level commands could introduce biases
- More details needed on annotation quality control measures