Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
Authors: Zeyu Dong
Yimin Zhu
Yansong Li
Kevin Mahon
Yu Sun
Venue: CoRL 2024 (2024)
Keywords: End-to-end Autonomous Driving
Large Vision-Language Model
Generalization
TL;DR: We proposed an architecture that combines LLMs with end-to-end driving models, characterized by low data requirements, strong generalization ability, and suitability for real-world closed-loop inference.

Abstract: Traditional autonomous driving methods adopt modular design, decomposing tasks into sub-tasks, including perception, prediction, planning, and control. In contrast, end-to-end autonomous driving directly outputs actions from raw sensor data, avoiding error accumulation. However, training an end-to-end model requires a comprehensive dataset. Without adequate data, the end-to-end model exhibits poor generalization capabilities. Recently, large language models (LLMs) have been applied to enhance the generalization property of end-to-end driving models. Most studies explore LLMs in an open-loop manner, where the output actions are compared to those of experts without direct activation in the real world. Other studies in closed-loop settings examine their results in simulated environments. In comparison, this paper proposes an efficient architecture that integrates multimodal LLMs into end-to-end real-world driving models in a closed-loop setting. The LLM periodically takes raw sensor data to generate high-level driving instructions. In our architecture, LLMs can effectively guide the end-to-end model, even at a slower rate than the raw sensor data, because updates aren't needed every time frame. This architecture relaxes the trade-off between the latency and inference quality of the LLM. It also allows us to choose a wide variety of LLMs to improve high-level driving instructions and minimize fine-tuning costs. Consequently, our architecture reduces the data collection requirements because the LLMs do not directly output actions, and we only need to train a simple imitation learning model to output actions. In our experiments, the training data for the end-to-end model in a real-world environment consists of only simple obstacle configurations with one traffic cone, while the test environment is more complex and contains different types of obstacles. Experiments show that the proposed architecture enhances the generalization capabilities of the end-to-end model even without fine-tuning the LLM.

Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
Zeyu Dong', Yimin Zhu', Yansong Li', Kevin Mahon Yu Sun 'Stony Brook University, $University of Illinois Chicago, $Sunrise Technology Inc.
Abstract: Traditional autonomous driving methods adopt a modular design, de- composing tasks into sub-tasks, including perception, prediction, planning, and control. In contrast, end-to-end autonomous driving, directly outputs actions from raw sensor data avoiding error accumulation. However, training an end-to-end model requires comprehensive dataset. Without adequate data, the end-to-end model exhibits poor generalization capabilities. Recently, large language models (LLMs) have been applied to enhance the generalization capabilities of end-to- end driving models. Most studies explore LLMs an open-loop manner, where the output actions are compared to those of experts without direct activation the real world. Other studies in closed-loop settings examine their results in sim- ulated environments. In comparison, this paper proposes efficient architecture that integrates multimodal LLMs into end-to-end real-world driving models in a closed-loop setting. The LLM periodically takes raw sensor data to generate high-level driving instructions. In our architecture, LLMs can effectively guide the end-to-end model, even at a slower rate than raw sensor data, because updates aren't 't needed every time frame. This architecture relaxes the trade-off between the latency and inference quality of the LLM. It also allows to choose from wide variety of LLMs improve high-level driving instructions and mini- mize fine-tuning costs. Consequently, our architecture reduces the data collection requirements because the LLMs do not directly output actions, and we only need to train a simple imitation learning model to output actions. In our experiments, the training data for the end-to-end model in real-world environment consists only simple obstacle configurations with one traffic cone, while the test environ- ment is more complex and contains multiple obstacles placed in various positions. Experiments show that the proposed architecture enhances the generalization pabilities of the end-to-end model even without fine-tuning the LLM.
Keywords: End-to-end Autonomous Driving, Large Vision-Language Model, Generalization
Introduction
Traditional autonomous driving methods [1] adopt the module design pattern, that is, the task of autonomous driving is decomposed into several subtasks: perception [2, 3, 4J, prediction [5, 6, 7], planning [8, 9, 10], and control [11, 12]. In contrast, end-to-end autonomous driving [13] takes raw sensor data directly and outputs action constraints for real-world inference.
The end-to-end model differs fundamentally from the module design in the context of training, i.e., it only requires sensor data labeled by expert actions instead of intermediate labels of the environ- ment [14, 15. 16] (c.g., lanes, traffic lights, etc.) or other vehicles [17, 18. 19] (e.g., speed, relative locations, driving patterns, etc.). These intermediate labels are much harder to obtain in practice.
"Corresponding author: yu. ww.suuuunriiseaitec.. com
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

1. The obsincies In the image are a blue fucyainiing ain and an orange mallo pore
Training Dataset
2. The uu recycling bin positioned nearer to the ieft wall the hallwey, while the orange traffic cona is more towards the sighty leaning right.
betweon
recycling
wall, orange
right
4. ampty space on right
between the orange
traffic cone the right larger
5. The direction the larger empty space
RIGHT
Tranting
rettucion
Model
Closedioop Evaluation
tramic cone
Figure 1: General idea of the architecture. Top left: the training dataset for the end-to-end model. The purple line represents the trajectories of the for this route. Top right: answer generated by ChatGPT-4o using the image in the bottom right. Botiom left: closed-loop evaluation without LLMs. Bottom right: evaluation on unseen scenarios with high-level instructions LLM still recognizes the new blue trash bin obstacle not included in the training dataset, evaluate viable empty space, and choose justifiable instructions.
unseen scenarios for end-to-end model LLMs. The
In addition, end-to-end methods can potentially avoid accumulation [13] caused by cascad- ing modules in modular design. End-to-end methods rely on data-driven approaches that require vast amounts of data to generalize effectively across diverse enviromment settings. Therefore, many previous end-to-end models have opted to gather extensive training data from a wide range of envi- ronments, often collected in simulators [20, 21J. Despite this, challenges persist when scaling from simple to complex scenarios that involve planning and reasoning, as discussed in Appendix A.
To address this issue, many recent works have explored the combination of language models with autonomous driving. It has been shown [22, 23] that small language models, for example, Bert or GPT, lack the generalization capability. In other words. they cannot generate future optimal trajectories based on the raw data for a new situation they have never encountered. Thus, Large Language Models (LLMs), especially multimodal LLMs, have been explored in autonomous driv- ing [22, 23, 24] and have shown extraordinary abilities to understand driving environments and properly reason about future actions. Multimodal LLMs offer a notable generalization capability when pre-trained for specific task, multimodal datasets encompass vast amount of diversity and have been shown to significantly enhance LLM performance across different tasks [25].
mixture of multimodal datasets. Compared to training in datasets designed
Only handful of studies have applied LLMSs to autonomous driving a closed-loop manner. Due the physical constraints and liability of real-worke driving, these researchers have to design, test, and evaluate driving models simulated environments that sensitive the issue of the slow response time of LLMs. Furthermore, these studies require fine-tuning the LLMs using significant amount of simulation data, which is impractical in real-world settings. Among these efforts, direct approach to adopting LLMs in end-to-end autonomous driving is to employ them as the end-to-end model [26], that is, the L.MM takes human instructions and outputs low-level actions.
LLMs to process the raw sensor data directly and generate actions. Many works explore this setting in open-loop environment where LLM outputs trajectories [27] or steering/throttles signals [28]. These outputs compared with an expert's 's trajectory instead of
An alternative approach

---

being applied to the real world or simulated environment for feedback control. Other efforts rely on L.M in autonomous driving and adopt prompt engineering [29, 26, 30] instead of fine-tuning with LLM to guide the language model to generate the desired outputs.
More recently, Shao al. [31] (LMDriiv)), Azarafza al. [32], and Paul et al. [21] fine-tuned multi- modal LLMs [33] and introduced the tuned LLMs to end-to-end autonomous driving in a closed-loop manner. In these approaches, the fine-tuned LLM takes raw sensor data and outputs future trajec- tories. One challenge of this approach is that the LLM model has long inference latency, and if every action of the vehicle needs one I.LM inference, it is impossible to use an LIM for real-world driving. However, slow LLM inference is not an issue for LMDrive because it uses the CARLA simulation environment. Another potential drawback of fine-tuning an LLM as the controller is catastrophic forgetling [34], property that the L.M may fit the new training data and potentially lose its generalization properties due to fine-tuning.
In this paper, we tackle these issues by liberating the LLM model from direct action controls and creating an architecture that integrates multimodal LLMs with end-to-end autonomous driving. We will assess the effectiveness of the proposed method in a closed-loop real-world environment. In our architecture, the multimodal LLM takes the multimodal sensor data and outputs high-level planning instructions such as LEFT, RIGHT. An end-to-end model (a neural network) then takes the sensor data and the instruction and outputs actions such as steering and throttle.
Fig. demonstrates the key idea of our architecture in an environment where the ego vehicle en- counters obstacles in a corridor. As shown in Fig. 1, the end-to-end model is only trained with an environment consisting of only single front obstacle. The ego vehicle learns to steer left or right avoid obstacles. The end-to-end model might not recognize additional obstacles in adjacent lo- cations. In this scenario, LLMs identify new objects and generate instructions by selecting wide clearance space to navigate around obstacles. During the evaluation stage, these instructions used to guide the end-to-end model even in scenarios outside its training dataset. In this way, we enhance the generalization and robustness of the end-to-end model with LLMs. See Section 3.1 for more details about this experiment.
Our architecture employs LLMs without fine-tuning. Instead, we utilize the Chain-of-Though (CoT) prompting method developed by Wei et al. [35]. In contrast to the previous work, our LLM focuses on generating high-level instructions (left, right, middle, etc) instead of individual actions (steering, throttle, etc). In doing so, we efficiently leverage the inherent abilities of LLMs for sce- nario understanding and reasoning and avoid the weakness of the LLM on specific calculations and inference latency. Conversely, generating actions would require fine-tuning the LLM on specific driving scenarios, resulting in data collection in complex environments. Another advantage of this design is that the LLM is allowed to run slower than the end-to-end model. The vehicle does not need to wait for each inference from the LLM to make single action by caching and applying the previous LLM command to the end-to-end model before generating the next instruction. The end-to-end model uses lightweight neural network that runs on edge devices or smartphones (for example, an iPhone in our robotic vehicle) and responds in milliseconds, meeting real-world infer- ence constraints and compensating for the slow response of the LLM.
In summary, we propose an architecture for autonomous driving that combines end-to-end au- tonomous driving methods with LLMs. This architecture leverages the contextual understanding of I.LMs to provide high-level instructions, enhancing the generalization and robustness of the end- to-end model. This combined model mimics human drivers who occasionally make deliberate deci- sions to alter driving patterns while following decisions instinctively with so-called muscle memory most of the time. The main contributions are as follows.
We are the first to integrate LLMS with end-to-end autonomous driving and apply in a woaldworld, closed-loop environment.
Our framework utilizes hybrid architecture that combines the rapid response capabili- ties of small end-to-end model achieve millisecond latency with the comprehensive
3

---

capacity of LLM for world understanding and inference, enabling great adaptation at the in-vehicle edge and embedding devices to new and previously inexperienced environments.
We only require the LLM to generate high-level instructions rather than low-level actions. Our experiment confirms that this approach relaxes the need to fine-tune the LLM while achieving the generalization capability necessary for end-to-end driving.
Our architecture only requires small amount of training data for the end-to-end model that is easy to collect solely from vision-based sensors, such as vehicle-mounted cameras and smartphones.
Related works: End-to-end autonomous driving: End-to-end autonomous driving [13, 36] has flourished since NVIDIA first introduced [37]. Unlike traditional autonomous driving, end-to-end autonomous driving outputs actions directly from sensor data. There are several methods for end-to-end autonomous driving, such the world model [38, 39], multi-sensor fusing [40, 20), trajectory based control [41, 42], and multi-task/imitttion learning [43, 44, 16, 45, 46]. However, the end-to-end autonomous driving model usually suffers from weak generalization capability. To ad- dress the issue, recent works [22] have explored the potential of using L.LMs autonomous driving improve the generalization capability of the end-to-end model.
LLM for autonomous driving: Most previous works applied LLMs for autonomous driving in an open-loop manner [27, 28], where the output actions or predicted trajectories are compared with experts without applying them to an environment. Recently, Shao et al. [31] explored LLM for autonomous driving in the closed loop. They applied fine-tuned multimodal LLM that takes the raw image and outputs actions. The experiment conducted a simulated environment. Paul et al. [21] and Azarafza al. [32] also developed methods for LLM in closed-loop manner. However, these methods require the LLM to generate outputs for every action taken by the ego vehicle, demanding quick response from the LLM. The authors tested their methods in simulated environments where the slow response problem can be overlooked. Instead, our proposed method does not require a quick response from the LLMs and is vetted in a real driving environment.
2 Methodology
In this study, we develop an autonomous driving system utilizing a car to navigate hallway, relying solely on images taken by a monoscopic camera. Our system captures frontal view images with the onboard camera and preprocesses images before passing them to the driving model to output the steering and throttle actions to control the car. While traditional end-to-end models often face generalization challenges inherent in data-driven approaches, LLMs have shown promise in learning rich representations and contextual understanding from extensive datasets. Inspired by the recent success of LLMs, we incorporated an LLM as an assistant in our end-to-end framework to mitigate the generalization problem.
2.1 Proposed architecture
The proposed architecture consists of two components: an end-to-end model and a pretrained LLM that requires no fine-tuning. The end-to-end model processes front-view images and outputs the corresponding actions, while the LLM provides high-level instructions based on the given images. The end-to-end model is trained respond to the environment efficiently following the high-level instruction. In Fig. 2, we use ChatGPT-4o [47] as an example, demonstrating how our model processes images from continuous camera streams and periodically receives high-level instructions from the LLM at intervals determined by the LLM's inference speed. Next, we discuss how to train an end-to-end model and combine it with LLMs.
End-to-end model The end-to-end model must make predictions in real time from the inputs of the image data and the instructions from the LLM. To make the end-to-end model suitable for taking both images alone and images with high-level instructions input, use the network architecture

---

Sensor Imput
Perciass siliin
Vision Transtormer
Por clsaa sdbuua
Prompt Input
Instructional
CoT Promøt
Identity the
2. Describe the position of the obstacle.
Vision-Language
space available
Compare the space paths.
5. Decide the future direction that leads
larger space.
data both the LLM and the end-to-end model. The end-to-end model outputs actions from sensor images and receives slower, high-level instructions from LLM due to its slower inference speed. This setup bridges the gap between fast-moving vehicles and the contextual insights and slow decisions from the LLM.
Figure 2: The proposed architecture inputs
High-level Query
Sub-queries
Ideutify any obstacle
image.
Describe the position of the obstacles
hallway.
Decide the future direction.
Describe the position of the empty space hetween the obstacles and the wall along the hallway.
Describe which empty space is larger.
Output the direction larger emply spuce LEFT, MIDDLE, RIGHT.
Table 1: CoT query example
and training method similar Hawke al. [16] and Shafiullah al. [48]. In Shafiullah al. [48], the action space is clustered into l different categories. Their model takes images input and uses MinGPT to predict the categorical probability and the per-category action values.
In our end-to-end model, we employ a pre-trained Vision Transformer (ViT) [49] instead of using MinGPT as the image backbone. We manually configured the action space categories instead of using the learned k-means clustering because the action space in our architecture consists only of steering and throttle. Each category assigned LLM instruction. Details on the encoding of the action space be found in Appendix B. We train the end-to-end model the dataset containing the input images and actions taken by human expert as labels, proposed by Hawke et al. [16]. The end-to-end model requires only minimal dataset including simple scenarios that illustrate the instructions provided by the LLM. Specifically, the end-to-end model is trained in an environment that contains only a single cone to avoid, and we use the planning capability of LLM to extend to more complex scenarios. More details about the training environment and the data collection process are discussed in Section 3.1.
LLM for zero-shot inference: As discussed earlier, our end-to-end model lightweight and trained on limited or simple scenarios, resulting in lack of generalization capability for more complex scenarios. To improve the generalization capability of the end-to-end model, an LLM is adopted to enhance the model's understanding of intrinsic scenarios by providing high-level instruc- tions. Instead of fine-tuning the LLM, we utilize the prompt engineering technique CoT [35]. CoT breaks down complex tasks into sequential intermediate reasoning steps. Table is an example of subqueries decomposed from a high-level instruction.
With the help of the LLM, the planning capacity navigate multiple obstacles is integrated into the end-to-end model. The performance of LLM with and without CoT in our experiment is discussed in Appendix D.

---

ChaRGPT
ChatGPT Assist Answer
ChaiGPT
Assist
Distacee
Assist
crange
Ivny, complently blocking passagee
Fimpty
Direction
Directin
emply
side the trafii There the wall
Larger Eipply Sprce:
the right side the hellway is large:
Direction of Larger Empty Space:
RIGHT
Sensor Imput
Input
Sensor Input
Sensor Input
End Model
End to End Model
Endito End Model
Model
Steering Throttle
Steering Thrortie
Steering Throttle
Steering: 0.172
Throtter 0.125
Time
2d
[+]
Figure 3: A closed-loop pipeline of the proposed architecture using ChatGPT-4o: The LLM takes the front- view image of the ego car with CoT prompts and generates the instruction. The end-to-end model then takes the previous M.M--siiite instruction, along with the real-time sensor input, and outputs steering and throttle real-time control. The inference time of the end-to-end model and the LLM are denoted as dl and I accordingly. The steering ranges from (leftmost) to (rightmost) and the throttle ranges between 0 to 1.
2.2 Closed-loop inference
The pipeline of our architecture in the closed-loop is demonstrated in Fig. 3 using ChatGPT-4o as an example. Since inference on the LLM takes longer than the end-to-end model, our end-to-end model uses the cached instruction from the previous LLM inference while waiting for the next instruction. This inference pipeline combines the world-knowledge of the LLM into the end-to-end model while still keeping the whole pipeline from suffering the slow inference speed of the LLM by making lightweight end-to-end model that runs fast even smartphone.
3 Experiment
Our experiment aims to demonstrate that an LLM, even without fine-tuning, improves the general- ization capability of an end-to-end model. Inspired by OpenBot [50, 51] and their early implemen- tation [52], we conducted experiments in a real-world setting on a self-driving robot that integrates OpenBot on a commercial off-the-shelf RC vehicle with a smartphone as the embedded system on- board. We chose the iPhone in our hardware implementation because of its excellent support for PyTorch. The RC car can reach a maximum speed of miles per hour. To avoid any hazard to people property, restricted its speed less than miles per hour. The smartphone onboard sends action signals to the car that subsequently adjusts its steering and throttle accordingly. The only sensor in the car is the back camera of the attached phone. The end-to-end model also runs on the smartphone, enabling it to send low-latency action signals directly to the car.
3.1 Environment setup
To show the generalization property, two different environment setups to train the end-to-end model and test the proposed architecture discussed in Section 2.

---

Training end-to-end model
environment
for
Testing environment for proposed chitecture
Figure 4: Training and testing environments.
Training environment for end-to-end models: The training environment is designed to be as simple as possible. As shown in Fig. 4a', a cone is placed directly in front of the ego car. The ego car can choose to turn left or right to avoid the cone. We manually control the ego car to drive left or right, storing images at 60 frames per second imput and the corresponding actions as labels to train the end-to-end model. The training details be found in Appendix B.
Testing environment for proposed architecture: As shown Fig. 4b, several cones and trash bins placed in zigzag pattern in front of the ego car. To avoid collision, the ego must turn right first to avoid the cones blocking the hallway's ' left. After passing these cones, several more placed right in front of the ego car, requiring the car to turn left to avoid a collision. This provides a challenging testing environment because the model has to reason and plan on what action to take to avoid the front obstacle without hitting the surrounding obstacles. Such a scenario never appears the training dataset. Besides the large and small cones, the environment also includes various distracting items, such as trash bins and doors. The width of the road also changes, becoming wider after passing the initial set of cones. This provides complicated testing environment to challenge the generalization capability of our proposed architecture.
3.2 LLMs and end-to-end models
We combine several LLMs with the end-to-end model. The LLMs considered LLaVA-LLaMA2-13E [53], LLaVA-LLaMA3 [54], MiniGPT-v2 [55], and ChatGPT-4o [47]. The ChatGPT-4o runs on the OpenAl's server, while other LLMs run locally with two NVIDIA RTX A6000 GPUs. The inference time of each model is summarized in Table 2. Note that the infer- ence time required by ChatGPT-4o is even shorter than the inference time some of the other models due to the superior computational resources provided by OpenAl compared to ours. The end-to-end model we utilized is ViT-B/16 [49]. More details discussed in Appendix B.
Model Name
Inference time (s)
LLaVA-LLaMAA2-13B
7.76 + 0.56
LLaVA-LLaMA3-SB
5.86 1.17
ChatGPT-40
7.09 + 2.80
MiniGPT-v2
7.30 1.24
Table 2: Inference time of each LLM
3.3 Results
We evaluate architecture based on different LLMs presented in Section 2.1 and Section 3.2 in the training and testing environments mentioned in Section 3.1. We also perform extended experiments on other types of obstacles and the experiment details and results can be found in Appendix C. We repeat 30 experiments in real-world scenario, and the experiment is counted as "success" if the vehicle passes around all the cones without hitting any of the cones or the wall. The success rate is calculated by the number of successful experiments divided by the total number of experiments. The results of experiments are summarized Table 3. The experiments confirmed that the end-to-end
'The figure is only for demonstration; the actual environments real world settings, simulations.
7

---

model can handle the scenario in its training dataset. However, for a more complicated environment shown in Fig. 4b, the end-to-end model only has a success rate of 40% because of a lack of training data on such scenarios. On the other hand, the LLM understands the complex environment and offers high-level guidance to steer the end-to-end model the correct direction. Nevertheless, the LLM does not always generate correct instructions and experiences hallucinations when the environment disrupted by rapidly changing lighting conditions. We will address these limitations of our driving model in Section 3.4.
Success Rate (%)
Model
Train
Test
LLaVA-LLaMA2-13B- + ViT
100
83
ChatGPT-4o + ViT
100
75
MiniGPT-v2+ ViT
100
75
LLaVA-LLaMA3-7B + VIT
100
63
ViT only
100
40
all values rounded into decimal place.
Table 3: Comparison of different LLM models.
3.4 Limitations
Even LLMs without fine-tuning demonstrate the ability to enhance the generalization capability of the end-to-end model, but there are still limitations. First, in scenarios with strong backlighting and reflection in the front image, the LLM struggles to identify the position of obstacles, resulting in incorrect instructions. This issue arises because the dataset used to train the LLM contains few im- ages with poor lighting conditions. Appendix D.1 provides examples of these failure cases. Second, maximize the ability of the LLM in obstacle identification, the CoT prompt must be designed specifically for the obstacle avoidance task. A sophisticated prompt design and an LLM with long contextual length required to tackle complex driving tasks.
4 Conclusion
We develop architecture that combines LLMs with an end-to-end model to improve the gen- eralization capability of the end-to-end model. The key distinction between our architecture and previously proposed architectures is that the LLM in our architecture only provides high-level in- structions rather than direct actions or trajectories. The end-to-end model uses both raw sensor data and high-level instructions from the LLM. These high-level instructions are received at a slower rate compared to the raw sensor data, allowing the end-to-end model to output actions even while the LLM still makes inferences. By integrating LLMS with the end-to-end model, our architecture relaxes data collection requirements and eliminates the need for rapid responses from LLMs. This design makes it feasible for real-world applications. Experiments are conducted in real-world en- vironment with a self-driving vehicle. The vehicle is equipped solely with the front-view camera as its sensor. We train the driving model in a simple yet common environment and deploy the model on the smartphone. The results show that the end-to-end model alone cannot navigate safely in complex environment that does not appear in the training data. However, when combined with LLMs, the end-to-end model navigates the complex environment successfully. Notably, the LLMs utilize are not fine-tuned; instead, we employ chain-of th-hhuth prompt engineering techniques to enhance performance. The results demonstrate that LLMs improve the generalization capability of the autonomous driving system, enabling it to adapt to more complex environments with sim- ple training data. However, LLMS still exhibit limitations, such as susceptibility to hallucinations when the environment changes, such as backlighting and ground reflection. This requires further investigation and a deeper understanding of L.M behavior.
8

---

Acknowledgments
This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Office of Nuclear Physics, under Award Num- ber DE-SC0019518.
References
[1] D. Parekh, N. Poddar, A. Rajpurkar, M. Chahal, N. Kumar, G. P. Joshi, and W. Cho. A Review on Autonomous Vehicles: Progress, Methods and Challenges. Electronics, 11(14):2162, Jan. 1122162.
2022. ISSN 2079-9292. 100110.330/eeecrroniee
[2] Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. Rus, and S. Han. BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation, June 2022.
[3] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, a.d Dai. BEVFormer: Learning Bird's- Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, July 2022.
[4] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2443-2451, Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-5 doi: 1et.1.0.11/11 1109///VR2420002200.0000
[5] S. Shi, L. Jiang, Dai, and B. Schiele. Motion Transformer with Global Intention Localiza- tion and Local Movement Refinement, Mar. 2023.
[6] X. Jia, L. Chen, P. Wu, J. Zeng. Yan, H. Li, and Y. Qiao. Towards Capturing the Temporal Dynamies for Trajectory Prediction: A Coarse-to-Fine Approach. In 6th Annual Conference on Robot Learning. Aug. 2022.
[7] X. Jia, Wu, L. Chen, Y. Liu, H. Li, and J. Yan. HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding, July 2023.
[8] M. Treiber, A. Hennecke, and D. Helbing. Congested Traffic States in Empirical Observations and Microscopic Simulations. Phys. Rev. E, 1055)18805 Aug. 2000. ISSN 1063-651X, 1095-3787. doi: 10. 1.0./
[9] Z. Zhu and H. Zhao. Survey of Deep RL and IL for Autonomous Driving Policy Learning. IEEE Trans. Intell. Transport. Syst. 23(9): (441)))4443-14006 Sept. 2022. ISSN 1524-9050, 1558- 0016. doi: 1..10.11100 10w.111//TITTS.CO 21010//TTTT.22111133777777
[10] D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta. Parting with Misconceptions about Learning-based Vehicle Motion Planning, Nov. 2023.
[11] A. O. Ly and M. Akhloufi. Learning to Drive by Imitation: An Overview of Deep Behavior Cloning Methods. IEEE Trans. Intell. Veh., 6(2): 209,, June 2021. ISSN 2379-8904, 2379- 8858. doi: 10. 1099111222000300022222
[12] M. Bain and C. Sammut. A Framework for Behavioural Cloning, pages 103-129. Oxford University PressOxford, Jan. 2000. ISBN 978-0-19-853867-7 978-1-383-02650- doi:
0193/0ss0978001983887777 000533667700330006
[13] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li. End-to-end Autonomous Driving: Challenges and Frontiers, Apr. 2024.
[14] A. Petrovskaya and S. Thrun. Model based vehicle tracking for autonomous driving in urban environments. In Robotics: Science and Systems, volume 2008, pages 1-8, 2008.

---

[15] M. Campbell, M. Egerstedt, J. P. How, and R. M. Murray. Autonomous driving in urban en- Approaches, lessons and challenges. Philosophical Transactions of the Royal So- ciery A: Mathematical, Physical and Engineering Sciences, 368(1928)44649 4672, Oct. 2010. doi:10 10i:10..098//se 1098//rta..210..0100
vironments:
[16] Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, and A. Kendall. Urban Driving with Conditional Imitation Learning, Dec. 2019.
[17] Y. Li and S. Han. Efficient collaboration with unknown agents: Ignoring similar agents with- out checking similarity. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, page 2363-2365, Richland, SC, 2024. Interna- tional Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.
[18] N. M. Negash and J. Yang. Driver behavior modeling toward autonomous vehicles: Compre- hensive review. IEEE Access, 11:22788-22822 2023. doi:10. 100,00/0//ACCCSSS.2232244999
[19] N. AbuAli and H. Abou-zeid. Driver Behavior Modeling: Developments and Future Direc- tions. International Journal of Vehicular Technology, 2016:66952791, Dec. 2016. ISSN 1687- 5702. doi: 15/11555911//657771
[20] H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu. Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer, Dec. 2022.
[21] P. Paul, A. Garg, T. Choudhary, A. K. Singh, and K. M. Krishna. LeGo-Drive: Language- enhanced Goall-oreented Closed-Loop End-to-End Autonomous Driving, Mar. 2024.
[22] Z. Yang, X. Jia, H. Li, and J. Yan. LLM4Drive: A Survey of Large Language Models for Autonomous Driving, Dec. 2023.
[23] S. P. Sharan, F. Pittaluga, V. K. B. G, and M. Chandraker. LLM-Assist: Enhancing Closed- Loop Planning with Language -aaseaggge-eedddddd-e Reasoning, Dec. 2023.
[24] H. Sha, Y. Mu, Y. Jiang, I. Chen, C. Xu, P. Luo, S. F. Li, M. Tomizuka, W. Zhan, and M. Ding- LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving, Oct. 2023.
[25] C. Cui, Y. Ma, Cao, Ye, Zhou, Liang, J. Chen, J. Lu, Yang, K.-D. Liao, T. Gao, E. Li. K. Tang, Z. Cao. T. Zhou, Liu, X. Yan, Mei. J. Cao, Z. Wang. and C. Zheng. A Survey Multimodal Large Language Models for Autonomous Driving. In 2024 IEEE/CVF Winter Conference Applications of Computer Vision Workshops ( (WACVW), pages 958-979, Waikoloa, HI, USA, Jan. 2024. IEEE. ISBN 9798350370287. doi: 10. 100.1099//AC 2024.00106.
[26] C. Cui, Ma, X. Cao, W. Ye, and Z. Wang. Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles, Sept. 2023.
[27] Mao, Y. Qian, H. Zhao, and Y. Wang. GPT-Driver: Learning Drive with GPT, Oct. 2023.
[28] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K.-Y. K. Wong, Z. Li, and H. Zhao. DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model, Oct. 2023.
[29] L. Wen, D. Fu, X. Li, X. Cai. T. Ma. P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao. DiLu: Knowledge-Driven Approach to Autonomous Driving with Large Language Models, Feb. 2024.
[30] D. Fu, X. Li. L. Wen, M. Dou, P. Cai, B. Shi, and Y. Qiao. Drive Like a Human: Rethinking Autonomous Driving with Large Language Models, July 2023.
10

---

[31] H. Shao, Y. Hu, L. Wang, S. L. Waslander, Y. Liu, and H. Li. LMDrive: Closed-Loop End-to- End Driving with Large Language Models, Dec. 2023.
[32] M. Azarafza, M. Nayyeri, C. Steinmetz, S. Staab, and A. Rettberg. Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving, Mar. 2024.
[33] H. Liu, C. Li, Q. Wu, and J. Lee. Visual Instruction Tuning, Dec. 2023.
[34] Y. Luo, Z. Yang. E. Meng, Y. Li, Zhou, and Y. Zhang. An Empirical Study of Catastrophic Forgetting Large Language Models During Continual Fine-tuning. Apr. 2024.
[35] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-Thougeee Prompting Elicits Reasoning in Large Language Models, Jan. 2023.
[36] H. Xu, Y. Gao. F. Yu, and T. Darrell. End-to-end learning of driving models from large- scale video datasets. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[37] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to End Learning for Self-Driving Cars, Apr. 2016.
[38] W. Zheng, W. Chen, Y. Huang, B. Zhang, Y. Duan, and J. Lu. OccWorld: Learning 3D Occupancy World Model for Autonomous Driving, Nov. 2023.
[39] Q. Li, X. Jia, S. Wang, and Yan. Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2), July 2024.
[40] R. Zhu, P. Huang, E. Ohn-Bar, and V. Saligrama. Learning Drive Anywhere, Sept. 2023.
[41] Wu, X. Jia, L. Chen, Yan, H. Li, and Y. Qiao. Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline, Oct. 2022.
[42] X. Jia, Wu, L. Chen, Xie, C. He, Yan, and H. Li. Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving, May 2023.
[43] Y. Hou, Z. Ma, C. Liu, and C. C. Loy. Learning to Steer by Mimicking Features from Hetero- geneous Auxiliary Networks, Nov. 2018.
[44] V. De Silva, X. Wang, D. Aladagli, A. Kondoz, and E. Ekmekcioglu. An Agent-Based Mod- elling Framework for Driving Policy Learning in Connected and Autonomous Vehicles. In K. Arai, S. Kapoor, and R. Bhatia, editors, Intelligent Systems and Applications, volume 869, pages 113-125. Springer International Publishing, Cham, 2019. ISBN 978-3-030-01056-0 978-3-030-01057-7.
[45] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end Driving via Conditional Imitation Learning, Mar. 2018.
[46] X. Jia, Y. Gao, L. Chen, J. Yan, P. I.. Liu, and H. Li. DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving, Aug, 2023.
[47] O. al. GPT-4 Technical Report, Mar. 2024.
[48] N. M. M. Shafiullah, Z. J. Cui, Altanzaya, and L. Pinto. Behavior Transformers: Cloning k modes with one stone, Oct. 2022. URL http://arxiv. hwwl:.or//gbba h2wp//bbs/200..... 11251. arXiv:2206.1 1251 [es].
[49] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale, June 2021. URL http: h /ttp:///aivvvv/aa wtwp://wwrrabbb///0.0.// 11929. arXiv:2010.11929 [cs].

---

[50] M. Müller and V. Koltun. Openbot: Turning smartphones into robots. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2021.
[51] M. Müller, S. Brahmbhatt, A. Deka, Q. Leboutet, D. Hafner, and V. Koltun. Openbot-fleet:
A system for collective learning with real robots. In 2024 IEEE International Conference Robotics and Automation (ICRA), 2024.
[52] Codevilla, M. Müller, A. López, Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 4693-4700, 2018. doi:10.1 1tt.111///AAA..11.884.4.... 8..60488..
[53] H. Liu, C. Li, Y. Li, and Lee. Improved baselines with visual instruction tuning, 2023.
[54] W. Wang. Llava-llama-3-8b: A reproduction towards Ilava-3 based on llama-3-8b Ilm back- bone, 2024.
[55] Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny. MiniGPT-v2: Large language model as unified interface for vision- language multi-task learning, Nov. 2023,
12

---

A Limitations of end-to-end network on extending to new scenarios
Training end-to-end model requires a massive amount of data enable the model to adapt to new environment settings, such as varying objects the road, weather and lighting conditions, road textures, and colors. Without a sufficient and diversified dataset, it faces challenges when extending from simple to complex scenarios (not shown in the training dataset) that involve planning and reasoning. In this section, we illustrate this limitation using simple yet representative experiment.
In the experiment, consider three-lane driving scenario, where the lanes designated as left, middle, and right. We create different scenarios by placing obstacles in one more lanes, denoted as to Saama,, where at, am, and represent the presence of an obstacle in the left, middle, and right lanes, respectively (with I indicating an obstacle and 0 indicating no obstacle). The training dataset collected in the scenario S01o. which represents an obstacle the middle lane. In this scenario, the car trained navigate around the obstacle by using either the left or right lane. We then evaluate the model on scenarios So1o. S110, and S011-
Scenario
Success rate (%)
Su10
100
S110
44
So11
78
Table Experiment results on out sample scenarios.
As shown in the experiment result in Table 4, the end-to-end model is not able to reliably extend to even a slightly modified scenario. Although it is possible to include more scenarios like S and So11 in the training dataset, there will always be edge cases that remain unaccounted for. This is why we introduce the VL.M to provide "world model" that enhances the planning and reasoning capabilities of the end-to-end model, enabling generalize from simple complex scenarios by decomposing complex behaviors into a of atomic actions.
B End-to-end network details
This section includes the implementation details of the end-to-end model discussed in Section 2. The end-to-end model is a neural network. The neural network takes the image and instructions from the LLM and outputs the steering and throttle actions. The input image is the front view of the car and taken by the ultra-wide camera the iPhone 15 Pro mounted on the car. The field of view of the camera 101 degrees, and the raw image size is 640 480. The image further cropped with rectangular area [top, bottom, left, right] [140, 330, 130, 510] and is resized to 320 160 before being sent to the end-to-end model. The input instruction
set of commands
(LEFT, MIDDLE, RIGHT), indicating which direction the car should to avoid the front obstacles if the input image indicates there are any. Also, to simplify the response to the LLM, we ignore any cases and recognize STRAIGHT as MIDDLE.
B.1 Neural network architecture
The image backbone for the end-to-end model is the pretrained ViT-B/16 model provided by Doso- vitskiy et al. [49]. The model is modified according to Shafiullah et al. [48] to include two prediction heads: one outputs the per-class action values, and the other outputs the classification probability. The per-class action-value output V is a matrix of size 3 x 2, where each row indicates one of the three commands in the instruction set, the first column is the steering value, and the second column is the throttle value. The classification probability output p is the probability density for each class.
13

---

B.2 Dataset collection
The dataset consists of 60 different routes. Each route consists of sequence of action pairs (image, steering, throttle) with an average length of 100. To train the end-to-end model to take the three different instructions LEFT, RIGHT, and MIDDLE, we collected three different types of routes in the training environments illustrated in Fig. 4a: the first and the second are to pass around the obstacle from the right and left, respectively, and the third is collected when there no obsta- cles in the front and the goes straight, Each route is labeled by its route type. We will discuss in the next subsection how these data are used to train the end-to-end network for different instruction outputs.
B.3 Training the end-to-end model
As discussed in Section B.1 and Section B.2, the neural network outputs three different actions, and the dataset also contains different types of route behaviors. In other words, each sample in the dataset is considered the tuple D = (Xing: Vs: Ut, Vc), where Ximg is the input image, is the steering value range [-1, 1], V1 is the throttle value range [0, I], and is the route label selected representing the three instructions. The neural network can be written as
from the set (V,p) fww; Xim)). where V is a matrix of x 2 representing the per-class action-value output, p is vector of representing the probability distribution of each class, and w is the weight of the neural network. Then the end-to-end model trained with the supervised learning method with the loss function defined
Lww; D) (Vy.,o -va²²
(Vyc.1 - yz)² k log Pye:
where the syper-paramettr weight the loss of the action value and the cross-entropy loss of the classification.
B.4 Evaluation
We evaluate the end-to-end model in real-worid, real-time, and closed-loop conditions. We conduct experiments with and without LLM assistance instructions. When high-level instructions and images are used as input, the prediction probability head discarded, and the L.MM provides the steering and throttle values. When only images are used as input, the actual instruction is sampled from the predicted probability head.
C Extended experiment: moving obstacles
We extend our experiment to more complex scenarios where moving obstacles (e.g. cars, pedestri- ans) are present. The settings for training and testing are similar to Fig. 4a and Fig. 4b. Instead of placing cones in the left or right lane, the obstacle is replaced with moving pedestrians or moving cars. We performed 10 repeated experiments on these testing scenarios, using both an online VLM (ChatGPT-44) and local VLM (LLava-LLaMA2), The success rates for each scenario are shown in Table 5.
VLM ChatGPT-40 LLaVA-LLaMA2
Scenario I (Car)
Scneario 2 (Ped) Scenario (Car+Ped)
50%
60%
33%
100%
80%
55%
Table 5: Comparison of different LLM models with moving obstacles.
D Prompt engineering and results
This section compares the performance across different prompting techniques. Table 6 shows the query context using naive prompting and CoT prompting. We use two different types of query
14

---

Prompt Type
Context
Naive
The image shows toy drives through hallway that might have obstacles.
Please output the future direction the car as LEFT, MIDDLE, or RIGHT.
CoT
A loy car drives through hallway that might have obstacles.
Please answer the following questions step by step:
1. Identify any obstacle in the image.
2. Descrihe the position of the obstacles in the hallway.
3. Describe the position of empty space between the obstacles and the hallway wall.
4. Describe which empty space is larger.
5. Output the direction of larger empty space as LEFT, MIDDLE, or RIGHT.
Table 6: Prompt content for naive, CoT
images where the car should go left or right to avoid the obstacles. Fig. 5a and Fig. 5b show examples of query images. Tables 10 are sample responses for Fig. 5a. Tables 11 14 are sample responses for Fig. 5b.
(
instrection: L.FT..
Ground RIGHT.
Example images with backlighting and reflections.
Figure S: Example images.
D.1 Limitations
This section discusses the case where the LLM might fail. Fig. 5c shows example image where the image contains intensive backlight and reflects light from the ground. In that case, the LLM may fail to generate the correct answer. Table 15 shows sample output for Fig. 5c using LLaVA- LLaMA2.
15

---

Prompt Type
Response
Naive
Based on the image, the toy car appears be moving towards the right side of the hallway.
CoT
1. There orange traffic cones in the hallway, which are obstacles.
2. The traffic cones are positioned in the middle of hallway, creating a pathway on either side.
3. The empty space between the obstacles and the hallway wall is on both sides of the cones, with the space the left side being larger than the space the right side.
4. The car should drive the right side of the obstacle, the space the right is smaller and would be more challenging to navigate.
5. The car should drive the left side of the obstacle, the space on the left is larger and would provide smoother path.
Table Response prompts for Fig. 5a by L.L
Prompt Type
Response
Naive
Based on the image, the toy car is currently in the middle of the hallway. It is not possible to determine the exact direction the car will take without more information or clearer view of the obstacles in the hallway.
CoT
Yes, there cone in the hallway.
2. The cone is located in the middle of the hallway.
The empty space between the cone and the hallway wall is larger on the right side.
4. The car should drive on the left side of the cone.
Table 8; Response of prompts for Fig. Sa by LLaVA -LLMM33
Type
Response
Naive
RIGHT.
CoT
1. There are two obstacles the hallway, one on left and one on the right.
2. The obstacles are positioned parallel to each other and are spaced about 3 feet apart.
3. The empty space between the obstacles and the hallway wall is approxi-
mately 6 feet long.
4. The empty space the left side of the obstacles is larger than the empty
space on the right side.
5.
The car should drive on the left side of the obstacles avoid hitting them.
Table 9: Response of prompts for Fig. Sa by MiniGPT-v2
16

---

Prompt Type
Response
Naive
The future direction of the toy should be to the LEFT. The hallway ahead is obstructed by traffic cones which requires the car to navigate around them.
CoT
Step I:
Identify any obstacle in the image
The image shows multiple red traffic cones as obstacles in the hallway.
Step 2: Describe the position of the obstacles in the hallway
The traffic cones are positioned in the middle and slightly to the right of the
hallway.
Step 3: Describe the position of empty space between the obstacles and the
hallway wall
There is empty space both on the left side and the right side of the obstacles.
The left side is between the obstacles and the left wall, and the right side is between the obstacles and the right wall.
Step 4: Describe which empty space is larger
The empty space on the left side between the obstacles and the left wall larger compared to the space on the right side.
Step 5: Output which side of the obstacle the car should drive as LEFT, MID-
DLE, or RIGHT
The car should drive on the LEFT side of the obstacles.
Table Response of prompts for Fig. Sa by ChatGPT-4o
Prompt Type
Response
Naive
Based on the image, the toy car appears be moving towards the right side of the hallway.
CoT
1. There are orange traffic cones in the hallway, which are obstacles.
2. The traffic cones are positioned in a line along the hallway, with the first cone closer to the camera and the last cone further away.
3. The empty space between the cones and the hallway wall is larger on the
right side of the image.
4. The empty space on the left side of the image smaller.
5. The car should drive on the right side of the obstacle, which is the side with the larger empty space.
Table 11: Response of prompts for Fig. 5b by LLaVA-LLaMA2
Prompt Type
Response
Naive
Based on the image, the toy car is currently in the middle of the hallway, and there are no visible obstacles in the immediate path. Therefore, the future direction of the car is likely to be straight ahead.
CoT
1. Yes, there orange and white cones in the hallway.
2. The cones are lined up in a row, with one cone placed in the middle of the hallway and cones placed on either side of the hallway.
3. The empty space between the cones and the hallway wall is larger on the right side of the hallway.
4. The car should drive through the larger empty space on the right side of the hallway.
5. The car should drive through the larger empty space on the right side of the hallway.
Table 12: Response of prompts for Fig. 5b by LLaVA-LLaMA3
17

---

Prompt Type
Response
Naive
RIGHT.
CoT
1. There are two orange cones in the hallway.
2. The obstacles are placed in zigzag pattern.
3. The empty space between the obstacles and the wall approximately 10
feet long.
4. The left side the obstacle is closer to the wall.
5. The car should drive on the right side of the obstacle.
Table 13: Response of prompts for Fig. 5b by MiniGPT-v2
Prompt Type
Instruction
Naive
To avoid the obstacles (traffic cones) in the hallway, the toy should move the LEFT, as it is the most clear path available.
CoT
1. ** Identify any obstacle in the image:**
The obstacles are orange traffic cones placed in the hallway.
2. ** Describe the position of the obstacles in the hallway:**
The traffic cones are positioned almost in the middle of the hallway but
wards the left side slightly.
3. **Decribe the position of empty space between the obstacles and the hall-
way wall:*
There is significant empty space on the right side of the hallway next the traffic cones.
There is a smaller empty space between the traffic cones and the left wall of the hallway.
4. **Decrribe which empty space is larger:
The empty space on the right side of the traffic cones is larger than the empty space on the left side.
5. **Output which side of the obstacle the car should drive LEFT, MIDDLE.
or *IGHT***
The should drive the RIGHT side of the obstacle.
Table 14: Response of prompts for Fig. 5b by ChatGPT-4o
18

---

Prompt Type
Instruction
Naive
The image shows the toy car approaching three traffic cones placed in the mid- dle of the hallway, partially blocking the middle path. Given the position of the obstacles, the future direction of the car should be RIGHT.
CoT
To analyze the provided image of a toy car driving through hallway with obstacles, let's s address the questions step by step:
1. **Identify any obstacle in the image:**
The obstacle the hallway is set of three orange traffic cones placed in row.
2. Describbe the position of the obstacles in the hallway:***
The traffic cones are positioned in the center of the hallway, forming barrier that extends across the middle part the hallway's width.
3. ** **Decribe the position of the empty space between the obstacles and the hallway wall:**
There is empty space to the left and right of the traffic cones. The left side of the hallway (from the perspective of the toy car) has some space between the cones and the wall with trash/recycling bins. The right side also has space, but there are bins near the wall as well.
4. ** Descriibe which empty space is larger:
The right side of the obstacle appears to have slightly larger empty space compared to the left side. The right side's space seems more unobstructed, especially considering the positioning of the bins which seem to be slightly farther apart away from the wall, giving a bit more clearance.
5. #*tpput which side of the obstacle the car should drive as LEFT, MIDDLE, or RIGHT:* **
The car should drive to the RIGHT side of the obstacles where there is more room.
So, the output is: **IIHH***
Table 15: Response of prompts for Fig. Sc by ChatGPT-40
19


Reviews:
[{"review_id": "m74sB9Tuu3", "reviewer": "Reviewer_XRbz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Use chain-of-thought reasoning to drive a car in OOD settings", "summary_of_recommendation": "Based on the limited novelty and weak experimental results I would recommend rejection. The authors would need to provide experimental results in significantly more complex scenarios, and significantly increase rigor and depth of analysis, for me to change my recommendation.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- It is claimed that \"even if cones replace trash bins, the car may still crash into the cones because one possible control pattern it learned from the training data dictates turning left when encountering cones\". How does the VLM+E2E hybrid architecture improve this? Are we assuming that the underlying E2E policy places more weight on the text instruction than on the image? What happens when the image and text instruction conflict?\n - Does the E2E model express any actual capabilities other than directly following what the LLM tells it to do? Perhaps a good additional baseline would be to directly control the car by the VLM's outputs (+some underlying \"blind\" controller) to demonstrate the importance of the E2E policy.\n - When is the VLM queried? In the right-left testing environment, does the E2E policy know that it should wait for the VLM's response before executing the second task, or is the robot just \"tuned\" to move slowly enough that this is not a problem? If the latter, does that mean the robot will be unable to react if the tow rows of obstacles are placed closer together?\n - Could the CoT query be generalized so as to be usable for more complex reasoning tasks? It would be nice to have a baseline with a more open-ended CoT task.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dSFz7JNWYu", "reviewer": "Reviewer_1ZPV", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a system for autonomous driving that leverages a multi-modal LLMs for high level reasoning along with an command-conditioned end to end model to execute low level controls. More specifically, an off-the-shelf LLM is given the image along with a chain-of-thought prompt to identify obstacles and provide one of three directions: left, right, middle.  The given command is then given to the end-to-end model which predicts the controls. The paper demonstrates the effectiveness of this approach in a real-world closed loop, but toy setting of a indoor robot with a configuration of obstacles.", "summary_of_recommendation": "While I appreciate the real world experiments, I believe the approach and setting are too simplified for the paper to provide much value to the research community.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- How would you extend the proposed method to support real world autonomous driving?\n- Why do you believe certrain LLMs outperform others? Is it well correlated with the performance of those LLMs on other tasks?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1xJpHkzMx", "reviewer": "Reviewer_Fw6g", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work explores several practical issues of deploying LLM for autonomous driving. They solve it by using LLM for high-level instruction and using an e2e model for low-level control. They deploy their algorithm in the real world.", "summary_of_recommendation": "To apply LLM for actual autonomous driving, the data issue and latency issue are non-negligible. This work starts an initial discussion and deploy their algorithm in the real world, which fits CoRL well. Thus, I give strong accept.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Missed reference in the end-to-end autonomous driving field:\n1. Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline. NeurIPS 22\n2. Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving. CVPR 23\n3. DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving. ICCV 23 Oral\n4. Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2). ECCV 24", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In brief, the strengths of the paper are, the use of LLM in autonomous driving, effective reasoning strategy, and end-to-end approach. Although there were concerns, during the rebuttal stage the authors have successfully addressed the weaknesses of the papers to clarify and significantly improve the quality of the publication.
Decision: Accept