Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs

Authors: zeyu  dong
yimin  zhu
yansong  li
kevin  mahon
yu  sun
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 04:56:51

Abstract: Traditional autonomous driving methods adopt modular design, decomposing tasks into sub-tasks, including perception, prediction, planning, and control. In contrast, end-to-end autonomous driving directly outputs actions from raw sensor data, avoiding error accumulation. However, training an end-to-end model requires a comprehensive dataset. Without adequate data, the end-to-end model exhibits poor generalization capabilities. Recently, large language models (LLMs) have been applied to enhance the generalization property of end-to-end driving models. Most studies explore LLMs in an open-loop manner, where the output actions are compared to those of experts without direct activation in the real world. Other studies in closed-loop settings examine their results in simulated environments. In comparison, this paper proposes an efficient architecture that integrates multimodal LLMs into end-to-end real-world driving models in a closed-loop setting. The LLM periodically takes raw sensor data to generate high-level driving instructions. In our architecture, LLMs can effectively guide the end-to-end model, even at a slower rate than the raw sensor data, because updates aren't needed every time frame. This architecture relaxes the trade-off between the latency and inference quality of the LLM. It also allows us to choose a wide variety of LLMs to improve high-level driving instructions and minimize fine-tuning costs. Consequently, our architecture reduces the data collection requirements because the LLMs do not directly output actions, and we only need to train a simple imitation learning model to output actions. In our experiments, the training data for the end-to-end model in a real-world environment consists of only simple obstacle configurations with one traffic cone, while the test environment is more complex and contains different types of obstacles. Experiments show that the proposed architecture enhances the generalization capabilities of the end-to-end model even without fine-tuning the LLM.

Main paper: 
 introduction: 
 Traditional autonomous driving methods [1] adopt the module design pattern, that is, the task of autonomous driving is decomposed into several subtasks: perception [2, 3, 4] , prediction [5, 6, 7] , planning [8, 9, 10] , and control [11, 12] . In contrast, end-to-end autonomous driving [13] takes raw sensor data directly and outputs action constraints for real-world inference.The end-to-end model differs fundamentally from the module design in the context of training, i.e., it only requires sensor data labeled by expert actions instead of intermediate labels of the environment [14, 15, 16] (e.g., lanes, traffic lights, etc.) or other vehicles [17, 18, 19 ] (e.g., speed, relative locations, driving patterns, etc.). These intermediate labels are much harder to obtain in practice.1. The obstacles in the image are a blue recycling bin and an orange traffic cone. 2. The blue recycling bin is positioned nearer to the left wall of the hallway, while the orange traffic cone is more towards the center, slightly leaning right. 3. There is an empty space between the blue recycling bin and the left wall, and another empty space between the orange traffic cone and the right wall. 4 . The empty space on the right side between the orange traffic cone and the right wall is larger. 5. The direction of the larger empty space is RIGHT. In addition, end-to-end methods can potentially avoid error accumulation [13] caused by cascading modules in a modular design. End-to-end methods rely on data-driven approaches that require vast amounts of data to generalize effectively across diverse environment settings. Therefore, many previous end-to-end models have opted to gather extensive training data from a wide range of environments, often collected in simulators [20, 21] . Despite this, challenges persist when scaling from simple to complex scenarios that involve planning and reasoning, as discussed in Appendix A.To address this issue, many recent works have explored the combination of language models with autonomous driving. It has been shown [22, 23] that small language models, for example, Bert or GPT, lack the generalization capability. In other words, they cannot generate future optimal trajectories based on the raw data for a new situation they have never encountered. Thus, Large Language Models (LLMs), especially multimodal LLMs, have been explored in autonomous driving [22, 23, 24] and have shown extraordinary abilities to understand driving environments and properly reason about future actions. Multimodal LLMs offer a notable generalization capability when pre-trained on a mixture of multimodal datasets. Compared to training in datasets designed for a specific task, multimodal datasets encompass a vast amount of diversity and have been shown to significantly enhance LLM performance across different tasks [25] .Only a handful of studies have applied LLMs to autonomous driving in a closed-loop manner. Due to the physical constraints and liability of real-world driving, these researchers have to design, test, and evaluate driving models in simulated environments that are not sensitive to the issue of the slow response time of LLMs. Furthermore, these studies require fine-tuning the LLMs using a significant amount of simulation data, which is impractical in real-world settings. Among these efforts, a direct approach to adopting LLMs in end-to-end autonomous driving is to employ them as the end-to-end model [26] , that is, the LLM takes human instructions and outputs low-level actions.An alternative approach uses LLMs to process the raw sensor data directly and generate actions. Many works explore this setting in an open-loop environment where LLM outputs trajectories [27] or steering/throttles signals [28] . These outputs are compared with an expert's trajectory instead of being applied to the real world or simulated environment for feedback control. Other efforts rely on LLM in autonomous driving and adopt prompt engineering [29, 26, 30] instead of fine-tuning with LLM to guide the language model to generate the desired outputs.More recently, Shao et al. [31] (LMDrive), Azarafza et al. [32] , and Paul et al. [21] fine-tuned multimodal LLMs [33] and introduced the tuned LLMs to end-to-end autonomous driving in a closed-loop manner. In these approaches, the fine-tuned LLM takes raw sensor data and outputs future trajectories. One challenge of this approach is that the LLM model has a long inference latency, and if every action of the vehicle needs one LLM inference, it is impossible to use an LLM for real-world driving. However, slow LLM inference is not an issue for LMDrive because it uses the CARLA simulation environment. Another potential drawback of fine-tuning an LLM as the controller is catastrophic forgetting [34] , a property that the LLM may fit the new training data and potentially lose its generalization properties due to fine-tuning.In this paper, we tackle these issues by liberating the LLM model from direct action controls and creating an architecture that integrates multimodal LLMs with end-to-end autonomous driving. We will assess the effectiveness of the proposed method in a closed-loop real-world environment. In our architecture, the multimodal LLM takes the multimodal sensor data and outputs high-level planning instructions such as LEFT, RIGHT. An end-to-end model (a neural network) then takes the sensor data and the instruction and outputs actions such as steering and throttle.Fig. 1 demonstrates the key idea of our architecture in an environment where the ego vehicle encounters obstacles in a corridor. As shown in Fig. 1 , the end-to-end model is only trained with an environment consisting of only a single front obstacle. The ego vehicle learns to steer left or right to avoid obstacles. The end-to-end model might not recognize additional obstacles in adjacent locations. In this scenario, LLMs identify new objects and generate instructions by selecting a wide clearance space to navigate around obstacles. During the evaluation stage, these instructions are used to guide the end-to-end model even in scenarios outside its training dataset. In this way, we enhance the generalization and robustness of the end-to-end model with LLMs. See Section 3.1 for more details about this experiment.Our architecture employs LLMs without fine-tuning. Instead, we utilize the Chain-of-Thought (CoT) prompting method developed by Wei et al. [35] . In contrast to the previous work, our LLM focuses on generating high-level instructions (left, right, middle, etc) instead of individual actions (steering, throttle, etc). In doing so, we efficiently leverage the inherent abilities of LLMs for scenario understanding and reasoning and avoid the weakness of the LLM on specific calculations and inference latency. Conversely, generating actions would require fine-tuning the LLM on specific driving scenarios, resulting in data collection in complex environments. Another advantage of this design is that the LLM is allowed to run slower than the end-to-end model. The vehicle does not need to wait for each inference from the LLM to make a single action by caching and applying the previous LLM command to the end-to-end model before generating the next instruction. The end-to-end model uses a lightweight neural network that runs on edge devices or smartphones (for example, an iPhone in our robotic vehicle) and responds in milliseconds, meeting real-world inference constraints and compensating for the slow response of the LLM.In summary, we propose an architecture for autonomous driving that combines end-to-end autonomous driving methods with LLMs. This architecture leverages the contextual understanding of LLMs to provide high-level instructions, enhancing the generalization and robustness of the endto-end model. This combined model mimics human drivers who occasionally make deliberate decisions to alter driving patterns while following decisions instinctively with so-called muscle memory most of the time. The main contributions are as follows.• We are the first to integrate LLMs with end-to-end autonomous driving and apply it in a real-world, closed-loop environment.• Our framework utilizes a hybrid architecture that combines the rapid response capabilities of a small end-to-end model to achieve millisecond latency with the comprehensive capacity of LLM for world understanding and inference, enabling great adaptation at the in-vehicle edge and embedding devices to new and previously inexperienced environments.• We only require the LLM to generate high-level instructions rather than low-level actions.Our experiment confirms that this approach relaxes the need to fine-tune the LLM while achieving the generalization capability necessary for end-to-end driving.• Our architecture only requires a small amount of training data for the end-to-end model that is easy to collect solely from vision-based sensors, such as vehicle-mounted cameras and smartphones.Related works: End-to-end autonomous driving: End-to-end autonomous driving [13, 36] has flourished since NVIDIA first introduced it [37] . Unlike traditional autonomous driving, end-to-end autonomous driving outputs actions directly from sensor data. There are several methods for end-to-end autonomous driving, such as the world model [38, 39] , multi-sensor fusing [40, 20] , trajectory based control [41, 42] , and multi-task/imitation learning [43, 44, 16, 45, 46] . However, the end-to-end autonomous driving model usually suffers from weak generalization capability. To address the issue, recent works [22] have explored the potential of using LLMs in autonomous driving to improve the generalization capability of the end-to-end model.LLM for autonomous driving: Most previous works applied LLMs for autonomous driving in an open-loop manner [27, 28] , where the output actions or predicted trajectories are compared with experts without applying them to an environment. Recently, Shao et al. [31] explored LLM for autonomous driving in the closed loop. They applied a fine-tuned multimodal LLM that takes the raw image and outputs actions. The experiment is conducted in a simulated environment. Paul et al. [21] and Azarafza et al. [32] also developed methods for LLM in a closed-loop manner. However, these methods require the LLM to generate outputs for every action taken by the ego vehicle, demanding a quick response from the LLM. The authors tested their methods in simulated environments where the slow response problem can be overlooked. Instead, our proposed method does not require a quick response from the LLMs and is vetted in a real driving environment.-A set of four orange traffic cones. 
methodology: 
 In this study, we develop an autonomous driving system utilizing a car to navigate a hallway, relying solely on images taken by a monoscopic camera. Our system captures frontal view images with the onboard camera and preprocesses images before passing them to the driving model to output the steering and throttle actions to control the car. While traditional end-to-end models often face generalization challenges inherent in data-driven approaches, LLMs have shown promise in learning rich representations and contextual understanding from extensive datasets. Inspired by the recent success of LLMs, we incorporated an LLM as an assistant in our end-to-end framework to mitigate the generalization problem.The proposed architecture consists of two components: an end-to-end model and a pretrained LLM that requires no fine-tuning. The end-to-end model processes front-view images and outputs the corresponding actions, while the LLM provides high-level instructions based on the given images.The end-to-end model is trained to respond to the environment efficiently following the high-level instruction. In Fig. 2 , we use ChatGPT-4o [47] as an example, demonstrating how our model processes images from continuous camera streams and periodically receives high-level instructions from the LLM at intervals determined by the LLM's inference speed. Next, we discuss how to train an end-to-end model and combine it with LLMs.End-to-end model The end-to-end model must make predictions in real time from the inputs of the image data and the instructions from the LLM. To make the end-to-end model suitable for taking both images alone and images with high-level instructions as input, we use the network architecture and training method similar to Hawke et al. [16] and Shafiullah et al. [48] . In Shafiullah et al. [48] , the action space is clustered into k different categories. Their model takes images as input and uses MinGPT to predict the categorical probability and the per-category action values.In our end-to-end model, we employ a pre-trained Vision Transformer (ViT) [49] instead of using MinGPT as the image backbone. We manually configured the action space categories instead of using the learned k-means clustering because the action space in our architecture consists only of steering and throttle. Each category is assigned an LLM instruction. Details on the encoding of the action space can be found in Appendix B. We train the end-to-end model on the dataset containing the input images and actions taken by a human expert as labels, as proposed by Hawke et al. [16] .The end-to-end model requires only a minimal dataset including simple scenarios that illustrate the instructions provided by the LLM. Specifically, the end-to-end model is trained in an environment that contains only a single cone to avoid, and we use the planning capability of LLM to extend to more complex scenarios. More details about the training environment and the data collection process are discussed in Section 3.1.LLM for zero-shot inference: As discussed earlier, our end-to-end model is lightweight and trained on limited or simple scenarios, resulting in a lack of generalization capability for more complex scenarios. To improve the generalization capability of the end-to-end model, an LLM is adopted to enhance the model's understanding of intrinsic scenarios by providing high-level instructions. Instead of fine-tuning the LLM, we utilize the prompt engineering technique CoT [35] . CoT breaks down complex tasks into sequential intermediate reasoning steps. Table 1 is an example of subqueries decomposed from a high-level instruction.With the help of the LLM, the planning capacity to navigate multiple obstacles is integrated into the end-to-end model. The performance of LLM with and without CoT in our experiment is discussed in Appendix D.-The traffic cones are placed in a row across the hallway, located towards the left half side but not completely blocking the passage.The pipeline of our architecture in the closed-loop is demonstrated in Fig. 3 using ChatGPT-4o as an example. Since inference on the LLM takes longer than the end-to-end model, our end-to-end model uses the cached instruction from the previous LLM inference while waiting for the next instruction. This inference pipeline combines the world-knowledge of the LLM into the end-to-end model while still keeping the whole pipeline from suffering the slow inference speed of the LLM by making a lightweight end-to-end model that runs fast even on a smartphone. 
position of empty space:: 
 -There is some empty space on the left side of the traffic cones between the cones and the hallway wall.-There is a larger empty space on the right side of the traffic cones between the cones and the hallway wall. 4. Larger Empty Space:-The empty space on the right side of the hallway is larger. The steering ranges from -1 (leftmost) to 1 (rightmost) and the throttle ranges between 0 to 1.Our experiment aims to demonstrate that an LLM, even without fine-tuning, improves the generalization capability of an end-to-end model. Inspired by OpenBot [50, 51] and their early implementation [52] , we conducted experiments in a real-world setting on a self-driving robot that integrates OpenBot on a commercial off-the-shelf RC vehicle with a smartphone as the embedded system onboard. We chose the iPhone in our hardware implementation because of its excellent support for PyTorch. The RC car can reach a maximum speed of 24 miles per hour. To avoid any hazard to people or property, we restricted its speed to less than ten miles per hour. The smartphone onboard sends action signals to the car that subsequently adjusts its steering and throttle accordingly. The only sensor in the car is the back camera of the attached phone. The end-to-end model also runs on the smartphone, enabling it to send low-latency action signals directly to the car.To show the generalization property, we use two different environment setups to train the end-to-end model and test the proposed architecture discussed in Section 2.1. Training environment for end-to-end models: The training environment is designed to be as simple as possible. As shown in Fig. 4a 1 , a cone is placed directly in front of the ego car. The ego car can choose to turn left or right to avoid the cone. We manually control the ego car to drive left or right, storing images at 60 frames per second as input and the corresponding actions as labels to train the end-to-end model. The training details can be found in Appendix B.Testing environment for proposed architecture: As shown in Fig. 4b , several cones and trash bins are placed in a zigzag pattern in front of the ego car. To avoid a collision, the ego car must turn right first to avoid the cones blocking the hallway's left. After passing these cones, several more are placed right in front of the ego car, requiring the car to turn left to avoid a collision. This provides a challenging testing environment because the model has to reason and plan on what action to take to avoid the front obstacle without hitting the surrounding obstacles. Such a scenario never appears in the training dataset. Besides the large and small cones, the environment also includes various distracting items, such as trash bins and doors. The width of the road also changes, becoming wider after passing the initial set of cones. This provides a complicated testing environment to challenge the generalization capability of our proposed architecture.We combine several LLMs with the end-to-end model.The LLMs we considered are LLaVA-LLaMA2-13B [53] , LLaVA-LLaMA3-8B [54] , MiniGPT-v2 [55] , and ChatGPT-4o [47] . The ChatGPT-4o runs on the OpenAI's server, while other LLMs run locally with two NVIDIA RTX A6000 GPUs. The inference time of each model is summarized in Table 2 . Note that the inference time required by ChatGPT-4o is even shorter than the inference time of some of the other models due to the superior computational resources provided by OpenAI compared to ours. The end-to-end model we utilized is ViT-B/16 [49] . More details is discussed in Appendix B.LLaVA-LLaMA2-13B We evaluate our architecture based on different LLMs presented in Section 2.1 and Section 3.2 in the training and testing environments mentioned in Section 3.1. We also perform extended experiments on other types of obstacles and the experiment details and results can be found in Appendix C. We repeat 30 experiments in a real-world scenario, and the experiment is counted as "success" if the vehicle passes around all the cones without hitting any of the cones or the wall. The success rate is calculated by the number of successful experiments divided by the total number of experiments. The results of all experiments are summarized in Table 3 . The experiments confirmed that the end-to-end model can handle the scenario in its training dataset. However, for a more complicated environment shown in Fig. 4b , the end-to-end model only has a success rate of 40% because of a lack of training data on such scenarios. On the other hand, the LLM understands the complex environment and offers high-level guidance to steer the end-to-end model in the correct direction. Nevertheless, the LLM does not always generate correct instructions and experiences hallucinations when the environment is disrupted by rapidly changing lighting conditions. We will address these limitations of our driving model in Section 3.4. Even LLMs without fine-tuning demonstrate the ability to enhance the generalization capability of the end-to-end model, but there are still limitations. First, in scenarios with strong backlighting and reflection in the front image, the LLM struggles to identify the position of obstacles, resulting in incorrect instructions. This issue arises because the dataset used to train the LLM contains few images with poor lighting conditions. Appendix D.1 provides examples of these failure cases. Second, to maximize the ability of the LLM in obstacle identification, the CoT prompt must be designed specifically for the obstacle avoidance task. A sophisticated prompt design and an LLM with a long contextual length are required to tackle complex driving tasks. 
conclusion: 
 We develop an architecture that combines LLMs with an end-to-end model to improve the generalization capability of the end-to-end model. The key distinction between our architecture and previously proposed architectures is that the LLM in our architecture only provides high-level instructions rather than direct actions or trajectories. The end-to-end model uses both raw sensor data and high-level instructions from the LLM. These high-level instructions are received at a slower rate compared to the raw sensor data, allowing the end-to-end model to output actions even while the LLM still makes inferences. By integrating LLMs with the end-to-end model, our architecture relaxes data collection requirements and eliminates the need for rapid responses from LLMs. This design makes it feasible for real-world applications. Experiments are conducted in a real-world environment with a self-driving vehicle. The vehicle is equipped solely with the front-view camera as its sensor. We train the driving model in a simple yet common environment and deploy the model on the smartphone. The results show that the end-to-end model alone cannot navigate safely in a complex environment that does not appear in the training data. However, when combined with LLMs, the end-to-end model navigates the complex environment successfully. Notably, the LLMs we utilize are not fine-tuned; instead, we employ chain-of-thought prompt engineering techniques to enhance performance. The results demonstrate that LLMs improve the generalization capability of the autonomous driving system, enabling it to adapt to more complex environments with simple training data. However, LLMs still exhibit limitations, such as susceptibility to hallucinations when the environment changes, such as backlighting and ground reflection. This requires further investigation and a deeper understanding of LLM behavior. 
 
 Appendix: Training an end-to-end model requires a massive amount of data to enable the model to adapt to new environment settings, such as varying objects on the road, weather and lighting conditions, road textures, and colors. Without a sufficient and diversified dataset, it faces challenges when extending from simple to complex scenarios (not shown in the training dataset) that involve planning and reasoning. In this section, we illustrate this limitation using a simple yet representative experiment.In the experiment, we consider a three-lane driving scenario, where the lanes are designated as left, middle, and right. We create different scenarios by placing obstacles in one or more lanes, denoted as to S a l amar , where a l , a m , and a r represent the presence of an obstacle in the left, middle, and right lanes, respectively (with 1 indicating an obstacle and 0 indicating no obstacle). The training dataset is collected in the scenario S 010 , which represents an obstacle in the middle lane. In this scenario, the car is trained to navigate around the obstacle by using either the left or right lane. We then evaluate the model on scenarios S 010 , S 110 , and S 011 .Scenario Success rate (%) As shown in the experiment result in Table 4 , the end-to-end model is not able to reliably extend to even a slightly modified scenario. Although it is possible to include more scenarios like S 110 and S 011 in the training dataset, there will always be edge cases that remain unaccounted for. This is why we introduce the VLM to provide a "world model" that enhances the planning and reasoning capabilities of the end-to-end model, enabling it to generalize from simple to complex scenarios by decomposing complex behaviors into a set of atomic actions.SThis section includes the implementation details of the end-to-end model discussed in Section 2.1.The end-to-end model is a neural network. The neural network takes the image and instructions from the LLM and outputs the steering and throttle actions. The input image is the front view of the car and is taken by the ultra-wide camera of the iPhone 15 Pro mounted on the car. The field of view of the camera is 101 degrees, and the raw image size is 640 × 480. The image is further cropped with a rectangular area [top, bottom, left, right] = [140, 330, 130, 510] and is resized to 320 × 160 before being sent to the end-to-end model. The input instruction is a set of commands {LEFT, MIDDLE, RIGHT}, indicating which direction the car should go to avoid the front obstacles if the input image indicates there are any. Also, to simplify the response to the LLM, we ignore any cases and recognize STRAIGHT as MIDDLE.The image backbone for the end-to-end model is the pretrained ViT-B/16 model provided by Dosovitskiy et al. [49] . The model is modified according to Shafiullah et al. [48] to include two prediction heads: one outputs the per-class action values, and the other outputs the classification probability. The per-class action-value output V is a matrix of size 3 × 2, where each row indicates one of the three commands in the instruction set, the first column is the steering value, and the second column is the throttle value. The classification probability output p is the probability density for each class.The dataset consists of 60 different routes. Each route consists of a sequence of action pairs (image, steering, throttle) with an average length of 100. To train the end-to-end model to take the three different instructions LEFT, RIGHT, and MIDDLE, we collected three different types of routes in the training environments illustrated in Fig. 4a : the first and the second are to pass around the obstacle from the right and left, respectively, and the third is collected when there are no obstacles in the front and the car goes straight. Each route is labeled by its route type. We will discuss in the next subsection how these data are used to train the end-to-end network for different instruction outputs.As discussed in Section B.1 and Section B.2, the neural network outputs three different actions, and the dataset also contains different types of route behaviors. In other words, each sample in the dataset is considered the tuple D = (X img , y s , y t , y c ), where X img is the input image, y s is the steering value range [-1, 1], y t is the throttle value range [0, 1], and y c is the route label selected from the set {1, 2, 3}, representing the three instructions. The neural network can be written as (V, p) = f (w; X img ), where V is a matrix of 3 × 2 representing the per-class action-value output, p is a vector of 3 representing the probability distribution of each class, and w is the weight of the neural network. Then the end-to-end model is trained with the supervised learning method with the loss function defined asL(w; D) = (V yc,0 -y s ) 2 + (V yc,1 -y t ) 2 -k log p yc ,where w is the hyper-parameter to weight the loss of the action value and the cross-entropy loss of the classification.We evaluate the end-to-end model in real-world, real-time, and closed-loop conditions. We conduct experiments with and without LLM assistance instructions. When high-level instructions and images are used as input, the prediction probability head is discarded, and the LLM provides the steering and throttle values. When only images are used as input, the actual instruction is sampled from the predicted probability head.We extend our experiment to more complex scenarios where moving obstacles (e.g. cars, pedestrians) are present. The settings for training and testing are similar to Fig. 4a and Fig. 4b . Instead of placing cones in the left or right lane, the obstacle is replaced with moving pedestrians or moving cars. We performed 10 repeated experiments on these testing scenarios, using both an online VLM (ChatGPT-4o) and a local VLM (LLaVa-LLaMA2). The success rates for each scenario are shown in Table 5 .VLM Scenario 1 (Car) Scneario 2 (Ped) Scenario 3 (Car+Ped)ChatGPT-4o 50% 60% 33% LLaVA-LLaMA2 100% 80% 55% This section compares the performance across different prompting techniques. This section discusses the case where the LLM might fail. Fig. 5c shows an example image where the image contains intensive backlight and reflects light from the ground. In that case, the LLM may fail to generate the correct answer. Table 15 shows a sample output for Fig. 5c using LLaVA-LLaMA2.Naive Based on the image, the toy car appears to be moving towards the right side of the hallway.CoT 1. There are orange traffic cones in the hallway, which are obstacles.2. The traffic cones are positioned in the middle of the hallway, creating a pathway on either side.3. The empty space between the obstacles and the hallway wall is on both sides of the cones, with the space on the left side being larger than the space on the right side. 4. The car should drive on the right side of the obstacle, as the space on the right is smaller and would be more challenging to navigate. 5. The car should drive on the left side of the obstacle, as the space on the left is larger and would provide a smoother path. Prompt Type Response Naive Based on the image, the toy car is currently in the middle of the hallway. It is not possible to determine the exact direction the car will take without more information or a clearer view of the obstacles in the hallway.CoT 1. Yes, there is a cone in the hallway.2. The cone is located in the middle of the hallway.3. The empty space between the cone and the hallway wall is larger on the right side. 4. The car should drive on the left side of the cone. 3. The empty space between the obstacles and the hallway wall is approximately 6 feet long. 4. The empty space on the left side of the obstacles is larger than the empty space on the right side. 5. The car should drive on the left side of the obstacles to avoid hitting them. Prompt Type ResponseThe future direction of the toy car should be to the LEFT. The hallway ahead is obstructed by traffic cones which requires the car to navigate around them.Step 1: Identify any obstacle in the image -The image shows multiple red traffic cones as obstacles in the hallway.Step 2: Describe the position of the obstacles in the hallway -The traffic cones are positioned in the middle and slightly to the right of the hallway.Step 3: Describe the position of empty space between the obstacles and the hallway wall -There is empty space both on the left side and the right side of the obstacles. The left side is between the obstacles and the left wall, and the right side is between the obstacles and the right wall.Step 4: Describe which empty space is larger -The empty space on the left side between the obstacles and the left wall is larger compared to the space on the right side.Step 5: Output which side of the obstacle the car should drive as LEFT, MID-DLE, or RIGHT -The car should drive on the LEFT side of the obstacles. Prompt Type Response Naive Based on the image, the toy car appears to be moving towards the right side of the hallway.CoT 1. There are orange traffic cones in the hallway, which are obstacles.2. The traffic cones are positioned in a line along the hallway, with the first cone closer to the camera and the last cone further away.3. The empty space between the cones and the hallway wall is larger on the right side of the image. 4. The empty space on the left side of the image is smaller. 5. The car should drive on the right side of the obstacle, which is the side with the larger empty space. Prompt Type Response Naive Based on the image, the toy car is currently in the middle of the hallway, and there are no visible obstacles in the immediate path. Therefore, the future direction of the car is likely to be straight ahead.CoT 1. Yes, there are orange and white cones in the hallway.2. The cones are lined up in a row, with one cone placed in the middle of the hallway and two cones placed on either side of the hallway.3. The empty space between the cones and the hallway wall is larger on the right side of the hallway. 4. The car should drive through the larger empty space on the right side of the hallway. 5. The car should drive through the larger empty space on the right side of the hallway. Prompt Type InstructionThe image shows the toy car approaching three traffic cones placed in the middle of the hallway, partially blocking the middle path. Given the position of the obstacles, the future direction of the car should be RIGHT.To analyze the provided image of a toy car driving through a hallway with obstacles, let's address the questions step by step: The figure is only for demonstration; the actual environments are real-world settings, not simulations. 


Decision: In brief, the strengths of the paper are, the use of LLM in autonomous driving, effective reasoning strategy, and end-to-end approach. Although there were concerns, during the rebuttal stage the authors have successfully addressed the weaknesses of the papers to clarify and significantly improve the quality of the publication.

Reviews: review: title: title: Review of Submission 257
paper_summary: paper_summary: Use chain-of-thought reasoning to drive a car in OOD settings
main_review: main_review: Strengths:
 - Effectively applies CoT reasoning to a robotics problem
 - Identifies an issue with just directly applying VLMs as controllers (inference latency) and proposes a hybrid architecture to rectify it.

Weaknesses:
 - The CoT prompt is highly specific to this exact scenario and likely could not (in its current form) reason through more realistic open-world challenges like pedestrian/vehicle interactions or semantic scene understanding
 - Only evaluated in relatively simple obstacle-avoidance scenarios.
 - Experimental evaluations and analysis are severely lacking; at the very least comparisons to stronger driving models and scaling results with additional data should be included.
 - Having only one train and one test scenario limits claims of generalizability.
 - The work has limited novelty; i.e. it is a fairly straightforward application of CoT-conditioning to a language-conditioned policy at the low level. Experimental results are not sufficiently rigorous or impressive to make up for this lack of novelty.
questions: questions: - It is claimed that "even if cones replace trash bins, the car may still crash into the cones because one possible control pattern it learned from the training data dictates turning left when encountering cones". How does the VLM+E2E hybrid architecture improve this? Are we assuming that the underlying E2E policy places more weight on the text instruction than on the image? What happens when the image and text instruction conflict?
 - Does the E2E model express any actual capabilities other than directly following what the LLM tells it to do? Perhaps a good additional baseline would be to directly control the car by the VLM's outputs (+some underlying "blind" controller) to demonstrate the importance of the E2E policy.
 - When is the VLM queried? In the right-left testing environment, does the E2E policy know that it should wait for the VLM's response before executing the second task, or is the robot just "tuned" to move slowly enough that this is not a problem? If the latter, does that mean the robot will be unable to react if the tow rows of obstacles are placed closer together?
 - Could the CoT query be generalized so as to be usable for more complex reasoning tasks? It would be nice to have a baseline with a more open-ended CoT task.
limitations: limitations: 3
review_summary: review_summary: Based on the limited novelty and weak experimental results I would recommend rejection. The authors would need to provide experimental results in significantly more complex scenarios, and significantly increase rigor and depth of analysis, for me to change my recommendation.
title: title: Initial Review
paper_summary: paper_summary: This paper presents a system for autonomous driving that leverages a multi-modal LLMs for high level reasoning along with an command-conditioned end to end model to execute low level controls. More specifically, an off-the-shelf LLM is given the image along with a chain-of-thought prompt to identify obstacles and provide one of three directions: left, right, middle.  The given command is then given to the end-to-end model which predicts the controls. The paper demonstrates the effectiveness of this approach in a real-world closed loop, but toy setting of a indoor robot with a configuration of obstacles.
main_review: main_review: Strengths: 
- While I believe the experimental setting is too simple to be referred to as "autonomous driving", it is nonetheless impressive to see the system tested in a real world, closed-loop setting with LLM inference run online.
- Using an LLM for high level reasoning along with a neural network for lower level controls is intuitively well-motivated. 

Weaknesses: 
- The **main weakness** of the paper is that the proposed method is built for a toy setting of navigating static obstacles in a particular indoor environment. It's unclear if the proposed approach has much practical value for a real setting with dynamic obstacles. Even without obstacles, real world driving would likely require more complex commands than LEFT, RIGHT, MIDDLE to be provided by the high level reasoning system. 
- The related works makes at least one incorrect statement: claiming NVIDIA introduced end-to-end autonomous driving (lines 121-122). The NVIDIA paper itself references ALVINN as the first end-to-end autonomous driving system introduced by Pomerleau in 1989. The section also misses important relevant works in end to end autonomous driving such as [1].
- The paper contains a reasonable amount of typos or grammatically incorrect sentences. Also, the word MIDDLE/STRAIGHT seem to be used interchangeably when referring to the command, which adds complexity for the reader. I would recommend additional proofreading for the final submission for improved clarity. 

[1] End-to-end Interpretable Neural Motion Planner, Zeng et al. CVPR 2021.
questions: questions: - How would you extend the proposed method to support real world autonomous driving?
- Why do you believe certrain LLMs outperform others? Is it well correlated with the performance of those LLMs on other tasks?
limitations: limitations: 2
review_summary: review_summary: While I appreciate the real world experiments, I believe the approach and setting are too simplified for the paper to provide much value to the research community.
title: title: A solid step towards applying LLM4AD
paper_summary: paper_summary: This work explores several practical issues of deploying LLM for autonomous driving. They solve it by using LLM for high-level instruction and using an e2e model for low-level control. They deploy their algorithm in the real world.
main_review: main_review: Applying LLM for autonomous driving has been widely discussed in the community. However, there are several common concerns:
1. Latency: LLM is heavy while autonomous driving requires high FPS decision making to ensure safety.
2. Numerical: LLM has inherent issues to handle numerical inputs and outputs due to the tokenization process, while in autonomous driving, the vectorized world and control trajectories/signals are numerical.
3. Necessity: If we simply use LLM for regression, it serves as an huge MLP, which could not utilize their reasoning ability well.

The authors solve them by only using LLM with raw sensor inputs for high-level instruction while using an end-to-end ad model for actual control.  In this way, LLM dose not need to process numerical inputs and outputs and the high-level instruction could be more naturallly integrated into language to utilize LLM's reasoning ability. At the same time, the adoptation of another e2e-ad model avoid the response requirements for LLM at every step.

Finally, they deploy their methods in the real world, which fits CoRL well.
questions: questions: Missed reference in the end-to-end autonomous driving field:
1. Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline. NeurIPS 22
2. Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving. CVPR 23
3. DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving. ICCV 23 Oral
4. Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2). ECCV 24
limitations: limitations: 3
review_summary: review_summary: To apply LLM for actual autonomous driving, the data issue and latency issue are non-negligible. This work starts an initial discussion and deploy their algorithm in the real world, which fits CoRL well. Thus, I give strong accept.