LASER: Script Execution by Autonomous Agents for On-demand Traffic Simulation

## Abstract

Abstract Autonomous Driving Systems (ADS) require diverse and safety-critical traffic scenarios for effective training and testing, but the existing data generation methods struggle to provide flexibility and scalability. We propose \tool , a novel framework that leverage large language models (LLMs) to conduct traffic simulations based on natural language inputs. The framework operates in two stages: it first generates scripts from user-provided descriptions and then executes them using autonomous agents in real time. Validated in the CARLA simulator, \tool successfully generates complex, on-demand driving scenarios, significantly improving ADS training and testing data generation.

## 1 Introduction

With the deep learning breakthrough, Autonomous Driving Systems (ADS) have made significant advancements in tasks such as occupancy prediction (Huang et al., ; Wei et al., ), trajectory prediction (Vishnu et al., ; Park et al., ; Gu et al., ), semantic scene completion (Li et al., ; Jiang et al., ; Cao et al., ) and world model (Wang et al., ; Zhao et al., ; Wang et al., ; Wen et al., ). For instance, Tesla’s Full Self-Driving (FSD) demonstrates exceptional performance in both common and complex tasks such as lane change, turn, merge, fork and detour for most types of roads including even curvy highways and roundabouts (Tesla, ).

The rapid development of end-to-end ADS research and application heavily relies on vast high-quality driving data. On one hand, a large dataset containing multi-modal data from all kinds of sensors (such as cameras, lidar and radar) is essential to train the underlying deep neural network (DNN) models. Taking FSD as an example, Tesla claims their camera-only models are trained with over 160 billion frames of driving data sampled from real-world scenarios, synthetic scenarios generated by simulators, as well as that from other sources (Tesla, ). On the other hand, ADS are highly safety-critical, which are required to be thoroughly tested under diverse scenarios, especially some rare, unanticipated ones, to guarantee their capability of handling emergence and avoiding accidents (Wikipedia, ).

The most straightforward approach to building datasets for ADS training/testing is to collect real-world traffic data through sensors such as vehicle cameras, which naturally reflect real distributions of the data and can be scaled up through crowdsourcing (Caesar et al., , ). However, this method is inefficient, as daily traffic often yields repetitive, trivial scenarios, while safety-critical events, which are rare and high-risk, are seldom included in the training set and thus hardly learned by the model (Feng et al., ).
Additionally, static data, such as vehicles captured in each frame, lack the flexibility to interact with or manipulate, preventing effective training/testing specific or customized scenarios.
Last but not least, online, interactive testing of ADS requires the actors (e.g., vehicles, pedestrians) to be reactive to the behavior of others, which is virtually infeasible for those collected from traffic data.

To address these limitations, another class of approaches is to generate the scenarios from traffic simulators, collecting synthetic data through high-fidelity sensors (Caesar et al., , ; Wei et al., ; Zhang et al., , ; Suo et al., ; Salzmann et al., ). These methods allow for creating customized driving scenarios tailored to specific needs, resulting in a controllable and editable dataset for training and testing models. Furthermore, they enable the rapid execution of thousands of diverse and targeted online tests by deploying virtual vehicles attached with trained models in the simulator, facilitating the identification of pitfalls and the resolution of exceptions before costly real-world settings.

Undoubtedly, generating driving scenarios with real-world traffic flows is demanding. Each dynamic object–regardless of a vehicle, bicycle, or pedestrian–exhibits its own time-varying motion patterns, which are often interdependent with those of other objects.
Mainstream methods for traffic simulation can be categorized as *rule-based* or *learning-based*. Rule-based traffic simulation employs analytical models to control vehicle movements (Lopez et al., ; Casas et al., ; Fellendorf and Vortisch, ), typically relying on fixed, predefined routes. This approach often results in highly repetitive scenarios with limited behavioral diversity.

In contrast, learning-based methods aim to replicate human trajectories from real-world driving logs to produce varied and realistic behaviors, which leverage techniques–such as imitation learning (IL), reinforcement learning (RL), deep learning (DL) and deep generative models–to generate diverse and realistic driving behaviors by utilizing real-world driving logs as demonstrations. We refer the readers to (Chen et al., ) for a comprehensive survey.
However, these methods generally face significant challenges in accurately modeling and generating human driving behaviors, often resulting in simplistic actions such as passing or merging (Suo et al., ). This is primarily due to three reasons.
(a) Limited and biased training data. Existing methods commonly rely on datasets such as Nuscenes, which contains only 1000 videos (Caesar et al., ). Learning high-level driving behaviors in complex, multi-agent environments must tackle the combinatorial explosion of input states, but the limited, imbalanced nature of the data makes hard to generalize to rare or unseen scenarios, commonly referred to as “long-tail" cases (Chen et al., ).
(b) Lack of alignment with human understanding. The behaviors generated by these models are often not aligned with natural language descriptions or human common sense, making them less interpretable and harder to customize for specific driving behaviors.
(c) Scenario generation in real-time. When generating interactive traffic scenarios online, these methods typically operate in an auto-regressive manner, where each step’s prediction builds upon the previous one. Without goal-oriented guidance, this approach can lead to trivial or ineffective behaviors, and the accumulation of prediction errors may result in catastrophic failures such as collisions or vehicles driving off-road (Xu et al., ).

Very recently, deep generative model-based methods (Wang et al., ; Zhao et al., ; Wang et al., ; Wen et al., ) provide a promising way for generating customized traffic data with world models. However, it remains to be a difficult task to generate *interactive* traffic scenarios with diverse, on-demand behaviors.

Recently, large language models (LLMs) and multi-modal language models (MMLMs) have demonstrated remarkable capabilities in common-sense reasoning, planning, interaction and decision-making, showcasing great potential to address the challenges mentioned above (Dubey et al., ; Huang et al., ).
We propose a new traffic simulation framework, named \tool (LLM-based scenArio Script gEnerator and ExecutoR) that leverages LLMs to create both intricate and interactive driving scenarios by generating readable scripts to guide step-by-step execution of each dynamic objects within the scenarios, which only requires simple natural language descriptions from the users in the first place.

As illustrated in Figure 1, we first translate user requirements to a master script, which then is converted to sub-scripts for each dynamic object, i.e. each executing actor in the scenario. Based on the rich domain-specific knowledge and advanced reasoning capability of LLM4AD (Wen et al., ; Shao et al., ), each actor’s lifespan is managed by an LLM-controlled autonomous agent that executes its sub-script in real-time. These agents make decisions about intermediate actions based on the current state of the environment at each simulation timestamp, all aiming to achieve their individual goals within the expected time frame.

Leveraging the common-sense and behavior understanding of LLM-controlled agents, we can perform top-down behavior-to-action script interpretation, as opposed to the previous bottom-up action-to-behavior accumulation. The LLM interpretation aligns language-specified behaviors with low-level actions, provides interpretability for the generation process and enables the on-demand generation of specific long-tail scenarios. With scripts highlighting their agendas, these LLM-controlled autonomous agents cooperate to achieve the tasks, generating on-demand behavior while avoiding accumulated prediction errors.

We design a task set consisting of 17 user requirements encompassing both long-tail and reasonable safety-critical scenarios. We evaluate \tool for on-demand script generation and execution on these tasks in the CARLA simulator (Dosovitskiy et al., ). The experimental results demonstrate that \tool can generate scripts based on user requirements effectively, with only 3.18% of the characters in the resulting executable script being inputted by the user to fulfill their demands. The experimental results also show that our approach can execute the script effectively and efficiently, with an average success rate of 90.48%, and usage of 1606.09 tokens per simulation second per agent. Furthermore, manual inspection confirms that our approach can successfully simulate various safety-critical scenarios which can be applied to ADS testing.

In summary, the primary contribution of our work is to propose an on-demand, interactive approach for traffic simulation, which includes a script generator and LLM-controlled autonomous agents as the executor. To the best of our knowledge, this is the first time to achieve on-demand scenario generation in ADS testing.

## 2 Related works

We have covered the related work on learning-based traffic simulation in Section 1. A recent trend in autonomous driving is to leverage LLMs which have demonstrated exceptional capabilities in human-like tasks such as common-sense understanding, planning, decision-making, and interaction (Dubey et al., ; Huang et al., ),
trained on vast datasets of trillions of tokens and images from the web. These models exhibit a deep reservoir of actionable knowledge, which can be harnessed for robotic manipulation through reasoning and planning (Wen et al., ). Recent research has explored the use of LLMs to develop autonomous agents that execute natural language tasks in interactive environments (Driess et al., ; Brohan et al., ; Belkhale et al., ). A notable example is RT-H, which enhances agent robustness and flexibility by decomposing high-level tasks into sequences of fine-grained behaviors, referred to as "language motions" (e.g., "move arm forward" followed by "grasp the can"). This approach effectively leverages multi-task datasets, significantly improving performance (Belkhale et al., ).

Substantial efforts have also been directed towards integrating LLMs with ADS, underscoring the models’ superior abilities in understanding and decision-making within driving scenarios (Sharan et al., ; Renz et al., ; Shao et al., ; Wen et al., ). For example, CarLLaVA achieved first place in the sensor track of the CARLA Autonomous Driving Challenge 2.0, surpassing the next-best submission by 32.6%. This success is attributed to its integration of the vision encoder LLaVA with the LLaMA architecture as its backbone (Renz et al., ). Additionally, LLM-Assist outperformed all existing learning- and rule-based methods across most metrics in the Nuplan dataset by leveraging LLMs’ common-sense reasoning to refine plans generated by rule-based planners (Sharan et al., ).

Further research has demonstrated the capability of LLM-integrated ADS to execute tasks based on natural language instructions, revealing its potential for modeling complex human driving behaviors (Wang et al., ; Shao et al., ). For instance, LMDrive showed that LLM-controlled driving agents could interpret and follow high-level driving commands, such as "Turn right at the next intersection," by aligning these instructions with vehicle control signals using a vision-language model as the foundation (Shao et al., ). More recently, DiLu demonstrated that with few-shot learning, LLMs could achieve results comparable to RL-based planners, significantly reducing the computational cost of deploying multiple LLM-controlled agents simultaneously (Wen et al., ).

## 3 Methodology

### 3.1 Framework

To achieve on-demand and interactive traffic simulation, we propose a framework called \tool, as illustrated in Figure 1. \tool consists of two stages, implemented by two modules respectively, i.e., script writer (Section 3.2) and \tool-Agent (Section 3.3). Unlike the previous learning-based methods that conduct generation and simulation simultaneously, our \tool framework first generates scripts that define logic-chained behaviors (LCB) with natural language instructions from the user requirements. It then executes the script by the real-time cooperation of \tool-Agents.

The two-stage framework detaches actor behavior from the scenario with a natural language script.
Compared to road data collection that collects on-road scenarios at the state level, our \tool framework records scenarios at the behavioral level, which enables dynamic and flexible execution during the simulation.
Compared to learning-based traffic simulation that generates scenarios at the state level, \tool generates scenarios at the behavioral level, this enables on-demand generation and top-down execution of language-specified behavior, and easy editing on actor behavior to improve performance.

To have a glance at the complexity of behaviors that \tool can simulate, we present an example of a user-defined scenario in Figure 1. The user requests a safety-critical scenario on the highway: Two cars are driving in the left lane ahead of the ego vehicle, which is in the right lane. Suddenly, the front car decelerates due to a mechanical failure. The rear car, unable to brake in time, switches to the right lane to avoid colliding with the front car, unaware that the ego vehicle is behind it. In this situation, the ego is responsible for quickly recognizing the potential for the rear car to change lanes due to the front car’s abrupt deceleration and must react by decelerating promptly to prevent a collision. We will demonstrate detailed implementations of the script writer and \tool-Agent modules with the example shown in Figure 1 in the following sections.

Figure: Figure 1: Framework of \tool
Refer to caption: x1.png

### 3.2 Script writer

Directly executing user requirements with autonomous agents is unstable. User requirements can be ambiguous, leading to varying behaviors across different execution attempts. Additionally, fulfilling the user requirement often requires coordination among multiple autonomous agents. Without a shared consensus on how to achieve this goal, these agents may act independently, potentially resulting in failure.

To address these challenges, we use script writer to translate user requirements into scripts. To ensure consistency across different execution attempts, the behaviors in the script must be detailed and concrete, with sub-scripts that contain detailed LCB instructions for each agent to execute. To ensure effective coordination, these sub-scripts must align under a single master script, serving as a unified consensus for all agents.

In our work, the generation of scripts follows a hierarchical chain-of-thought (CoT) manner (Wei et al., ) to enhance the common-sense reasoning and planning ability of LLM as GPT-4o does (OpenAI, ). The script writer first generates a master script that outlines the framework of the story, then further generates sub-scripts that contain detailed LCB instructions for each actor based on the master script. This ensures behavior consistency over agents and executions. The scripts are written in natural language, to enable easy editing of the behaviors. The generating procedure of the master script and sub-scripts for individual agents are shown in the following paragraphs.

Master-script Generation. Based on the user’s initial requirements and (optionally) a map description to specify the surrounding layout, the script writer generates a master script using a CoT approach. First, the script writer prompts the LLM to generate a story that aligns with the user’s requirements. The LLM then outlines key stages in the sequence of events, starting with the initial state, where each stage acts as a direct cause or prerequisite for the next. For example, given the user requirements illustrated in Figure 1, the following master script is produced:

Sub-scripts Generation.
Based on the master script, the script writer queries the LLM to generate sub-scripts containing detailed, step-by-step LCB instructions for each individual actor. These sub-scripts clearly outline the specific actions each actor must perform, using natural language to chain behaviors logically. Each action is paired with a termination condition, specifying when the task is complete and when the next action should begin, along with the reasoning behind it. This ensures that the behavior not only aligns with the overall narrative but also allows for flexible execution by the autonomous agent.

The resulting sub-scripts are structured similarly to movie scripts. They begin with an initial state, defining the actor type (e.g., truck, car, or pedestrian), and then break down into several sequential steps. Each step includes an action, termination conditions and a reason, all of which work together to link the actor’s behavior with logical decision-making. An example of a sub-script is shown in Figure 1. The initial state specifies the actor’s lateral and longitudinal position, as well as its initial speed. The action defines a simple, concrete motion that can be easily executed by our \tool-Agent, such as merging into the leftmost lane. The termination condition outlines measurable criteria, which may depend on the actor’s own behavior or interactions with others, for example, when the longitudinal distance to another vehicle is within 2 meters. Finally, the reason clarifies the rationale behind each action, enhancing the agent’s understanding and enabling more flexible execution, especially during interactions with other actors.

### 3.3 \tool -Agent

The second stage of grounding the sub-scripts into execution is achieved through the collaboration of \tool-Agents. To facilitate the comprehension of language instructions and behavior, we employ LLM-controlled driving agents. These agents, designed to operate autonomously, execute the sub-scripts step by step based on real-time environmental observations, working together to bring the entire scenario to life.
Since fine-tuning LLMs with vehicle control signals and applying LLMs to learning-based planners both require substantial computational resources during runtime (especially when managing multiple agents), we integrate each agent with an LLM-based decision module alongside a rule-based planner.
The LLM-based decision module, equipped with common-sense, plays a crucial role in converting language-based LCB instructions into executable actions. Every 0.5 seconds, the \tool-Agent encodes the environmental scenario into a descriptive format, integrates sub-script LCB instructions to create a prompt, and queries the LLM for an executable decision. This decision is then carried out by the rule-based planner.

LLM-based Decision Module.
This module processes the scenario description along with LCB sub-script instructions, generating an executable decision every 0.5 seconds. It begins by checking if the termination condition for the current step is met. If the step is complete, it transitions to the next one. The module then predicts an executable decision based on the step’s instructions and the scenario description, which includes parameters such as target speed, lane change direction and lane change delay.

LLM brings a common-sense understanding to translate language instructions to executable decisions. Instead of directly outputting vehicle control signals where LLMs do not excel, this higher-level decision-making process offers better alignment with the model. This approach enables zero-shot grounding of language-specified behaviors more effectively.

To enhance LLM’s comprehension of the current traffic environment, we encode the surrounding traffic conditions into a standardized textual scenario description, following the approach outlined in DiLu (Wen et al., ). This description includes all essential information for decision-making, such as the number of available lanes, the positions, speeds and lane-change statuses of both the subject vehicle and surrounding vehicles.

Rule-based Planner.
It outputs vehicle control signals to execute the decisions made by the LLM-based decision module at every frame. It tracks a path consisting of waypoints on the map and uses PID control (Wikipedia, ) to regulate speed. Whenever the rule-based planner receives a new decision from the LLM-based decision module, it generates a new path based on the current position, following the lane change direction and the delay specified in the decision. At each frame, the planner tracks the waypoints on the path, while the PID controller calculates vehicle’s steering and throttle.

This lightweight design allows the control of multiple agents simultaneously, enabling their complex behaviors and interactions. The rule-based planner functions as a humble executor of the LLM’s decisions, without incorporating safety constraints such as maintaining distance from other vehicles. This design ensures that the agent can exhibit alarmingly realistic behaviors.

## 4 Evaluation

In this section, we present a comprehensive evaluation of script writer and \tool-Agents for on-demand traffic simulation.
All code and the results are available on an anonymous repository.
(^1^11[https://njudeepengine.github.io/LASER/](https://njudeepengine.github.io/LASER/))

### 4.1 Setting

User requirements. We design 17 scenario generation tasks, each representing a complex traffic execution requirement that is challenging to capture through traditional road collection or existing simulation methods. (cf. Table 3 in Appendix A.1.1 for details). For example, tasks such as “Accident", “Ambulance", and “reckless Driving" present long-tail scenarios that are rarely encountered. “Swerve" and “Three in Line 1" depict reasonable safety-critical situations where even experienced drivers could make mistakes. To navigate these scenarios effectively, one must be able to anticipate signs of an impending accident.

Setup. Our experiment utilizes CARLA 0.9.15 (Dosovitskiy et al., ), a widely used open-source simulator for closed-loop ADS testing. Built on the UE4 engine, CARLA offers realistic graphics, a variety of vehicle and pedestrian models, and diverse maps. We assess each task across three road segments from Town04, Town05, Town06 and Town10, conducting 20 simulations per segment for effectiveness (totaling 60 simulations per task) and 5 simulations per segment for efficiency (totaling 15 simulations per task).

All experiments employ GPT-4o as the LLM, which queries every 0.5 seconds. Appendix A.1.3 shows an example of LLM reasoning process. To evaluate interactions between \tool-Agents and ADS in safety-critical scenarios, we use our \tool-Agents to test the end-to-end ADS InterFuser (Shao et al., ), which ranked #1 in the CARLA challenge 2022 among the open source models.

Road Segments. We evaluate each task on three highway segments with varying lane numbers (including one curved segment) and three urban segments (including a curved road) to assess the generalizability of \tool.

Metrics. The effectiveness of script writer is measured by the user involvement percentage, defined as the average proportion of user-provided characters (excluding spaces and newlines) in the final executable script. We assess script execution success rate as the number of traffic simulations meeting user requirements divided by the total number of simulations conducted. Generated traffic simulations are manually reviewed against criteria outlined in Table 4 (in Appendix A.1.3). Efficiency is evaluated using token cost, defined as the number of tokens used per simulation second per agent, and time cost, representing the real-world simulation time per simulation second.

**Table 1: Evaluation of user involvement in script generation. (* indicate safety-critical tasks)**
| Task | User involvement percentage $\downarrow$ |
| --- | --- |
| Accident | 5.43% |
| Ambulance | 4.41% |
| Caught in Pincer | 0.47% |
| Failed at Start | 7.36% |
| Newbie Lane Change 2 | 0.35% |
| Cut-in* | 1.50% |
| Swerve* | 3.15% |
| Three in Line 1* | 2.74% |
| average | 3.18% |

Figure: Figure 2: A case of script generation result. (A: Action T: Termination Condition R: Reason)
Refer to caption: x2.png

### 4.2 Evaluation on script generation

To evaluate the effectiveness of on-demand script generation, we select eight user requirements related to long-tailed scenarios from the task set. For each task, script writer generates five scripts, which we then manually refine until the scripts successfully fulfill the user requirements.

The experimental results are summarized in Table 1, while Figure 2 visualizes a selected case for the task “Swerve". The results indicate that script writer effectively generates on-demand scripts, with an average user involvement percentage of just 3.18%.
Most inaccuracies in script writer ’s outputs stem from imprecise numerical values, such as positions and speeds. Additionally, there are instances where script writer overlooks steps implied by the user requirements. For example, in the “Failed at Start" task illustrated in Figure 4 (Appendix Section A.1.2), the bus should initially move in the left lane while the car stops in the right lane, before both vehicles change lanes simultaneously. However, script writer incorrectly bypasses this step, causing the bus to change lanes at the start.

### 4.3 Evaluation on script execution

To evaluate the effectiveness and efficiency of script execution, we select eight user requirements from our task set that involve complex interactions, including four tasks focused on safety-critical scenarios. For each task, script writer generates a script based on the user requirement, which we then manually modify to ensure it meets the requirements fully. Scripts are executed by our \tool-Agents to assess their effectiveness and efficiency.

Figure: Figure 3: The visualized results for script execution. (* indicate safety-critical tasks)
Refer to caption: x3.png

Effectiveness. The experimental results are shown in Table 2. The results indicate that \tool-Agents successfully execute the scripts, achieving an average success rate of 90.48%. Most errors arise from inaccuracies in numerical comparisons, such as positions and speeds, while some mistakes result from hallucination.

Efficiency. The efficiency results are presented in Table 2, revealing an average of 1,606.09 tokens and 7.87s for generating a one-second simulation. In the Ambulance task, the inclusion of multiple dummy agents to simulate congestion significantly increased the input tokens needed to describe other vehicles’ states, leading to a higher token cost. Similarly, the Reckless Driving and Sudden Jaywalker tasks require lengthy scripts for the actors, resulting in elevated input token counts. Using GPT-4o API service, generating a 40-second simulation with 3 \tool-Agents incurs a cost of approximately $1. The majority of the time cost is attributed to querying the LLM, with a 10-second simulation taking around one minute to generate.

**Table 2: Results of effectiveness and efficiency for script execution. (* indicates safety-critical tasks)**
| Task | Road type | Execution success rate $\uparrow$ | Token cost $\downarrow$ | Time cost $\downarrow$ |
| --- | --- | --- | --- | --- |
| Accident | highway | 100% | 1,556.08 | 9.63 |
| Ambulance | highway | 96.43% | 2,387.81 | 15.63 |
| Bus | urban | 91.67% | 1828.40 | 7.02 |
| Reckless Driving | highway | 44.07% | 2333.84 | 7.11 |
| Cut-in* | urban | 98.33% | 1305.83 | 5.76 |
| Sudden Jaywalker* | urban | 100% | 2130.40 | 7.55 |
| Swerve* | highway | 100% | 732.37 | 5.60 |
| Three in Line 1* | highway | 93.33% | 573.99 | 4.62 |
| average |  | 90.48% | 1606.09 | 7.87 |

## 5 Limitations

Manual description of map layout. \tool’s execution relies on manually formatting map layouts for scenario generation. This approach can lead to inaccuracies and inefficiencies, particularly in complex environments. Implementing automated map interpretation (e.g., querying image-to-text models with the initial frame) could greatly enhance the framework’s scalability and accuracy.

Lack of automatic search for scenario details. The current system necessitates user-in-the-loop revision for scripts. While this allows tailoring the generation that aligns with users’ intentions, it restricts the system’s capacity to autonomously generate a multitude of test cases with the same initial goals but varying details. Developing a more intelligent script writer  capable of automatically searching for reasonable and elaborate scenario details poses a challenge due to the knowledge gap between off-the-shelf LLMs and the specific requirements of the simulation environment.

Computational overhead for real-time execution. Integrating LLM-controlled agents with real-time execution in complex environments incurs significant computational overhead, especially when scaling the simulation to multiple agents. Future enhancements could focus on optimizing interactions between the LLM-based decision-making module and the rule-based planner, aiming to reduce latency and computational load while maintaining high performance and decision accuracy.

Generalization to real-world scenarios. Although our framework demonstrates strong performance in simulated environments, its ability to generalize to real-world driving scenarios may hinge on the simulation’s fidelity. Ensuring that virtual agents accurately mimic human driver behavior across diverse global contexts remains an ongoing challenge.

## 6 Conclusion

In this paper, we introduce \tool, a novel approach that leverages LLMs to generate on-demand traffic simulations. Our two-stage framework separates scenario generation from real-time execution, providing greater flexibility, scalability, and customizability compared to traditional simulation methods. By utilizing LLM-controlled agents, \tool offers a more human-like interpretation of driving behaviors, ensuring coherent and realistic interactions within the simulated environment.
The experimental results demonstrate that \tool effectively meets diverse user requirements for both general and safety-critical driving scenarios, showcasing high accuracy and adaptability in scenario creation. Overall, the proposed approach represents a significant advancement in on-demand traffic simulation for ADS training and testing.

## Appendix A Appendix

### A.1 Experiment details

#### A.1.1 Task set

We present our task set of 17 user requirements in Table 3.

**Table 3: Task set (* indicates safety-critical tasks)**
| Task | User requirement |
| --- | --- |
| 1. Accident | An accident occurred on the highway. One car turned left to avoid it, but with the left lane occupied, the following car turned right. |
| 2. Ambulance | On the highway, an ambulance is driving straight at high speed. Multiple vehicles in the same lane move to both sides to make way for it. |
| 3. Bus | A bus switched from the middle lane to the bus stop, then started again and changed lanes back. |
| 4. Caught in Pincer | A car to the left is overtaking, forcing the ego vehicle to decelerate, while a car behind speeds up, pressuring it to accelerate. |
| 5. Cut-in* | A car overtakes the VUT and then slows down. |
| 6. Failed at Start | A car parked in front of the bus station started, changed lanes, and collided with the bus that was changing lanes to park. |
| 7. Meet in Mind 1 | Two cars simultaneously change lanes to the middle lane. |
| 8. Meet in Mind 2 | Two cars, unable to see each other, simultaneously attempt to overtake a car in the middle. |
| 9. Merge Alternately | Two cars collided in the right lane at a road intersection, and several cars in the rear weaved into the left lane. |
| 10. Newbie Lane Change 1 | A car in front on the right changes lanes without accelerating, causing a collision with the ego vehicle. |
| 11. Newbie Lane Change 2 | A truck is to the left of the ego vehicle, and a car in front on the right changes lanes without accelerating, causing a collision with the ego vehicle. |
| 12. Reckless Driving | A reckless driver repeatedly overtakes other cars on the highway. |
| 13. Sudden Jaywalker* | The VUT was moving in the left lane behind a stopped truck. As it approached, a pedestrian emerged from behind the truck. |
| 14. Surrounded | Four police cars surround the criminal’s vehicle from the front, back, left, and right. |
| 15. Swerve* | There are two cars ahead of ego on the left lane. The front car suddenly decelerates, and the rear car swerves to the right to avoid a collision. |
| 16. Three in Line 1* | There are two cars in front of the VUT. The car at the very front slows down, and the second car, to avoid a collision, changes lanes to the right. |
| 17. Three in Line 2 | The front car suddenly hit the stopped car ahead, and the ego car collided with the front car. |

#### A.1.2 Experiment details of script generation

We present the visualization for the execution of some representative scripts in Figure 4. Further examples are given in Appendix A.2 for readability.

Figure: Figure 4: The visualized results for script generation.
Refer to caption: x11.png

#### A.1.3 Experiment details of script execution

Criteria for reviewing simulations.
We present the criteria for reviewing simulations to evaluate the effectiveness of script execution in Table 4.

**Table 4: Criteria for reviewing simulations (* indicate safety-critical tasks)**
| Accident | The two cars correctly change lanes according to their scripts before colliding with the stopped accident vehicles. |
| --- | --- |
| Ambulance | As the ambulance approaches, vehicles in the same lane move aside one by one to make way, without any collisions. |
| Bus | The bus changes lanes and stops at the bus station within a ±2.5m range, then starts again and changes lanes back. |
| Reckless Driving | The reckless car successfully overtakes the first car from the left lane and the second car from the right lane. |
| Cut-in* | The car successfully overtakes the front car then slows down. |
| Sudden Jaywalker* | The jaywalker steps out from behind the truck as the ego vehicle approaches. |
| Swerve* | The front car decelerates to a stop. The rear car changes into the ego vehicle’s lane, nearly hitting the ego vehicle before colliding with the front car. |
| Three in Line 1* | The front car decelerates to a stop, while the rear car changes lanes to the right before colliding with the front car, and then drives away. |

Example of LLM decision module’s reasoning.
Take the “rear_car" in the “Swerve" task as an example. Following the script outlined in Figure 2, the agent maintains its speed while cruising on the highway. Once the “front_car" decelerates below 5 m/s, the agent immediately changes lanes to the right. To achieve this, we encode the sub-script and the state of the “front_car" in the user prompt and query the integrated LLM. The LLM automatically evaluates whether the termination condition has been met based on the state of the “front_car" and makes decisions according to the current action specified in the script.

### A.2 Additional Tasks

We present visualized results for additional tasks in our task set in Figure 5.

Figure: Figure 5: The visualized results for additional tasks.
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: - Missing references regarding natural language to driving behaviors: [R1][R2][R3], especially R3 is extremely relevant in the sense of using LLMs to write code for traffic simulation. It would be nice to also include some of them as baselines for comparison.
- The metric of user involvement doesn't seem convincing. Could the authors provide other metrics that better capture user involvment? Perhaps something like engagement time for user to make the modification, number of issues fixed, task completion rate, etc.
- Does the user get to run the simulation when "debugging" the generated script?
- In table 2, could the authors provide more informative metric other than execution success rate? Perhaps something that measures the generated simulation actually being intended (e.g., the simulated scenario actually involve a swerving car leading to the ego-car changing its behavior or even crashing).
- There is no baseline. [R3] can be a suitable baseline for comparison as this paper follows extremely similar setup. Others like [R1][R2] can also be potential baselines as they take language as inputs. Some of the aspects to be compared can be diversity of the scenarios, how safety critical can the simulated environments be, number of simulation, etc.

Reference

[R1] Language conditioned traffic generation

[R2] Language-guided traffic simulation via scene-level diffusion

[R3] Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models
soundness: 1
strengths: - The paper is easy to follow.
summary: This paper introduces LASER, an LLM-powered approach for dynamic traffic simulation, allowing users to create complex, interactive driving scenarios using simple natural language prompts. LASER translates these prompts into master scripts, which are then broken down into sub-scripts for each actor in the scenario. LLM-controlled agents manage these actors, making real-time decisions based on environment states to achieve specific goals. Tested on the CARLA simulator, LASER achieved a 90.48% success rate with minimal user input. The proposed method enables on-demand, safety-critical scenario generation for ADS testing, marking a first in on-demand scenario creation in autonomous driving simulations.
weaknesses: - The proposed method is not fully automated and still requires human in the loop.
- Defining user involvment with average proportion of user-provided characters does not fully make sense since users may need to read through a lot more before even provide sensible and effective modification. In such a case, the actual user involvment may be greatly underestimated.
- Execution success rate in table 2 is defined as the scripts being successfully executed. However, it doesn't indicate whether the generated simulation actually makes sense or is useful.
- There is no baseline at all and it is hard to judge how effective the proposed method is.
- The technical contribution of the paper is extremely limited. Further addressing some of the mentioned limitation (e.g., manual description of map, lack of automatic search) may be a good direction.
- The number of generated cases (17) is quite limited. It is hard to (i) be convincing that the proposed method can be to certain degree scalable and (ii) have the statistics of quantitative results sufficiently convincing.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.	How do the authors design the task set in table 3? Is there any justification for selecting these tasks about the coverage and risk level?
2.	Which LLM does the authors use? Did the authors try different LLMs and see a large performance gap between them?
soundness: 2
strengths: 1.	The idea of using LLM to aid the generation of scenarios is interesting and promising. Since LLMs have general knowledge about driving and traffic, they should help generate risky and long-tail scenarios. This gives them the potential to overcome problems that exists in pure data-driven scenario generation methods.
2.	The paper is generally well-written and well-organized. Figure 1 clearly shows the components and pipeline of the proposed method.
summary: This paper identifies the problem that existing scenario generation methods struggle to provide flexibility and scalability, which may limit the effective training and testing of autonomous vehicles. Therefore, the authors propose LASER, a novel framework that leverage large language models to conduct traffic simulations based on natural language inputs. Specifically, there are two stages during the generation: (1) it first generates scripts from user-provided descriptions and then executes them using rule-based planner in real time.
weaknesses: 1.	The 4th and 5th paragraphs in the introduction section discuss literature of traffic scenario generation in simulation. However, a lot of relevant works are missing. I understand that it is not necessary to review related works in the introduction section, but this review is also missing in the related work section with “We have covered the related work on learning-based traffic simulation in Section”. The authors can find more relevant papers in the following survey papers [3][4][5]. For example, there are recent papers that using LLM to generate scenarios [1][2], which I believe are highly related to this paper.
2.	Based on my understanding, the proposed method is very similar to LCTGen [1], which converts natural language to some intermedium representation (the script in this paper) and then execute the script. The main difference is that this paper uses a rule-based planner while LCTGen uses a learned trajectory generator. In addition, this paper uses Carla as the sensor simulator while LCTGen only considers 2D simulation with BEV representation. Considering the similarity and differences, I feel the novelty of the proposed method is somehow limited. 
3.	As a generation task, the quantitative evaluation is important. In this paper, I think only two main metrics are considered, namely User involvement percentage and Execution success rate. However, I think they are mainly correlated to user instructions and prompt design. We still need metrics to evaluate the realism, diversity, and instruction following capability of the method, which I believe are important for indicate whether the generated scenarios are useful or not.
4.	The current system requires human users in the loop to revise the script. I appreciate that the authors mentioned this in the limitation section. This seems a strong limitation since the labor requires from human may reduce the benefit of using this system. A human user may directly use similar API interface and tools to quickly compose a scenario without using LLM.
5.	It is not clear how the generated scenarios benefit the testing of autonomous vehicles. Either using the scenarios to train a policy or evaluate a trained policy would provide insights on the value of the proposed system. Since this paper uses Carla, which provides high-quality sensor simulation, it would be good to see how some end-to-end driving model perform in the generated scenarios. 

---
[1] Tan, Shuhan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuehl. "Language conditioned traffic generation." arXiv preprint arXiv:2307.07947 (2023).

[2] Zhong, Ziyuan, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. "Language-guided traffic simulation via scene-level diffusion." In Conference on Robot Learning, pp. 144-177. PMLR, 2023.

[3] Ding, Wenhao, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. "A survey on safety-critical driving scenario generation—A methodological perspective." IEEE Transactions on Intelligent Transportation Systems 24, no. 7 (2023): 6971-6988.

[4] Zhong, Ziyuan, Yun Tang, Yuan Zhou, Vania de Oliveira Neves, Yang Liu, and Baishakhi Ray. "A survey on scenario-based testing for automated driving systems in high-fidelity simulation." arXiv preprint arXiv:2112.00964 (2021).

[5] Schütt, Barbara, Joshua Ransiek, Thilo Braun, and Eric Sax. "1001 ways of scenario generation for testing of self-driving cars: A survey." In 2023 IEEE Intelligent Vehicles Symposium (IV), pp. 1-8. IEEE, 2023.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Can you respond to each of the weaknesses point I mentioned?
soundness: 1
strengths: +the application can be potentially useful for augmenting traffic data.

+the paper is overall clear and provides relevant details of the proposed approach.
summary: This paper introduces LASER which leverages ChatGPT-4o to generate traffic simulations. The method contains two stages: converting user description into a step-by-step plan for each agent and then using LLM-controlled agents to convert the plan into executable waypoints for a low-level PID controller. To evaluate LASER, the authors evaluate the success execution rates of 8 scenarios.
weaknesses: -missing related work on using LLM on generating traffic behavior e.g., [1] [2] [3].

-the proposed approach is incremental compared with existing work (e.g. [3]).

-scripts still require manual revision and thus greatly limit its scalability. The number of total generated evaluation scenarios is limited.

-diversity of the generated scenarios is not evaluated.

-no evaluation on the added values of the generated scenarios has been conducted. 

-compared with learning based controllers, using a PID controller to directly control a vehicle can differ from human driving behavior (e.g., in terms of acceleration, jerk, etc.)

[1] Language Conditioned Traffic Generation, CoRL 2023
[2] Language-Guided Traffic Simulation via Scene-Level Diffusion, CoRL 2023
[3] Promptable Closed-loop Traffic Simulation, CoRL 2024

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How is the observation generated? Is it heuristic?
2. Is there communication between the autonomous agents during execution? In cases where one agent's action is dependent on other agents, is it all resolved in the script-writing phase, i.e., everything has to be specifically written down as scripts?
3. What happens if one of the agents is a self-driving car following some policy? In the case where all cars are controlled by LASER, I don't see much benefit in closed-loop simulation compared to open-loop generation.
soundness: 3
strengths: 1. I think the application of LLM to translate language descriptions of traffic scenarios into detailed scripts is interesting and reasonable
2. The simulation is carried out in closed-loop, which adds to the fidelity of the result
3. The paper is overall well-written and easy to follow
summary: This paper proposes a framework named LASER that generates traffic simulation conditioned on language description. First, an LLM is called to create scripts for the simulation, including a high-level script for the whole scene and sub-scripts for individual agents including detailed task descriptions.

Then an autonomous agent executes the scripts by invoking LLM to give high-level commands and following the command with a rule-based controller. 

The simulation is carried out in Carla. The metrics used to measure success are 1) the percentage of human modification in the final executable script, 2) the number of tokens and wall clock time for executing a 1-second simulation 3) the success rate of the simulation measured by whether the agents follow the language description.
weaknesses: 1. I think the fact that humans need to keep modifying the scripts generated by LLM is concerning. It's unclear whether the use of LLM actually saves time.
2. Following the previous comment, the metric of Effectiveness seems very objective, i.e., what if one person is better than another person at modifying the LLM-generated scripts? I think a more reasonable metric is to measure how often the script can work without any human modification.
3. There lacks comparison to some existing works such as LCTGen, CTG.

Meta-review: This paper presents LASER, a framework that leverages large language models (LLMs) to generate traffic simulation scenarios from natural language descriptions. The approach has two stages: first generating scripts from user descriptions, then executing them with autonomous agents in real-time in the CARLA simulator.
Strengths:
+ The application of LLMs to traffic scenario generation is a potentially useful direction
+ The paper is generally clear and well-organized
+ The closed-loop implementation in CARLA adds realism

Major Weaknesses:
- Limited novelty compared to existing work like Text-to-Drive and LCTGen that also use LLMs for traffic scenario generation
- High reliance on manual script revision undermines the claimed benefits of automation
- Very limited evaluation (only 17 scenarios) with no comparison to baselines
- Lack of metrics for scenario quality, diversity and usefulness
- Insufficient analysis of how the generated scenarios benefit autonomous vehicle testing

The paper has significant limitations in technical novelty, methodology and evaluation. Multiple reviewers noted strong similarities to existing work and raised concerns about manual intervention requirements that limit scalability. The evaluation lacks meaningful comparison to relevant baselines and provides limited evidence of practical value. Given these substantial weaknesses and no author response addressing these concerns, I recommend rejecting this paper.

Decision: Reject