ReGen: Generative Robot Simulation via Inverse Design
Authors: Phat Tan Nguyen
Tsun-Hsuan Wang
Zhang-Wei Hong
Erfan Aasi
Andrew Silva
Guy Rosman
Sertac Karaman
Daniela Rus
Venue: ICLR 2025 Poster (2025)
Keywords: generative simulation
robot
autonomous driving
large language model
inverse design
Primary area: applications to robotics, autonomy, planning
TL;DR: ReGen generates simulations from behavior by inferring plausible simulated environment where the behavior could have occurred through inverse design.

Abstract: Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains labor-intensive. In this paper, we introduce ReGen, a generative simulation framework that automates this process using inverse design. Given an agent's behavior (such as a motion trajectory or objective function) and its textual description, we infer the underlying scenarios and environments that could have caused the behavior.
Our approach leverages large language models to construct and expand a graph that captures cause-and-effect relationships and relevant entities with properties in the environment, which is then processed to configure a robot simulation environment. Our approach supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinct sensing modalities, such as braking due to faulty GPS signals. 
We demonstrate our method in autonomous driving and robot manipulation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learning for improved generalization and robustness.

Published as a conference paper at ICLR 2025
ReGen: GENERATIVE ROBOT SIMULATION VIA INVERSE DESIGN
Phat Nguyen', Tsun-Hsuan Wang' Zhang--aaee Wei Hong', Erfan Aasi', Andrew Silva', Guy Rosman", Sertac Karaman', Damiela Rus "Massacuusetts Institute of Technology {peterng, , tssnw, zwhong, eaasi, ser tac, rus] edu
Toyota Research Institute
(andrew. si silva, guylaagg...... crrosmmnnetrrr global
ABSTRACT
Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains a labor-intensive process. This paper introduces ReGen, a generative simulation framework that automates simulation design via inverse design. Given robot's behavior-such jective function-ang its textual description, ReGen infers plausible scenarios and environments that could have caused the behavior. ReGen leverages large language models to synthesize scenarios by expanding directed graph that en- codes cause-and-effect relationships, relevant entities, and their properties. This structured graph is then translated into a symbolic program, which configures and executes a robot simulation environment. Our framework supports (i) aug- menting simulations based ego-agent behaviors, (ii) controllable, counterfac- tual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinet sensing modalities, such braking due to faulty GPS signals. We demonstrate ReGen in autonomous driving and robot manipu- lation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable gener- ation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learn- ing for improved generalization and robustness. Please check our website here: https://regeet gi io/.
motion trajectory or an ob-
INTRODUCTION
Simulated environments play a vital role in validating robotic systems and provide platforms for robots to acquire complex skills, including autonomous driving (car, 2020; Amini al., 2022; Gulino et al., 2024), manipulation (Zhu et al., 2020; James et al., 2020; Nasiriany et al., 2024a), and locomotion (Rudin al., 2022; Makoviychuk al., 2021). Unlike real-world learning testing. simulations offer access to privileged states, enable unlimited exploration, and support large-scale parallel computation-ale without the need for heavy investment in robotic hardware. Classical simulation methods often rely on manually crafted environments and predefined scenarios, which require significant human expertise and effort both setup and maintenance Brockman (2016); Müller et al. (2018). These traditional approaches, while effective, are often limited in flexibility and scalability compared to the newer techniques. more recent paradigm, generative simulation, leverages generative artificial intelligence (AI) to automate the creation of simulations, greatly re- ducing the human effort and tedious process typically involved. Promising progress has been made in asset generation (Wang et al., 2023a; Siddiqui et al., 2024), scene layout design (Höllein et al., 2023; Yang et al., 2024), and task and environment creation (Wang et al., 2024d; 2023c).
Generative simulation offers the potential to create infinite environments for robots learn and be tested in. However, previous methods often encounter significant limitations in generating low- level control-such as a trajectory-from high-level textual descriptions, constraining the diversity
"This work is supported by Toyota Research Institute (TRT). It, however, reflects solely the opinions and conclusions of its authors and TRI any other Toyota entity

---

Published as a conference paper at ICLR 2025
Ego-agent Behaviors
Inverse Deeign
Simulated Environments
The vehiccle stopped abrupplly et an Intersection
Stop for oar tumnng kio
Yiekd lo ambulance
Broka for falaan bus
Hallfer police chuse
The robet opened the door
Cheok for deliery
Allowr petto enter
Open the door for venstattion
See off guest
Figure 1:
Given a robot behavior (such as trajectory or an underlying objective function) and its textual description, ReGen generates simulated environments that could have caused the behavior.
and complexity of the robot simulations. For instance, these approaches often require training a new policy for each simulation based newly generated reward function, making this process the primary computational bottleneck. We leverage the insight that behaviors are relatively limited compared the diverse environments in which they occur. For instance, the abrupt stopping of a self-driving car can apply to various contexts, such as red traffic light, a pedestrian stepping into the road, or an approaching police car with its siren on (see Figure 1). To address this, we draw inspiration from inverse design (Molesky et al., 2018), concept widely used in computational design that starts with desired properties outcomes. For example, given target flow pattern, the geometry of fluidic device is optimized accordingly. In generative simulation, we propose generating or designing environments conditioned on the agent's behavior. The benefits of this framework can be investigated from different perspectives:
validation perspective. Generative simulation via inverse design enables the conditional genera- tion of simulated environments based on specific robot behaviors. This approach facilitates the unit testing of robotic systems, where the unit defined at the behavior level'. By tailoring environments to specific behaviors, set of diverse yet relevant contexts, rather than relying on random or uncontrolled scenarios.
systematically evaluate how well a robot performs under controlled
An augmentation perspective. A natural extension from validation is incorporate failed test cases into the robot's learning pipeline as a form of data augmentation. This approach strengthens the robustness of the robot's behavior across diverse and relevant environmental contexts. Addition- ally, this inverse approach can be seen as a method for augmenting existing simulated environments. Given a simulated environment and robot policy exhibiting a relevant behavior, inverse design en- ables the sensible addition or removal of entities based on the context. This is achieved by reasoning backward from the robot's behavior identify what elements in the environment are relevant.
In this work, we propose an inverse design approach for generative simulation. Given robot's behavior-such as motion trajectory or an objective function-alone with their textual descrip- tion, our method generates relevant simulated environments. We represent the environment as directed graph, where nodes correspond to events, entities, or properties, while edges capture causal relationships (e.g.. car stopping "due to" ambulance) or dependencies (e.g., a siren on an ambu- lance). This structured representation is then transformed into a symbolic program that configures and executes robot simulation environment. In summary, contribute:
An inverse design approach for generative simulation demonstrated in autonomous driving and manipulation with abilities to (i) augment simulations based on ego-agent behaviors. (ii) generate controllable, counterfactual scenarios, (iii) reason about agent cognition and mental states, and (iv) handle distinct sensing modalities, such as braking due to faulty GPS signals.
method for synthesizing simulatable scenarios using LLM- -iMddddded graph search to construct a directed graph that encodes causal relationships, entities, and properties, which is then converted into an executable symbolic program for robot simulation.
Extensive experiments that showcase greater diversity of generated environments compared to ex- isting simulations, controllable generation corner cases for safety-critical applications like driv- ing, and superior complexity of generated environments, which produces vision-language-acoo datasets that are more challenging to vision language models (VLMs) than existing datasets.
'The level of granularity at which the hehavior should be defined is beyond the scope of this paper and remains open research question for future exploration.
2

---

Published as a conference paper at ICLR 2025
Algorithm 1 Graph Expansion
Input: Obetavior E Vevent Output: G E g
procedure POTENTIALLY CONNECT TO NODE(G,U,..
Vcaadiddtee = node proposal
G, _addeddd = edge_constee www.cconnsttccccc construction(r, Veandidate: G)
return G, !/aandiaate- edge_added
end procedure
Initialize: G = (} E 9; G f- Ubehavior
while input_degree(1i E G) < do
Vevent ~ for input degree( G) <
G, eventt potentially connect to node(G, -eventy....)
break whenever by the user.
end while for
(Uevent I G, Vevent do
G. Veutity. addee_dded wta.eenniallyyyonnee..t.t soonoode(( no, Vevenns....) edgeeadded added then for All possible property types do
won.eeeellcccntt.t..omoomoooii.. G, Upopenyy potentially connect. to node G, Ventity ...)
end for
end end for
METHOD
We present ReGen, framework that takes as inputs behavior B = {T, R}, where T is motion seecttrr-ggneratee using motion primitives- and a reward function R. along with its textual description and a graph-based representation of the simulator database Dasee. to generate plausible simulations. For example, given "vehicle changed lanes," our method simulates scenarios such as yielding to ambulance, overtaking slow truck, and merging into open lane. We assume access to a simulation engine, such as CARLA (car, 2020) for autonomous driving or PyBullet (Coumans & Bai, 2016-2021) for manipulation, along with database Dasset For further details on implementation, see (Appendix A.
At high level, the method begins by synthesizing plausible scenarios through graph search using an L.M,, where directed graph iteratively expanded by introducing new plausible contexts and fine-grained details, such entities and their properties (Section 2. 1). Next, for scenario generated through the graph, we leverage an LLM to generate a symbolic program that formulates high-level constraints, serving verifier ground the scenario in simulation (Section 2.2).
2.1 INVERSE DESIGN VIA GRAPH EXPANSION
The inverse design process in Algorithm 1 begins by representing the input behavior description as a leaf node. The graph g is then expanded backward through two atomic steps: node pro- posal and edge construction, applied to different types of nodes and edges. Both steps employs the 11--40-2024-08 08-06 model with parameters temp=0 and top-p=0. A key perspective is that graph expansion systematically explores the LLM's knowledge space, uncovering and organizing latent information into a structured graph.
Node Proposal is the process of generating candidate nodes Veandidate [Veeatt Veatity Vpropeny) that may connect to the existing graph via source node Usource- We consider three types of nodes: event, entity, and property nodes. First, event nodes Vevent represent causal variables (e-g., an event "yielding to ambulance" is cause of another event "ego-vehicle stopping")). We use an LLM to propose candidate event nodes LLM (v E Vevent Vsourcee prior), treating the source node as the effect. The prior is optional natural language description that injects additional context and can be sourced from various inputs, such as user preferences or another LLM. Next, entity nodes Ventity represent static objects (e.g, debris) dynamic actors (e.g., ambulance) and proposed from a fixed set of supported assets Daseet in the simulation engine Veminy C D Dasset- Lastly, property nodes Upopeeyy define attributes for each entity, including elements proposed by LLM,, such as
3

---

Published as a conference paper at ICLR 2025
location (e.g. "in front of the ego"). or possible states retrieved from D For example, candidate property nodes for a traffic light represent all its possible states in the simulator, such as "red," "green," "yellow", and "off". These proposed nodes serve as candidates for the edge construction step. See Appendix A. 1.2 for examples and full prompts in Appendix A.5.1.
source node Usource by evaluating possible connections simultaneously. rather than pairwise (Jiralerspong et al., 2024). Formally, it involves mapping LLM (Usoureey Veandidate) - {True, False). where candidatte {VI, {UI, U2,..., ... represent the sets of candidate nodes from the node proposal step. Edges are then constructed for all {u, E Veandidase : LLM( Usource: 14)] True). This mapping leverages an LLM as a general classifier to evaluate the plausibility of each candidate connection in the context of the source node. Our graph expansion considers three types of edge constructions: event-to-event, entity-to-event, and property-to-entin edges, the LLM validates direct causal relationships using common- sense reasoning- for example, an emergency vehicle behind the ego-vehicle may cause the ego to pull over, while one ahead may not. For entity-to-event edges, the LLM selects relevant entities, while for property-to-entity edges, connections are determined by supported properties relevant to the entity's 's role. The edge con- struction process performs rejection sampling by discarding implausible connections and ensuring only edges corresponding
Edge Construction determines which candidate nodes should be connected
with the order indicating the direction. For event-to-event
available entity or property in Daset added the graph. See Appendix A.1.3 for examples and Appendix A.5.2 for full prompts.
Graph Expansion is iterative process that augments the graph through successive node propos- als and edge construction. The process begins with the initialization of the graph G = (V,&), where the initial sets of nodes V }CB} contains the input behavior description L V Vevent- In the node proposal stage, candidate event nodes are generated for all event nodes that lack an incoming edge, i.e., nodes without defined cause. Edges are then established by assessing the plausibility of connections between nodes. This staged approach incorporates new context into the graph, pro- moting scenario diversity, while enforcing logical correctness through common-sens reasoning. At this stage, the graph contains only event nodes, forming a causal graph with a directed acyclic struc- ture. Users can specify a stopping criterion, such as a maximum number of nodes or graph depth; otherwise, the process continues indefinitely. Subsequent steps involve adding entity and property nodes to incorporate fine- ame-grained details into the graph through a similar graph expansion process. The complete graph expansion process is outlined in Algorithm 1.
2.2 GROUNDING TO SIMULATED ENVIRONMENT
Given a subgraph g S 9 that forms a unique sequence of event nodes connected by event-to-event edges, the goal is to generate a simulation that specifies: (i) the initial state of the environment qu, defined by the preconditions Pmit. (ii) the motions of all dynamic actors, governed by the transition function d Q x Σ + Q of the abstract states Q, and (iii) the terminal conditions, characterized by the post conditions P Piem that define the symbolic constraints for the final states, including success or failure criteria, represented by the accepting states F S Q. Inspired by other work (Nguyen et al., 2024), we convert y into a FSM configured with the same model and hyperparameters as in the graph expansion step. The FSM, formally defined as tuple FSM (Q. E, 6, go, F),, serves as a symbolic representation of the task plan and is used to (i) verify its feasibility prior to execution, and (ii) trigger state changes based on simulation context, such as opening a car door when the ego-vehicle is nearby.
The set of abstract states Q are high-level code abstractions, which bridges abstract reasoning with low-level simulation states. These abstractions are implemented as executable code compatible with the simulation engine. For example, an abstract state such as "yielding to ambulance" provides an abstraction for checking whether the ambulance is nearby and whether the ego-vehicle is sta- tionary: check_car1 www.caarrrnrarr___ee s_ne_____as__ 2 aambulance', 'ego') ') & is_stationary 'ego'). For further examples see Appendix A.2.
3 EXPERIMENTS
In this section, we start in Section 3.2 with qualitative analysis that demonstrates examples and capabilities of the inverse design approach; in Section 3.3, we compare diversity of the generated simulation against existing benchmarks or simulations; in Section 3.4, we showcase ReGen can be used for effective corner case generation; in Section 3.5, we demonstrate that the complexity
4

---

Published as a conference paper at ICLR 2025
Ego-vehicle Changes Lanes Sile
Din
Crange
umargency vific
Marping itto sper
Ansiding
Ego-vehicle Slows Down
Ego-vehicle Stops Abruptly
Ovv Smoure
Delve Aower
GINCTR
Eucilen
Figure 2: ReGen for driving. Given a behavior such as "changing lanes," " our method can generate diverse simulated environments in which the behavior could have occurred, such as changing lanes to "yield for an emergency vehicle", "overtaking a truck", "merging into an open lane", or "avoiding debris." The bottom row illustrates additional cases: "ego-vehicle slows down <- avoids collision" and "ego-vehicle stops f- halts for an opening car door."
Tum on the faucet
top
vegutable
plant
Close the window
Block outsiide oder
Protect
candiess
Keep out unmanted gaze
Figure 3: ReGen for manipulation. Given robot manipulator behavior such as "turning on the faucet," our method generates diverse simulated environments with contextually appropriate scenar- ios, including "collecting water," "rinsing a dish," and "watering plant." " In cases where certain physical effects such temperature changes in the "thawing frozen vegetables" scenario are not explicitly modeled, our method simplifies the process by retrieving and appropriately placing ob- jects in the environment.
of our generated environments allows us to create Asioo-language-action datasets that pose greater challenges to VLMs compared existing datasets.
3.1 EXPERIMENT SETUP
Driving, For autonomous driving, we utilize the CARLA simulator (Dosovitskiy et al., 2017). We selected six key ego-motion behaviors: forward stopping abruptly in the middle of the road, and stopping after making right turn. Each behav- ior is defined in natural language and mapped to a predefined route. encoded in an XML file that defines the start location, target waypoints, and speed. The ego-vehicle follows this route gener- ated by an A search algorithm, with longitudinal and lateral PID controllers for speed and steering control. To simulate other agents, we use motion primitives such as stationary (location) and dri reiiinnnoowwrdsstaal
forward, changing lanes, stopping an intersection,
speed). These agents also use A for path planning and PID controllers for low-level control. Parameter values (e.g., start locations, speeds) are determined using Google's CP-SAT solver (Perron & Furnon, 2019) to ensure compliance with the FSM's s constraints.
Manipulation. For manipulation tasks, use the PyBullet simulator (Coumans & Bai, 2016-2021) and selected 10 example reward functions from RoboGen (Wang et al., 2024d) for general tasks such closing the window and opening the door. In PyBullet, behavior is limited stationary interactions as the environment consists of non-articulated meshes that do not support dynamic motion. We use Google's ss the CP-SAT solver determine parameter values (e.g. object location), and employ Soft Actor Critic (SAC) (Haarnoja et al., 2018) for policy training.
5

---

Published as a conference paper at ICLR 2025
1. Mental State (e.g. Distracted Driver)
3. Counterfactual Generation
U-s
1-10s
1-15s
Brake lights on
Brake lights off
Green light
Green light
light
stope ligtt?
2. Reasoning in different sensing modalities: (e.g. GPS Jamming)
Different
t-Cs
Ege-vehidee
driver
jamming.
Whet i the emergeecy veriide wndt in The carra lune?
Figure 4: Emergent capabilities of ReGen. (1) Our method can simulate scenarios that capture the mental states or decision-make process of actors- for example, a distracted driver failing to move despite green light. (2) Our method can model reasoning across different sensing modalities, such as simulating a scenario with GPS jamming with noisy GNSS measurements in CARLA. (3) Our method enables counterfactual scenario generation by perturbing the constructed graph with "what- if" questions-for example, changing from "the front car stops with brake lights" to "the front car with broken brake light stops and thus brake light being off.'
3.2 QUALITATIVE ANALYSIS
Generative Simulation via Inverse Design. We demonstrate the inverse design approach of ReGen by generating diverse simulations from a single behavior, showcasing qualitative results across driv- ing and manipulation domains. As shown for driving in Figure 2, when provided with behavior such as "changing lanes," our method can simulate scenarios such as "yielding for the ambulance," "overtaking a slow vehicle," "avoiding debris on the road." Unlike prior methods, which often re- quired extensive data collection or expert-curated policies to create such scenarios, we demonstrate that our approach can generate these variations by simply reconfiguring the environmental context with respect to the given robot behavior. For instance, as shown in Figure 7 in the Appendix shows that ChatScene primarily generates collision avoidance scenarios, while DriveCoTT and DriveLM focus mainly on general driving scenarios. In contrast, ReGen supports diverse task scenarios, in- cluding "picking up passengers," "stopping for pedextrians." " and "yielding to emergency vehicles." This capability can potentially be applied to augment the simulation of the original robot learning pipeline, further enhancing the robustness the learned behaviors.
Similarly, in the manipulation domain in Figure 3, given an input such "turn on the faucet," our method can infer new action verbs and their corresponding purposes, such as "thawing frozen vegetables." "wattrrig a plant," or "washing dishes." This capability extends beyond simulating actions to also capturing the underlying intent or goal behind each action.
Simulating Mental States. Accurately capturing subtle mental states and decision-making pro- cesses from real-world driving datasets inherently challenging. ReGen provides framework for simulating nuanced mental states, such distracted driver at an intersection (see Figure (1)). Previous work, like DriveCoT (Wang et al., 2024c), employs rule-based expert policies to control vehicles and generate ground truth labels for reasoning processes. However, this approach can introduce extraneous variables that obscure causal relationships. For example, in a DriveCoT scenario labeled as yielding at an intersection, 16/17 annotations correctly attributed the stop to a traffic sign, but missed the emergency vehicle's 's influence as contributing factor. ReGen mitigates this by reusing scenes and applying targeted interventions to simulate distinct cognitive processes, providing greater control in modeling mental states.
Reasoning with Different Sensing Modalities. ReGen extends its capabilities by reasoning over multiple data modalities, including vision, language, and other distinct sensor inputs such as GNSS (see Figure 4 (2)). By leveraging large language models (LLMs), ReGen can provide reasoning over these different modalities to simulate complex scenarios that influence the decision- making of the ego-driver. For instance, invoking functions such as add gnss noise() allow for the simulation of GPS jamming, connecting sensor noise to abstract concepts such as signal interference.

---

Published as a conference paper at ICLR 2025
Method
Number of Scenarios
Embedding Diversity
SelfBleu Diversity
NHTSA Crash Report
24
0.1381
0.4350
Zero-shot poppt-3--turbo-oo-
0.1509 0.01
0.0945 0.03
Zero-shot (got-40-2824-88--00
0.1680 0.02
0.6082
Zero-shol (top-p=1, temp=1)
0.1767 0.04
0.7814 0.06
Zero-shol (top-p=0, temp=8)
0.1493 0.01
0,4143 0.02
ChatScene
40
0,1214
0,2945
DriveLM
696
0,1135 = 0.00
0.4731 + 0.01
ReGen (Ours)
24
0.2268
0.7377
Table Simulation diversity for driving. The baselines include NHTSA typology (National High- way Traffic Safety Administration, 2007), zero-shot LLM method, ChatScene (Zhang et al., 2024) (few-shot), and DriveLM (Sima et al., 2024). Apart from expert driving or exclusively safety-critical scenarios, our method can generate a broader range of environments, e.g., yielding emergency ve- hicles, navigating intersections with malfunctioning traffic light, thus achieving better diversity.
Methods
Environments
Unique Reward F Functions
Embedding Diversity 1
Bchavior-100 100
100
100
0.5513 + 0.01
RLBench
106
106
0,5819 0.01
GenSim
152
152
0.4350 + 0.01
RohoGen (manipulation)
46
46
0.5787 + 0.01 0.5536
RoboGen (subset)
10
10
ReGen (Ours)
38
10
0.6560
Table 2: Simulation diversity for manipulation. We compare to Behavior- 100 (Srivastava al., 2021), RLBench (James et al., 2019), GenSim (Wang et al., 2024b), and RoboGen (Wang et al., 2024d). Our method augment simulation in an orthogonal axis that changes environmental context, e.g.. "opening the door to let the pet in" or "opening the door to pick up delivery". as opposed to purely skill-driven environments, e.g., "opening the door", thus achieving better diversity.
Counterfactual Generation. Our method can generate counterfactuals, such as modifying brake lights or adjusting the initial location of surrounding vehicles, to improve explainability in multi- modal foundation models (Figure (3)) For example, by altering brake lights, we can test whether the model infers speed from visual motion cues or relies brake lights shortcut, addressing the limitation of multimodal foundation models identified in prior work (Sreeram et al., 2024).
3.3 SIMULATION DIVERSITY
In this section we conduct a series of experiments evaluate the diversity of generated simulations against extensive baselines in Table and 2. To quantify the diversity in terms of task semantics and scene configurations, we use text diversity metrics, following approaches from (Wang et al., 2024d; Nguyen al., 2024). Specifically, 2018; Papineni et al., 2002) and embedding similarity with Sentence-BEr BERT (Reimers & Gurevych, 2019). For each method, we sample equal to the smallest sample size among the baselines and compute their similarity score, repeating this process 10 times and reporting their average diversity score similarity.
assess diversity using metrics such BLU (Zhu al.,
Driving. Table shows the scenario diversity results in the driving domain. We compare our method against CARLA Leaderboard 2.0. scenarios, which cover traffic scenarios based on the NHTSA ty- pology (National Highway Traffic Safety Administration, 2007). ChatScene (Zhang et al., 2024) employs few-shot prompting of large language models to generate diverse safety-critice scenarios, while DriveLM (Sima et al., 2024) provides graph annotations from the NuScenes dataset (Caesar al., 2020). Additionally, we evaluate the diversity of zero-shot methods across varying top-p and temperature settings, as well as with different L.LM models, including gpt-4o-22224-8- 2824-08-06 and 3. 5-tuurttoo01., Our method consistently outperforms all baselines in scenario diver- sity, as measured by embedding similarity and Self-BLEU scores. Methods like ChatScene focus exclusively on safety-criticn scenarios. while DriveLM is limited to general driving scenarios. Al- though increasing the top-p and temperature settings slightly improves diversity for the zero-shot baseline, our method achieves superior diversity even with conservative settings (top-p 0 and temperature 0), demonstrating that the improvements not merely result of tuning these pa-
7

---

Published as a conference paper at ICLR 2025
CharScenne
Metric
Method
Base Traffic Scenarios
Scenario Descripton Diversity
SO
TO
LaneC
VP
0.25
L.C
0.30
0.09
0.87
0.83
AdySim
0.51
0.33
0.86
0.87
0.20
CS
0.45
0.61
0.89
0.87
CR 1
Adv
0.50
0.31
0.78
0.82
6.15
ChatScene
0.89
0.70
0,95
0.79
0.10
ReGen (ours)
0.90
0.83
0.96
0.77
0.05
Table 3: Collision Rate in SafeBench. SO means Straight Obstacle. TO means Turning Obstacle. LaneC means Lane Changing. VP means Vehicle Passing. Baselines include LC (Ding et al., 2020), AdvSim (Wang et al., 2023b), CS (Wang et al., 2023b), AdvTraj(Cao et al., 2022), and ChatScene (Zhang et al., 2024). Our method can produce cor- ner cases that lead to a higher collision rate (CR).
Alruptly
(1e2-wtand ars]
Adversarial Policy
Figure 5: Diversity of Corner Cases. Com- pared to ChatScene (Zhang et al., 2024), our method generates more diverse corner cases via reasoning about different causes.
rameters. In contrast to the baselines, our approach can simulate a broader range of scenarios, such as yielding to emergency vehicles or navigating intersections with malfunctioning traffic lights. The success rate of our scenario generation is 80%, with a detailed breakdown provided in Table 6 in the Appendix. Most failure cases are due to overly strict FSM constraints that require multiple con- ditions to be satisfied simultaneously. Although semantically correct, it imposes unnecessary strict satisfiability requirements. Further discussion can be found in Appendix A.4.2.
Manipulation. Table 2 shows the simulation diversity in the manipulation domain. Since the task deseriptions were short we only report the embedding diversity since n-gram based metrics are not applicable. We compare our results against baselines such as RoboGen (Wang et al., 2024d), GenSim (Wang et al., 2024b), Behavior- (Srivastava et al., 2021). and RLBench (James et al., 2019). In prior work, generating a manipulation simulation often required learning new task either through reward design (Wang et al., 2024d) or by using expert demonstrations (Wang et al., 2024b). We demonstrate using our inverse design framework that we can generate simulations by reusing learned behaviors from these methods to simulate new scenarios. We use 10 reward functions from RoboGen, each generating variants of environments. With success rate of 78%, end up having 38 environments. The most failure cases are invalid reasoning of articulated objects, e.g., a shelf-opening event with the asset not being able to open. Among all baselines, our method achieve the highest diversity. This is mainly because prior works mostly focus simulating environment corresponding to a single skill, often within similar contexts e.g., "open the door" and "close the door". while method augments simulation in orthogonal axis such as "open the door to let the pet in." However, unlike driving where fine-grained control on many entities especially dynamic actors is possible, our approach simplifies manipulation tasks by limiting actions to retrieving and placing entities in contextually appropriate locations (c.g., the dog has to be outside the door). Elements not supported by the simulation engine- such as simulating the motion of a pet walking in are skipped. We leave these (better articulated objects and more flexibility of simulation engine for manipulation) future research.
3.4 CORNER CASE GENERATION
In Figure 5, we demonstrate our method's ability to generate diverse corner case scenarios for test- ing in safety critical scenarios, detailed in Table We highlight ReGen's ability to reuse an existing adversarial policy behavior in more diverse context. For comparison, we evaluated against ChatScene (Zhang et al., 2024), which uses LLM few-shot prompting and converts text into simula- tion via text-to-scemic. For fair comparison, we use the same adversarial policy behavior as theirs but change the context of the environment. We consistently achieved greater diversity than ChatScene across all adversarial policy behaviors. While ChatScene introduces only minor variations, such as a pedestrian crossing in front a car or vending machine, our method captures broader range of cause-and-effect relationships. For the "pedestrian walking" behaviors, our method generates unique causes, such as a group of protesters causing a collision or a single pedestrian forcing an- other vehicle to stop abruptly in front of the ego-vehicle. These scenarios underscore the greater diversity enabled by our approach. We observe that both methods achieve greatest diversity for the behavior "drive forward," " as it can be applied in a broader range of contexts, such as driving fast or
8

---

Published as a conference paper at ICLR 2025
Accuracy (%)
Method
GPT-4o 0.87
GPT-4-Turbo
Claude 3.5 Sonnet
DriveCoT
0.80
0.80
DriveLM
0.90
0.83
0.77
ReGen (Ours)
0.63
0.53
0.50
Table 4: Complex Vision-languaag--action Dataset from Our Simulated Environments. We com- pare with existing driving-related reasoning datasets, DriveCoTT (Wang et al., 2024c) and DriveLM (Sima et al., 2024). The accuracy of reasoning about the correct ego-vehicle actions is reported. Our dataset consists of more complex scenarios compared to baselines with mostly urban driving maneuvers, thus posing greater challenges to existing VLMs.
slow, accelerating, and decelerating. In contrast, both methods exhibit lower diversity for left and right turns, as these scenarios offer fewer plausible contextual variations.
We then evaluated these generated scenarios against ChatScenc and other adversarial policy learning baselines including Learn-to-Collidde (LC) (Ding et al., 2020), AdvSim (Wang et al., 2023b), Carla Scenario Generator (CS) (Wang al., 2023b), Adversarial Trajectory Optimization (Cao et al., 2022) to test whether the cornerccase simulations led to higher collision rates. The results, presented in, Table 3, only includes the collision rates, as the final score incorporates driving infractions, which fall outside the scope of generating collision scenarios. To run the benchmark, we converted our scenarios into scenic files, required in the SafeBench benchmark (Xu et al., 2022). This process was done manually as generating scenic files is outside the scope of our work.
Both our method and ChatScene outperform other baselines by leveraging LLMs to intelligently spawn adversarial agents. We further validate by human inspection that all generated environments do not place actors in the locations that cause inevitable collision, namely too close to the ego-car. However, our method surpasses ChatScene by diversifying the underlying causes of challenging scenarios. While ChatScene primarily generates scenarios where an object crosses in front of the ego-vehicle (e.g., pedestrian crossing
car merging), our method introduces more complex variations, such as a group of pedestrians or a vehicle traveling in the opposite direction. Empirically, these scenarios are significantly more challenging for the driving policies, such as requiring larger steering adjustments to avoid groups of pedestrians.
3.5 PROBING MULTIMODAL FOUNDATION MODELS
In this section, we evaluate how well the shie-offthh-arr vision language models (VLMs) can reason about the vision-language-action dataset produced by the generated simulation from our methods, compared to existing datasets. We aim to (i) demonstrate a scalable possibility of multi-modal data synthesis with complex reasoning and (ii) use the complexity of the produced dataset as an indirect measure to demonstrate the complexity of the generated simulation. We conduct experiment in the driving domain; the goal of the VLMs is to process sequence of images along with textual context and question, and answer most plausible actions to be taken by the ego-vehicle. Specifically, we assess the VLMs' ability to infer the desired action in our generated driving scenarios and compare their performance with two literature baselines: DriveCoT (Wang et al., 2024c) and DriveLM (Sima et al., 2024). In DriveCoT the authors used based expert policies to control ego and generated ground-truth labels for reasoning processes, while in DriveLM they employed graph annotations on NuScenes dataset (Caesar et al., 2020) as a large scale real-world driving dataset. From each dataset, we randomly select 30 simulation traces, together with the ground-truth desired ego action for each one. As part of the preprocessing, we extract three consecutive key frames from each simulation trace, where the last key frame corresponds to the timepoint where the desired ego action has been recorded. Since here we are solely evaluating the planning capability of VLMs, we also provide some privileged information, such as the location and speed of all entities in the scene, to the VLMs. Therefore, the three key frames, along with their corresponding privileged information that are parsed from the recorded log files, are provided to the VLMs, and they are prompted to identify the desired action for ego. The VLMs tested in this evaluation include GPT-4o (OpenAI, 2024), GPT-4-turbo (Achiam et al., 2023), and Claude 3.5 Sonnet (Anthropic, 2024). The success rates of the VLMs in inferring the desired actions are presented in Table 4. As shown in Table 4, the success rate of VLMs in inferring the desired ego actions for our dataset is significantly lower compared to the DriveCoT and DriveLM datasets. This difference is due to the greater diversity of

---

Published as a conference paper at ICLR 2025
scenarios generated by method compared to those in DriveCoT and DriveLM. In the baseline datasets, the desired ego actions are primarily common urban driving maneuvers, such as stopping or moving forward, whereas our method produces scenarios with wider range of ego actions.
We observed that VLMs frequently responded with deceleration as the default action, consistent with findings in prior work (Sreeram et al., 2024). For instance, in lane change scenarios such as "avoiding debris", "overtaking a slow vehicle". "merging into an open lane", or "swerving to avoid wrong-way driver" the VLM often suggested that the ego-vehicle should brake and stop behind obstacles instead of performing logical lane change to avoid them. These outcomes suggest that VLMs generally struggle in environments with nuanced spatial and situational reasoning. In contrast, DriveCoT (also using the CARLA simulator) generates scenarios where objects appear directly in the ego-vehicle's path, requiring a deceleration response. In such cases, the VI.Ms' biases align with the expected behaviors for those scenarios. These observed failures underscore a limitation of off-the-shelff VLMS in reasoning about complex driving scenarios. Conversely, the scenarios in DriveLM were less challenging for VLMs, as they mostly comprised general driving scenarios, as shown in Appendix 1.
4 RELATED WORK
Robot Simulation. Simulation has played a critical role for general robotics. Simulated environ- ments for driving (car, 2020; Amini et al., 2022; Gulino et al., 2024), manipulation (Zhu et al., 2020; James et al., 2020; Nasiriany et al., 2024a), and locomotion (Rudin et al., 2022; Makoviychuk et al., 2021; Wang et al., 2023d), have each contributed significantly their robotic subfield. Their use for verification has its own community and field of research (Kleijnen, 1995; Pace, 2004; Corso et al., 2021), in parallel significant efforts for leveraging such approaches for policy training (Muratore et al., 2022; Wang et al., 2022: Loquercio et al., 2019). Our work focuses on a new paradigm of robot simulation that uses generative models for more scalable simulation construction.
Generative Simulation. Generative simulation methods have emerged as powerful tool for auto- matically creating diverse and realistic scenarios in robot manipulation (Wang al., 2023c; Man- dlekar et al., 2023) and more general robotic tasks (Wang et al., 2024d; Nasiriany et al., 2024b). Our work shares the same goal of automating the process of constructing simulation with the power of generative AI. Uniquely, we follow an inverse design approach which is more tailored for validation and augmentation use cases, complementing existing approaches with an orthogonal axis.
LLMs for simulation. Specific approaches for diverse data generation (Shiroshita et al., 2020; Sinha et al., 2020) within simulation environments, including those that leverage language models (Zhong et al., 2023; Elmaaroufi et al., 2024; Aasi et al., 2024) are playing an important role in how simulation environments are used for both verification and system training. More broadly, our work relates to the rich set of different approaches harnessing LLMs for reasoning about, and planning in, domains such as robotics (Zeng et al., 2023; Wang et al., 2024a) and autonomous driving (Cui et al., 2023). These include aspects such language- based planners (Song al., 2023; Liu et al., 2023; Mao et al., 2023), and simulation environments (Zala et al., 2024), among others.
5 CONCLUSION
We present an inverse design approach for generative simulation, demonstrated in autonomous driv- ing and manipulation. Using LLMs and simulation engines, we construct and expand graphs that capture cause-and-effect relationships and relevant entities, which are then converted into simu- lated environments. With the extensive experiments, we demonstrate capabilities of our method via qualitative analysis, superior diversity compared to existing simulation, more effective corner-case generation, and more complex laaioo-lnggaag-acttinn dataset synthesis than current dataset.
Limitation. There are properties that our method can reason about but are not fully simulatable due to limitations of the simulation engine. For example, without simulating temperature, we cannot measure the progress of "thawing the frozen vegetable". This limitation is more common in manip- ulation tasks than in driving, where robots primarily interact with other actors, allowing for more flexible simulation. Another challenge is handling articulated objects, such as reasoning about open- ing a shelf, which cannot be simulated with the available assets; more details in Appendix A.4.2.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
CARLA leaderboard 2.0 scenarios, 2020. URL Accessed: 2024-09-29.
Erfan Aasi, Phat Nguyen, Shiva Sreeram, Guy Rosman, Sertac Karaman, and Daniela Rus. Gen- erating out-of-distribution scenarios using language models, 2024. URL https://arxiv .rrg/ 0oss22111.1555
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Alexander Amini, Tsun-Hsuan Wang. Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus. Vista 2.0: An open, data-driven simulator for multimodal sens- ing and policy learning for autonomous vehicles. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2419.2426. IEEE, 2022.
Anthropic. Claude 3.5 sonnet. Accessed: September 15, 2024.
http:/ppp...ccom////ccc///c/ssae 3-5- sonnet, 2024.
Brockman. Openai gym. arXiv preprint arXiv:1606.0155: 2016.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving, 2020. URL. httpps://aaxxi.rrr///////1.....
Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, and Marco Pavone. Advdo: Realistic 08744.
adversarial attacks for trajectory prediction, 2022. URL https:
Anthony Corso, Robert Moss, Mark Koren, Ritchie Lee, and Mykel Kochenderfer. A survey of algo- rithms for black-box safety validation of cyber-physical systems. Journal of Artificial Intelligence Research, 72:377-428. 2021.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybulle org, 2016-2021.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. A survey on multi- modal large language models for autonomous driving. arXiv [cs.Al], November 2023.
Wenhao Ding, Baiming Chen, Minjun Xu, and Ding Zhao. Learning collide: An adaptive safety- wttp://wwwwv....rror//rbbreea 01197.
critical scenarios generating method, 2020. URL
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator, 2017. URL httpps///arrkv..or///////1.....1
Karim Elmaaroufi, Devan Shankar, Ana Cismaru, Marcell Vazquez-Chanlatte Alberto Sangiovanni- Vincentelli, Matei Zaharia, and Sanjit A Seshia. Generating probabilistic scenario programs from natural language. arXiv preprint arXiv:2405.037700 2024.
Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, et al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems, 36, 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018. URL https://
arxiv.org/ htt.v.orrgabb//80000 abs//881s /00.///1.1.......... 01290.
Lukas Höllein, Ang Cao. Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Ex- tracting textured 3D meshes from 2D text-to-image models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7909-7920, 2023.

---

Published as a conference paper at ICLR 2025
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark and learning environment, 2019. URL https: http://arxivv.org////////ss 12271.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment, IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. Efficient causal graph discovery using large language models, 2024. URL https:/arrkv..orgaass//// 01207.
Jack PC Kleijnen. Verification and validation of simulation models. journal of operational research, 82(1):145-16) 162, 1995.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+p: Empowering large language models with optimal planning proficiency, 2023. URL
https://arxiv. orgoorgggaoooo wtt.oorg/////a/aeeoo abs//304./222/////////// 1tt.//224......... 11477.
Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza. Deep drone racing: From simulation to reality with domain randomization. JEEE Transactions Robotics, 36(1):1-14, 2019.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models, 2024. URL https:///rxivv https://arxxi..og//bb////2 2310. 12931.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108 10470, 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations, 2023. URL https:///aaa org/ abss/315 17596.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt, 2023. URL https://arxiv.ot http://aaxxvv..rg/aaa/ wttp://///a////...00//// 01415.
Sean Molesky, Zin Lin, Alexander Y Piggott, Weiliang Jin, Jelena Vucković, and Alejandro W Rodriguez. Inverse design in nanophotonics. Nature Photonics, 12(15):59,,770 2018.
Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, and Vladlen Koltun. Driving policy transfer via modularity and abstraction. arXiv preprint arXiv: 2018
Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot learning from randomized simulations: A review. Frontiers in Robotics and AI, 9:799893, 2022.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for gener- alist robots. arXiv preprint arXiv:2406.02522. 2024a.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation everyday tasks for gener- alist robots, 2024b. URL
http///xiii..or/////////.// /bs/ 02523.
National Highway Traffic Safety Administration. Pre-crash scenario typology for crash avoidance research. Technical report, United States Department of Transportation, National Highway Traffic Safety Administration, April 2007. URL https: //wp:: http:/wwwwttttaaaat/ttt../.. nhtsa. gov/ si tes/nhtsa..... httes/hhtta.ooo/lles/////. pre-crash scenario typology-final pdf version 5-2-00 2-07. 07.dff.
Phat Nguyen, Tsun-Hsuan Wang. Zhang Wei Hong, Sertac Karaman, and Daniela Rus. Text- to-drive: Diverse driving behavior synthesis via large language models, 2024. URL https: wwwaxii.org/abs///e. 043000
12

---

Published as a conference paper at ICLR 2025
OpenAl. Hello gpt-4o. ber 13, 2024.
http://hhhllogtttooo//oo 2024. Accessed: Septem-
Dale Pace. Modeling and simulation verification and validation challenges. Johns Hopkins APL. technical digest, 25(2):163-172, 2004.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguisties, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- tics. doi: 10.3115/107308. 1073135. URL https:
Laurent Perron and Vincent Furnon.
Or-tools.
https://devellopers gttp://eeeeeeprr..gooooolloeoe com/
optimization/, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https://arxiv. org/ abs/1908. 10084.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. Conference on Robot Learning, pp. 91-100. PMLR, 2022.
Shinya Shiroshita, Shirou Maruyama, Daisuke Nishiyama, Mario Ynocente Castro, Karim Hamza- oui, Guy Rosman, Jonathan DeCastro, Kuan-Hui Lee, and Adrien Gaidon. Behaviorally diverse traffic simulation via reinforcement learning. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 3103-2110. IEEE, 2020.
Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Gar- reau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov. et al. Meta 3d assetgen: Text-to-mesh generation with high-quality geometry, texture, and pbr materials. arXiv preprint arXiv:2407.02445, 2024.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering, 2024. URL httpss://aran www. abs/2312/22222 14150.
Aman Sinha, Matthew O' O'Kelly, Russ Tedrake, and John C Duchi. Neural bridge sampling for eval- uating safety-critical autonomous systems. Advances in Neural Information Processing Systems, 33:6402-6416 2020.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Llm- planner: Few-shot grounded planning for embodied agents with large language models, 2023.
URL httph:ttt//aata /ttp:://axxvvv..e 222..
Shiva Sreeram, Tsun-Hsuan Wang, Alaa Maalouf, Guy Rosman, Sertac Karaman, and Daniela Rus. Probing multimodal Iims world models for driving, 2024. URL https://arxiv. arg/abs/ 2405.05956.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach. Roberto Martin-Martin, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments, 2021. URL https://arxiv.or ortp.//////......... 03332.
Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12619-12629, 2023a.
Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, Yincheng Yao, Xuan Liu, Huaqin Zhao, Zhengliang Liu, Haixing Dai, Lin Zhao, Bao Ge. Xiang Li, Tianming Liu, and Shu Zhang. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv (cs. ROJ, January 2024a.
13

---

Published as a conference paper at ICLR 2025
Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles, 2023b. URL https://arxiv.. wwwggab/////a// 06549.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang. Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. XrXiv preprint arXiv:2310.01361, 2023c.
Lirui Wang. Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language 01361.
models, 2024b. URL https: wwwpaaawwv..oog///baaaaee
Tianqi Wang, Enze Xie, Ruihang Chu, Zhenguo Li, and Ping Luo. Drivecot: Integrating chain-of- thought reasoning with end-to-end driving. arXiv preprint arXiv:2403. 16996, 2024c.
Tsun-Hsuan Wang, Alexander Amini, Wilko Schwarting, Igor Gilitschenski, Sertae Karaman, and Daniela Rus. Learning interactive driving policies via data-driven simulation. In 2022 Interna- tional Conference on Robotics and Automation (ICRA), pp- 7745-7752. IEEE, 2022.
Tsun-Hsuan Wang, Pingchuan Ma, Andrew Everett Spielberg, Zhou Xian, Hao Zhang, Joshua Tenenbaum, Daniela Rus, and Chuang Gan. Softzoo: A soft robot co-design benchmark for locomotion in diverse environments. arXiv preprint arXiv:2303.0955 2023d.
Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zack- ory Erickson, David Held, and Chuang Gan. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 51936-51983 PMLR, 21-27 Jul 2024d. URL https: httpp://proccedinggs mlr pres////s// wang24cc.
Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang. Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. Advances in Neural Information Processing Systems, 15:25667-25633 2022.
Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16227-16237, 2024.
Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, and Mohit Bansal. Envgen: Generating and adapting environments via Ilms for training embodied agents, 2024. URL https:///axxiiiorrr .rg/ abs/2403. 1204.
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226 2023.
//400 2405. 14062. Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledge sufety-critical scenario gen- eration for autonomous vehicles, 2024. URL
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guided traffic simulation via scene-level diffusion. Conference on Robot Learning, pp. 144-177. PMLR, 2023.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models, 2018. URL http:://axxvv.orrgg./ee 1802. 01 886.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 GRAPH EXPANSION DETAILS
A.1.1 ASSET DATABASE
The simulator database, Dasset = (V, E), is represented as a directed graph, where each node e E V corresponds to simulatable asset, such sensors (e.g. GPS, temperature, humidity), agents (enti- ties with a dynamic model), objects (static entities without dynamic model), properties (e.g.. siren, door, light color), and behaviors of other agents (e.g., driving forward, standing still, walking). Each edge E L' represents relationship between two assets, such as (siren + ambulance), in- dicating that the siren is property of the ambulance. property is node that has least one outgoing edge, while agents and objects only have incoming edges. We provide an example of D Dasset for the CARLA simulator and PyBullet:
Code Example 1: Asset Database for CARLA Simulator (Passet)
node[" v ["bicycle", "sedan", "ambulance"]
node[ttraffic"" [" traffic light"]
node["behavior"] = ["constant
Lanes...
node[" light"] ["rre//gree//yellooooffff
(andeulanan"""" ["siren", "behavior"]
mode["bicycler ["behavior"]
node[ [edda"" ["behavior", "front door"]
node["siren" ["on/off"]
front door"] ["open/closed"]
constant speed"] ["ending location", "starting location", "target speed"]
Here, an entity such as an ambulance, has properties such as ['siren', 'behavior']. In CARLA, th siren property have states-or off-inndicted by /. Meanwhile, the behavior property, such as constant speed, includes an ending location property. This location, for example, 'in front of the ego-vehicle," can be queried from the LLM.
Code Example 2: Asset Database for PyBullet (Passet)
mode["statil objects"] ["desk lamp", "tv", "trash can", "ceramic cup", "book", "toy", "children", "adult", ...] [ode["behavior"] = [ "sstatioary/sstaiioaay"
node[ node[" ("stationary"" ["location"]
In the PyBullet simulator, the behavior is limited to stationary because the environment consists soley non-articulated meshes, which do not support dynamic motion.
A.1.2 NODE PROPOSAL
Event nodes. Candidate nodes can be generated either by an LLM or provided by the user. For instance, in Example given a source node, the LLM generates plausible events, such as 'stopping because there is a jaywalker.' In another example, the prior involves 'police car,' the generated events might include scenarios like 'police chase.' However, not all nodes proposed during the node proposal stage are valid causes. For instance, 'animal on the road' might invalid not simulatable, 'a jaywalker in another city' would be logically implausible. During edge construction, each proposed node validated to ensure represents a plausible cause of the source node.
15

---

Published as a conference paper at ICLR 2025
Example 1: Event nodes (Vevent)
Source node: "The eeo-vel stopped abruptly" ***Example #1***
Prior: null
Candidate event nodes (proposed by LLM): ["a jaywalker walked in front", "animal on the road", "emergency vehicle approaching from behind", "debris ahead"]
x**Example #2***
Prior: "police car"
Candidate event nodes (prosposed by LLM): : ["road block", "police chase", "arrest", ...]
x**Example #3***
Candidate event nodes (proposed by user): [*a tree fell in front", "a jaywalker in another city", "a cyclist changed lanes"]
Entity nodes From the asset database Daset, the candidate nodes include all entities listed D assset starting from the node node[ 'vehicles"], In the implementation, also append additional entities, such as pedestrians and static objects. The appropriate vehicle for the source node will be selected during the edge proposal stage, detailed in Appendix A.1.3.
Example 2: Entity nodes (Ventity)
Source node: "Emergency vehicle approaching from behind" Candidate event nodes: ["bicycle", "ambulance", "sedan"]
Property nodes In Example 1, we verify whether the variable exists in the databaes. For instance, the variable 'siren' found Dasset: allowing us retrieve its values such as 'on' and 'off,' as detailed in Appendix A.1.1. 1. In Example 2, the variable 'location' does not have predefined values (i.e. no 'location' key in node) so we query the LLM to generate all possible locations. However, not all generated locations may be valid. The appropriate nodes are selected during the edge proposal stage.
Example 3: Property nodes (V property)
Event-to-event graph: "ego-vehicle stopped abruptly <- emergency vehicle
approaching from behind"
Entity-to-event graph: "emergency vehicle approaching from behind ambulance"
****xample 1***
Source node: "siren"
Candidate event nodes (from asset database): ["on", "off"]
***Example 2***
Source node: "start location"
Candidate event nodes (proposed by LLM): ["behind the ego-vehicle adjacent lane", "behind the ego-vehicle on same lane", "in front ego-vehicle on adjacent lane", "in front of ego-vehicle on same lane"]
A.1.3 EDGE CONSTRUCTION
Given a list of candidate nodes (from Appendix A.1.2), the edge construction process is used to select plausible nodes and eliminate unlikely ones. For event-to-event edges, a node such as "a jaywalker in another city" can be excluded because the causal relationship "ego-vehicle stopping abruptly + a jaywalker in another city" is implausible. For entity-to-event edges, only nodes such as "ambulance" are selected from the available entities, they are relevant to the source node "emergency vehicle approaching from behind." cases where the source node "tree falls in front." and corresponding "tree" entity exists in the simulator, no edges created. Finally, for property-to-entiy edges. relevant simulatable properties are selected, such as "siren" for an ambulance.
16

---

Published as a conference paper at ICLR 2025
Example 4: Event-to-event (Eevent)
Source node: "ego-vehicle stopped abruptly" Candidate nodes: [*a jaywalker walked in front", "animal the road". "emergency vehicle approaching from behind", "debris in the road", fell front", jaywalker another city", cyclist changed lanes"]
x*Chosen nodes**: ["a jaywalker walked in front", "animal on the road", "emergency vehicle approaching from behind", "debris in the road", "a tree fell front", cyclist changed lanes"]
Nodes not chosen**; ["a jaywalker in another city"] Generated graphs**:
1. ego-vehicle stopped abruptly <- jaywalker walked in front ego-vehicle stopped abruptly <- animal the road
ego-vehicle stopped abruptly <- emergency vehicle approaching from behind
4. ego-vehicle stopped abruptly <- debris in the road
5. ego-vehicle stopped abruptly <- tree fell in front
ego-vehicle stopped abruptly <- cyclist changed lanes
Example 5: Entity-to-event (Centity)
***Example ****
Source node: "Emergency vehicle approaching from behind"
Candidate event nodes: ["bicycle", "ambulance", "sedan")
*Chosen nodes*: ["ambulance"]
*Nodes not chosen*: ["bicycle", "sedan"]
*Generated graphs*:
emergency vehicle approaching from behind <- ambulance
***Example 2***
Source node: "Tree fell in front"
*Chosen nodes*: []]
*Nodes not chosen*: ["bicycle", "ambulance" "sedan"]
Return that this event cannot simulated in CARLA
Example 6: Property-to-entity (Eproperty)
Event-to-event graph: "ego-vehicle stopped abruptly <- emergency vehicle
approaching from behind"
Entity-to-event graph: "emergency vehicle approaching from behind ambulance"
***Example 1***
Source node: "siren"
Candidate event nodes (from asset database): ["on", "off"]
*Chosen nodes*: ["on"]
*Nodes not chosen*: ["off"]
*Generated graphs*:
ambulance siren
***Example 2***
Source node: "start location"
Candidate event nodes (proposed by LLM): ["behind the ego-vehicle adjacent lane", "behind the ego-vehicle on same lane", "in front ego-vehicle on adjacent lane", "in front of ego-vehicle on same lane"]
*Chosen nodes*:
["behind the ego-vehicle adjacent lane"]
*Nodes not chosen*: ["behind the ego-vehicle on same lane", front of ego-vehicle on adjacent lane", "in front of ego-vehicle on same lane"] *Generated graphs*:
1. behind the ego-vehicle same lane <- starting location <-- ambulance
17

---

Published as a conference paper at ICLR 2025
A.2 GROUNDING TO SIMULATION DETAILS
The graph expansion process produces graph that defines enviromment and describes the nario. This graph serves as input to the LLM to generate the Low-Level State Translator (LLST), which bridges abstract reasoning with physical state transitions in order track states. This tracking crucial for defining constraints that align with the intended scenario we aim to simulate. For ex- ample, the abstract state "Ambulance Approaching" defines constraint that requires the ambulance to be behind the ego-vehicle and in motion.
Code Example 3: Generated Graph
causal_grapp [ A'mulaance approaching from behind', stopped on left lane']
'Ego-vehicle abruptly
entities [[' [C'name": 'ambulancel', 'type': 'agent', "entity_nane "ambulance', 'behavioral_ prooorrippppeeeiess { ('action': 'Vehicle drives straight entire time' "starting location': 'behind the ego vehicle in the right lane', 'ending location': "in front of vehicle in the right [), ('name": 'ego-vehicle', "type': 'agent', 'entity_name': 'ego-vehicle', cataavioral_ropertti ('action': "Vehicle drives straight and suddenly stops }}}
Code Example 4: Low-Level State Translator
def
agent_name) -> None:
if agent__name = "anbblaaceel":
# State: Ambulance Approaching
if behind_ wehhind_ehiclleeggent.nno "ego-vehicle") and cuurreenlyy.oovinneeeeeen....nne wwwppddatesttteeAalbaaee
Approaching", agent_name, True)
# State: Ambulance Close to Ego
"ego-vehicle")
ss_currently_movinngeeee (ag_etitnnnggg_t_aamm)))
s"tbuteteaullaccceeeeeee self urfa___uddee_tttttee staate" Ambulance Clase to Ego", agent_name, True)
These abstract states are subsequently used to construct finite state machine (FSM), incorporating transitions that capture the temporal dynamics of the scenario and encode temporal logic. For ex- ample, in this scenario, the abstract state "Ambulance Approaching" must occur and must precede the state "Ambulance Passing Ego."
Code Example 5: Finite State Machine
fsm [[(' ambulance1',
'Ambulance Approaching'), ('ego-vehicle', 'Ego
Driving Steady'))
[(' (('ambulancel' 'Ambulance Close to Ego') )},
[[ [''ego-vehicle', "Ego Braking )},
[(" [''ego-vehicle' "Ego Stopped Abruptly )}, [(' ('aambulace1', 'Ambulance Passing Ego >}]
Given a FSM, we use Google's CP-SAT solver to find solutions for the variables such as the x, coordinates of the start and end positions (No, Vo. Fr, Ur), well the speed, such that it satis- fies the constraints imposed by the FSM. For instance, the behavior of the ambulance, defined as "drive straight," is generated Dr DriveStraights ambulance'. To, yo. Nr, yr, speed). The simulation considered valid only if it terminates in a state that satisfies the terminal condition of the FSM.
start": ("x": -25, "y": 4), "end": ("x": 80, "y": 4), "speed": 40,
Full Config Example. We provide the full example of the generated scenario config file below:
18

---

Published as a conference paper at ICLR 2025
Code Example 6: Scenario Config Example
narrative
"An ambulance approached from behind, promoting the vehicle to stop abruptly, allowing the ambulance pass safely."
entities [('name": 'anbulancel', 'type': "agent', "entity_name': "ambul aabbulance,, catpavioral_prppettie { ('action': "Vehicle drives straight the entire time', "starting location': 'behind the ego vehicle in the right lane', "ending location': "in front of ego vehicle in the right lane')), ('name': 'ego-vehicle', 'type': 'agent', 'entity_name': 'ego-vehicle', 'properties': O2 Meehaioorll_propeee properties'' ('action': 'Vehicle drives straight and suddenly stops' }}}
vehicles [["name": "ambulance1",,
"start": ("x": -25, "y": 4), "end": ("x": 80, "y": 4), "spped_raanel" [40, 40],
"blueprint_id": weriice..oord.ce anbulaance"".
"driving_policy": "drive forward",
"type": "dynamic",
"heading": 0)]
causal_graph d'Ambulance approaching from behind', 'Ego-vehicle abruptly stopped on left lane']
fsm = [[( ambaaaance''' "Ambulance Approaching'). ('ego-vehicle', 'Ego Driving
Steady )},
[(' ambulance1', 'Ambulance Close to Ego')],
[(' [''ego-vehicle'
"Ego Braking )
[('ego-vehicle'
"Ego Stopped Abruptly') ))]
[(" (('ambulancel', "Ambulance Passing Ego )]]
class (eteaaannge(((tttttMMaagggan
def __init__seef,, obj_name, world):
super (oo__._itt__o_________________ get_object_states// world)
def teeeet__ttttttrrcckke agent_name "ambulance1":
agent_name) -> None:
State: if Ambulance Approaching cww.urrennly.movinnggeeennnnnnne self welf._update_steee s"ateetnmuulaccc Approaching", agent_name, True)
"ego-vehicle") and
State: Ambulance Close to Ego
if are_close barsse_byyggeee baggeett_name, "ego-vehicle") and is suurrently_movinnggeeeeee
self self._uudate__tattee saataesssttteeeeacccccacaa Close Ego", agent_name, True)
State: Ambulance Passing Ego
if fight_in__ronttegget is cwurrrently.moovinnageentttt
"ego-vehicle") and sel stt.uudddtte_ttteaaaaallaae
Passing Ego", agent_name, True)
elif agent_name = "ego-vehicle":
State: Ego Driving Steady
if serriving_steayyaggent
self..update_statce
Driving Steady", agent_name, True)
State: Ego Braking if is ss_braakingaaget_nnneee selff_.uddate_statee"" Braking", agent_name, True)
State: Ego Stopped Abruptly
if is self selff__udate__sttte Stopped Abruptly", agent_name, True)
return None
19

---

Published as a conference paper at ICLR 2025
A.3 ABLATION
Accuracy of edge creation. In Table 5, we present an ablation study evaluating the accuracy of edge creation during the graph expansion stage. For event-to-event edges, we assess whether the LLM can correctly identify causal and non-causal variables given the input behavior, testing only depth of 1. For entity-to-event edges, we evaluate whether the LLM can accurately identify simulatable events and select corresponding assets from the database. The results shown in Table consistently demonstrate high performance for wenttto-event and entity-to-even edges across both domains, with accuracy exceeding 0.90. The consistency in LLM responses across trials can be attributed to the temperature and top-p hyperparameters being set to O. Failure cases for entity- to-event stem from the LLM's misunderstanding of the simulator's capabilities,
described in Dasset For example, given a graph such as "ego-vehicle stopping +- flood to knees," the LLM might propose changing the weather (treated misaligns with the constraints of the simulation engine. Finally, for property-to-entee evaluate the LLM'' ability select the most plausible location. While the LLM performs well simpler properties, such as determining whether a siren should be on, its accuracy decreases for properties requiring complex spatiotemporal reasoning. In our driving experiment, the LLM was tasked with selecting end locations for two entities across two scenarios, with each entity having possible locations. We observed significantly higher variance in its responses.
entity) to "flooding." While this is valid in general, it
edges, we
Accuracy
Precision
Recall
FI Score
Driving
0.98 + 0.04
1.00 + 0.00
0.97 + 0.06
0.98 + 0.03
Event-to-Event
Manipulation
0.98 + 0.04
1.00 + 0.00
-.97=0000
0.98 + 0.02
Driving
0.91 0.02
0.86 + 0.02
0.98 0.03
0.92 0.01
Entity-to-Event
Manipulation
0.94 + 0.02
1.00 + 0.00
0.89 + 0.03
0.94 + 0.02
Driving
0.81 + 0.09
0.73 + 0.11
0.90 0.03
0.80 0.08
Property-to-Event
Manipulation
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
Table 5: Accuracy of Edge Creation
100% changes
changes
7-
40% changes
causal changes
Density
0.2
0.4
0.6
0.8
Scenario Description Diversity (Pairwise)
Figure 6: Controllability
Eliciting Diversity Controllability. Figure 6 shows the pairwise diversity distribution of gener- ated scenarios, measured as the proportion of scenarios with unique event-to-even nodes, excluding the input behavior node, value of 100% indicates that all compared scenarios have distinct causes, while 0% means they share the same cause but vary in properties, such as their start location or behavior. We observe that scenario diversity increases as the causes vary. Notably, the bimodal distribution suggests that introducing new causal graph introduces greater diversity by exploring broader cause-and-effed relationships, while changes to the property graph result in more subtle variations. By explicitly introducing new causal variables, we guide the model to explore more diverse range plausible outcomes, uncovering interactions that are otherwise not immediately apparent, but are entirely plausible.
20

---

Published as a conference paper at ICLR 2025
Re-usability. Beyond generating diverse simulations, method allows for the efficient reuse of existing behaviors. In Table 2, we show that the diversity manipulation tasks generated by method surpasses all baselines, RoboGen tasks. Although RoboGen supports a broad range of tasks, our method further increases task diversity by 18.50%. Previous methods encountered bottleneck in simulating actions from text due to the need to design a unique reward function for every task (Wang et al., 2024d; Ma et al., 2024; Nguyen et al., 2024). Our method addresses this by changing the context reflect different higher-level goals, enabling the creation of new tasks while reusing the same task-specific reward function. For instance, the task "opening door" can be augmented convey new narratives such as "ventilating room," "letting the guest out," or "letting the pet in." This contextual dimension is a unique capability of ours that addresses the limitations of simulating actions from text.
with the use of only 10 reward functions- subset of
21

---

Published as a conference paper at ICLR 2025
A.4
CASE STUDY
A.4.1 GENERATED SCENARIOS
Driving. Our method provides broader coverage of driving scenario categories than existing ap- proaches, as shown Figure 7. While methods like ChatScene primarily focuses on safety-critica scenarios and DriveLM covers general driving tasks, our approach generates diverse range of sce- narios that encompass both safety-critical and general driving situations, such as "yielding to an emergency vehicle" or "picking up passenger."
100
Categeries
Percentage of Scenarios (9S)
23.0%
Colliiio Ausiidana
Construction Zone
80
Following Trffiic
Following Itafiic Signals
23.9%
17:29
Safety
Picking Up Pascenger
40
Congastion
Emergency Vitice
$1.70
13.2%
8.15 DriveLM Datasets
ChatScone
DriveCoT
Ours
Figure 7: Categories of Scenarios (driving)
A.4.2 FAILURE MODES
Most failures in simulating the scenarios are from overly strict FSM constraints that required mul- tiple conditions to be satisfied simultaneously. For example, in the scenario "an object falling off truck causing the ego-vehicle to stop," successfuly found solutions where the box falls to the ground and the ego-vehicle brakes. However, the scenario became infeasible because the FSM also required the delivery truck to exit the scene the exact moment the ego-vehicle stopped. While this requirement is not inherently incorrect, relaxing the constraint- -atrraint-allooingg the delivery truck's ss exit to occur after the ego-vehicle stopped-wond make the scenario feasible.
Code Example 7: Failure case (driving)
fsm
[[('delivery_tree) truck', 'Approaching Intersection'), ('box', 'On Truck'), ('ego_vehicle" 'Driving Steady') (('delivery_truct 'In Intersection'), ('box', 'Falling')) [('box', 'On Ground"), ( rego_vehicle', "Braking')] ))], [''delivery_truck' "Exiting Intersection'). ('égo_vehicle', 'Stopped" )]]
Scenarios involving the ambulance posed additional challenges due to the constraints imposed by its vehicle dynamics. Specifically, the ambulance's slower acceleration made it difficult to overtake another vehicle within a short distance.
In the manipulation domain, most failures occurred in scenarios requiring objects to be placed inside articulated objects. For example, all scenarios involving the behavior "the robot opens the table door" failed because objects could not be placed inside the table closet. Unlike in the driving domain, manipulation scenarios did not involve dynamic actors. Consequently the FSM constraints in manipulation were less complex, avoiding the issues encountered in driving.
However, the manipulation domain presents unique challenges and limitations to manipulation. For instance, scenarios scenarios such as "rotate lamp head to remove screen glare" require determin- ing object orientations-an open problem due to the arbitrary canonical orientations of meshes in simulation. Similarly, simulating other modalities like temperature sound not currently feasi- ble in PyBullet. For example, tasks such as "thawing frozen vegetables" cannot simulate changes in temperature. While our method correctly identifies relevant objects retrieve, such as "frozen vegetables." these limitations underscore gaps in current simulation capabilities.
22

---

Published as a conference paper at ICLR 2025
Input Behavior
Scenario Name
Feasihle/Infeasille of FSM
Traflic light malfunctin
Feusible
1.2) Distracted driver late start
Feasible
Driving forward from being stationary
13) Intersection congestion
Feasible
1.4) Pedestrian jaywalking
Feasible
1.5) Let emergency vehicle puss at intersection
Feusible
2.1) Traffic congestion due lane closure
Feasible
2.2) Letting emergency vehicle pass
Infeusible
Slowing down
2.3) A large truck ahead stopped abruptl
Feusible
2.4) A vehicle cutting in
Feusible
2.5) Speed enforcement
Feusible
3.1) Road assistance
Fensihle
3.2) Elder walking the street
Feusible
Stop abruptly while driving forward
3.3) Accident ahead
Feasible
ambulance
Feasible
3.5) Parked car door open
Feasible
4.1) Protest on the streets
Infeasible
4.2) Parked car intersection corner
Feasible
Stop abruptly after taking
4.3) Police checkpoint
Feasible
4.4) Letting the ambulance
Infeasible
4.5) Picking up passenger
Feasible
5.1) Ambulance entering intersection
Feasible
5.2) Sudden traffic signal change
Feasible
Stop abruptly while crossing intersection
5.3) Another vehicle running red light
Feusible
5.4) Object falling out of truck
Infeasible
5.5) Police chase
Feasible
6.1) Debris in front
Feasible
6.2) Slow traffic
Infeasible
Changing lanes while driving forward
6.3) Yielding for emergency vehiele
Infeasible
6.4) Lane closure
Feasible
6.5) Driver going in the wrong direction
Infeasible
Table 6: Breakdown of scenario feasibility for driving
23

---

Published as a conference paper at ICLR 2025
Input Behavior
Scenario Name
Feusible/Inlcasbble of FSM
Block unplcasani
Feasible
1.2) Minimize outside noise to watch movie
Feasible
1.3) Control ambient lighting
Feasible
The robor closes window
1.4) Block outside construction site noise
Reasible
1.5) Avoid prying eyes
Feasible
contidentality during private conversation
Infeasible
1.7) Block direct sunlight profect indoor plants
Feasible
2.1) Improve ventiliation
Feasible
2.2) Check delivered package
Infeasible
The rohot opens the door
2.3) Bid farewell and let the guest out
Feasible
Allow
Feasible
3.1) Bake cake
Feasible
The rhhor adjusts the temperature
3.2) Roast vegetables
Preheating oven
Infensible
Feasible
3.4) Warning a ceramic cup
Infeasible
1) Cleon dirty dishes
Feasible
4.2) Watering plants
Feasible
4.3) Filling water bottle
Reasible
Thawing frozen vegetables
Feasible
The robot adjusts the water flow
4.5) Washing mixed fruits in bowl
Reasible
4.6) Soaking sponge
Feusible
4.7) Wash broccoli sink
Feasible
4.8) Cleaning coffee mug
Reasible
Looking for
Infeasible
5.2) Showing table content to somcone
Infeasible
The rohot open the table doors
5.3) Retrieve
sight
Infeasible
5.4) Look for hiding pet
Infeasible
6. Clearing the flocr after playtime
Feusible
The storing an storage
away toys gaast
Feasible
6.3) Removing choking hazard near pets
Feasible
6.4) Store detergent out of children's reach
Foasible
7.1) Retrieve ingredients for cooking
Infeasible
The zobot is retrieving an item from fridge
7.2) Clearing expired items
Infeasible
Offering food for visitor
Feasible
8.1) Brighten the desk read book
Feasible
The rohor is turning lamp The display screen
8.3) Setting up small photography shoot avoid shadows
8.2) Showcas: an artwork
Feasible
Feasible
8.4) Sctup working environment
Feasible
9.1) Remove glare from display
Feusible
9.2) Sharing presentation during meeting
Feasible
10.1) Scarching for an item
Infeasible
The pull
Looking documents
Infeasible
10.3) Finding pencil
Infeasible
11.1) Unpacking boxes
Feasible
The rohor is retrieving an item from hox
11.2) Chocking content of box
Infeasible
11.3) Inspection warehouse
Feasible
Retrieving
person
Infeasible
12.1) Transporing soil for gundening
Infeusible
The robot is carrynng bucket
12.2) Fetching water for mopping the floor
Feasible
12.3) Grabbing backet fill sand
Feasible
Table 7: Breakdown of scenario feasibility for manipulation
24

---

Published as a conference paper at ICLR 2025
A.5
FULL PROMPTS
A.5.1 NODE PROPOSAL
Prompt:
Event Node Proposal Prompt
Input variables: causal_graph
Prompt:
You are expert in driving scenarios.
In this scenario there is (causal_graph).
Please provide a list of all the plausible scenarios that caused Make sure it is precise. Provide the final answer as comprehensive list of plausible scenarios in the following format within the tags <Answer>... </Answer>:
cause_name: cause description
Answer: Let's think step by step.
Example 7: Event Node Proposal LLM Response
**Input**:
causal_graph ("ego-vehicle stopping abruptly"]
**Output**:
Sure, lets think through plausible scenarios step by step.
<Answer>
**Peestrrian crossing**; pedestrian unexpectedly crosses in front of the ego-vehicle.
**Vhiill cutting inex: Another vehicle unexpectedly cuts in front of the ego-vehicle.
**Mechanical failure**: A sudden internal failure such as brake malfunction leads to abrupt stopping.
**Emergency vehicle**; An emergency vehicle suddenly appears signals the ego-vehicle to yield.
*Cyclist lane violationt*:
cyclist suddenly moves into the ego-vehicles
path, </Answer>
Prompt: Property Node Proposal Prompt
Input variables: node_name, mndiidate_values
Prompt:
expert in driving scenarios.
In this scenario there is [eeusal_graph) The entities in the scenario are: [entities_name)
Please provide a list
the possible ms for the entities the scenario, excluding the ego-vehicle. Make sure is precise. Provide the final answer as a comprehensive list of possible [node_name]s in the following format within the tags
entity_name: [' (node_name))1, (node_name)2* ...]
Answer:
Let's think step by step.
25

---

Published as a conference paper at ICLR 2025
Example 8: Property Node Proposal LLM Response
**Input**:
node_name "starting location"
causal_graph ["emergency vehicle", "ego-vehicle stopping abruptly") entities [( name": anbulance1', 'type': 'agent, dentity_name": 'ambulance', "behavioralL.ropertiess ()),
('name": 'ego-vehicle',
'type": 'agent', 'entity_name": 'ego-vehicle', @ehavioral__ropertiie ('action': 'Vehicle drives straight and suddenly stops' }}]
**Output**;
Sure, lets think through plausible locations step by step.
<Answer>
ambulancel: ['in the right lane behind the ego-vehicle', "in the right lane in front of the ego-vehicle', 'in the left lane behind the ego-vehicle', 'in e
the left 1ane in front of the (rrehhccee<<<Ammmm ]</Answer>
A.5.2 EDGE CREATION
Prompt: Property Node Proposal Prompt
Input variables: node_name, candidate_valuee entities
Prompt:
You are expert in driving scenarios.
possible (node_name]] for each entity are: Seanddddateeallesseeettt""""""""
for entity in entities}
What are all the possible (node_name)s for each entity in the scenario? To answer this, first, please summariz ize the details of each entities in the scenario. Then, check each see it is known. For each, start by stating everything that is known about all the entities, then check if it is plausible given what is known, finally give your conclusion. Think step by step. You must not assume additional actions beyond what is explicitly described the behavior. You must also assume that the actions are executed fully. Your evaluation needs to be the following format:
possible outcome given what
**Name of )*ome_name]**
Known:
Analysis: (think step by step)
Contradictions to what is known: (think step by step)
Conclusion:
Finally, provide final answer as a list locations following
formats within the tags "[node_name)2" entity_name: [' ["(node_name)1',
Here are some tips help you answer the question:
You may assume that the vehicles can break traffic rules as long as it is
plausible in real life (realistic). However, the vehicles action must not violate the behavior described.
The [node_name]s selected can only from the list of possible {node_naid [nmee_name]s provided.
Answer:
Let's think step by step.
26

---

Published as a conference paper at ICLR 2025
Example 9: Property Node Proposal LLM Response
**Input**:
node_name "starting location"
causal_graph ["emergency vehicle", "ego-vehicle stopping abruptly left lane"]
entities
[( name": 'aabulance1',
"type' 'agent',
"entity_name': 'ambulance', 'tehavioral_properries: (
('name": 'ego-vehicl',
'type": 'agent', "entity_name", 'ego-vehicle', "behavioral__ropertiee
('action': 'Vehicle drives straight and
suddenly stops'))]
candidate values { ("ambulance1": ['in the right 1ane behind the ego-vehicle', in the right lane in front of the ego-vehicle', 'in the left lane behind the ego-vehicle' 'in the left lane in front of the ego-vehicle']]
**Output**:
Sure, lets think through plausible locations step by step.
<Answer>
ambulance1: ['in the right behind the ego-vehicle A//sserr>>>
27


Reviews:
[{"review_id": "VAZkdwKfkR", "reviewer": "Reviewer_q9yg", "summary": "This paper introduces ReGen, a generative simulation framework that automates the process of constructing simulations using inverse design.  Given an agent’s behavior (such as a motion trajectory or objective function) and its textual description, ReGen infers the underlying scenarios and environments that could have caused the behavior.  This approach leverages large language models (LLMs) to construct and expand a graph that captures cause-and-effect relationships and relevant entities with properties in the environment, which is then processed to configure a robot simulation environment.  ReGen is demonstrated in autonomous driving and robot manipulation tasks, generating more diverse and complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. ReGen presents an inverse design approach for generative simulation, which allows for the creation of diverse and complex simulated environments based on agent behavior and textual descriptions.\n2. ReGen generates more diverse and complex simulated environments compared to existing simulations, as demonstrated in autonomous driving and robot manipulation tasks.\n3. ReGen enables controllable generation for corner cases, which is important for safety-critical applications like autonomous driving.", "weaknesses": "1. The evaluation of ReGen is primarily focused on diversity and complexity, with less emphasis on the realism and physical accuracy of the generated simulations.\n2. ReGen heavily relies on LLMs, which can be computationally expensive and may not always generate semantically accurate or physically plausible scenarios due to LLMs.\n3. The applicability of ReGen to other robotics domains beyond autonomous driving and robot manipulation is not extensively explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OO6uIiCA8R", "reviewer": "Reviewer_HrM2", "summary": "This paper presents ReGen, which, given an existing agent's behavior and motion trajectory, generates simulated environments counterfacturally that explains the possible causes and preconditions of this agent trajectory via LLM and VLM. In this way, authors are able to augment simulation data that allows the training of more robust models and policies. The authors then compare their approach to previous baselines in self driving (Carla simulator) and in robot manipulation (PyBullet), demonstrating that their approach effectively generates more diverse simulation environments.", "questions": "Please address all weaknesses listed above.\n\nMinor:\n- Fig 4 caption: \"desipte\" -> \"despite\"\n- \"table {x}\" should be \"Table x\"; \"figure {x}\" should be \"Figure {x}\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Diversifying and augmenting existing simulated environments and trajectories through counterfactural generation is an important research problem. It provides a promosing way to improve the robustness and safety of agents and policies under corner cases and unexpected scenarios.", "weaknesses": "The paper suffers from significant weaknesses in writing clarity and the depth of analysis, specifically:\n- The methodology section 2 does not detail the models, the model versions, and the important hyperparameters used for prompting LLM / VLM for counterfactual generation. Some of the details are deferred to page 9, which should have been in much earlier in the paper.\n- No prompt examples for LLM / VLM are provided in the appendix, making the method unreproducible. No concrete, detailed examples for the entire process of constructing Carla and PyBullet environments through counterfactual generation (following algorithm 1) are provided. \n- Algorithm 1 will very likely result in infinite loop, as there will likely always be at least 1 node with input degree < 1 (unless there are circular cause and effect dependencies).\n- L320 - ReGen is able to generate simulated environments by augmenting visual observations like GPS measurement jamming. However there is no detailed description of how the visual observations are augmented. It seems that some tool use ability of LLM is invoked.\n- The experiments only demonstrate that ReGen outperforms baselines, but **why** ReGen outperforms the baseline is unclear. There are no ablations in e.g., prompting to answer this question. For example, why does ReGen outperform ChatScene? Is it due to better prompting and / or the better VLM used (note that no prompts are provided throughout the paper)? Is the comparison with baseline fair, with the same specific versions of GPT4o / GPT4 / Claude 3.5?\n\nEdit: Thanks authors for the rebuttal! The revision has significantly improved the quality of the paper and I'm increasing my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UF7rOdgNIW", "reviewer": "Reviewer_x8VP", "summary": "This paper introduces ReGen, a generative simulation framework that automates the creation of robot simulation environments using inverse design. ReGen takes a robot's behavior and textual descriptions as input, then uses large language models to construct and expand causal graphs that capture relationships between events, entities, and their properties, which are then converted into executable simulation environments. The framework is implemented and validated in both autonomous driving and robot manipulation tasks, demonstrating capabilities in augmenting simulations based on ego-agent behaviors, generating counterfactual scenarios, reasoning about agent cognition, and handling different sensing modalities. The experimental results show that ReGen generates more diverse environments compared to existing simulations, effectively creates corner cases for safety-critical applications, and produces more challenging vision-language-action datasets for vision language models.", "questions": "1. Regarding the LLM implementation: Could you provide more details about the specific prompting strategies used for graph expansion? How do you ensure consistency and reliability in the LLM output, and how do you handle cases where the LLM generates invalid or inconsistent relationships? \n2. Regarding the failures: The paper reports success rates of 80% for driving and 78% for manipulation. Could you provide a detailed analysis of the failure modes? \n3. About simulation constraints: How do you systematically identify and handle cases where desired scenarios exceed the capabilities of the simulation engine? Is there a formal process for determining which aspects of a scenario can be reasonably approximated and which must be excluded?\n4. About generated task completeness. How do you guarantee that the generated tasks are feasible for a robot or autonomous agent? i.e., that there is a practical solution to the generated tasks.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "While I do not see a clear strength in terms of novelty. However, the authors provide a comprehensive evaluation against multiple baselines, and thorough implementation across autonomous driving and manipulation tasks to demonstrate the applicability of the proposed method.", "weaknesses": "The main contribution of this paper is vague. The use of LLMs to generate simulated task environments for robotic tasks cannot be considered as a major contribution. The proposed method of using LLMs for graph expansion and simulation generation appears to be a straightforward extension of existing work such as [a, b]. I do not see a convincing argument that states the technical contributions of the proposed approach other than some prompt engineering. The \"inverse design\" is essentially a rebranding of standard goal-conditional generation approaches, where the goal is guided by the inverse design and feed into the LLMs. \n\nThe evaluation of the proposed method is not thorough enough, with only brief mentions of success rates without analysis of failure modes, no comparison with human-designed scenarios, and unclear metrics for scenario complexity beyond embedding diversity. Furthermore, the paper inadequately addresses fundamental simulation constraints, as many proposed scenarios can't be fully simulated due to engine limitations, but there is no systematic approach for dealing with these limitations or assessing their impact on practical utility. \n\n[a] Wang, Yufei, et al. \"Robogen: Towards unleashing infinite data for automated robot learning via generative simulation.\" arXiv preprint arXiv:2311.01455 (2023). \n[b] Zhang, Jiawei, Chejian Xu, and Bo Li. \"ChatScene: Knowledge-based safety-critical scenario generation for autonomous vehicles.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n------\n\nAdditional comments:\n\nGiven that direct responses are no longer possible on the original [thread](https://openreview.net/forum?id=EbCUbPZjM1&noteId=yeUwITDALH), I am sharing my reply here.\n\nTraining policies in complex environments demands significantly greater computational resources and effort (e.g., see [c]). The fact that your policy was obtained in just 2 minutes suggests the task and environment were overly simplified comparing to actual scenarios. I have concerns about whether your proposed method would remain effective when scaled to more complex tasks and environments.\n\nRegarding your comments about the primary contribution, I remain unconvinced. The work does not demonstrate any novel or innovative applications of LLMs. Therefore, I will keep my original rating.\n\n[c] Gu, Jiayuan, et al. \"Multi-skill mobile manipulation for object rearrangement.\" arXiv preprint arXiv:2209.02778 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7mQmN9e0W", "reviewer": "Reviewer_hjAh", "summary": "This paper presents ReGen, a generative simulation framework that automates the creation of robot simulation environments through inverse design. The approach leverages large language models (LLMs) to construct and expand graphs capturing cause-and-effect relationships and environmental properties, which are then converted into executable simulation environments.", "questions": "1. In line 161, what description is used when inferring the property node from the source node?\n2. In line 155, is there an example that clarifies what the prior represents exactly?\n3. This paper demonstrates a plausible generation ability; however, it lacks analysis of the entire generated graph. Could you provide the complete graph generated by ReGen?\n4. Is there an issue with the brackets in line 182? It seems there are multie understandings here. A more precise notation or explanation is needed.\n5. When expanding the event-to-event nodes, if the LLM generates events that the simulator cannot execute, how does the system handle this?\n6. ReGen exhibits strong capabilities in generating diverse scenarios, yet realism is not thoroughly addressed in the large-scale experiments. Could you elaborate on the realism of the generated scenarios? If the generated scenarios are not reasonable, off-the-shelf VLM understanding may perform poorly, as observed in Table 3.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Appealing core functionality: leveraging inverse design with large language models to generate realistic and diverse simulation environments by capturing cause-and-effect relationships and relevant environmental properties. \n- Great experiments to show the strong capability to generate diverse and corner-case of the scenarios.", "weaknesses": "1. The paper's novelty analysis is insufficient:\n- The causality reasoning and diversity aspects appear to be inherent properties of the LLMs rather than unique contributions of the framework\n- The generation capabilities seem heavily dependent on the underlying simulator's features\n\n2. There is no analysis of the generated graph, only the results are shown, making it unclear whether the main contributions come from the LLM rather than the proposed graph structure. Additionally, there is no ablation study on the graph; for instance, it is unclear what would happen without edge construction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)