Title: FactorSim: Generative Simulation via Factorized Representation
Venue: NeurIPS 2024
Accepted: yes
Keywords: generative simulation

POMDP

Large Language Models
Abstract: Generating simulations to train intelligent agents in game-playing and robotics from natural language input, user input, or task documentation remains an open-ended challenge. Existing approaches focus on parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents. Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce a generative simulation benchmark that assesses the generated simulation code’s accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment (i.e., accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.

FACTORSIM: Generative Simulation via Factorized Representation

Abstract

Generating simulations to train intelligent agents in game-playing and robotics from natural language input, from user input or task documentation, remains an open-ended challenge. Existing approaches focus on parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents. Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce a generative simulation benchmark that assesses the generated simulation code's accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment ( i.e. , accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.

1 Introduction

Simulations hold significant potential for training agents to perform real-world tasks where data collection is costly, dangerous, or infringes on individual privacy. A major bottleneck in harnessing the potential of simulations at scale for agent training is the cost of designing and developing them, especially when we need a distribution of simulations that meet detailed design specifications to train more generalized policies. In this paper, we aim to generate coded simulations given text specifications. Code provides a natural interface for users to inspect, modify, and debug the simulation. It also allows us to craft diverse environments for Reinforcement Learning (RL) purposes.

Generating full simulations in code to train agents from a text prompt is an under-explored challenge. Previous works focus on parts of this challenge, including reward function design [22], hyperparameter tuning [24], and task configuration while relying on an existing simulator [35]. These methods use large language models (LLMs) to generate the components of simulations specified as code. However, when faced with large and detailed contexts, LLMs often generate simulations that ignore or fail to adhere to parts of the input prompt [21]. This issue is not solely due to the limitations of

FactorSim

Step 1: ....

Step 2: introduce red dot...

. . .

Step N: ....

Chain of Thought

Documentation

agent

position

red dot

position

Implement Factors

// Transition functions

collision_logic()

// Rendering functions

render_red_dot()

model

handle

key press

collision

logic

render

red dot

Factored POMDP

controller

view

While main game loop:

// Input handling functions

handle_key_press()

Policy Training

reward on unseen testing

environment over training

Generated simulations

Soore: 15

Game Over!

Press R to Restart

WaterWorld

In Wtertird the aent, abe cird, nuet navipte around d woril capuring grencirces

xtile aoiding red one.

al dhe green-cirdles have been cpturet.

Atar capture a dircke itwill respuwn i

Valid Actions

osertine.

Terminal states (game_over)

The game ends when al fe groen cirles huve

usit ixp Ag piangtir 4

Rewards

(asses a neprtive rwan of-1

existing LLMs but also suggests that some form of decomposition is always critical as we scale up the number of components in simulations. We ask the question: can we exploit the inherent structure (e.g., having a game loop that handles agent actions, updates internal game states accordingly, and displays the game states to the users through a rendering process) of coded simulations to generate them better?

We propose FACTORSIM, a framework that takes an arbitrary language specification as input and outputs a full simulation that can be used to train RL agents. The key idea of FACTORSIM is to decompose the input prompt into a series of steps and then use a factored Partially Observable Markov Decision Process (POMDP) representation to reduce the context needed for each generation step. To realize FACTORSIM, we use the model-view-controller software design pattern to structure the generation process. Consider generating a coded simulation of WaterWorld; see Figure 1. The game consists of an agent (blue circle) traveling in a 2d world, capturing food (green circle) while avoiding enemies (red circle). Our method first decomposes the game description into multiple steps to be implemented. For example, a step instruction could be 'Introduce red dot enemies that can be controlled with arrow keys. Give the player a -1 reward when the agent collides with an enemy'. We first select the context needed for this functionality to be implemented, e.g. , positions of existing agents. Subsequently, FACTORSIM generates (at most) three functions: one to handle player input (i.e., handle_key_press , the controller component), one to implement the collision logic (i.e., collision_logic , the model component), and one to update the rendering function (i.e., render_red_dot , the view component). Limiting the context during each step of the simulation generation process allows FACTORSIM to focus on the task at hand while avoiding hallucinating non-existent functions or modifying code not meant to be changed.

To evaluate the task of full simulation generation, we propose a new Generative Simulation 4 benchmark with accompanying success metrics. One set of success metrics is the pass rate in automated system tests. Commonly used in game development, these system tests programmatically assess whether the behavior of the generated simulation adheres to the specifications given in the input prompt. The second success metric assesses the value of the generated simulations for transfer learning in an RL setting. This evaluates how well agents trained on a set of generated simulations can generalize to held-out environments that satisfy the design specifications provided in prompts. Generalization to unseen environments is crucial for many applications, including transferring robotics

policies learned in simulation to the real world. This benchmark consists of 8 RL environments with varying levels of difficulty. In addition to evaluating our method on the benchmark we introduced, we further assess FACTORSIM's ability to generate robotic tasks on the dataset published by GenSim [35]. We demonstrate the value of our method, FACTORSIM, on both the benchmark task suite and GenSim's dataset, showing performance superior to baseline alternatives.

In summary, our contributions are three-fold. First, we propose FACTORSIM, a framework for generating coded simulation with a factor graph of a POMDP as a principled way to reduce context dependence. Second, we introduce a new generative simulation benchmark by adapting an existing RL benchmark [33], and demonstrate FACTORSIM's superior results against baselines in terms of code correctness (i.e., prompt alignment), ability to facilitate zero-shot generalization and human evaluation of the simulations. Third, we demonstrate that FACTORSIM can be applied to generating simulation tasks for robotics, outperforming existing approaches.

2 Related Work

We aim to generate simulations for training agents to generalize to previously unseen environments. Recent work has investigated this in the context of learned neural world models and LLM-generated code for simulations.

World models simulate the dynamics of a given environment and use this as a proxy environment for agent training, rather than interacting with a ground truth simulator [8]. Several approaches have demonstrated the value of learning world models as part of general algorithms that can learn to play a variety of games (AlphaZero [30], Muesli [13], and DreamerV3 [9]). Other efforts use a large set of offline data to learn a world model that is subsequently used for agent training, including for autonomous vehicle driving (GAIA-1 [14]), robotic manipulation (UniSim [40]), and 2D platformer games (Genie [5]). We generate world models as code as they are more interpretable, modular, and easily modified or extended by humans-key advantages we believe are important for their use in authoring large-scale or complex simulations.

LLMs have generated many parts of simulations for game playing and robotics. In (RL) games, LLMs have been used to generate game levels [34, 31], to choose parameters for an existing simulator [44], and to assist humans in creating full games [3]. In robotics, LLMs have been used to generate reward functions, task specifications, and specific components like scene configurations within robotics tasks. Many works such as RoboGen [37], Holodeck [41], and Gen2Sim [16] build on top of existing simulators and use a series of prompts to generate interactable 3D environments to train agents. GenSim [35] starts from a human task library and iteratively generates and tests new tasks to generate robotic manipulation tasks. Other efforts have focused on generating reward functions for tasks [22, 19, 23]. Eureka [22] uses feedback from agent training to refine reward function specification. Our approach is able to generate both the simulator dynamics and reward functions and can be applied to both robotics and games.

As noted above, LLMs can struggle to handle complex tasks: this has prompted research into different ways to structure LLM reasoning. Chain-of-Thought (CoT) prompting demonstrated LLM performance can be substantially boosted by prompting the LLM to break a single task into multiple steps with either few-shot examples [38] or zero-shot [18]. Subsequent work has developed a variety of techniques to improve LLM reasoning through multi-step reasoning prompts: checking for consistency among multiple reasoning paths [36], interleaving reasoning and tool use (ReACT [43]), using tree data structures to guide the LLM reasoning process (Tree-of-Thought [42]), or formulating reasoning as a tree search process [12, 46]. Approaches for general code generation include decomposing the task into functions to subsequently generate (Parsel [45]), generating code to reach a series of intermediate execution states (ExeDec [28]), and using a multi-agent framework to generate, test, and refine code (AgentCoder [15]). Other efforts optimize the prompts for given tasks, using evolutionary search (EvoPrompt [7]) or defining generalized declarative programming frameworks with modular optimization algorithms [17]. Our approach generates code by leveraging a factorized representation specific to simulations to reduce the input context needed for different reasoning steps; it can be used in conjunction with approaches for general code generation, such as generating tests as a form of self verification.

3 FACTORSIM: Generating Simulations via Factorized Representation

A simulation is a structured system of modules connected by events and responses. Our framework, FACTORSIM, generates code using LLMs by exploiting this structure to construct a simulation progressively. Our key insight is that, by generating a simulation step-by-step while only selecting the relevant context information needed for each step , we can effectively reduce the reasoning capacity needed for each step, leading to simulations that adhere more closely to the input requirements.

In this section, we describe our method for generating Turing-computable simulations. First, we describe simulations that can be modeled as a Partially Observable Markov Decision Process (POMDP). Second, we use Chain-of-Thought (CoT) to decompose an input prompt describing the desired full simulation into a series of prompts describing different components to be implemented. Third, we introduce a factorized POMDP representation that exploits the inherent modularity of coded simulations. Refer to Algorithm 1 and Figure for an overview of FACTORSIM alongside an illustrative example.

3.1 Modeling Simulation as POMDP

A Partially Observable Markov Decision Process (POMDP) is used to represent a coded simulation. Formally a POMDP is represented as a tuple M = ⟨ S, A, O, T, Ω , R ⟩ where S is a set of states, A is a set of actions, O is a set of observations, T : S × A → ∆ ( S ) is a transition probability distribution, Ω : S → ∆ ( O ) is an observation function, and R : S × A × S ′ → R is the reward model 5 .

We aim to generate a simulation from a prompt Q text. In this paper, we are particularly interested in the case where Q text comprises detailed design specifications such that the resulting simulation could be used to train agents, though our method applies to any prompt for defining a simulation. In our experiments, Q text is a paragraph of text around 10 sentences specifying this simulation.

AnillustrativeexampleofFactorSim

Input:

Output:

detailed descriptions of the simulation

we aim to generate

PROMPT 1: Decompose the specification into

a series of modules to be implemented.

(see appendix 1isting 1)

...Instruction at Step k (qx):

Instruction at Step 1(q):

Instruction at Step2(q2):

Instruction at Step k (q) :

Relevant States : s [Zk]

blue circle,allowing players to control

respawn when it collides the blue agent.

it with arrow keys. The red puck should

Introduce a blue agent rendered as a

PROMPT 2: Identifies and compiles a list of

relevant state variables.

(see appendix 1isting 2)

red_puck_position =

red_puck_speed = ...

blue_agent_position =...

red_puck_radius = ...

Relevant States:s[Z]

blue_agent_radius =...

blue_agent_speed-...

red_puck_position = ..

red_puck_speed = ...

red_puck_radius = ...

Taking select context of the current

blue_agent_speed -...

blue_agent_position =

blue_agent_radius =

implement new functions into the codebase

codebase as input, PROMPT 3,4,and 5

New codebase:Mk+1

PROMPT 3:(appendix 1isting 3)

blue_agent_handle_input(.... event) :

Retrieve functions from

based on the Factorized

the current codebase

POMDP representation

PROMPT 4:(appendix 1isting 4)

PROMPT 5:(appendix 1isting 5)

def blue_agent_render():

Relevant Functions : T[Zk]

Input Event handinge

def red_puck_respamn(state_manager) :

red_puck_position = ...

Render game conponents

blagnt_d(st,)

3.2 Chain of Thought

We first decompose the prompt Q text into a series of steps using Chain of Thought [38], each describing a module of the simulation to be implemented. Following similar formulation as in [26], this can be thought of as marginalizing over a step-by-step plan variable ( q 1 , . . . , q k ) using N Monte Carlo samples:

p is a probability estimation model (i.e., an LLM in our experiments), and M ′ is the resulting code that fully specifies a simulation. In practice, we only produce a single plan N = 1 .

Intuitively, this process breaks the prompt into sub-tasks. After we sample such a plan of K steps, we generate the simulation progressively. Given an existing POMDP M and a natural language specification q , we update the POMDP to reflect the changes specified.

where M k +1 is the POMDP (simulation as code) after the k -th step is implemented, and M K +1 is the final simulation. While Chain-of-Thought prompting allows LLMs to avoid having to generate code for all simulation logic at once, the complexity of each step still grows with k due to the expanding codebase. This task remains challenging because LLMs must comprehend the code and accurately identify where modifications are needed. Acknowledging the limited reasoning ability of LLMs, we ask: can we further decompose the p ( M k +1 |M k , q k ) into simpler distributions to reduce the complexity of each prompt?

3.3 Decomposition by Factorized Representation

Naively, we could further decompose a step of the generation into several steps, each focused on generating a different component of the POMDP:

However, this still requires the LLMs to take the entire simulation ( M k ) as context, which could be over hundreds of lines of code in our experiments. Empirically, we observe that many failed generations can be attributed to LLMs attending to or modifying parts of the input context unrelated to the prompt.

To reduce the input context needed for each generation step, we propose to use a factored POMDP representation to remove the dependence on the full previous POMDP as context. For instance, given an existing simulation M k of red, green, and blue agents, to implement the kth -step instruction q k : respawn the red agent when it collides with the blue agent , we only need context regarding the respawn logic of the red agent and the positions of the red and blue agents. Code regarding the green agent or the rendering logic would be unnecessary context.

To formalize our approach, we first introduce notation common to the literature [25, 32]. Suppose we have a POMDP with a state space factored into n state variables S = S [1] × . . . S [ n ] and Z is a subset of indices Z ⊆ { 1 , 2 , . . . , n } , we define the scope set S [ Z ] := ⊗ i ∈ Z S [ i ] as the state space spanned by the subset of state variables. For example, if Z = 1 , 3 , 4 , then S [ Z ] defines a state space defined by S [1] × S [3] × S [4] . We denote a state in the scoped state space S [ Z ] as s [ Z ] . Below, let us formally define a factored POMDP.

Definition 3.1. A factored POMDP is a POMDP with both factored transition distribution and factored reward function. A transition probability distribution T of a POMDP with discrete action space is factored over its state space S = S 1 × . . . S n with scopes Z 1 , . . . , Z m if, for all s ∈ S , a ∈ A there exist some { T i } m i =1 in the space of all possible transition distributions on the state space S and action space A , such that,

A reward function R of a POMDP is factored over S = S 1 × . . . S n with scopes Z 1 , . . . , Z l if, for all s ∈ S , a ∈ A there exist some { R i } l i =1 in the space of all possible reward functions on the state space S and action space A , such that,

A factored POMDP can be represented as a factor graph 6 with two types of nodes: state variables (i.e., S i ) and factors (i.e., T i or R i ), functions of (state) variables. Our idea is to reduce context dependence by structuring the code using a factored POMDP representation and treat each generation step as expanding a factored POMDP with new state variables and new factors . During every step q k , we first select a set of relevant state variable indices Z k . Then, we select existing factors that have overlapping scope with the selected set of state variables as context, which we denote as T [ Z k ] and R [ Z k ] . That is, we can reduce the dependence on the previous simulation M k and rewrite Equation 3-6 to the following:

Note that Z k can only consist of state variable indices in the state space S k +1 . In practice, we achieve this by encouraging the LLM to select a minimal set of relevant states Z k in the prompt.

We find that the term 11 is most prone to error, likely because the most complicated functions of a simulation are state transitions. Motivated by this observation, we propose to adopt the model-viewcontroller design pattern for structuring these prompts. Instead of prompting LLMs to update the state transition function first and then update the reward function, we prompt the LLMs to update the action-dependent part of the state transition function (i.e. the Controller component) and then the

action-independent part (i.e., Model ). We treat the reward model as part of the state transition function that updates a score state variable. That is, T ( s ′ | s, a ) = T ( s ) ( s ′ | s ) T ( a ) ( s | s, a ) where T ( a ) ( s | s, a ) denotes the part of the state transition function that handles how actions affect the states and T ( s ) ( s ′ | s ) denotes the part of the state transition function that how states are updated every step. This gives us our final algorithm as illustrated in Algorithm 1.

In Algorithm 1, colors indicate the corresponding components of the model-view-controller pattern. Red highlights the controller , corresponding to parts of the state transition dependent on user-input actions.Green shows the model , corresponding to parts of the state transition function that are not dependent on user-input actions. Blue shows the view component, updating the observation function that acts as the 'renderer' of the state space.

4 Experiments

In this paper, we consider two types of simulations: 2D Reinforcement Learning (RL) games and robotics tasks in a physics engine. We also introduce a new benchmark to evaluate generative simulation methods. Our experiments are designed to test three hypotheses. First, FACTORSIM generates simulations with better prompt alignment , which we evaluate through system tests and human evaluations. Second, FACTORSIM enables better zero-shot transfer by training RL agents in the simulated generated environments. Third, FACTORSIM's strengths in generating robotic tasks .

4.1 RL Game Generation

To answer our first two hypotheses, we propose a new benchmark that includes all 2D games from the PyGame Learning Environment 7 [33]: Flappy Bird, Catcher, Puckworld, Pixelcopter, Pong, Snake, Waterworld, and Monster Kong. For each RL game, the input prompt consists of the game's online documentation. Since most game documentation is incomplete, we manually supplement them with additional details (see Appendix). This ensures that our method and the baselines do not hallucinate any missing game information, allowing for a fair evaluation across all methods.

Following common practices in game development, we design system tests to verify that the generated simulations follow the specified logic programmatically. These tests simulate actions like key presses and mouse clicks and check if the game states are updated correctly. Refer to the Appendix for more details.

Baselines For baselines, we compare to three methods using a closed-source (GPT-4 [1]) and an open-source LLM (Llama-3 [2]). The first approach prompts the LLM with all contexts at once, which we denote as the vanilla method. The second approach uses self-debugging [6], where the model retries generating the code when provided error messages from running the code (up to 10 times in our experiments). A third approach combines Chain-of-Thought [38] (CoT) reasoning with self-debugging, where the LLM generates code incrementally, processing one instruction at a time. Additionally, we incorporate AgentCoder [15] as a baseline. CoT with self-debugging is an ablation study of our method that acts without the factored POMDP representation.

Code Generation Evaluation

Table 1 shows the results for the baselines and our method. FACTORSIM outperforms all baselines in 7 out of 8 games. Additionally, we compare performance and LLM token usage across various methods using GPT-4 (Figure 3). While the vanilla baseline uses the fewest tokens, it only achieves moderate accuracy. Additionally, combining Chain-of-Thought (CoT) reasoning with self-debugging results in the highest token usage but only

1.0

Passed

GPT-4 w/ self debug

0.8

GPT-4 w/ FactorSim

Tests

0.6

System

■

Snake

Catcher

0.4

Pixelcopter

Pong

0.2

+

Puckworld

Waterworld

%

★

Monster Kong

0.0

0

40000

80000

120000

TokenUsage

marginally improves accuracy over iterative self-debugging. FACTORSIM achieves the highest accuracy with modest token usage, indicating that the decomposition of tasks reduces the need for extensive iterative debugging.

Empirically, we find that certain prompts, when tested on baselines without the decomposition mechanism in FACTORSIM, are prone to syntax or runtime errors that the LLMs cannot self-debug. This is particularly evident with Llama-3 (vanilla) and Llama-3 self-debug, which perform poorly as they generate highly similar incorrect implementations, ignoring the logic specified in the prompts even when the temperature is set to 1. We hypothesize that this behavior is due to the model having a strong prior for how certain games, like Pong and Flappy Bird, should be implemented, ignoring the prompt specifications. This 'mode collapse' issue of LLMs could be caused by over-fitting in the instruction tuning stage [10].

While AgentCoder iteratively refines code, it performs poorly because it relies on a test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark, the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard "role-based" Chain of Thought decompositions, and that it is non-trivial to generate simulations from complex textual specifications.

Zero-shot Transfer Results Additionally, we test FACTORSIM by training a PPO [27] agent on 10 generated environments for 10 million steps and zero-shot test it on the 'ground-truth' environment implemented in the original RL benchmark (Figure 4). The rewards are linearly scaled such that 0 corresponds to the performance of a random policy and 1 corresponds to the performance of a PPO agent trained for 10 million steps on the "ground-truth" environment. FACTORSIM achieves notably better zero-shot transfer results as a result of generating code that adheres more closely to the prompt specification. We also observe that the errors FACTORSIM made tend to be more spread out across different components of the simulation. In contrast, many baselines suffer from failure modes concentrated in a specific aspect of the generation (e.g., incorrectly implementing the collision logic) that significantly hampers the ability of a set of generations to facilitate zero-shot transfer.

Human Study Evaluation Automated systems tests cannot holistically capture some aspects of game playability such as rendering a usable user interface. To address this limitation we conducted a human study where users were asked to play the generated games and evaluate their playability. Over 320 human evaluations (40 per game) we find FACTORSIM generates more functional and playable games, compared to the strongest baseline GPT-4 CoT with iterative self-debugging (Figure 5). More details can be found in the Appendix.

4.2 Robotics Task Generation

We evaluate on GenSim's [35] 50-task benchmark of robotics tasks in the CLIPort framework [29]. Refer to Figure 6 for an overview of our experimental setting. We compare FACTORSIM with the best-performing methods in generating code that specifies tasks (object states and reward structure) that can be used to train robots. Analogous to the game generation experiment, we use FACTORSIM

1.0

Trainingontestingenvionments

Reward

0.8

Normalized

0.6

GPT-4w/self-debug

GPT-4CoTw/self-debug

0.4

GPT-4W/FACTORSIM

0.2

0.0

catcher

puckworld

waterworld

pixelcopter

flappy bird

pong

snake

monster_kong

Check for Syntax Correctedness

Task completion Check

(whether the task can be completed

by an expert oracle agent)

Can you generate the following task

"Stack Boxes Column"?

Human Verification (whether the

demonstrate aligns with the prompt)

State Variables

FactorSim

Generated Task

A subtask function

Runtime Verification

(whether the task can be

instantiated in the physics engine)

Expert demonstration obtained  from

the language specification of the

target task)

Lass StacideeesCeluen(Task):

_init_,(stf):

fs

Statefanager + type(*Stateanager*, Cobject,). 0)

tuper()-,nit,_()

InitLalLLze the stats_anager attrSutes that

sunage aL1.

sef.statengerain_target,pese * (05, 0.0, 0.0), , 0, 0, 1,)

fsteftb

self.aditsosal_resetO

sef.stateer,fistx - *xtnlatef*

seLf.state_eanager,second_box_size = (0.07, 0.07, 0.07)

seL.f.state,eanager.,second,boxcolar + "bLse*

sef.state,nger,secox_f - *exbextelate.g*

sef.state_sanager,thdrdbox_sze = (007, 0.07, 0.07)

sef.state,anager,.thrd,bxcolor + "btue*

sfstate,anager.thidbx_df - *ex/bex-tenp/late.gndf*

seLf.nax_steps = 15

self.lang,teptate  *euentiatly stack a set mb

seLf.stateanaer,first,box_sLe + (0.17, 0.17, 0.17)

tablete withvt thee teeling over,

box_color  sef.statemnger,frst_bo_coan

# Steg 1. ketrieve the box size, esler, and yRgf peth

box,size = sef.statenager fist,x_size

# Steg 2. Set a randos position for the initial. p/Lacee

initiat_bes_pese * seLf.grt,rande_pese(e, bex,si)

targetese  sef,stateanager,tarete

lle′l. vse the nain_target_gose attribste fros state

Step 3. The target pose for the first bex Is on the taolatap

Step 4. Ad the first box to the environent st the Initial pose

59 + PI°x99

bpx_urdf, initia_b_m, coartits.oos[xctar]

addsbjeet(

se.f.8dd,goa1(

Steg 5: Define the goa/l, fer this suttask, The goal, Is t9 f

5se[,1],

Tasks that FactorSim successfully generated but

all other baselines failed.

Build Lamp Post

Build Picnic Table

InsertEl1 Along Squre Path

Color Coordinated Cylinder Tower

to modularize the code generation process into subtasks and have LLMs generate each subtask using only a set of necessary states as context. More details can be found in the Appendix.

Baselines & Metrics We compare our method with the multiple GenSim baselines: vanilla (one-shot), Chain-of-Thought (topdown), and Chain-of-Thought (bottom-up). Adopting the same set of metrics, we evaluate all methods on a sequence of pass rates on 'syntax correctness', 'runtime-verified', and 'task completed'. A generated task is considered 'completed' if a coded oracle agent could collect 99% of the total reward half of the time.

We empirically found that the "task completion rate" is an imperfect metric for evaluating the success of a generated task. A task deemed "complete" by the oracle agent may fail to adhere to the prompt. For example, when asked to generate a task "build a wheel," a method might

FactorSim

CoT w/ self-debug

Fully Playable

Unplayablebutrenderedproperly

Playable with issues

Unplayable -crashes or fails to launch

21.2%

28.7%

24.7%

10.8%

42.9%

11.2%

21.6%

38.9%

produce a task specification that involves rearranging blocks into a structure that does not resemble a wheel. To address this, we introduced a metric of the 'human pass rate'. This involved manually inspecting runtime-verified tasks to determine if the task specifications aligned with the prompt descriptions (see Appendix).

LLama-3

1.0

0.8

Rate

Success

0.6

0.4

0.2

0.0

Runtime Pass Rate

TaskCompletionRate

Human Pass Rate

Runtime Pass Rate

TaskCompletionRate

Human Pass Rate

Gensim-vanilla

Gensim CoT(topdown)

FactorSim

Task:

BuildHouse

1.Place Base

2.Place Walls

4.Place Roof

Task:

InsertEll

Along

1.Place Red Ell

2.Place Blue Ell

3.Place Green Ell

4.PlaceYellowEll

Results & Discussion FACTORSIM outperforms baselines in generating tasks with a higher runtime pass rate and better human evaluation scores, indicating improved prompt alignment (Figure 7). Task completion rates are generally low for all methods due to the limitation of the oracle agent. For example, tasks like "Build Ball Pit" (fill a container with balls to create a ball pit) often fail because the balls roll out of the visible area of the oracle agent, not because the generated task is invalid. FACTORSIM performs particularly well on tasks that specify spatial relationships (e.g., "on top of," "left of," "outside of") between objects, such as the "build House" example in Figure 8. This improvement is likely due to the decomposition process, where for each step, instead of addressing a combination of multiple spatial relations all at once, FACTORSIM attends to a smaller context, allowing each spatial relation to be addressed separately.

5 Conclusion & Future Work

We have proposed FACTORSIM as an approach to generate full simulations as code that can train agents while adhering to detailed design requirements specified as a text prompt. We also introduce a benchmark suite of eight RL environments to evaluate generative simulation methods.

Generating complex simulations in code is challenging, and we anticipate numerous opportunities to extend the simulation generation process. There is substantial room to address larger-scale, more complex games, and robotics environments that require code bases beyond what can be used effectively in the context window of existing LLMs. We also see great potential to accelerate RL agent training by generating code that can be accelerated on GPU devices. Our robotic simulation results will benefit from further investigations to demonstrate transfer to real-world environments. We have only addressed single-agent simulations, leaving the extension of our method to multi-agent settings to future work. In the future, we also plan to incorporate information from the agent training process to automatically modify the generated simulation environment for enhanced agent learning and generalization. Taken together, we believe the generation of full simulations as code will be an important step toward enhancing the capabilities of LLMs to support the development of generalized RL agent policies.

Acknowledgments and Disclosure of Funding

This work was in part supported by the Stanford Institute for Human-Centered Artificial Intelligence (HAI), the Stanford Center for Integrated Facility Engineering (CIFE), NSF CCRI #2120095, AFOSR YIP FA9550-23-1-0127, ONR N00014-23-1-2355, ONR YIP N00014-24-1-2117, and Google.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

AI@Meta. Llama 3 model card. 2024.

Asad Anjum, Yuting Li, Noelle Law, M Charity, and Julian Togelius. The Ink Splotch Effect: A case study on ChatGPT as a co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games , 2024.

Craig Boutilier and David Poole. Computing optimal policies for partially observable decision processes using compact representations. In Proceedings of the National Conference on Artificial Intelligence , pages 1168-1175, 1996.

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint arXiv:2402.15391 , 2024.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In 12th International Conference on Learning Representations , 2024.

Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In 12th International Conference on Learning Representations , 2024.

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Sil Hamilton. Detecting mode collapse in language models via narration. arXiv preprint arXiv:2402.04477 , 2024.

Eric A Hansen and Zhengzhu Feng. Dynamic programming for pomdps using a factored state representation. In AIPS , pages 130-139, 2000.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4214-4226. PMLR, 18-24 Jul 2021.

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1: A generative world model for autonomous driving, 2023.

Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010 , 2023.

Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. Gen2Sim: Scaling up robot learning in simulation with generative models. arXiv preprint arXiv:2310.18308 , 2023.

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In 12th International Conference on Learning Representations , 2024.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199-22213, 2022.

Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In 11th International Conference on Learning Representations , 2023.

Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning , pages 3053-3062. PMLR, 2018.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 12:157-173, 2024.

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931 , 2023.

Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: Language model guided sim-to-real transfer. 2024.

Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596 , 2023.

Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored MDPs. Advances in Neural Information Processing Systems , 27, 2014.

Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems , 36, 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Kensen Shi, Joey Hong, Manzil Zaheer, Pengcheng Yin, and Charles Sutton. ExeDec: Execution decomposition for compositional generalization in neural program synthesis. arXiv preprint arXiv:2307.13883 , 2023.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 894-906. PMLR, 08-11 Nov 2022.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Shyam Sudhakaran, Miguel González-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. MarioGPT: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems , 36, 2023.

István Szita and András L˝ orincz. Optimistic initialization and greediness lead to polynomial time learning in factored MDPs. In Proceedings of the 26th annual international conference on machine learning , pages 1001-1008, 2009.

Norman Tasfi. Pygame learning environment. https://github.com/ntasfi/ PyGame-Learning-Environment , 2016.

Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games , pages 1-8, 2023.

Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. GenSim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01361 , 2023.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In 11th International Conference on Learning Representations , 2023.

Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, and Tsun-Hsuan Wang. Towards a foundation model for generalist robots: Diverse skill learning at scale via automated task and scene generation. arXiv preprint arXiv:2305.10455 , 2023.

Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In 12th International Conference on Learning Representations , 2024.

Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodied ai environments. CVPR , 2024.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Abhay Zala*, Jaemin Cho*, Han Lin, Jaehong Yoon, and Mohit Bansal. EnvGen: Generating and adapting environments via llms for training embodied agents. arXiv preprint arXiv:2403.12014 , 2024.

Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions. Advances in Neural Information Processing Systems , 36:31466-31523, 2023.

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023.

A Societal Impact

This work can be applied broadly to many types of simulations, including robotics, autonomous vehicles, and other autonomous systems. Such systems have the potential for both positive and negative societal impact (e.g., harmful dual use). As researchers, we must critically evaluate such applications and promote beneficial ones. In this work we have focused on simulations with potential positive social impact, particularly in supporting the development of robots able to operate in human environments like households or manufacturing facilities.

The methods we present generate simulations that can be used to train agents to perform tasks. One risk with generated simulations is for training agents in an unintended manner. By generating simulations specified as code we mitigate this concern by making the behavior of the simulation explicit and inspectable by humans. Further, our approach is better able to guide LLMs to generate code that matches input design specifications compared to baseline methods, reducing the risk of LLMs inadvertently producing undesirable functionality. We believe this can help enhance the reliability of generated simulations while offering strong editing and control capabilities to humans.

The potential negative environmental impact of the compute for using our technique is small. We have shown our technique consumes less tokens than comparable methods to yield equally good results. Thus our method can be seen as a way to reduce computational needs when using LLMs for tasks like creating simulations. Compared to systems that use a neural world model our approach benefits from the relatively lower computational costs of running simulations in code compared to running large neural models for simulation.

B Additional details of our experiments

All experiments are done on a workstation with 8 Nvidia A40 GPUs and 1008G of RAM. For our code generation experiments, one generation (i.e., generation of one training environment) takes around 30 seconds to 5 minutes. For our Reinforcement Learning experiments, one trial of training (i.e. training on a set of environments for 10M steps in total) takes around 3-5 hours to complete. In all of our experiments, GPT-4 refers to the OpenAI's 'gpt-4-1106-preview' model, GPT-3.5 refers to OpenAI's 'gpt-3.5-turbo' model, and Llama-3 refers to the open-sourced 'meta-llama-3-70b-instruct' model that can be found on huggingface.

For the zero-shot transfer RL experiment, we supply all methods with the reference "controller" (i.e., the same key press/mouse click leads to the same thing). We do this because language descriptions of such can be very ambiguous (e.g., a description 'a key press leads the bird to flap its wings' can imply a change in position, velocity, or acceleration). In our experiments, we generate 10 environments and filter out those that cannot be properly executed with a random policy.

All the RL experiments are implemented in RLLib [20] 8 . The PPO agent is trained with a batch size of 10,000, and an SGD minibatch size of 2048. Our agent used a fully connected network with hidden layers of sizes (4, 4) and post-FCNet hidden layers of size 16, all employing ReLU activation functions. The policy network uses an LSTM with a cell size of 64 to incorporate previous actions but not previous rewards. Over the course of the 10 million training steps, 20 checkpoints were saved, with the best zero-shot performance on the testing environment reported.

C Additional details of FACTORSIM

We provide code in the supplementary material. Here we provide the prompts used in FACTORSIM.

Listing 1: The first decompositional prompt used in FACTORSIM.

Listing 4: The prompt for the Model component (as defined in the Model-View-Controller) utilized in the FACTORSIM.

Listing 5: The prompt for the View component (as defined in the Model-View-Controller) utilized in the FACTORSIM.

FactorSim

Target Tasks

Syntax

Runtime

Task

Human

Syntax

Runtime

Task

Human

Syntax

Runtime

Task

Human

Syntax

Runtime

Task

Human

BuildDogHouse

yes

no

no

no

no

no

no

no

yes

yes

no

no

yes

yes

no

no

BuildLampPost

yes

yes

yes

yes

yes

no

no

no

yes

no

no

no

yes

no

no

no

BuildNewsstand

yes

yes

no

no

yes

no

no

no

yes

no

no

no

yes

no

no

no

BuildBench

yes

no

no

no

yes

no

no

no

yes

no

no

no

yes

no

no

no

BuildPicnicTable

yes

yes

yes

yes

no

no

no

no

yes

yes

no

no

yes

yes

no

no

BuildBicycleRack

yes

no

no

no

yes

no

no

no

yes

no

no

no

yes

no

no

no

BuildPicnicBasket

yes

no

no

no

yes

no

no

no

yes

yes

no

no

yes

yes

no

no

BuildCylinderStructure

yes

yes

yes

yes

no

no

no

no

yes

no

no

no

yes

no

no

no

BuildBridge

yes

yes

yes

no

yes

no

no

no

yes

no

no

no

yes

no

no

no

BuildCar

yes

yes

no

no

yes

yes

no

no

yes

yes

yes

no

yes

yes

yes

no

BuildTwoCircles

yes

yes

no

no

yes

no

no

no

no

no

no

no

no

no

no

no

BuildWheel

no

no

no

no

yes

no

no

no

yes

yes

no

no

yes

yes

no

no

Aggregated Pass Rate

92%

58%

33%

25%

75%

8%

0%

0%

92%

42%

8%

0%

92%

42%

8%

0%

D Additional details and results for the robotics task generation experiment

In this section, we provide the prompts we used for FACTORSIM in the robotics task generation experiment. The prompts for the baselines can be found in the GenSim paper 9 [35].

To conduct human evaluation, we begin by observing the oracle agent attempting to solve the task. If the oracle agent successfully completes the task, we then assess whether the resulting goal states align with the input task prompt. If the oracle agent fails to solve the task, we investigate the reason for the failure. Often, the cause is apparent, such as the target container being too small or not having the right color of objects for the task. These are marked as failures. For cases where it is clear that the limitation lies in the oracle agent's ability, or when the reason for failure is not immediately apparent, we manually inspect the code for the task specification and base our decision on both the code and our observation of the oracle agent's attempt at solving the task.

Listing 7: The state change prompt.

E Additional details for the proposed generative simulation benchmark

This section provides the prompts for all 8 RL games in the benchmark.

Listing 9: The prompt for the game Catcher .

Create a catcher character , represented as a rectangle , positioned at the bottom and the middle of the screen.

Allow the player to control the catcher 's horizontal movement using the left and right arrow keys on the keyboard.

There should always be exactly one ball on the screen at all times. The ball should be visually distinct and easily recognizable.

Make the ball move downwards at a steady pace towards the catcher. The speed can be

Listing 10: The prompt for the game Flappy Bird .

Create a bird character , visually represented as a simple rectangle within the game window. Introduce gravity , causing the bird to continuously fall slowly. Allow the bird to 'jump' or accelerate upwards in response to a player 's mouse click , temporarily overcoming gravity. Periodically spawn pairs of vertical pipes moving from right to left across the screen. Each pair should have a gap for the bird to pass through , and their heights should vary randomly. If the bird makes contact with the ground , pipes or goes above the top of the screen the game is over. Implement the following scoring system: for each pipe it passes through it gains a positive reward of +1. Each time a terminal state is reached it receives a negative reward of -1. When the game ends , display a "Game Over!" messagea and stop all the motion of the game. Show the current score in the top-left corner of the screen during gameplay. Ensure the game has no predefined end and that new pipes continue to generate , maintaining consistent difficulty as the game progresses.

Listing 11: The prompt for the game Snake .

Create a snake character represented as a series of connected pixels or blocks. Initially , the snake should be a single block (i.e. the head) that moves in a specific direction within the game window. Allow the player to control the snake 's movement using arrow keys. The snake should be able to turn left or right , but it should not be able to move directly backward. Eg. if its moving downwards it cannot move up. The movement of the snake should be continuous in the current direction until the player provides new input. Ensure that the snake moves one grid unit at a time. Implement a basic food system where one food item appears randomly on the screen. When the snake consumes the food by moving over or colliding with it, the snake 's length increases , and the player earns points. It recieves a positive reward , +1, for each food the head comes in contact with. While getting -1 for each terminal state it reaches. If the head of the snake comes in contact with any of the walls or its own body , the game should end. Incorporate a scoring system , displaying the current score on the screen during gameplay. The score should increase each time the snake consumes food. Ensure that the game has no predefined end, allowing the snake to continue growing and the difficulty to increase over time. New food items should appear after the snake consumes one. Provide an option for the player to restart the game after it ends. Display a "Restart" option on the game over screen to allow the player to play again.

Listing 12: The prompt for the game Pixelcopter .

Listing 13: The prompt for the game Pong .

Listing 14: The prompt for the game Puckworld .

Listing 15: The prompt for the game Waterworld .

F Additional details for the human study experiment

In this section, we first provide details for the experiment and then the instructions we gave to human participants in our human study, along with the user interface. We also provide the detailed results of this evaluation for all games in Figure 9.

Human participants were asked to play and evaluate the generated games given the prompt while excluding factors such as aesthetics or difficulty. They rated the games on a scale of 1 to 4, where 4 indicates a fully playable game, 3 is a playable game with some bugs or flaws that hinder gameplay experience, 2 is an unplayable game (i.e., no interactivity) with correctly rendered UI, and 1 is a game that crashes or fails to launch.

Playability Ratings

FullyPlayable

Playablewithissues

Unplayablebutrenderedproperly

Unplayable

Pong (FactorSim)

Pong (Baseline)

Waterworld(FactorSim)

Waterworld(Baseline)

36.0%

23.5%

29.2%

17.6%

5.9%

4.0%

4.0%

17.6%

23.5%

5.9%

29.2%

25.0%

52.9%

52.9%

56.0%

16.7%

Puckworld(FactorSim)

Puckworld (Baseline)

0.0%

36.8%

8.7%

26.1%

0.0%

45.5%

11.1%

66.7%

15.8%

21.7%

36.4%

43.5%

22.2%

47.4%

0.0%

18.2%

MonsterKong(FactorSim)

MonsterKong (Baseline)

Catcher(FactorSim)

Catcher (Baseline)

5.9%

13.0%

36.4%

5.0%

55.0%

11.8%

64.7%

47.8%

5.0%

4.5%

30.4%

4.5%

17.6%

35.0%

54.5%

8.7%

Snake (FactorSim)

Snake (Baseline)

PixelCopter(FactorSim)

PixelCopter(Baseline)

4.5%

45.5%

31.6%

0.0%

21.1%

8.7%

13.6%

10.5%

43.5%

31.6%

21.1%

34.8%

36.4%

15.8%

68.4%

13.0%

FactorSim: Generative Simulation via Factorized Representation

Choose a game:

Waterworld

Replay

Prompt:

Create a character for WaterWorld, visually represented as a blue circle, that can move freely within the game window using arrow keys.

Introduce a dynamic environment with equal number of green and red circles.

Make the green and red circles move randomly around the screen.

Implement a scoring system,where the player earns pointsforeach greencircle captured and deduct oneforeachred circle.

Display the current score in the top-left corner of the screen during gameplay.

When the bluecircle captures a circle,make it respawn in arandom locationon the screenaseitherred orgreen.

Ensure the game continues until all green circles have been captured. Once all green circles are captured, display a "Game Over! message and stop allmotion in the game.

Provide anoptionfortheplayertorestartthegame afteritends,creatingaloopforcontinuousgameplay.

WaterWorld Game-Work-Microsoft Edge

https://cs.stanford.edu/~sunfanyun/factorsim/game_directory/waterworld/33/build/web/index.html

Score: 0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to the abstract and the introduction section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to the experiment section and our supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the supplementary material and our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to the supplementary material and our code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all assets used in our experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Please refer to the supplementary material.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve sensitive data of study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents FACTORSIM, a framework that converts any language specification into a complete simulation for training RL agents. FACTORSIM decomposes the input prompt into steps and uses a factored Partially Observable Markov Decision Process (POMDP) to minimize the context needed for each generation step. It also introduces a method to benchmark generative simulation and demonstrate the capability for FACTORSIM to be used in robotics setting.\n\nSTRENGTHS:\n1. Develop a robust pipeline for constructing game environments from language descriptions, which could significantly enhance the scalability of training generalist agents.\n2. Formalize the framework as a Partially Observable Markov Decision Process (POMDP), reducing the need for full context during generation and improving outcomes.\n3. Demonstrate the potential of this method to generalize to embodied scenarios.\n\nWEAKNESSES:\n1. The evidence for generalizing to embodied scenario is limited.\n2. The successful rate in Table 1 and Figure 3 is low. Could there be some potential way to improve it?\n\nQUESTIONS:\nI have one primary concern: how could this be applied to the field of robotics and embodied AI?\n\n**From this concern, several questions arise:**\n\n- How is the background (or scenario) generated within this pipeline? In a 2D game setting, detailed descriptions generate the scenarios, but this can become extremely tedious as scenes become more complex, such as in an embodied environment. While language-guided scene generation could be a solution, how will it fit into the POMDP framework?\n- The framework addresses three main components: controller, model, and view (rendering). In robotics, these aspects are typically handled by a physics simulation. How will this framework further contribute to the field of robotics? Currently, the paper shows potential for task generation in tabletop scenarios only.\n- I am still unclear on how the Robotics Task Generation part is achieved by this pipeline.\n\n**Some suggestions:**\n\n- While it might be challenging to address this concern with experiments during the rebuttal period, more discussion on approaches and challenges would be beneficial.\n- Any additional experiments that could demonstrate the pipeline's usage in robotics or embodied AI could help.\n\nLIMITATIONS:\nThe limitations are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a LLM prompting method to generate full game / robot simulations in code based on text descriptions.  Given a long text description, the method first utilizes an LLM to decompose it into multiple sentences, and then use them to iteratively generate and update simulation code. For each iteration, the code is generated and updated separately as three modules, i.e., controller, model and view. The update happens in a factorized way - the authors use the LLM to identify relevant state and context to avoid feeding the full generated code into LLM.\n\nIn experiments, the method is evaluated on game and robot simulation code generation benchmarks. The method shows superior results against other LLM prompting baselines in generating valid simulation code that aligns with text description.\n\nSTRENGTHS:\n- The proposed method exploits the structure of simulation to modularize and factorize code generation. This strategy significantly improves LLM's capability to generate full simulation code.\n- The method is comprehensively evaluated on game and robot benchmarks.\n- The paper is well written and easy to follow.\n\nWEAKNESSES:\nThe major contribution of the paper seems to be a prompting technique crafted for the specific task of simulation code generation. While such a technique does improve performance on the task, my concern is it is neither fundamental nor sufficiently novel. The proposed prompting technique highlights two key designs:\n- modularize simulation code generation manually, which aligns with the common practice to manually decompose a complex task into sub-tasks for LLMs to handle more effectively.\n- extract relevant portion of code for LLM to consume and update, which is also an implementation-wise design that many works have already incorporated.\n\nWhile the paper writes factorized POMDP formulations, they don't seem to make a difference on how the prompting method is implemented. So I'm concerned that the contribution of this paper is more as a practical application rather than a general method or framework.\n\nQUESTIONS:\nI'm curious what the failure modes of FactorSim is like.\n\nLIMITATIONS:\nSee Weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a factorized approach to generate simulated games via LLM code synthesis. The code idea is that one doesn't need to generate the entire code at once, but rather generate different part of a POMDP game, such as controller, model, and view. The generated simulation allows RL policies to train on top. The authors introduced a benchmark to evaluate the proposed framework and show good results in terms of prompt alignment, transfer ability and human evaluation.\n\nSTRENGTHS:\nThe paper investigates an important problem, simulation generation. The evaluation over the mentioned environments is solid, spanning from automated tests to human evaluations.\n\nWEAKNESSES:\n1. The paper is poorly written. I have hands-on experience with almost all important concepts mentioned in the paper, yet still have a hard time understanding the paper, and have to read again and again including some code. Rather than talking about abstract terms like POMDP / factorization first, I think the authors can start easy with intuitions and explanations. The figure can also be improved. The main method figure shall spend more time showing what's special about \"Factored POMDP\" compared to prior methods. The benchmark claim should have its own section. The motivation is not clearly narrated either.  The world model section in related work doesn't seem to fit there.\n\n2. On of the main contributions the authors listed in the introduction section is a benchmark. However, I think this benchmark seems to lack the technical depth I was expecting as a standalone contribution. I feel it's just a set of small metrics rather than benchmark.\n\n3. The paper just lacks the level of technical contribution that meets my criteria for a Neurips paper. While there are many other prompting papers like CoT, ToT, the problem the paper is trying to solve is also very specific. \n\n4. While I have experience with both LLM and robotics, I believe the authors should not put Robotics as primary area, but NLP or code synthesis community.\n\nQUESTIONS:\nIn figures like figure 6, is the human pass rate based on the previous stage e.g. only executable code. \n\nIt seems that on open source model like llama 3, gensim with CoT is very close to factor sim. Can you explain the insights?\n\nLIMITATIONS:\nThe author discussed the limitations of automated evaluation by conducting a human study. No outstanding negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an LLM-based method for generting code for simulations. After generating the simulation of famous games based on their manual and description, the authors show that policies trained in these environments transfer well to the real games.\n\nSTRENGTHS:\n- **S.1 Great results.** I think that the results from Fig.3 are very impressive. Zero-shot transfer is very hard, and doing so much more reliably than vanilla GPT-4 is impressive.\n- **S.2 Overall good idea.** The idea to deconstruct the game development task into M-V-C makes a lot of sense to me. I just thought that's already kind of captured in the POMDP formulation. \n- **S.3 Good presentation.** The overall presentation and writing are good, although there is much left to be desired in terms of implementation details.\n\nWEAKNESSES:\n- **W.1 Implementation details and relationship to formulas.** I'm happy that there was code provided with the submissio and I hope that it will be released publicly because based purely on the main body of the paper, the method is not reproducible. Including the prompts in the appendix helps but I wish they were commented a bit more on why certain phrases and sections are there. And I'm wondering if the theory that's presented in the paper holds water wrt the actual instructions in the appendix. Because as far as I understand, there aren't any restrictions on what code the LLM can generate for each component, right? Also, the paper mentions graphs every so often and I don't know how they fit into this. I also think the context selection is crucial to your method and from the main body of the paper, it's completely unclear how that's implemented.\n- **W.2 Missing examples.** Along a similar idea, I'd have loved some examples throughout the paper to illustrate what some of these instructions actually mean.\n- **W.3 Unclear robotic experiments.** The robotic experiments seem to be more of an afterthought in the paper, and it's unclear what existing assets there are, what control code is assumed given, what camera parameters are assumed, etc. \n- **W.4 Unclear input mapping.** The appendix mentions that the controller is given or that button presses always mean the same thing. I completely don't understand what's meant by that.\n\nOverall, I think the paper shows a great idea and is probably beneficial to the community, but some work should go into tweaking the main body of the paper and making the method more clear and reproducible.\n\nQUESTIONS:\nI don't have any major questions wrt the work. There were a couple of points throughout the paper in the methods section, where I asked myself why this is relevant, but then this was cleared up a paragraph later.\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}