FactorSim: Generative Simulation via Factorized Representation
Authors: Fan-Yun Sun
Harini S I
Angela Yi
Yihan Zhou
Alex Zook
Jonathan Tremblay
Logan Cross
Jiajun Wu
Nick Haber
Venue: NeurIPS 2024 poster (2024)
Keywords: generative simulation
POMDP
Large Language Models
Primary area: robotics
TL;DR: We propose a framework for generating simulations in code to train RL agents and introduce a new benchmark to showcase its efficacy.

Abstract: Generating simulations to train intelligent agents in game-playing and robotics from natural language input, user input, or task documentation remains an open-ended challenge. Existing approaches focus on parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents. Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce a generative simulation benchmark that assesses the generated simulation code’s accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment (i.e., accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.

FACTORSIM: Generative Simulation via Factorized Representation
Fan-Yun Sun Stanford University
S. I. Harini Stanford University
Angela Yi Stanford University
Yihan Zhou Stanford University
Alex Zook Nvidia
Jonathan Tremblay
Logan Cross
Nvidia
Stanford University
Jiajun Wu Stanford University
Nick Haber
Stanford University
Abstract
Generating simulations to train intelligent agents in game-playing and robotics from natural language input, from user input or task documentation, remains an open-ended challenge. Existing approaches focus parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents, Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce generative simulation benchmark that assesses the generated simulation code's ' accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment (i.e., accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.
Introduction
Simulations hold significant potential for training agents to perform real-world tasks where data collection is costly, dangerous, infringes on individual privacy. A major bottleneck in harnessing the potential of simulations at scale for agent training is the cost designing and developing them, especially when need a distribution of simulations that meet detailed design specifications to train more generalized policies. In this paper, we aim to generate coded simulations given text specifications. Code provides natural interface for users to inspect, modify, and debug the simulation. also allows us to craft diverse environments for Reinforcement Learning (RI.) purposes.
Generating full simulations in code to train agents from text prompt is an under-explored challenge. Previous works focus on parts of this challenge, including reward function design [22], hyperparame- ter tuning (24], and task configuration while relying
existing simulator (35] These methods
use large language models (LLMs) to generate the components of simulations specified as code.
However, when faced with large and detailed contexts, LLMs often generate simulations that ignore
or fail to adhere to parts of the input prompt [1]. This issue is not solely due to the limitations of
Work done while Harini S I was an intern at Stanford.
"Corespondence to sunfanyun @ ees.stt stanford edu 3 'Project website: https://cs..sanne stanford. edu/ sunfanyun/faactorsim/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Chain of Thought
FactorSim
g Generated simulations
Step
Factored POMDP
Step
Context Solection
Stop
dot position
positiion
handll key
tollision logic
Documentation
Raseicold
Policy Training
Irplenent Factoxs
ZeALR on unceen testing envi ronsent over training
Mhile main game loop:
contzollex
// Irput handling dunctions handle, in Jey, p.epes (
#odel
Transition functions collisson_ingte]]
view
1f Rendaxing functions render_ red _doo))
Figure 1: Overview of FACTORSIM. FACTORSIM takes language documentation as input, uses Chain- of-Thought to derive series of steps to be implemented, adopts Factored POMDP representation to facilitate efficient context selection during each generation step, trains agents on the generated simulations, and tests the resulting policy on previously unseen RL environments.
existing LLMs but also suggests that some form of decomposition is always critical as we scale up the number of components in simulations. We ask the question: can we exploit the inherent structure (e.g., having game loop that handles agent actions, updates internal game states accordingly, and displays the game states to the users through a rendering process) of coded simulations to generate them better?
We propose FACTORSIM, a framework that takes an arbitrary language specification as input and outputs a full simulation that can be used to train RL agents. The key idea of FACTORSIM is to decompose the input prompt into a series of steps and then use a factored Partially Observable Markov Decision Process (POMDP) representation to reduce the context needed for each generation step. To realize FACTORSIM, we use the model-vvewwccontroller software design pattern to structure the generation process. Consider generating coded simulation of WaterWorld; see Figure The game consists of an agent (blue circle) traveling in a 2d world, capturing food (green circle) while avoiding enemies (red circle). Our method first decomposes the game description into multiple steps to be implemented. For example, a step instruction could be "Introduce red dot enemies that can be controlled with arrow keys. Give the player -1 reward when the agent collides with an enemy". We first select the context needed for this functionality to be implemented, e.g., positions of existing agents. Subsequently, FACTORSIM generates (at most) three functions: one to handle player input (i.e., handle_key_prees the controller component), one to implement the collision logic (i.e., collision_logee the model component), and one update the rendering function (i.e., render_red lii__reddda dot, the view component). Limiting the context during each step of the simulation generation process allows FA to focus on the task hand while avoiding hallucinating non-existent functions or modifying code not meant to be changed.
To evaluate the task of full simulation generation, we propose new Generative Simulation bench- mark with accompanying success metrics. One set of success metrics is the pass rate in automated system tests. Commonly used in game development, these system tests programmatically assess whether the behavior of the generated simulation adheres to the specifications given in the input prompt. The second success metric assesses the value of the generated simulations for transfer learn ing in an RI. setting. This evaluates how well agents trained on a set of generated simulations can generalize to held-out environments that satisfy the design specifications provided in prompts. Gen- cralization to unseen environments is crucial for many applications, including transferring robotics
WWe adopt this term from 39 to refer automated simulation generation to train agents within.

---

policies learned in simulation to the real world. This benchmark consists of 8 RL environments with varying levels of difficulty. In addition to evaluating our method on the benchmark we introduced, we further assess FACTORSIM'S s ability to generate robotic tasks on the dataset published by Gen- Sim [35]. We demonstrate the value of our method, FACTORSIM, on both the benchmark task suite and GenSim's s dataset, showing performance superior to baseline alternatives.
In summary, our contributions are three-fold. First, we propose FACTORSIM, framework for generating coded simulation with a factor graph of a POMDP as a principled way to reduce context dependence. Second, we introduce a new generative simulation benchmark by adapting an existing RL benchmark [33, and demonstrate FACTORSIM'S 's superior results against baselines in terms of code correctness (i.e.. prompt alignment), ability to facilitate zero-shot generalization and human evaluation of the simulations. Third, we demonstrate that FACTORSIM can be applied to generating simulation tasks for robotics, outperforming existing approaches.
2
Related Work
We aim to generate simulations for training agents to generalize to previously unseen environments. Recent work has investigated this in the context of learned neural world models and Len-rrrertted code for simulations.
World models simulate the dynamics of given environment and use this as a proxy environment for agent training, rather than interacting with a ground truth simulator g] Several approaches have demonstrated the value of learning world models part of general algorithms that learn to play a variety of games (AlphaZero (30), Muesli [3], and DreamerV3 (9). Other efforts use a large set of offline data to learn a world model that is subsequently used for agent training, including for autonomous vehicle driving (GAIA-1 (44,, robotic manipulation (UniSim (40), and 2D platformer games (Genie (5). We generate world models code as they more interpretable, modular, and easily modified or extended by bumans-kee advantages we believe are important for their use in authoring large-scale or complex simulations.
LLMs have generated many parts of simulations for game playing and robotics. In (RL) games, LLMs have been used to generate game levels (34 31), to choose parameters for existing simulator 44] and to assist humans in creating full games 3. In robotics, LLMs have been used to generate reward functions, task specifications, and specific components like scene configurations within robotics tasks. Many works such as RoboGen (37], Holodeck [41], and Gen2Sim [16] build on top of existing simulators and use series of prompts generate interactable 3D environments train agents. GenSim (35] starts from a human task library and iteratively generates and tests new tasks to generate robotic manipulation tasks. Other efforts have focused on generating reward functions for tasks (22) 19 23). Eureka (22] uses feedback from agent training to refine reward function specification. Our approach is able generate both the simulator dynamics and reward functions and can be applied to both robotics and games.
As noted above, LLMs can struggle to handle complex tasks: this has prompted research into different ways to structure LLM reasoning. Chain-of-Thought (CoT) prompting demonstrated LLM performance can be substantially boosted by prompting the LLM to break a single task into multiple steps with either few-shot examples [38] or zero-shot (18]. Subsequent work has developed a variety of techniques improve L.M reasoning through multi-step reasoning prompts: checking for consistency among multiple reasoning paths [36], interleaving reasoning and tool use (ReACT (43), using tree data structures guide the LLM reasoning process (Tree-of-Thought [42)). or formulating reasoning as a tree search process [12 [6). Approaches for general code generation include decomposing the task into functions to subsequently generate (Parsel (45)), generating code to reach series of intermediate execution states (ExeDec (28). and using multi-agen framework to generate, test, and refine code (AgentCoder (15)). Other efforts optimize the prompts for given tasks, using evolutionary search (EvoPrompt (7) or defining generalized declarative programming frameworks with modular optimization algorithms [77. Our approach generates code by leveraging factorized representation specific to simulations to reduce the input context needed for different reasoning steps; it can be used in conjunction with approaches for general code generation, such as generating tests as form of self verification.
3

---

FACTORSIM: Generating Simulations via Factorized Representation
A simulation is structured system of modules connected by events and responses. Our framework, FACTORSIM, generates code using LLMs by exploiting this structure to construct a simulation progressively. Our key insight is that, by generating simulation step-by-step while only selecting the relevant context information needed for each step, we can effectively reduce the reasoning capacity needed for each step, leading to simulations that adhere more closely to the input requirements.
In this section, we describe our method for generating scribe simulations that can be modeled a Partially Observable Markov Decision Process (POMDP). Second, we use Chain-of-Ti Thouhtt (CoT) to decompose an input prompt describing the desired full simulation into series of prompts describing different components to be implemented. Third, introduce factorized POMDP representation that exploits the inherent modularity of coded simu- lations. Refer to Algorithm and Figure for an overview of FACTORSIM alongside an illustrative example.
simulations. First, we de-
Algorithm 1: FACTORSIM
Input: Quau, natural language description of simulation, and an L.M
Output: a turing-connn computable simulation represented as a POMDP M' = (8,
Initialize Factored POMDP Mi (S1.4,0, 21, RR)) where
-S1 := {&score)
-A the set all keyboard inputs
- is identity function, i... Ti(s' s, 1/s
Ri (s, a, = S
Bucoe
// Chain of Thought
Derive step-by-step plan (q)......4)) qu) ~ p(y...... ... Ques)
Eq. D
for each step, or module qu do
State space update & context selection p(Sx+1, [Zx]
Eq. 9 10
// Controller component update
Action-dependent state transition model update:
// Model component update
Action-independent state transition model update: p ((T(())
Ze, ( [Ze).
// View component update Observation model update: p(2++11 1S [Zx].
Eq.
Me+1 (Sx+1, A, Ok+1, Te+1, ax+1, R1} where O is the new observation space defined by Sa+1 and 22+1, and 1 (8' (8 = ((() (s' (8) T(e) 2+110 (s a))))
end
Return the final simulation M' t Ma+1
3.1 Modeling Simulation POMDP
Partially Observable Markov Decision Process (POMDP) is used to represent coded simulation. Formally POMDP is represented as a tuple M = (S, A, O, Ω2 R) where $ is a set of states, A is of actions, O is set of observations, T S x
- ((S) is transition probability distribution, Q: S (O) an observation function, and : S x S' -> is the reward model [
We aim to generate simulation from prompt Qiest In this paper, are particularly interested in the case where Quet comprises detailed design specifications such that the resulting simulation could be used to train agents, though method applies any prompt for defining simulation. In our experiments, Qieat is a paragraph of text around 10 sentences specifying this simulation.
'We omit the discount factor and the initial state distribution æ in the formulation for brevity. In our experiments, generated alongside the states S.

---

An illustraiive example FactorSim
Input:
Output:
estalled Rnccuutions te sinalcie
sacin anlleaaied lsee apanni butting
Jutraniee of Step kis)
Instrnation Steph (k))
Selraut Static (Pu)
ceteeren the ipwe wame trppan shen bollion
perzst
sileen state veristin)
agait.
Untevant Stadee: a [44]
deplmeent tee Basiiaan anno the celesase
codebase Ma
PRceert Cxpeediia listing 23
host
the Farteclard
(apereti Bisting
11307 Iappendic Ltarinn 5)
Selvoct Fraciion. TIA
Figure 2: An illustrative example of how the five main prompts in FactorSim correspond to formulation in Algorithm
Note that the function red puck_respar is retrieved as part of the context to Prompt 3, 4, and 5 because it modifies the state variable red red_puck_ lhcpuuusccttonn state variable LLM identified as relevant in prompt 2.
3.2 Chain of Thought
We first decompose the prompt Quxt into a series of steps using Chain of Thought [38., each describing module of the simulation to be implemented. Following similar formulation as in (2)., this can thought of as marginalizing over step-by-step plan variable (qı...., .9k) using N Monte Carlo samples:
(MM))))e))
N Σ
P(M' (0)
(8) 9K
where (0)")
. p(n...... .K Qrea); (1)
p is a probability estimation model (i.e., an LLM in our experiments), and M' the resulting code that fully specifies simulation. In practice, we only produce single plan N 1.
Intuitively, this process breaks the prompt into sub-tasks. After we sample such a plan K steps, generate the simulation progressively. Given existing POMDP M and natural language specification q, we update the POMDP to reflect the changes specified.
p(M (MM++199 . 4K)
II p(M qc)
where M8+1 is the POMDP (simulation as code) after the k-th step is implemented, and MK- is the final simulation. While Chain-of-Thought prompting allows LLMs to avoid having to generate code for all simulation logic at once, the complexity of each step still grows with A due to the expanding codebase. This task remains challenging because LLMs must comprehend the code and accurately identify where modifications are needed. Acknowledging the limited reasoning ability of LLMs, we ask: can we further decompose the [MM+1|MM 9k) into simpler distributions to reduce the complexity of each prompt?
3.3 Decomposition by Factorized Representation
Naively, we could further decompose step of the generation into several steps, each focused on generating different component of the POMDP:
p(M++1 MM qx)
=p(S++1 |M gx)-
p(T |S+11, qk).
(4)
(Ra++ |S++1, Tat1, Me,
(5)
p( 18 Te41: R Ma. 9s)
(6)
5

---

However, this still requires the LLMs to take the entire simulation (M₂) as context, which could be over hundreds of lines of code in our experiments. Empirically, we observe that many failed generations can be attributed to L.Mss attending to modifying parts of the input context unrelated to the prompt.
To reduce the input context needed for each generation step, we propose to use a factored POMDP representation remove the dependence on the full previous POMDP as context. For instance, given an existing simulation Ma of red, green, and blue agents, to implement the kth-step instruction qu: respawn the red agent when it collides with the blue agent, we only need context regarding the respawn logic of the red agent and the positions of the red and blue agents. Code regarding the green agent or the rendering logic would be unnecessary context.
To formalize our approach, we first introduce notation common to the literature
25 32 Suppose
have POMDP with state space factored into state variables $ S(1] ...5 and Z is subset of indices Z _ (1,2.....1) .n), we define the scope set S[z] == Øiez i Sli] as the state space spanned by the subset of state variables. For example, if Z = 1, 3, 4, then SiZ) defines a state space defined by S[1] x S[3] x S[4]. We denote state in the scoped state space S|Z| as s [Z]. Below, let us formally define a factored POMDP.
Definition 3.1. factored POMDP is a POMDP with both factored transition distribution and factored reward function. A transition probability distribution T' of a POMDP with discrete action space is factored over its state space S₁ x...S, S, with scopes Zi,..., Zm if, for all 8 E S, a E A there exist some {T.),= in the space of all possible transition distributions on the state space S and action space A, such that,
T(sss,a)
II T. (s oZ).
A reward function I of POMDP is factored over S S ....s, S, with scopes Zi,..., Zi if, for all E S, E A there exist some { (R) in the space of all possible reward functions the state space S and action space A, such that,
R(s,a) = = Ra (s
(8)
A factored POMDP can be represented factor graph f with two types of nodes: state variables (i.e., St) and factors (i.e., T; or Rt), functions of (state) variables. Our idea is to reduce context dependence by structuring the code using a factored POMDP representation and treat each generation step as expanding a factored POMDP with new state variables and new factors. During every step Ok. first select set of relevant state variable indices Zμ. Then, select existing factors that have overlapping scope with the selected set of state variables as context, which we denote T[Zx] and R Zz). That is, we can reduce the dependence on the previous simulation M₁.
and rewrite Equation 3 6 to the following:
p(M++1 |M gb)
9x)-
update state space
(9)
P( [7x] (8k+1.ga)-
identify relevant state variables
(10)
p( (TTx+1 [[[x], [Zz,,, A qx.))
update state transition function
(11)
p( R R[Zz), (Z,),,, gx)
update reward function
(12)
p(2 [Zk]
update partial observation function
(13)
Note that Zu can only consist of state variable indices in the state space SE+1 In practice, we achieve this by encouraging the LLM to select a minimal set of relevant states Z₁ in the prompt.
We find that the term I is most prone to eиror, likely because the most complicated functions of a simulation are state transitions. Motivated by this observation, we propose to adopt the model-view- controller design pattern for structuring these prompts. Instead of prompting LLMs to update the state transition function first and then update the reward function, we prompt the LLMs to update the action detioo-deeendeett part of the state transition function (i.e. the Controller component) and then the
MMore precisely, a factor graph of Dynamic Bayesian Network (DBN) M

---

Table 1: Percentage of system tests passed by different methods of generating 2D RL games.
of rystem possed.
Flappy Bird
Catcher
Snake
Pixelcopter
Pong
Puckworld
Waterworld
Monster Kong
Mistra--B..Issree
0.00
0.00
0.00
(1.00
0.00
0.90
0.00
0.00
Llama-3
L w/ self debug
0.15
0.41
0.28
0.19
0.03
0.44
0.22
0.31
Llama-3 CoT w/ self dehug
0.20
0.39
0.25
0.21
0.16
0.50
0.42
0.35
GPT-3.5
0.19
0.39
0.37
0,38
0.22
0.33
0.34
0.19
GPT-4
0.35
0.35
0.42
01.4
0.25
0.34
0.46
0.21
GPT-4 debug
GPT-4 w/ AgentCoder
0.18
0:45
0.27
0.43
0.43
0.33
0.20
0.23
GPT-4 CoT w/ self debug
0.30
0.51
0.39
0.53
0.64
0.47
0.50
0.34
LJamaa w/ FACTORSIM (ours)
0.55
0.54
0,30
0.41
0.38
0.58
0.27
0.35
GPT-4 w/ FACTORSIM (ours)
0.78
0.66
0.44
0.78
0.61
0.81
0.62
0.44
part (i.e., Model). We treat the reward model as part of the state transition function that updates a score state variable. That is, T(s'la a) = Tx (s' (s) (s|s, a) where T(a) (s]s, a) denotes the part of the state transition function that handles how actions affect the states and T()) (s'(s) denotes the part of the state transition function that how states are updated every step. This gives us our final algorithm as illustrated in Algorithm 1.
In Algorithm 1, colors indicate the corresponding components of the model-view-ccnntoolle pattern. Red highlights the controller, corresponding to parts of the state transition dependent on user-input Green shows the model, corresponding to parts of the state transition function that dependent on user-input actions. Blue shows the view component, updating the observation function that acts the "renderer" of the state space.
actions. Green
4 Experiments
In this paper, consider two types of simulations: 2D Reinforcement Learning (RL) games and robotics tasks in a physics engine. We also introduce a new benchmark to evaluate generative simulation methods. Our experiments are designed to test three hypotheses. First, FACTORSIM generates simulations with better prompt alignment, which we evaluate through system tests and human evaluations. Second, FACTORSIM enables better zero-shot transfer by training RL agents in the simulated generated environments. Third, FACTORSIM's strengths in generating robotic tasks.
4.1
RL Game Generation
To answer our first two hypotheses, we propose new benchmark that includes all 2D games from the PyGame Learning Environment 33: Flappy Bird, Catcher, Puckworld, Pixelcopter, Pong, Snake, Waterworld, and Monster Kong. For each RL game, the input prompt consists the game's online documentation. Since most game documentation is incomplete, we manually supplement them with additional details (see Appendix). This ensures that our method and the baselines do hallucinate any missing game information, allowing for fair evaluation across all methods.
Following common practices in game development, we design system tests to verify that the generated simulations follow the specified logic programmatically, These tests simulate actions like key presses and mouse clicks and check if the game states are updated correctly. Refer the Appendix for more details.
Baselines For baselines, compare to three methods using closed-source (GPT-4 M and an open-source LLM (Llama-3 (2]). The first approach prompts the LLM with all contexts at once, which we denote as the vanilla method. The second approach uses self-debuggnn [6], where the model retries generating the code when provided error messages from running the code (up to 10 times in experiments). A third approach combines Chain-of Thoughh (38) (CoT) reasoning with self-debugging, where the LLM generates code incrementally, processing one instruction at a time. Additionally, incorporate AgentCoder [15] baseline. CoT with self-debugging is an ablation study of our method that acts without the factored POMDP representation.
We exclude the sole 3D game Raycast Maze and leave 3D game generation future work.
7

---

GP14
Code Generation Evaluation
% of System Tests Possed
GPT- wu self delug
GPT a Cot wi debug GPT-4 su FactorSim
Table[[shove the results for the baselines and our method. FAC- ToRSIM outperforms all baselines in 7 out of games. Addition- ally, we compare performance and LLM token usage across various methods using GPT-4 (Figure D. While the vanilla baseline uses the fewest tokens, it only achieves moderate accu- racy. Additionally, combining Chain-of-Thouue (CoT) reason- ing with self-debugging results in the highest token usage but only marginally improves accuracy over iterative self-debugging, FACTORSIM achieves the highest ac- curacy with modest token usage, indicating that the decomposition of tasks reduces the need for extensive iterative debugging.
Flappy
Fteelloptat
Ponig
huckword
Wattrword
Kong
42000
BAN00
120:00
Token Usage
Figure 3: Performance and token usage analysis of GPT-4-based methods. Ellipses correspond to 90% confidence intervals for each algorithm, aggregated over all RL games.
Empirically, find that certain prompts, when tested baselines without the decomposition mechanism in FACTORSIM, are prone to syntax or runtime errors that the LLMs cannot self-debug. This is particularly evident with Llama-3 (vanilla) and Llama-3 self-debug, which perform poorly as they generate highly similar incorrect implementations, ignoring the logic specified in the prompts when the temperature is set to 1. We hypothesize that this behavior is due the model having strong prior for how certain games, like Pong and Flappy Bird, should be implemented, ignoring the prompt specifications. This "mode collapse" issue of LLMs could be caused by over-fitting in the instruction tuning stage [10
While AgentCoder iteratively refines code, it performs poorly because it relies on test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark. the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard "role-based" Chain of Thought decompositions, and that it is non-trivial to generate simulations from complex textual specifications.
Zero-shot Transfer Results Additionally, we test FACTORSIM by training PPO 27 agent on 10 generated environments for 10 million steps and zero-shot test it on the "ground- touudd-ttth" environment implemented in the original RL benchmark (Figure 4 The rewards are linearly scaled such that O corresponds to the performance of random policy and corresponds to the performance of PPO agent trained for 10 million steps on the "ground-truth" environment. FACTORSIM achieves notably better zero-shot transfer results as result of generating code that adheres more closely to the prompt specification. We also observe that the errors FACTORSIM made tend be more spread out across different components of the simulation. In contrast, many baselines suffer from failure modes concentrated specific aspect of the generation (e.g., incorrectly implementing the collision logic) that significantly hampers the ability of a set of generations to facilitate zero-shot transfer.
Human Study Evaluation Automated systems tests cannot holistically capture some aspects of game playability such rendering usable user interface. To address this limitation conducted human study where users were asked to play the generated games and evaluate their playability. Over 320 human evaluations (40 per game) we find FACTORSIM generates more functional and playable games, compared to the strongest baseline GPT-4 CoT with iterative self-debugging (Figure More details can found the Appendix.
4.2 Robotics Task Generation
We evaluate on GenSim's 35
50-task benchmark of robotics tasks the CLIPort framework [29 Refer to Figure 6 for an overview of our experimental setting. We compare FACTORSIM with the best-performing methods in generating code that specifies tasks (object states and reward structure) that can be used to train robots. Analogous the game generation experiment, use FACTORSIM

---

10
Training on testing envionments
Normallzzo lizd Roward
GPT4 w/ debug
GPT-4 CoT selu-
GPT-4 w/ FACTORSIM
61
waterworld
pixelcopter
Mappp
pong
monster
Figure 4: Zero-shot transfer results on previously unseen environments (i.e., environments in the original RL benchmark (33))
Pactorsie
Tasks that Factortin successfully generated but all othez baselines failed.
"Stck Boow Calime"?
Geoocated Talk
suioo Lap fost
Lartie rectication to
One tea os map
Sicget 1M0
Audzass Fanseed
Teart dhog fgem Pats
Diw Sardiadee Syllake Towr
NEM selatlec sTrate HAGEW wil U1RMOT
Figure 6: Left: overview of our robotics task generation experimental setting. Right: Tasks successfully generated using FactorSim, which all other baselines fail on.
to modularize the code generation process into subtasks and have LLMs generate each subtask using only set of necessary states as context. More details can be found in the Appendix.
Baselines & Metrics We compare our method with the multiple GenSim baselines: vanilla (one-shot), Chain-of-Thought (topdown). and Shain-of-Thoughh (bottom-up). Adopting the same set of metrics, we evaluate all methods on sequence of pass rates on "syntax correct- ness", cruntime-veriffes. A generated task is considered "completed" if a coded oracle agent could collect 99% of the total reward half of the time.
rully Playable
Unplayable rendered properly
Plyoble with soues
Unployable crashes es or fails to launch
CaT self-debug
Factorsin
28. 7%
and "task completed"
35 9%
11.2%
We empirically found that the "task completion rate" is an imperfect metric for evaluating the success of generated task. A task deemed "complete" by the oracle agent may fail to ad- here to the prompt. For example, when asked to generate task "build wheel," a method might produce a task specification that involves rearranging blocks into a structure that does not resemble wheel. To address this, we introduced a metric of the "human pass rate". This involved manually inspecting uutime-veriffe tasks to determine if the task specifications aligned with the prompt descriptions (see Appendix).
Figure 5: Human evaluation results on the gener- ated simulations of FACTORSIM and the strongest baseline (i.c., GPT-4 CoT w/ self-debug), aggre- gated over all RL games.

---

GPT44
LLama-3
Success
Syntox Pass Rate
Buniime POsS Rate
Back Completion Rafe
Pasz Rate
Systax Pasa Rate
Runline Posa Rate
Completton Rate
Humnn Poss Rate
Gensim vanilla
Gensim (topdown)
Gensim (bottomup)
FactorSim
Figure 7: Performance of FACTORSIM and GenSim 33 baselines in generating robotic tasks.
Task:
Build House
Walls
Place Chimney
Task:
Insert
Along
Square
1. Place Red Ell
Place Blue Ell
3. Place Green Ell
4. Place Yellow Ell
Figure 8: This figure illustrates two input task prompts and the corresponding sequence of subtasks decomposed by FACTORSIM.
Results & Discussion FACTORSIM outperforms baselines in generating tasks with higher runtime pass rate and better human evaluation scores, indicating improved prompt alignment (Figure 7 Task completion rates generally low for all methods due the limitation of the oracle agent. For example, tasks like "Build Ball Pit" (fill a container with balls to create a ball pit) often fail because the balls roll out of the visible area of the oracle agent, not because the generated task is invalid. FACTORS STM performs particularly well on tasks that specify spatial relationships (e.g.. "on top of." "left of," "outside of")) between objects, such the "build House" example in Figure 8 This improvement is likely due to the decomposition process, where for each step, instead of addressing combination of multiple spatial relations all once, FACTORSIM attends to a smaller context, allowing each spatial relation to be addressed separately.
Conclusion & Future Work
We have proposed FACTORSIM an approach to generate full simulations as code that can train agents while adhering to detailed design requirements specified text prompt. We also introduce benchmark suite of eight RL environments evaluate generative simulation methods.
Generating complex simulations in code is challenging, and anticipate numerous opportunities to extend the simulation generation process. There is substantial room to address larger-scale, more complex games, and robotics environments that require code bases beyond what can be used effectively in the context window of existing LLMs. We also see great potential to accelerate RL agent training by generating code that can be accelerated on GPU devices. Our robotic simulation results will benefit from further investigations to demonstrate transfer to wal-world environments. We have only addressed single-agent simulations, leaving the extension of our method to multi multi-agent settings to future work. In the future, we also plan to incorporate information from the agent training process to automatically modify the generated simulation environment for enhanced agent learning and generalization Taken together, we believe the generation of full simulations code will be an important step toward enhancing the capabilities of LLMSs to support the development of generalized RL agent policies.
10

---

Acknowledgments and Disclosure of Funding
This work in part supported by the Stanford Institute for Human- Cenaan-Centeeddd Artificial Intelligence (HAD), the Stanford Center for Integrated Facility Engineering (CIFE), NSF CCRI #2 #2120095, AFOSR YIP FA9550-23-1-0127, ONR 4-0014-23-1-2355, ONR YIP N00014-24-1-2117, and Google.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.00874, 2023.
[2] AI@Meta. Llama 3 model card. 2024.
[3] Asad Anjum, Yuting Li, Noelle Law, M Charity, and Julian Togelius. The Ink Splotch Effect: A case study ChatGPT as co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games, 2024.
[4] Craig Boutilier and David Poole. Computing optimal policies for partially observable decision processes using compact representations. In Proceedings of the National Conference on Artificial Intelligence, pages 1168-1195 1175, 1996.
[5] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint OrXiv:2402.15391, 2024.
[6] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In 12th International Conference on Learning Representations, 2024.
[7] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In 12th International Conference on Learning Representations, 2024.
[8] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[9] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 104104, 2023.
[10] Sil Hamilton. Detecting mode collapse in language models via narration. arXiv preprint vrXiv:2402.04477 2024.
[11] Eric A Hansen and Zhengzhu Feng. Dynamic programming for pomdps using factored state representation. In AIPS, pages 130-139. 2000.
[12] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[13] Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre. Theo- phane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4214-4226. PMLR, 18-24 Jul 2021.
[14] Anthony Hu. Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoscev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-I: 1: A generative world model for autonomous driving, 2023.
[15] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010 2023.

---

[16] Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. Gen2Sim: Scaling up robot learning in simulation with generative models, arXiv preprint arXiv:2310.18300 2023.
[17] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In 12th International Conference Leurning Representations, 2024.
[18] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
[19] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In 1/th International Conference on Learning Representations, 2023.
[20] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez. Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053-3062. PMLR, 2018.
[21] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni. and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157 2024.
[22] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[23] Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: Language model guided sim-to-real transfer. 2024.
[24] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimieGen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310. 17596, 2023.
[25] Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored MDPs. Advances in Neural Information Processing Systems, 27, 2014.
[26] Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint 170i
[28] Kensen Shi, Joey Hong. Manzil Zaheer, Pengcheng Yin, and Charles Sutton. ExeDec: Execution decomposition for compositional generalization in neural program synthesis. arXiv preprint vrXiv:2307.13883 2023.
[29] Mohit Shridhar, Lucas Manuelli, and Dicter Fox. CLIPort: What and where pathways for robotic manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 894 004.900... PMLR, 08-11 11 Nov 2022.
[30] David Silver, Thomas Hubert, Julian Schritwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, (2226419):1100-111 144, 2018
[31] Shyam Sudhakaran, Miguel González-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. MarioGPT: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36, 2023.
[32] István Szita and András I Lörinez. Optimistic initialization and greediness lead to polynomial time learning in factored MDPs. In Proceedings of the 26th annual international conference on machine learning, pages 1001-1008, 2009.
12

---

[33] Norman Tasfi.
Pygame learning environment. PyCame-Learning Environment, 2016.
https://github.c http:///gg.huub..ontttssttttttt cwt.tt/o/////////c/cc/cc/ccotttt
[34]
Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games, pages 1-8, 2023.
[35] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. GenSim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01356 2023.
[36] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistnncy improves chain of thought reasoning in language models. In / 1thh International Conference on Learning Representations, 2023.
[37] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang. Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint urXiv:2311.014555 2023,
[38] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. tfthout prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 244777 2022.
[39] Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, and Tsun-HSuun Wang. Towards foundation model for generalist robots: Diverse skill learning at scale via automated task and scene generation. arXiv preprint arXiv:2305. 10455, 2023.
[40] Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In 12th International Conference on Learning Representations, 2024.
[41] Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison- Burch, Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodied ai environments. CVPR, 2024.
[42] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[43] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[44] Abhay Zala*, Jaemin Cho Han Lin, Jaehong Yoon, and Mohit Bansal. EnvGen: Gen- erating and adapting environments via ilms for training embodied agents. arXiv preprint arXiv:2403.1 /2144 2024.
[45] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algo- rithmic reasoning with language models by composing decompositions. Advances in Neural Information Processing Systems, 36:31 -3553,,,,, 2023.
[46] Andy Zhou, Kai Yan, Michal Shlapentokh Siaapentokh-Rothman, Haohan Wang, and Yu- Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning language models, 2023.
13

---

A
Societal Impact
This work can be applied broadly many types of simulations, including roboties, autonomous vehicles, and other autonomous systems. Such systems have the potential for both positive and negative societal impact (e.g.. harmful dual use). As researchers, we must critically evaluate such applications and promote beneficial ones. In this work we have focused on simulations with potential positive social impact, particularly supporting the development of robots able to operate human environments like households or manufacturing facilities.
The methods we present generate simulations that can be used to train agents perform tasks. One risk with generated simulations is for training agents in an unintended manner. By generating simulations specified code we mitigate this concern by making the behavior of the simulation explicit and inspectable by humans. Further, our approach is better able to guide LLMs to generate code that matches input design specifications compared to baseline methods, reducing the risk of LLMs inadvertently producing undesirable functionality. We believe this can help enhance the reliability of generated simulations while offering strong editing and control capabilities to humans.
The potential negative environmental impact of the compute for using our technique is small. We have shown our technique consumes less tokens than comparable methods yield equally good results. Thus our method can be seen as a way to reduce computational needs when using I.I.ss for tasks like creating simulations. Compared to systems that use a neural world model our approach benefits from the relatively lower computational costs of running simulations in code compared to running large neural models for simulation.
B Additional details of our experiments
All experiments are done a workstation with 8 Nvidia A40 GPUs and 1008G of RAM. For our code generation experiments, one generation (i.e., generation of one training environment) takes around 30 seconds to 5 minutes. For our Reinforcement Learning experiments, one trial of training (i.e. training on a set of environments for 10M steps in total) takes around 3-5 hours to complete. In all of our experiments, GPT-4 refers to the OpenAl's "gpt-4- 11-11-166preeew model, GPT-3.5 refers to 1lama-3-70b-- 7--0b-innttrutt
OpenAl's "gpt-3.5-tur 3.5tttb-t model, and Llama-3 refers to the soer-soureed model that can be found on huggingface.
For the zero-shot transfer RL experiment, we supply methods with the reference "controller" (i.e., the same key press/mouse click leads to the same thing). We do this because language descriptions of such can be very ambiguous (e.g., a description "a key press leads the bird to flap its wings" can imply change in position, velocity, acceleration). In our experiments, we generate 10 environments and filter out those that cannot be properly executed with a random policy.
All the RL experiments are implemented in RLLib 20 The PPO agent is trained with batch size of 10,000, and an SGD minibatch size of 2048. Our agent used a fully connected network with hidden layers of sizes (4, 4) and post-FCNet hidden layers of size 16, all employing ReLU activation functions. The policy network uses an L.STM with a cell size of 64 incorporate previous actions but not previous rewards. Over the course of the 10 million training steps, 20 checkpoints were saved, with the best zero-shot performance on the testing environment reported.
C Additional details of FACTORSIM
We provide code in the supplementary material. Here we provide the prompts used in FACTORSIM.
Listing 1: The first decompositional prompt used in FACTORSIM.
Given unstructured
deacription, decompose the gane apecification into steps or modules. Each step or nodule should contain at most one input event transitional logic, and rendering logic. If on model the gane as a Narkov Decision Process (MDP), the steps, or nodules of the gane, should share little atate variables as posaible.
handling,
Please provide the response in the folloving format: <r jeon
14

---

"steps": [
"Step Describe first nodule/step",
"Step 2: Describe the second podule/step "Step Describe the third module/step";
1.
" "explanationa" "To make
deconposition don 't niss any important game machanica, explain where each module/stap fits in the gane ta logic."
0)
For exanple:
eeon
"steps": : [
" "Step Introduce that the balloon is by default alvays falling doun. Allay users gravity to nove the balloon with arrow keya.",
rendered
rectangle. Implement
"Step 2: Implenent scoring systen that rewards the human player for returning the ball back (i.e. making contact with the ball). Display the current score Diaplay
top of the reachea a score of and provide an option for the player to reatart the dieplayed."
Over!" neasage
player
gane
"Step S: Implement & "Gaee Over' screen that is displayed when the player reaches of 5. Allov the player
the from this screen."
J.
"explanationa". "The balloon
gane. and movement are handled in the first nodule. The scoring systen is implemented Over'
character
rendering
second nodule keep track logic is implemented in the last module to provide clear end to the gane."
player progress. The
>
The unstructured spacification of the gane 15: aggane..eeeiiicationn
Please provide the structured steps in the format of the JSON above.
Easure that each step is separable part of the gane that can be implemented as possible.
independently
You most likely don't need to decompose the gane inte more than 5 steps. Hovavar, the inportant thing implenentation.
steps accurately describe the gane
most important thing step (e.g., the balloon should not be abla to off the screan).
sure
decomposition
any logic
Note that the order the steps is the order that these modules vill be called in the
gane loop. Ensure that the gane described exauple, the reset position logic should be inplenented after the collision detection logic.
inplemented sequentially. For
Listing 2: The state context selection prompt used in FACTORSIM.
gane designer building eingle-playy gane Pygane by modeling gane Markov Decision Process (MDP).
relevant specific feature requested by the gane designer.
identify compile
implament
The gane already has the following state space implementation: er pythoo
import pygame
import sys
import randon
(etate_nanager_coe)
# nex variables will be added here: seriable_description
self Gelf.([(varabll__nnnn] = (tvartable
Please provide the state variables in the folloving format within a JSDN object: 1 json
(
H
"variable_nane" "Name of the variable',
)),
15

---

"aor_atate_
"variable_ deraaibeddecriptioon "Description
variable",
"variable_name" "Name of the variable', variable, one (iitt, float, str, baol, tuple
typp:" "Type listh}',
"variable_vall value": "Value of the variable, e.g. 100, 0.5, 'balloon', tuple ((256, ) True, [10, 50], [(('' 100, 'y': 2003}]",
gane designer's request {query).
Here are the dos and don'ta for this request:
The list "relevant_ seryraat_tttt_eariieee variables" should contain the of the existing variables that are relevant to the game designer 's request.
Please
variables that contains both existing variables that you think are relevant and new state variablea.
aingle
of
software engineer
request by implementing function takes these variables aa imput, ensure all the variables needed implement the request are included.
implement
It is okay include variables that don end up being used the implementation because redundant state variables will be filtered out later. rendering
there components be rendered. Color should never be white since the background is white.
provide
(a.g.,
provide Sprite, Surface, Don 't introduce variables using existing variables (e.g., self bird - self. pipe_eize/2), all state variables should be independent of each other.
variables.
handle
Always provide default value randonness will be implemented later.
variable should be chosen randonly. The
"variable_
Alvaya provide non-eupty default value so the softvare engineer can infer how the variable can be accessed. offecte up3)
empty
(e.g.r .png, jjgg) Prieritize reusing the existing state variables much as possible. For example, if xe have 'position_x' and "position_y" of character, do not give ne another variable "positions" list format.
not
inage
Additionally, you may add nas state variables to the 1ist necessary. Please only
if
if necessary.
Listing 3:
The prompt for the Controller component (as defined in Model-View-Contree utilized in the FACTORSIM.
gane designer Markov Decision Procesa (NDP). Your task is detect key/nouse input and update the state variables accordiugly according to a feature requested by the game designer. The gana the falloving inplementation already:
building songle-playee game Pygame modeling gane as
crypython
importt pygame
import
import
(state__amager_code
existing input event handling functions (existing irw.ring_iimleeentttiioo
logic function will
if the functioa is already implemented, it will replaced with the implementation
main():
State_asnager StateManager
running
chile running:
event
()
if pnaetiion_deerrpttion))
pygame... QUIT:
running False
[(function_aan=) (atnctionaaae))atttteaaaaaeeee event)
pygame. pygame.quit
if ..nane.. main__".
pygame.ing init() main ()
114
16

---

To ansure the implementation is correct, please also implneent an unit test for the function. Please implement folloving request fron gane designer your ansver in the folloving format: er jaon
"fuaction iaaetti "(function_aeze"
seaactio__eecripption "(fonction_ "function "function_
"the pygane implementation of the function, including the first line the function definition', "unit_test": "the unit teat cade for the function'
Hore dos and don 'ts for this request:
Note that the implementatin of the function shuold only have tuo argunents (1.e.
nanager
The function inplementation ahould involve checking user imput with event (1.a. avent.
type and event event.key).
Niminize the aumber of functions added while neeting the gane designer requirements.
Hovever, make sure to always give the full implenentation the function.
Include only the essential details requested that are not requested.
gane designer. Do add things
defined
state nanager.
introduce
variables.
Only KEYDOW! events will detected. Do not rely on KEYUP events. bounds
Check for indez bero Check far divide-by zero errors.
with any lists arrays being used. leave any code incomplete. denonstration code implementation. Be sure all code is fully implemented.
placeholder
provide
Listing The prompt for the Model component (as defined the Model- View-Controller) utilized in the FACTORSIM.
gane designer building single-player game Pygame by modeling gane as Markov Decision Process (NDP). Your task is to define and cade new state transition functions according to the feature requested by the game designer. The gane the folloving implementation already:
erypytoon
import pygame
import import
(stata_nomager_cod)
existing transition functions
function
if the function is already inplemented, it will replaced with the now inplementation
def main O):
sate__anager StateNanager running True
chila running:
event pygame. poll
event type pygane... QUIT: QUIT:
running
((function, peaactiio_deeccrpptiool (ffunctioa_nane) Pygame. quit()
if
mai........
pygame.ing init() main()
To ensure the inplementation is correct, please also implneent an unit test for the function. Please implement the felloving request fron the gane designer return your
the folloving fornat:
jeon [{
"fuaction_pame"" "[function_naze)"
ceneeciioodeecrppttion "function, iar.rioonnnmmmeetaatttaae "the pygame impienantiioon of the function, including the first line of the function definition', "unit_test"; "the unit test code for the function'
r4

---

Hare are the dos and don 'ts for this request:
Only implement things
pertain to updating the gane like input event handling and UI components will be handled separately. only
variables. ааpecta
assential detaila requested by gane deaigner. Do not things that are not requested.
These state transition functione will be called in every iteration of the main game add conditional the function itaelf.
loop. If you
the function, please implement in
Note functions.
added
transition
Please use the state variablea defined in the state nanager. Do not introduce new state variables.
Check for indez out bounds errore with any lists or arrays being used. Check far divide-Br Do not leave any code incomplete. Do not leave placeholder valuea. Do not provide demonstration
inplementation.
code fully implemented.
Listing 5: The prompt for the View component (as defined in the Model-View-Controlle utilized in the FACTORSIM.
The gane designer is building single player game in Pygame modeling this gane as Markov Decieion Process (NDP). Your tack is add rendering functions that decide how state variables rendered UI feature requested by the game designer. gana the following inplementation already: ppython
the according to the
import Fygame import import randon
(atate__nmager.code
existing rendering functions trender_coda)
the nex function vill be here
the function already implemented, it will replacad with the implementation
main():
state_asnager StateManager
clock pygame.tine. Clock()
running True
while running:
action Pygane. event event.poll() poll
if action.typ: type == pygane QUIT: running
state transitional logics
onitted for bravity
state_ _aate_nangerr... acrsen..rren.nr. fill ((255, 255, 255))
code rendering as components
['fuuctiio__dosciittta
diaplay.
state paateer clock. ticx(etate_manager:
Pygame. quit((
if _ane.__ -- "__main__":
pygame.init()
pygane... display.p....... sep_iin ("")
main()
ensure implementation correct, please implneent unit function. Please inplement the folloving request fron the gane designer and return your ansver in the following format:
114 jeon
(
"function io_eion_____;;____" "[function_name)",
"fuaction seaeecionndecrriptinn
"(function ('suctiio__dsccipptioaa.
"function_ itr.rioonnnnpleeetttttiieoe
pygane inplementation function, including first. line of the function derinition', "unit_test"; "the unit test code for the function'
)
18

---

CINEE TEAQUT Thsteaeeg
30h
60
Buili ampiPoo
376
305
376
576
375
100
800
80
305
88
376
18
110
305
370
Builfflrrdh
376
306
575
80
80
80
98
305
576
183
88
80
300
100
80
80
300
110
315
BulddBrdge
3.5
300
370
80
300
550
80
20
520
330
703
80
30
303
700
700
303
Aggragenal Rase
62%
3075
3374
15%
7574
374
125
05
92%
324
57 05
925
175
$25
075
Table 2: Additional Experimental Results: FactorSim outperforms other Chain Thought baselines by large margin assembly tasks. Syntax indicates the task passes the syntax check. Runtime indicates that the task can run in the physics simulator. Task indicates whether the task can be completed by the oracle agent. Human indicates whether the completed task matches the input prompt specification.
Here are the dos and don 'te for this request: Only implement things that pertain to hov state variables randered UI compenents on the screen. Other aspacts like input avent handling and atate transition will be handled separately.
Please make sure that all of the state variables ramain unchanged in the rendering functions.
Include only the essential details requested the designer. Do not add thinge that are not requested.
These rendering functions will be called every iteration of the main gane loop. It you vant to add conditional logic to the function, please implement it in the
function
Note that the background color of the screen is shite white UI companents will not be visible. Do not fill the screen with white again in the rendering functiona. functien will be added the end of the list of rendering functions. Please use the state variables defined in the state manager. Do not introduce new state variables.
Note that the
Check for index out of bounds errors with any lists or arrays being used. divide-by-aa zer- errors.
Do not call pygane. "ygame.display. set_mode UI functions. Only call it once outaide of any VI function that is called multiple times.
Do leave any code incomplete. demonstration code implementation. Be sure all coda is fully implemented.
leave placeholder values. Do provide
D Additional details and results for the robotics task generation experiment
In this section, we provide the prompts we used for FACTORSIM in the robotics task generation experiment. The prompts for the baselines can be found in the GenSim paper 35
To conduct human evaluation, we begin by observing the oracle agent attempting to solve the task. If the oracle agent successfully completes the task, then assess whether the resulting goal states align with the input task prompt. the oracle agent fails to solve the task, we investigate the reason for the failure. Often, the cause is apparent, such as the target container being too small or not having the right color of objects for the task. These are marked as failures. For cases where is clear that the limitation lies in the oracle agent's ability, when the reason for failure is not immediately apparent, we manually inspect the code for the task specification and base our decision on both the code and our observation of the oracle agent's attempt solving the task.
Listing 6: The chain of thought prompt.
The gane designer is building single -paayer game in Pygame modeling this gane as Markov Decision Process (NDP). Your task is datact kay/nause input update atate variables accordingly according to feature requested the gane designer. The gane has the following implementation already: er "yython import pygame
import sys import randon
(state_ Lecate_nanager_coee
"hnps://gthib.co///niwwwCCa
19

---

existing input handling functions ceristingg.iplemennttee
logic function
if the function is already implemented, it will be replaced with the nox implementation
def main():
stato_aenager StateNanager
running True
chile running:
pygame. poll
if srant.type =- pygame. QUIT:
running
([Eauttioo.ammo)))acttt..... Pygame. quit()
event)
_anee....... main.....
init( nain() To
please also implneant unit for function. Please implement the folloving request fron the gane designer and return folloving fornst:
the implenentation
your
erejan Jaon
{{
"function_nanee "(function_aaze)""
"function ceaection_ddette description":
"function seanctio_inplleee int.einnnnnneeettateeene
pygane inplementation function, including first line of the function definition', "unit_test";
code function'
}
Hare are the dos and don tt for this request:
Note that implementation
function shuold only
argunents (i.e.
state aansger and event).
function implementation should type and event key).
checking user input
(i.e. event.
Mininize the number of functione added while neeting the gane designer 's requirements,
Hovaver,
always give the full inplenentation the function.
Include only the essential details requested by the gane designer. Do not add things
not requested.
Please use the state variables defined in the state manager. Do not introduce new state
Only KEYDONN events zill be datacted. Do not roly KEYUP aventa. Check for index out bounds errors with any lists or arrays being used.
Check far divida-by
Do not leave any code incomplete. Do not leave placeholder values. Do not provide demonstration
implenentation.
code fully implemented.
Listing 7: The state change prompt.
Objective:=* Python progran generates variables
simulation tack. The state variables should be able to depict the final target environment
.+Tack Details:**
"TARGET_
Goal: "TAKG DESCRIPIITPT "TASX_ASSET.
><Reuurrmmntt:::
Return python function called "get_atatt_variablles variables' which returns list of StateVariableData objecte. should be able to take the result this function plug
should dieplay target environment will look like for the given tack. folloxs:
"StatolaarialeDDaaa dataclass
edataclaas
class StateVarisbleData::
variablo_deseripiion. str
varieble_uame: str varisble_urdf: str variable_aize: Tuple[float fleat, variable_color: str
srriable_tarett.ppe List[Tuple [Tople [float, float, float], Tuple [fleat, float, float]]]
criaabl_anourse str
20

---

Inportant
Iten sizes and pasitions must reflect realistic setting. Ensure logical spatial neither everlap nor float unnaturally. Account Items diuensions and sizes when deternining their placenent.
relationahips; itena
veriable_aamne is unique name for every state variable.
"variablo_size" should be list of 3 floats, representing the size the three dimension [x, 7, 2]. target,
object along
posea, containining the 3-element pasition vector, and 4-element quatarnion rotation. The position vectora should within the following boundarisa: I0.25, 0. fer the axie, (-0.5, 0.5] for the y-axie, and [0.01, 0.3] for the 2-axis. For single object, the target pase should only contain one elenent, like (1[0.52, 0.02, 0.001).
pose
[0, 0, illl. For multiple objects, the target pase like Itfo.52, 0.02, 0.001], to, 0, 0, il1, [[0.48, 0.02, 0.001l, [o, D, 0, 1111. something part conething the robotic agent should be noving. For example, if the task is ball into the container, the container should be static.
multiple items,
"static" describes
object
environment
*Example Input and Expected Dutput:
Taak: 'build_a_car
Goal: "Construct simple car structure using blocks and cylinders.* ("ber/box-teaplatee urdf", "eylinder/eylinder teaplate -eepllde--eeellateeeeeeee urdf") Expected output "rpython
def wel.ettt_variaallee.
List (StatttVaarraalleaaaal car_pose ((0.5, 0.0, 0.0), (0,0,0,1)) fixed poze
base_length 0.04
taee_target_poee [(utile (pery(car_poee (base_length base_length 0.001) ), car_pose [1]),
(utils apply(cars (car_poee, (-base_length / 2, base_length 2, 0.001)), car_pose [1])]
wheel_length [lutlle apply (car_pose,
wheel, vheel_ wargetrrggepae.
wheel_length / 2, vheel_length 2,
0.001)), car_pose [1]).
(utils. (utils..ppll
(-Kheel_lengte / 2, wheel_length 0.001)), car_pose [1]),
(utils. apply(car_pase
whees_lengthh / -whecl_langth 0.001)), car_pose [i]),
(utils apply pos__ (-sbeel_length / -Wheel_ 0000)), car_pose [1])]
return [
StateVariialeeData
aarbbbleddderrrphhssn*hhhhhh usad build the base",
variable_nnme=-bbass 0.02),
veriiabe__ize=-00.22
"ariablee_color=rrad""
uxrrable_.ergettppae-baa..
variable_anountt2.
stationary=False.
StateVariableData
"ariabll__descrippttoa===" to the base",
vartablee_nme=="hheal""
v4r100le..112--000022, 0.02, 0.02],
"ariablee.colo=="blle""""""
sagee__ppeereee___ggepp.... targett_oeeee stationary-False.
variable_anoust44
StateVariableDetac
variabble_ddscrtp:::onnmtb of the car",
variable__omne="bodd"""
variable_size-OD.66 0.02, 0.02),
http:aaardddbbbbbbb-aaaaaaaeee.d urdf", tartable_.olor==bblee
variable_.urdd-bborbbbx
[ variable_anoutt11 stationary=Palse.
variable.target.ppoar
-xExaple Input and Expected Dutput: Task: 'build_a_circle
21

---

Goal: "Construct circle using 6 red blocke"
URDFs: btcck//loc..urd
Expected output
"pyytonn
[aat[BaattVarriaaaaaaaaa]
block_size (0.04, 0.04, 0.04)
red_circle_pases circle_radsus 0.1
circle_center (0, block_ [2] angles = linspaca (0, nt. 6, andpoint=False)
circle_pose ((0.4, 0.3, 0.0), (0, 0, 1)) fixed pose
# Define initial and target pases for the red and blue circles. for angle in angles: pas (circle_centar [o] circle_radius up (opcgle)))))) circle_radius ep.ein(angle),
circle_center [i]
circle_centar [2])
block_pose (utils. apply applle.apply (circle.pose, pos), circle_pose [1])
red_circle_poeee append (block_pose)
return [
StateWarisbleData(
variabie__dsccriptton===- used to build the base",
"ari
varieble_cizo=block_.seee
varabbl..rdd-biic///ll....
"arsabbleccolo="rrad",
variable_anount
stationary=False.
)
Listing 8: The subtask code generation prompt.
Obbjettive:** designing training plan
environment to complete a task "AAKGET_TASK___ multiple
TASK, NAME This task will be completed in manages rebotic move the composition of assets from one state to another, ultimataly achieving the ideal state that completes the task.
from
this step, you asked generate aubtask function 'SUBTASK_NANE MAMME' Refer to the existing template and use the exiating generate Python caate_variable_fo.. Str Srror_sBBTSKK_AAA and ceubtark_codo_for_Bte Betr_foorSBBTS__AAAA and return the
involvea
your
state_
the given JSON format.
.+Task Overview:**
Nane: 'TARGET_TASK_
Taak Description: 'TARGET D
Subtaek Descriptions: 'TARGET
Current Subtask Name: 'SUBTASK_NAN
Dascription: "SURTASK_DESCRIPTIII
>*Eriating
SUBTASK SUSTASK_CODE,TEMTE TEMPLATE
>4bbtaak Requirenents:
Generate Python code addressing the specified eubtask based the provided description and initial variables.
Adhere guidelines:
specified you just reviezed, avoid libraries, already defined.
and conment on the code for clarity.
nanager
subtask
Subtaes function important Notes:**
Only one 'self, "self.add add_ should be used, and it should not be used on state variables static-True.
The 'natches' argument in the nethod called 'self, 'self, add_ should alvays be aumpy array.
Do not use libraries, functiona, and assets that don't 't knav.
not create additional functions inside the subtask function, only return one function.
Do not add state variables marked as "static Truc* to the anvironnent using fer. add_object',
22

---

make up any other functions from When passing color using 'env. add_abject', renember pass using C iten_color]. Good exanple is: 'eav. add_ baase_blcck_uee base base_block.pere. iten
color=utila rrrd)) Bad example ia: fenv add_eb urrr.r,, base_ base_block_pose. 'red')'
subtask code, you need create Initiel pose should NOT be very close to pase determines where the object will be placed before the treining starts. Target determines the ideal right in the simulation. You ahould priaritiza using the target_pose from the state nanager
pasa the same the target pose. Initial
target pase.
vhere the robot sill receive reward if placed
Only use get aat_rrandom_pos for initial pasa (pasitiio and rotation).. "get_randon.pose obj_size)* geta random colliaion- object pose within vorkspace bounda. param obj_size: (3, contains the object size in x,y,2 dimensione. return: translation (3, ), rotation (4, ). You should pass the obj_size directly to the got_randon_pose function, block_size). Bad exauple is: aelf rew.retttdddoopaaaeeeee. get_randon_ pose (block_eige)
self wtt...aaddo..ooseeee of single element. Good example #lock_size(0] or env.
It are generating need to check it using other helper function like 1i.ppoe__iihhn.boue copy it.
from 'gat_randon__oos posse" It will be within bounda. You don
Do not
'mp, copy'
It you need initial rotation, use roet_ to get the 'poze' first. Then use example is 'ratation
pasa [t] up. np. 'pose
De initiate it other ways, (pose res).. (3, 3)
environment
simulation.
DO NOT USE other unlieted way to create 'pose', In self add_gosl, make Each 'env. abjectt call will create a nav object id. If only one object is needed in "aelf.add_ge
to
1aaa..revrr-1.//SBBA.. COUFT"
this aubtask, then pasa objects are needed, create 1ist that contains all the needed object ids then pass 'obje object_id,liet' Nake sure you include all the arguments to aelf.add_goal 'obja', 'matchen', targ_poses', 'replace', 'rotations", 'metric', 'parans', 'step_nax_revard" langaage_goo
'abja [abject_id]
multiple
For 'self add_geal "s argument 'natches', it should be 'natches=pp anea((n, n)) 'n' represents total called & times then it should be set as 'natches-ep giren
'env', E.g. 'env objectt objecta added satchee-up.ones((( ones 4))*,
have unknovn variables.
variables
creating the subtask.
assume any
Only three functions are available frem 'utile'; atilssapply" 'utils. matt\22__t_oollrrte and ' utila',
include triple quotes ("**) your code, only 's' comments.
If the asset of this subtask involves 'zone', make zure that pose of the zone should for creating target position
not be noved, it's usually zone_size = [0.12, 0.12, urdf"
E.g:
sone gone_urdf 'zone/zono.. urde"
zone_colors ['yalloz', 'blus', 'green
sone_poses -]]
color zone_colors:
zone_poss (zone_urdt, zone_pose, 'fixed', color-utils.cOLORS (cooos add__dddecddeeeeeoeeceeeeeeceeed
gelff.get_randon.. pone zone_size)
zone_poses. pow.pppee..pppeed (zone__ooo)_)
If the subtask iavolves updating the ratation, you may call 'utils. www.tuuV_tt_oooeooooo tullexx22 function
1. Bere's
def quatXYZN_ to_t (qutte/iuuntriion.xoooco
pylint: disable-invalid quaternion 3-paraneter rotation.
ana Abstraction for converting
help us Quaternion should be in order for pybullet.
rotation paramatarization use.
Arga:
nuateraion_xy2v in xyzv order, tuple of 4 floate
Returns:
rotation: 3 3-paranater rotation, zyz order, tuple of floata
q quaterniin, xxzx
quaternion_ ([q g[0], q[1), [[2]])
euler_zzy euler. quateeuirreeeeeeeee ( (quaternion_ sxaterrion__xy,.. axes-' euler_xyz [ealer_ (1), ouler_zxy [2]. ealer_2xy [0]) return suler_xyz
**Exmplle Input and Expected Dutput:**
Task Nane: 'build_a_car'
Tark Description: 'Conetruct a simple car structure using blocks and cylinders.' "['uuidd_car_baee Build base
simulation
environnent.". "build_car body: Build the body of the car in the aimulation environnent." cuiid_car_whhelll Build the wheels the car in the simulation environment.""
23

---

Subtask Name: "baild_car__ee base' cas_
Subtask Description: 'Build the base of the car in the simalation environment. Existing template:
import numpy up
from paporttteess.... taak import Task
from cliport. import utile
class MyBuildCar (Task):
Construct simple car structure using blacks and eylinders."* "**
def _.init__ (self);
inper()...intt_....
initialize the
attributes that manage all the variables
this taak StateNanager ("pStaateWaaaeerr" (object,),
self. self.atate_managee = StataManager
[[O.5, 0.0, 0.01, 10, o. o, 111
self. "elf.ax__tepp 15
self.stare_aangger
sstare_anggrr.baoe..ii base_size (0.04, 0.08, 0.02)
self.state_nanngeer caaeecorrassecoorrr
"green"
self.state__mat hatiiaannnggrrbbaee... baserbbaeewwue base_urdf "box/box- urdr*
self.etate_msnagee: wwweer.bbae.... anout
self. Solf.state_managee anchor_bhaaaaaeeee
[((0.52, 0.02, 0.001), (0, 0, 1)),
((0.48, 0.02, 0.001), (0, 0, 0, 1))
self ool..additioaa_rreeet(
(placcholder..uuttss.ccom
reeef, env):
super reset (env)
self. wwwpeeefrret.stppeee.....
Expected output: "pyyhho build_ear car callde_ee_bbseeeeef (ssef env):
# Build the base of the car in the simulation anvironment. Let solve this problem byep-by-step.
# Step 1. Retrieve the (the car's pose, the base langth, and the base size) building.
initialize wawsger..aas..iiz. base_size base_size self state state__ base_ self.state_nanager statt._anager.bbae..cm base_color hase_ameunt aelf-atate__ane anchor_base.pp
base_amount
self, state_ anchor..bboorr.ooooorr
# Step 2. Setting the base block URDF. base, prte_aah_pp self state_nangge base_urdf
base_block,uref self fill_temmplate bbabbeblock_ureeeeeeeeebeeeeeeee ('DIM': base_size))
# Step Adding base blocks to the scene base_blocks
idx r2):
baae_block_poos self self rew.roo pose (eae (ase_size)
base_ base_block_id-
add_object (wwse_blockkuueeeeuebbeeeao block_urdf, base_block.pose. color=utils. COLORS("red base_blocks. append (base_block_idd
Step Setting the goal to create the base the car by positioning tea red blocks by
self, add_ obje-bace_blockt.
matches-np.ond ones ones ((base_
base_anount))
targ.poses-anchox hat.ssaaanccoo.aaeepeeeee..
replace-False,
rotationa=True,
netric="poae",
parans-Hoze.
ecap_nox.revard-l./
language.goal= Firstly,, create the base of the car positioning two red blacks side by side."
REFLECT_PROKT
**Output Format:*x
Use SSUBAASK_ NANE as the function name.
INPORTANT doublecheck your code folloving everything in inportant

---

E
should Additional details for the proposed generative simulation benchmark
This section provides the prompts for all 8 RL games in the benchmark.
Listing 9: The prompt for the game Catcher.
Create catcher character, represented rectangle, positioned at the betton and the middle of the screen.
Allow the player
control the catchar's horizental movement using the left and right arrov xeys on the keyboard.
There should alvaye
screen catcher. apeed can visually distinct and easily recognizable.
exactly one
steady pace constant increase gradually the game progresses.
the ball move dounwards
Detect collisions between the catcher and the ball. When the catcher catches a ball, player score, spaun left corner of the screen.
increnent
and diaplay this
top-
Give the player
is nissed by catcher and reaches botton of the screen, decrease the player 's life count by one. player's
lives.
gane when
reach zero. Display a
Over!" message
temporarily halt ganeplay but dont terminate the game.
Provide an optica for the player to click the acrean to restart the gane after the "Game is displayed.
Over" Continuously generate new balle after each catch or miss, ensuring endlese ganeplsy. Optionally, increase
gane's difficulty gradually by speeding up ball fall or reducing the size of the catcher as the player 's score increases.
Listing 10: The prompt for the game Flappy Bird.
character, visually represented as simple rectangle within gane vindow.
Introduce gravity, causing the bird to continuously fall zlovly. Allow the bird 'junp' accelerate upuards reaponse player 's click, temporarily overconing gravity.
Periodically ераип pairs vertical pipes noving from right left across screen. Each pair should have vary randonly.
for the bird to pass through, and their heights should
If the bird nakea contact with the ground, pipes goea above the top of the screen the gane is over
Impl ieeentnt the following scoring systen:
pipe passea through gains positive reward of +1. Each time a terminal state is reached it receives negative zevard
When the gane ends, display "Game Dver!" messagea and stop all the motion of the gane. Shox the currant score in the top-left corner of the screen during ganeplay. Ensure the game has predefined end and that pipes continue generate, maintaining consistent difficulty as the game progresses.
Listing 11: The prompt for the game Snake.
Create enake character represented
series of connected pizels blocks. Initially the anake should be a single block (1.e. the head) that noves in a specific direction
game vindov.
Allov the player to control the saake's novement using arros keys. The snake should be able to nove directly backuard. Eg.
abla turn
or right, if its moving dounwards it cannot nave up.
The novement
provides the snake should be continuous input. Ensure that the snake Implenent basic food systen where one food iten appears randonly on the screen.
the current direction until the player
When the easke consunes the
by noving over or colliding with it, increases, and the player earns points. It recieres positive rexard, +1, for each comes in contact with.
length
food
gatting
each terminal state
reaches.
If the head of the snake cones in contact with any of the valls or its oun body, the gane should end.
Incorporate scoring systen, displaying the current acore on the screea during ganeplay. should increase
snake consunes food.
Ensure that the gane has no predefined end, alloving the snake to continue groving and the difficulty to increase over time. New food itema ahould appear after the anake consunes one.
Provide an option for the player to restart the gane after it ende. Dieplay a "Restars" option the game sereon allos the player play again.
Listing 12:
The prompt for the game Pixelcopter.
Create a coptor character represented as large white square that remains fixed horizontally
descend vertically within gane
25

---

Introduce gravity mechanice, causing the copter continuously descend slovly. Enable the player to ascend whan the player clicks the mouse, alloving it to nomentarily counteract gravity
upuarda.
Create obstacles in the shape of cavern. Construct the cavern using series of top of boundaries of the gane vindos, triggaring the end of the gane upon collision.
vartically aligned rectangular barriera positioned both the botton the screen. Eusure the adjacent obstacles are of sinilar length to naintain consistently smooth "tunnel" effect.
Implenent collision detection to detect vhen the copter collides with obstacles the
Display
gane ends due to collision, balting all movenent within the gane and pronpting the player to restart.
Over!" nessage prominently
Create a scoring system that zevarda the player based how far the copter travela through the Show the current score in the top left area of the screen.
without colliding with obstacles.
Ensure the gane
predefined and and that maintaining consistant difficulty aa the gane progresses
obstacles continue generate,
Allow player start gane
collision.
Listing 13:
The prompt for the game Pong.
Create a paddle character for the human player, reprasented as rectangle, positioned on left side the Allow the hunan player to control the paddle's 's vertical novenent using the and dovn velocity added it allox smooth novenents. rectangle,
arrev keys. The paddle has Implement paddle character for the CPU opponent, also represented poaitioned on the opposite sida the screen.
Introduce ball that
with speed. The ball should bounce off the paddles and the top and batton walls of the game vindow. left right
If the ball goes off center and its direction.
its position the screen,
centrol its paddle's vertical movenent autonomously track ball. Detect collisions betzeen the ball and the paddles. When the ball collides with a paddle,
make it bounce off in the oppoaite direction.
Implement scoring systen that revards the human player for returning the ball back (i.e making contact with the bal).
Display the current score
top of screen. alloving for contiauous play.
gane
predefined end
Display 'Game for the player to reatart the gane after the "Game Over" screan is displayed.
message
player reaches score
provide option
Listing 14: The prompt for the game Puckworld.
an ageat character, visually represented as screen. The agent should be novable in any direction based on user input. Implement green that noves randonly around the acrean, serving as the target the agent to navigate tovards. Introduce red puck, a larger entity that slovly follows the agent 's novements on the
circle, positioned
Allov the player to control the agant's 's novement using arroz keys or another specified imput nethed.
Implement scoring systen that positively revards the agent proportionally to the closeneas betusen the agant and the grean dot, panalizes the agant for proxinity to the red puck.
Display the current score in the top-left corner of the screen during gameplay. Ensure the game Upon reaching the graen dot, rolacate it to neg randon position, maintaining the player.
predefined end, alloving for endless ganeplay.
challenge
Listing 15: The prompt for the game Waterworld.
Create a player character visually represented as a blue circle that can nove freely within the gana vindow using Introduce a dynamic environment with equal number of green and red circles. Make the green red circles nove randonly sround the screen.
keys.
Implenent scoring systen, where the player earns points for each green circle captured and deduct one for each red circle Display the
the top-left corner of the When the player captures a circle, nake it respaon in a randon location on the screan as either red green.
during gameplay.
Ensure the gane continues until all green circles have been captured. Once all green circles are captured, display
Over!" message stop all notion the gane
Provide an option for the player to restart the gane after it ends, creating a loop for continuous ganoplay.
Listing 16: The prompt for the game Monster Kong.
26

---

you to you Urite code for 2d gane viexed from the side where the character walk and jump. Let the character nove left or right using the 'a? and 'd' keys. Let the character jump key.
using
Create level by arranging stationary platforms above the ground. Nake sure the the platforn height.
character junp
Let the character stand the ground platforms but fall othervise. Start the player on the ground.
Add princess character that the character
reaching her position. Place top of the fireballs nove through the platforns. These fireballs serve as obstacles that the player nust avoid.
the princess on ane of the platforms.
Implenent fireballe
randon places
let
Touching fireball should deduct 25 points from the player's 's score and then to lose a life. The game ends if the player loses fifteen lives. Scatter points each.
randonly around game vindov that player collect for 8 princess randon Display the current score and remaining lives in the game vindov.
Axard player points platform when the player rescues her.
rescuing princess.
F Additional details for the human study experiment
In this section, we first provide details for the experiment and then the instructions we gave to human participants in our human study, along with the user interface. We also provide the detailed results of this evaluation for all games in Figure
asked to play and evaluate the generated games given the prompt while excluding factors such as aesthetics or difficulty. They rated the games on a scale of to 4, where indicates fully playable game, 3 is playable game with some bugs flaws that hinder gameplay experience, 2 is an unplayable game (i.e., no interactivity) with correctly rendered UI, and 1 is game that crashes fails to launch.
Human participants
Listing 17: The instructions we give to human participants to our human study.
Melcone your study! Your task Select the gane you vant to generate and click the button "Gencrate" to generate games. night wait
evaluate AI Al-generated games.
gane load for aaconda.
game intentionally sloxed doxn, making night react them!
Thus, whan you or preaa kay/batton, the
expected.
Please click "Randon Ganarate" to generate a gane.
If the gane doeen't 't load (black screea), gelect "1 unplayable." difficulty Please don 't take aesthetics into account.
Please not
gane.
want to assess whether the elementa are rendered accurately for gameplay purposes shile excluding considerations related sesthetics, everlapping, duplicated UI componenta.
Given the proupt as shovn, please judge the playability of the game. Pully Playable: significant buga.
game is generally playable from start
Playable but with sone flavs: the gane is sonexhat playable (interactable), but there issues (inaccurate logic glitches) that impair the gameplay.
2: Not playable: no interactivity but the VI seena to be rendered correctly. Unplaysble: the gane canaot started,
immedistely upon launch.
27

---

Figure 9: Human study results on all games.
Playability Ratings
Fully Playable
Playable
Unplayablle but rencered properly
Unplayable
Pong (FactorSim)
Pong (Baseline)
Waterworld (FactorSim)
Waterworld (Baseline)
10.0%
2539
29.2
APM
4.5%
27 8%
20
52 9%
9%
56.8%
Flappy Bird (FactorSim)
Flappy Bird (Baseline)
Puckworld (FactorSim)
Puckworld (Baseline)
35 8%
8.7%
0.5%
21.7%
17.8% d%
00%
10.2%
Monster Kong (hactorSim)
Monster Kong (Baseline)
Catcher (FactorSim)
Catcher (Baseline)
20.49
19%
35.09
41.0%
3.0%
30.0%
4.5%
17.0%
13.0%
54.5%
Snake (FactorSim)
Snake (Baseline)
PixelCopter (FactorSim)
PixelCopter (Baseline)
21.1%
0.7%
23.00
5%
22.99
8.2%
O 5%
22.2%
21.4%
15 its
41/4%
28

---

FactorSim: Generative Simulation via Factorized Representation
Prompt:
WoterWoood sually recresented dircie, with equalin, gresmand redddnederddte
gaino ucing brow keys.
introcucia dynarac
grown and red move randortly around screen.
Irplement a acoris spoams wtnew the planee warra po itta for euchgraan circec cactured and dioduct the for cuchiieldcccle..
Dirplty the ourrent coone the tap-left comor the screcn during parnppiiy.
When true
mare
acresnus
Driun contennt have captured ricen captumed, diapley "Gemne theer" meesagevanni allmotoo Frovide
Score: 0
Figure 10: Human study interface screenshot.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the abstract and the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer the experiment section and our supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
Please refer to the supplementary material and our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
coiioooooolcyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.cc/pulli//guiii wwwti//ggue////eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the supplementary material and our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::://///is/eeeeeeeee http:/pp///rrpps../ppppopo///opa httprpps..cppolllchhhhcieeieeeed
Question:
Answer: [Yes]
Justification: The research conducted in our paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all assets used in experiments.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Please refer to the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve sensitive data study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "MCEFt85gR1", "reviewer": "Reviewer_EKr6", "summary": "The paper introduces an LLM-based method for generting code for simulations. After generating the simulation of famous games based on their manual and description, the authors show that policies trained in these environments transfer well to the real games.", "questions": "I don't have any major questions wrt the work. There were a couple of points throughout the paper in the methods section, where I asked myself why this is relevant, but then this was cleared up a paragraph later.", "limitations": "The authors adequately address the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **S.1 Great results.** I think that the results from Fig.3 are very impressive. Zero-shot transfer is very hard, and doing so much more reliably than vanilla GPT-4 is impressive.\n- **S.2 Overall good idea.** The idea to deconstruct the game development task into M-V-C makes a lot of sense to me. I just thought that's already kind of captured in the POMDP formulation. \n- **S.3 Good presentation.** The overall presentation and writing are good, although there is much left to be desired in terms of implementation details.", "weaknesses": "- **W.1 Implementation details and relationship to formulas.** I'm happy that there was code provided with the submissio and I hope that it will be released publicly because based purely on the main body of the paper, the method is not reproducible. Including the prompts in the appendix helps but I wish they were commented a bit more on why certain phrases and sections are there. And I'm wondering if the theory that's presented in the paper holds water wrt the actual instructions in the appendix. Because as far as I understand, there aren't any restrictions on what code the LLM can generate for each component, right? Also, the paper mentions graphs every so often and I don't know how they fit into this. I also think the context selection is crucial to your method and from the main body of the paper, it's completely unclear how that's implemented.\n- **W.2 Missing examples.** Along a similar idea, I'd have loved some examples throughout the paper to illustrate what some of these instructions actually mean.\n- **W.3 Unclear robotic experiments.** The robotic experiments seem to be more of an afterthought in the paper, and it's unclear what existing assets there are, what control code is assumed given, what camera parameters are assumed, etc. \n- **W.4 Unclear input mapping.** The appendix mentions that the controller is given or that button presses always mean the same thing. I completely don't understand what's meant by that.\n\nOverall, I think the paper shows a great idea and is probably beneficial to the community, but some work should go into tweaking the main body of the paper and making the method more clear and reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing constructive feedback. We are glad you share our excitement about FactorSim's performance on the challenging task of zero-shot transfer. We appreciate your feedback and would like to address your questions.\n\n# Missing examples\nThank you for your great suggestion! Following your advice, we have included two new figures in the pdf file attached to our general response. One is to illustrate FactorSim with a concrete example, and the other will explain our robotics experiment in more detail. See more detail below.\n\n# The connection between FactorSim and the formulations\nWhile LLMs are free to generate code, our formulation in the paper (i.e., Algorithm 1) directly informs what is included in the prompt as context. To clearly illustrate how FactorSim's prompts are constructed and map to the theory presented in Algorithm 1, we made a new figure and attached it to the PDF of our general response. FactorSim uses five prompts, as highlighted in grey boxes in the newly added figure. The first prompt corresponds to Equation 1 in our main paper. It decomposes the long and complex simulation specification into a list of steps to be implemented sequentially. The output from that is a list of instructions. For example, a step instruction could be \"Introduce a blue agent rendered as a blue circle, allowing players to control it with arrow keys. The red puck should respawn when it collides with the blue agent.\" Then, at each step, FactorSim would add and modify the code of the existing codebase to incorporate the instructed change.\n\nDuring each step, FactorSim uses the second prompt (i.e. Context Selection Prompt, as shown in the figure) to select a set of state variables from the list of all state variables in the current codebase. This step corresponds to Equations 9 and 10 in Algorithm 1 in our main paper. The output of this prompt consists of a list of state variables, including state variables that are considered to be relevant to this module and new state variables that LLMs deem necessary to add to the system. After the list of context states is obtained, we will use it to retrieve the relevant functions in the codebase. A function is defined as relevant to a state variable if the state variable is used in the function. Our factorized POMDP representation essentially arranges a codebase into a (hyper-)graph where state variables are nodes and functions are hyperedges. After we obtain the list of relevant states and functions, they are used as context for the existing codebase in the following prompts the LLMs to generate the \"model,\" \"view,\" and \"controller\" part of the module. Note that not all steps require all three components; for instance, in our robotics experiment, only the \"model\" is generated as the \"view\" and \"controller\" are determined by the robot and the physics engine.\n\n# Clarification on our robotics experiment\n\nThe paper GenSim proposed a benchmark to evaluate how well LLMs can design and implement simulation tasks, as well as improve their performance in task generation. We follow the same experimental setup described in their paper, with the addition of a new metric—\"human pass rate\"—to assess prompt alignment.\n\nIn our experiments, we apply FactorSim to the benchmark proposed by the paper GenSim [2]. The benchmark, built on top of CLIPort [3], is designed to investigate how well Large Language Models (LLMs) can design and implement simulation tasks and how to improve their performance on task generation. This benchmark includes a list of tasks described in the following format:\n\n- task-name: build-car\n- task-description: Construct a simple car structure using blocks and cylinders.\n- assets-used: [block/block.urdf, ball/ball-template.urdf]\n\nThe assets needed are provided in the prompts. The robot control and camera parameters are assumed to be given. Refer to Figure 2 in the attached PDF for an overview of our robotics experiment. We apply FactorSim to generate robotics tasks in a very similar way to how we generate RL games, as both can be modeled as POMDPs. \n\nTo further demonstrate FactorSim's effectiveness in generating robotics tasks, we curate a list of 12 assembly-related tasks and show that FactorSim outperforms all Chain-of-Thought baselines used in GenSim by a large margin due to its ability to factorize the multi-step process into subtasks and generate these subtasks in separate steps, by using only the relevant context needed. Due to space limitations, please find more details in our response to reviewer 6Px5.\n\n# Unclear input mapping\nIn our zero-shot transfer experiment, we train RL agents on generated code and test them on the original game implementation in the Pygame Learning Environment benchmark (PLE) [1]. In the Pygame Learning Environment (PLE), an action is defined as a \"keyboard button.\" When we say, \"The same button always means the same thing,\" we mean that a specific keyboard key consistently maps to the same in-game action during RL agent training. For example, pressing the spacebar in Flappy Bird will always trigger the flap action in training and testing environments. We will rewrite the sentence in our paper for better clarity. Thank you for pointing it out!\n\n# Reproducibility\nWe are committed to releasing the code publicly. We will ensure that the code repository includes comprehensive documentation and examples to facilitate the reproduction of our results. We have sent our code, with instructions on how to run it, via an anonymous link to the AC, following the conference guidelines.\n\n### References\n\n[1] Tasfi, Norman. \"Pygame learning environment.\" GitHub repository (2016).\n\n[2] Wang, Lirui, et al. \"Gensim: Generating robotic simulation tasks via large language models.\" ICLR 2024\n\n[3] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. \"Cliport: What and where pathways for robotic manipulation.\" Conference on robot learning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9ZZyIxBtg", "reviewer": "Reviewer_p3c7", "summary": "The paper proposed a factorized approach to generate simulated games via LLM code synthesis. The code idea is that one doesn't need to generate the entire code at once, but rather generate different part of a POMDP game, such as controller, model, and view. The generated simulation allows RL policies to train on top. The authors introduced a benchmark to evaluate the proposed framework and show good results in terms of prompt alignment, transfer ability and human evaluation.", "questions": "In figures like figure 6, is the human pass rate based on the previous stage e.g. only executable code. \n\nIt seems that on open source model like llama 3, gensim with CoT is very close to factor sim. Can you explain the insights?", "limitations": "The author discussed the limitations of automated evaluation by conducting a human study. No outstanding negative societal impact.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper investigates an important problem, simulation generation. The evaluation over the mentioned environments is solid, spanning from automated tests to human evaluations.", "weaknesses": "1. The paper is poorly written. I have hands-on experience with almost all important concepts mentioned in the paper, yet still have a hard time understanding the paper, and have to read again and again including some code. Rather than talking about abstract terms like POMDP / factorization first, I think the authors can start easy with intuitions and explanations. The figure can also be improved. The main method figure shall spend more time showing what's special about \"Factored POMDP\" compared to prior methods. The benchmark claim should have its own section. The motivation is not clearly narrated either.  The world model section in related work doesn't seem to fit there.\n\n2. On of the main contributions the authors listed in the introduction section is a benchmark. However, I think this benchmark seems to lack the technical depth I was expecting as a standalone contribution. I feel it's just a set of small metrics rather than benchmark.\n\n3. The paper just lacks the level of technical contribution that meets my criteria for a Neurips paper. While there are many other prompting papers like CoT, ToT, the problem the paper is trying to solve is also very specific. \n\n4. While I have experience with both LLM and robotics, I believe the authors should not put Robotics as primary area, but NLP or code synthesis community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We are glad that you find this an important problem and that you find our evaluation solid and comprehensive.\n\n# Clarification of our Motivation and Novelty and why we chose Robotics as the primary area\nRecent advancements in foundational models have demonstrated their value in cognitive planning in robotics. The use of these models for obtaining lower-level policy is relatively underexplored, with many works focusing on direct control outputs [6, 7]. Our work leverages foundational models to generate simulations for training (i.e., Generative Simulation), a promising approach for training RL and robotics agents [1, 2]. However, LMs often fail at generating large amounts of complex code, leading to incorrect simulations and significant distributional shifts in downstream environments.\n\nBy representing our codebase as a factorized POMDP, we can decompose the code into simpler chunks, reducing the need for extensive context. This method improves the accuracy of generating simulations with complex logic, outperforming all Chain-of-Thought-based baselines in generating prompt-aligned coded simulations. In zero-shot transfer experiments, more accurate code generation for simulations proves pivotal for the downstream training of RL and robotics agents. FactorSim’s zero-shot performance reaches 50% of direct training results in 5 out of 8 environments, whereas baselines show significant transfer in only 1-2 environments. Given that most robotics tasks can be formulated as a POMDP, our method could significantly impact the robotics community.\n\n# Presentation\nThank you for your detailed feedback on our presentation. We understand the importance of legibility and have made the following improvements to our paper:\n- We included intuitive explanations for our method. We propose Factored POMDP representations to model an existing simulation codebase as a hypergraph, with nodes as state variables and hyperedges as functions relating to one or more state variables. Based on instructions for module additions, LLMs first select relevant state variables, then include only these variables and related functions in subsequent prompts.\n- We added a new figure to demonstrate how our formulation directly corresponds to prompt construction, with general explanations and concrete examples.\n- We included two new figures to explain our robotics task generation experiment and show visualizations of tasks that FactorSim successfully generates, which all other baselines fail to achieve.\n- We moved the world model part of the related section to the appendix.\n\n# Clarification on our Benchmark \nPygame Learning Environment [3] is a Reinforcement Learning benchmark used in various existing works [4,5]. We extend these environments to include prompts with detailed specifications of game logic paired with system tests to test for “precise” prompt adherence ability in simulation code generation. When coupled with the original RL environments, our extension enables the evaluation of not only the accuracy of the generated code but also its usefulness in solving RL tasks (i.e., zero-shot transfer).\n\n\n# Additional experiments on the benchmark\nWe additionally ran AgentCoder [8], a SOTA code generation method that uses a multi-agent system to perform CoT prompting, on our benchmark. \n| RL Env| GPT-4 w/ AgentCoder | GPT-4 w/ FactorSim |\n|----------------------|:----------------:|:----------------:|\n| Flappy Bird | 0.18 | 0.78 |\n| Catcher | 0.45 | 0.66 |\n| Snake | 0.27 | 0.44 |\n| Pixelcopter | 0.43 | 0.78 |\n| Pong | 0.43 | 0.61 |\n| Puckworld | 0.33 | 0.81 |\n| Waterworld | 0.20 | 0.62 |\n| Monster Kong | 0.23 | 0.44 |\n\nAdditionally, we added two Atari games to our benchmark to increase the difficulty of our tasks. Here are provide some preliminary results:\n\n| RL Env | GPT-4 | GPT-4 w/ self debug | GPT-4 w/ Factor Sim |\n|----------------------|:----------------:|:----------------:|:----------------:|\n| Breakout | 0.13 | 0.10 | 0.40 |\n| Space Invaders | 0.10 | 0.14 | 0.28 |\n\nThis benchmark allows us to investigate whether classical RL environments can be solved through the paradigm of generative simulation. \n\n# In figures like figure 6, is the human pass rate based on the previous stage e.g. only executable code.\n\nYes. These four metrics (i.e. \"syntax-correct\", \"runtime-verified\", \"task-completed\", \"human-pass\") have an incremental structure from syntax to runtime to successfully generate demonstrations to human verification where failing the former metrics implies failing on the latter metrics.\n\n# Explain the insights re: open source models’ performance\nIn our experiments, llama3 performs very well on generating specific tasks but exhibits instability, performing exceptionally on some tasks while poorly on others. Empirically, we observe that it resorts to \"memorization\" more, suggesting that if the memorized code is correct, it excels at generating those tasks.\nGenSim with CoT, both bottom-up and top-down, are competitive baselines that use a chain of 4-7 prompts. Despite this, FactorSim outperforms it, demonstrating its superior performance in task generation.\n\n\n[1] Xian, Zhou, et al. \"Towards generalist robots: A promising paradigm via generative simulation.\" \n\n[2] Katara, Pushkal, Zhou Xian, and Katerina Fragkiadaki. \"Gen2sim: Scaling up robot learning in simulation with generative models.\" \n\n[3] Tasfi, Norman. \"Pygame learning environment.\" GitHub repository (2016).\n\n[4] Riemer, Matthew, et al. \"Learning to learn without forgetting by maximizing transfer and minimizing interference.\" ICLR 2019.\n\n[5] Tucker, Aaron, Adam Gleave, and Stuart Russell. \"Inverse reinforcement learning for video games.\" \n\n[6] Wang, Yen-Jen, et al. \"Prompt a robot to walk with large language models.\"\n\n[7] Nasiriany, Soroush, et al. \"Pivot: Iterative visual prompting elicits actionable knowledge for vlms.\"\n\n[8] Huang, Dong, et al. \"Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2vEqc8TkT", "reviewer": "Reviewer_yVST", "summary": "The paper proposes a LLM prompting method to generate full game / robot simulations in code based on text descriptions.  Given a long text description, the method first utilizes an LLM to decompose it into multiple sentences, and then use them to iteratively generate and update simulation code. For each iteration, the code is generated and updated separately as three modules, i.e., controller, model and view. The update happens in a factorized way - the authors use the LLM to identify relevant state and context to avoid feeding the full generated code into LLM.\n\nIn experiments, the method is evaluated on game and robot simulation code generation benchmarks. The method shows superior results against other LLM prompting baselines in generating valid simulation code that aligns with text description.", "questions": "I'm curious what the failure modes of FactorSim is like.", "limitations": "See Weakness", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method exploits the structure of simulation to modularize and factorize code generation. This strategy significantly improves LLM's capability to generate full simulation code.\n- The method is comprehensively evaluated on game and robot benchmarks.\n- The paper is well written and easy to follow.", "weaknesses": "The major contribution of the paper seems to be a prompting technique crafted for the specific task of simulation code generation. While such a technique does improve performance on the task, my concern is it is neither fundamental nor sufficiently novel. The proposed prompting technique highlights two key designs:\n- modularize simulation code generation manually, which aligns with the common practice to manually decompose a complex task into sub-tasks for LLMs to handle more effectively.\n- extract relevant portion of code for LLM to consume and update, which is also an implementation-wise design that many works have already incorporated.\n\nWhile the paper writes factorized POMDP formulations, they don't seem to make a difference on how the prompting method is implemented. So I'm concerned that the contribution of this paper is more as a practical application rather than a general method or framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time to review our paper and for providing insightful feedback. We are glad to learn that you find our paper to be well-written and comprehensively evaluated.\n\n# Novelty\nWe present a novel method for generating coded simulations that allows for efficient context selection; our method outperforms the Chain-of-Thought baseline in generating complex code, e.g., RL environments by a significant margin (See Table 1 in the main paper) and robotics tasks (See Figure 3 in the main paper and Table 1 in the pdf attached to our general response). More specifically, our method differs from the existing Chain-of-Thought methods by introducing a principled way to select context from the codebase, based on the factorized POMDP representation. It can be applied to any code generation task where the target generation can be modeled as a Partially Observable Markov Decision Process (POMDP), which is a flexible representation. The key advantage of our proposed representation is that it helps us model dependencies between different state variables and functions, instead of having to consider the entire codebase when making modifications to it or having to retrieve context code snippets based on some pre-defined similarity scores (e.g., akin to Retrieval Augmented Generation). To further showcase the novelty and effectiveness of FactorSim, we ran an additional SOTA baseline that uses a \"role-based\" chain of thought to perform code generation.\n\n\n| RL Env| GPT-4 w/ AgentCoder | GPT-4 w/ FactorSim |\n|----------------------|:----------------:|:----------------:|\n| Flappy Bird | 0.18 | 0.78 |\n| Catcher | 0.45 | 0.66 |\n| Snake | 0.27 | 0.44 |\n| Pixelcopter | 0.43 | 0.78 |\n| Pong | 0.43 | 0.61 |\n| Puckworld | 0.33 | 0.81 |\n| Waterworld | 0.20 | 0.62 |\n| Monster Kong | 0.23 | 0.44 |\n\nWhile AgentCoder claims to refine code iteratively, it performs poorly because it relies on a test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark, the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard \"role-based\" Chain of Thought decompositions, and that it is non-trivial to generate simulations from complex textual specifications.\n\n# Failure modes and Limitations\nThe improvement of FactorSim stems from better context provided in the prompts, allowing the LLMs to be more focused during code generation. Below, we discuss one failure mode and one limitation of FactorSim. The main failure mode we observe with FactorSim is when the context is selected incorrectly, leading to incorrect implementation. However, we find that the benefit of FactorSim outweighs the occasional errors LLMs make when selecting contexts, as shown in our experiments.\n\nIn our experiment of generating robotics tasks, we find that all baselines, including our method, often ignore physical constraints necessary for the task to be completed by a robot. It is difficult for LLMs to consider context related to these \"constraints\" necessary for the task to be completed without being explicitly prompted. For example, if the LLM is prompted to generate the task \"build a bridge\", LLMs might generate a \"bridge\" block that is too small to span the two bottom items' distance. When the LLM is prompted to generate the task \"put the ball in the container,\" the generated task might consist of a base container that is much smaller than the size of the ball. We leave this to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldQp8QFXvj", "reviewer": "Reviewer_6Px5", "summary": "This work presents FACTORSIM, a framework that converts any language specification into a complete simulation for training RL agents. FACTORSIM decomposes the input prompt into steps and uses a factored Partially Observable Markov Decision Process (POMDP) to minimize the context needed for each generation step. It also introduces a method to benchmark generative simulation and demonstrate the capability for FACTORSIM to be used in robotics setting.", "questions": "I have one primary concern: how could this be applied to the field of robotics and embodied AI?\n\n**From this concern, several questions arise:**\n\n- How is the background (or scenario) generated within this pipeline? In a 2D game setting, detailed descriptions generate the scenarios, but this can become extremely tedious as scenes become more complex, such as in an embodied environment. While language-guided scene generation could be a solution, how will it fit into the POMDP framework?\n- The framework addresses three main components: controller, model, and view (rendering). In robotics, these aspects are typically handled by a physics simulation. How will this framework further contribute to the field of robotics? Currently, the paper shows potential for task generation in tabletop scenarios only.\n- I am still unclear on how the Robotics Task Generation part is achieved by this pipeline.\n\n**Some suggestions:**\n\n- While it might be challenging to address this concern with experiments during the rebuttal period, more discussion on approaches and challenges would be beneficial.\n- Any additional experiments that could demonstrate the pipeline's usage in robotics or embodied AI could help.", "limitations": "The limitations are addressed adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Develop a robust pipeline for constructing game environments from language descriptions, which could significantly enhance the scalability of training generalist agents.\n2. Formalize the framework as a Partially Observable Markov Decision Process (POMDP), reducing the need for full context during generation and improving outcomes.\n3. Demonstrate the potential of this method to generalize to embodied scenarios.", "weaknesses": "1. The evidence for generalizing to embodied scenario is limited.\n2. The successful rate in Table 1 and Figure 3 is low. Could there be some potential way to improve it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are glad that you share the excitement of the implication of this to the potential of scaling up the training of embodied generalist agents.\n\n# How can this be applied to the field of robotics and embodied AI?\nWe present a novel method for factorizing simulation codebases, based on factorized POMDP representation, which allows for efficient context selection. Since most robotics tasks can be formulated as a POMDP, we believe that our method can beneficial in the field of robotics and embodied specifically in assembly related tasks at they require many dependent steps. To further demonstrate the superiority of our method, we also conduct additional experiments on 12 assembly-related tasks where FactorSim outperforms all Chain-of-Thought baselines used in GenSim by a large margin due to its ability to factorize the multi-step process into subtasks and generate these subtasks in separate steps, by using only the relevant context needed. Please find more details in the following section.\n\n# Details on the robotics experiment\n\nRefer to Figure 2 in the newly attached pdf for an overview of our robotics experiment. Below, we explain our experimental settings in more detail and how we apply FactorSim to generate robotics tasks.\n\nThe paper GenSim [1] proposed a benchmark to investigate how well Large Language Models (LLMs) can design and implement simulation tasks and how to improve their performance on task generation. This benchmark includes a list of tasks described in the following format:\n\n- Task: Build-Car\n- task-name: build-car\n- task-description: Construct a simple car structure using blocks and cylinders.\nassets-used: [block/block.urdf, ball/ball-template.urdf]\n\nAfter the task code is generated, GenSim builds on top of CLIPort, evaluates the tasks by feeding them through a series of checkers that validate \"syntax-correctness,\" \"runtime-verification,\" and \"task-completion.\"We followed the same experimental setting as described in their paper, except that we introduced an additional metric, \"human-pass-rate\" to measure for prompt alignment. This “human pass rate” checks whether the generated task adheres to the input prompt upon completion.\n\nFactorSim achieved state-of-the-art performance on their benchmark. To clarify how we apply FactorSim to the benchmark, we first describe how the POMDP is defined in this setting:\n\n- States: object states in the environment (color, object pose, size, scale)\n- Reward function: a set of functions that define how the rewards are given (e..g., a reward of X is given when the blue cube is in the bowl). This is analogous to the scoring functions are generated in the case of RL game generation.\n- Transitional function and Observation function: The underlying physics simulation and the robot assumed in CLIPort handle these aspects, so we don't generate them.\n\nWe apologize if we didn’t make it clear how we applied FactorSim to generate robotic tasks. To summarize, a robotic task is no different than a RL game in that the final robotic task can be modeled as a POMDP. While the transition dynamics in this case are often handled by a physics simulator, we still need to generate the “reward function.” The only part left to generate is essentially the reward function, in the form of goal states. Objects in the scene and their arrangements are essentially state variables, and the functions are essentially functions that operate on these state variables to define whether a state should be given rewards. This is analogous to how in RL games we implement a reward function that takes in a set of state variables and updates them.\n\n# Additional robotics task generation experiment\nTo further showcase FactorSim's effectiveness in generating robotics tasks, we form a list of 12 assembly tasks (i.e. build related tasks) and by combining the assembly tasks in GenSim's benchmark and by asking GPT-4 to generate additional assembly tasks such as \"BuildLampPost\" or \"BuildDogHouse\". As shown in Table 1 of the attached pdf to our general response, FactorSim outperforms all baselines by a large margin, including GenSim's two Chain-of-Thought baselines. We also provide visualizations of tasks that FactorSim is able to generate, demonstrating collection from using the oracle agent in the accompanying figure. These tasks use primitive assets like blocks and balls to form more complicated structures. We believe that scene generation fits into the POMDP framework, similar to what we achieved in our robotics task generation experiment.\n\n# Can we improve upon the results in Table 1 and Figure 3\nThe low success rates testify to the difficulty of the tasks in the benchmark. We additionally ran AgentCoder [1], a SOTA code generation method that uses a multi-agent system to perform CoT prompting, on our benchmark. Please find the results in the General Response. While AgentCoder claims to refine code iteratively, it performs poorly because it relies on a test designer agent and a test executor agent to write quality test cases. However, due to the complexity of the tasks in our benchmark, the test designer agent tends to write incorrect or infeasible tests, leading to negative feedback. This points to FactorSim being an improvement over the standard \"role-based\" Chain of \nThought decompositions.\n\n# How is the background generated?\nIn Pygame learning environments, the game states are represented as a non-visual state, as exemplified by the documentation of the Catcher game: https://pygame-learning-environment.readthedocs.io/en/latest/user/games/catcher.html.\n\n[1] Huang, Dong, et al. \"Agentcoder: Multi-agent-based code generation with iterative testing and optimisation.\" arXiv preprint arXiv:2312.13010 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the insightful feedback provided on our paper. We are encouraged to find that the reviewers found our paper well presented, comprehensively evaluated, achieved impressive zero-shot results, and shared our excitement about its applicability to training generalized agents. Below, we address some of the points raised by more than one reviewer and summarize the improvements we have made. We respond individually to points specific to each reviewer.

# Clarification of our contribution
With our goal of generating simulations that include long and complex prompt logic for downstream training in mind, our approach is to generate code for various simulation modules step-by-step, modifying and expanding the codebase. Every prompt has two components: the context (i.e., the existing codebase and how the new function will be used in conjunction with it) and the task specification (i.e., what to implement). Existing works showed that Retrieval-Augmented Generation can improve LLM’s performance, while irrelevant contexts can hurt [1,2]. FactorSim constructs the "context" portion of the prompts dynamically in a principled manner. While we agree that task decomposition or Chain-of-Thought (CoT) are proposals of existing work, our method for factorization and representation of dependencies for such code generation tasks are novel to the best of our knowledge. 

To clarify our method, we provide a new figure in the pdf, illustrating how the formulation in Algorithm 1 directly corresponds to how the prompts are constructed. FactorSim consists of five prompts, highlighted in gray boxes. FactorSim first decomposes the long and complex prompt into a series of steps. Then, at each step, FactorSim modified the codebase according to the step instruction. If the codebase is written in an object-oriented paradigm, almost the entire codebase (i.e., 120 lines of code) would have to be included in the prompt as context since this logic pertains to many entities in the game. Instead, FactorSim maintains a factorized POMDP representation of the codebase, which is essentially a graph with state variables as nodes and functions as hyperedges:

| Name| Type|
|-|- |
|score| State Variable  |
| game_over| State Variable |
| green_puck_position| State Variable |
| red_puck_speed| State Variable|
| red_puck_radius| State Variable |
| blue_agent_position| State Variable  |
| …   | State Variable  |
| red_puck_respawn| Function |
| check_game_over_condition | Function|
| …| Function |

FactorSim uses PROMPT 2 to select a set of relevant state variables and then retrieve functions that pertain to the set of relevant context state variables according to the maintained graph structure. FactorSim constructs the “context” portion of the following code generation prompts with just the relevant state variables and the functions that modify one or more of the relevant state variables. In this example, instead of providing 120 lines of the codebase to the "context" part of the prompt, FactorSim only needs to include around 23 lines of code to the code generation prompts (i.e., prompt 3, 4, and 5 in Figure 1). Refer to Figure 1 for an overview of FactorSim, alongside an illustrative example.

# Broader Implications of our contribution
We believe in the significance of our work and FactorSim's broader implications. First, we demonstrate LLMs' ability to self-refine their prompt contexts for improved performance, unlike existing works that rely on similarity-based metrics to retrieve relevant code context [4]. 
Second, we believe the result of our zero-shot transfer experiment has important implications. Factorsim's zero-shot performance achieves 50% of the performance of training directly on the testing environments in 5 out of 8 Reinforcement Learning environments, compared to the best baseline's 1-2 out of 8. This suggests the importance of automated task generation and its potential for scaling up the training of generalist embodied agents.

# Improvements we have made:
1. **New Figure**: include a new figure to explain FactorSim, with concrete examples, and how our formulation directly corresponds to how the prompts are constructed. Refer to Figure 1 in the pdf.

2. **Additional code generation experiments**: We include a comparison between FactorSim and the SOTA code generation method (i.e., AgentCoder [3]). More detail on the experimental result can be found in our response to reviewer p3c7.

|Env|GPT-4 w/ AgentCoder| GPT-4 w/ FactorSim|
|-|:-:|:-:|
|Flappy Bird|0.18|0.78 
|Catcher|0.45|0.66|
|Snake|0.27|0.44| 
|Pixelcopter|0.43 | 0.78 |
|Pong | 0.43 | 0.61 |
|Puckworld | 0.33 | 0.81 |
|Waterworld | 0.20 | 0.62 |
|Monster Kong | 0.23 | 0.44 |


3. **Details on our robotics experiments**: We include a new figure that overviews how our robotics experiments are conducted and how FactorSim is utilized. Refer to our response to reviewer 6Px5.

4. **Additional robotics task generation experiments**: refer to Table 1 in the pdf and our response to reviewer 6Px5.

5. **Benchmark Extension**: We have extended our benchmark to include two classical Atari RL environments (i.e., breakout and space_invaders), which are included in the code we submitted. Results can be found in our response to reviewer p3c7.

6. **Reproducibility** To mitigate concerns regarding reproducibility, we have also sent an anonymous link to our codebase with instructions on how to run our code to the AC in an official comment, following the conference guidelines.

References:

[1] Shi, Freda, et al. "Large language models can be easily distracted by irrelevant context."

[2] Levy, Mosh, Alon Jacoby, and Yoav Goldberg. "Same task, more tokens: the impact of input length on the reasoning performance of large language models."

[3] Huang, Dong, et al. "Agentcoder: Multi-agent-based code generation with iterative testing and optimisation."

[4] Zhang, Fengji, et al. "Repocoder: Repository-level code completion through iterative retrieval and generation."

Meta-review: This paper introduces FACTORSIM, an LLM-based method for generating simulation code from language input. The core concept is to iteratively generate and update different parts of a POMDP game, such as the controller, model, and view.

**strengths**

* Importance of the problem that this paper tackles and interesting idea. 

* Extensive evaluation and impressive results including zero-shot transfer.

**weaknesses/suggestions**

* Clarification on robotics applications: Although the original explanation of the robotic experiments was unclear, the authors have addressed the concerns raised by several reviewers.

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera-ready version.
Decision: Accept (poster)