ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games
Authors: Ruoyao Wang
Graham Todd
Xingdi Yuan
Ziang Xiao
Marc-Alexandre Côté
Peter Jansen
Venue: EMNLP 2023 Main (2023)
Keywords: text games
code generation
simulation
TL;DR: We assess language models' ability to create world models via text-game generation, introducing ByteSized32, a corpus of Python text games for testing this. GPT-4 produced runnable games in 27% of cases, with potential for future improvement.

Abstract: In this work we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks.  We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code.  To facilitate this task, we introduce ByteSized32, a corpus of 32 reasoning-focused text games totalling 20k lines of Python code.  We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases.  When allowed to self-reflect on program errors, game runnability substantially increases to 58%.  While evaluating simulation fidelity is labor intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high-degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.

BYTESIZED32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games
Ruoyao Wang', Graham Toddf, Xingdi Yuan o Ziang Xiao
Mare-Alexandre Côté Peter Jansen
University of Arizona New York University
Microsooft Research Montréal
Johns Hopkins University {ruoyaowang, pejansen)@arizona. edu
gdrtoddenyu. edu {eric. yuan, ziangxian ao, macote eemi Gemi crosoft.comm
Abstract
Input Prompt
this work we investigate the capacity of language models to generate explicit, inter- pretable, and interactive world models of sci- entific and common-senss reasoning tasks. We operationalize this text games, expressed as hundreds of lines PYTHON code. To facilitate this task, we introduce BYTESIZED32!,
ByteSized32 Game
Task Specification
task of generating
SD
GPT-4
corpus of 32 reasoning-focused text games totalling 20k lines of PYTHON code. We empirically demon- strate that GPT-4 can use these games as tem- plates for single-shot in-context learning, suc- cessfully producing runnable games on unseen topics in 28% of cases. When allowed to self- reflect on program errors, game runnability substantially increases 57%. While evalu- ating simulation fidelity is labor intensive, we introduce a suite automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability. showing a high-degree of agreement with expert human ratings. We pose this as challenge task to spur further development at the juneture of world modeling and code generation.
Generated Game
Reflection
Evaluation
Validity
Compliance
Winnability
Alignment
Figure 1: An overview of our text game generation and evaluation process. The model, here GPT-4, generates game using in-context learning with prompt consisting of (1) a single highly- templattd example game, and (2) the task specification for the target game to generate. Generated games are self-reflected by providing the model with error output from PYTHON interpreter that detects syntactic and API issues. The generated game is then evaluated by an automated evaluation harness, as well as manually by human evaluators, measure its technical validity, specification compliance, physical reality alignment, and winnability.
1 Introduction
Simulating the world through mental models is a crucial component of human problem solving, in- ference, and cognition (Barsalou, 1999; Buckner and Carroll, 2007; Addis et al., 2009). Large lan- guage models (LLMs) have demonstrated precur- sors of this ability, such as encoding wide range of common-seae sense world knowledge from their train- ing data (Li et al., 2022a). Similarly, LLMs have been used directly as interactive world simulators in text games like AI DUNGEON (Walton, 2019), where their capacity to predict tokens context is leveraged to convert natural language user inputs (e.g. open treasure chest) into plausible environ- mental observations (e.g. you open the chest and find within glittering sword).
In this work, we examine instead whether LLMs can be used generate explicit and task-specific world models expressed as code, providing more formal and interpretable method to examine a model's 's world knowledge. We operationalize this as a problem of generating the complete PYTHON source code of an interactive text game that centers around a particular common-sent task, such as washing dishes with dishwasher building a campfire. Although interactive multi-step simu- lation of even modest tasks typically requires sev- eral hundred lines of code, we show that it pos- sible for LLMs to generate these simulations us- ing single-shot in-context learning. This is accom-
Code: gi thub.

---

plished by providing the heavily-templlaed source code of an existing text game as imput, and tasking models with adapting the template to novel spec- ification, as shown in Figure The template pro- vides a consistent, scalable, and general-purpose code architecture by hierarchically decomposing the simulation into object classes and sub-classes (e.g. device and container), which can be instan- tiated to make specific game objects (e.g. stove and jug). The template also offers example imple- mentations of common actions (e.g. acti activating devices or opening containers) and scoring func- tions that automatically detect task progress. The contributions of this work are:
for studying embodied agents, owing their rel- ative simplicity compared to full 3D simulations and ability to model complex and abstract tasks (Jansen, 2021; Li et al., 2021). While early text game research focused on testing agents on a small set of extant "interactive fiction" games like Zork, recent approaches have leaned towards procedu- rally generating a wider set of simple text-based games in order to evaluate agents' ability to gen- eralize (Côté et al., 2018; Urbanek al., 2019; Shridhar al., 2020; Wang et al., 2022). These frameworks typically rely on hand-crafted rules and templates programmatically arranged novel configurations, though some efforts leverage exter- nal data sources (Barros et al., 201 and generative language models (Fan et al., 2019) as well. In con- trast, in this work require models to produce novel text game as complete program, expressed as PYTHON code, using only a single existing game for reference.
We present BYTESIZED32, a corpus of 32 world models (expressed as text games in PYTHON) cen- tered around tasks that require common-sense rea- soning. The corpus includes 20k lines of code (including detailed comments), and is suitable for both in-context learning producing fine-tuned models.
Code Generation: As large language models have become more capable, interest in their ability to generate working snippets of program code has only grown. Several recent datasets have been proposed to facilitate this research, covering a wide range of programming languages and prob- lem types (Yu et al., 2018; Lin et al., 2018; Austin et al., 2021; Chen et al., 2021). Contemporane- ously, improvements in model architccture and training have led to impressive gains in code gen- eration (Chen et al., 2021; : Nijkamp et al., 2022; Li et al., 2022b; Fried et al., 2023). The GPT- 4 language model (OpenAI, 2023). in particular, has sparked an interest in the use of prompting for code generation tasks, technique which has led to advancements problem decomposition (Pourreza and Rafiei, 2023) and self-debugging by reflect- ing on errors (Chen et al., 2023; Olausson et al., 2023). Despite these gains, however, existing code generation benchmarks tend to require short and relatively simple programs. In contrast, here mod- els must generate hundreds of lines of PYTHON code to generate complete and accurate task simu- lations. Similarly, we show that self-reflection substantially increase the runnability of even large simulations.
2. We develop a suite of metrics to assess the qual- ity of generated games, including measuring tech- nical aspects of the code, whether a game contains required content, how accurately game models the physical world, and whether game winnable. We show that most of these metrics can be auto- mated with a high agreement to gold human ratings, dramatically reducing the manual labor required to evaluate mener-generated simulations.
3. We show that a model with large input con- text, GPT-4, can produce runnable text games for unseen tasks in 28% of cases using in-context learn- ing alone. When allowed self-reflect on its own generated code combined with PYTHON interpreter errors that assess syntax issues or API compliance, the model dramatically increases performance, gen- erating runnable simulations in 57% cases.
4. We empirically demonstrate that while cur- rent best-generated games frequently include task- critical objects and actions, they only accurately model the physical world 51 % cases, while be- ing winnable in only 38% of cases. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.
3 The BYTESIZED32 Corpus
2
Related Work
To support the task of generating simulations the form of text games, we construct a corpus of highly-templated text games written in PYTHON
Text Games and Virtual Environments: Inter- active text environments are an attractive choice

---

BYTES1ZED32 Corpus Statistics (per game)
Playthrough of a training set game
Lines of PYTHON code
618.1
Task Descriptin: Your task boll water.
Lines of comments
198.1
Tokens
6792
Observation:
yourself in kitchen.
you see:
yoursell
Action verbs
9.8
stove that currently and has nothing sink that is emply
Valid actions
306.6
Object classes
5.4
pot that is empty peanut butter an orange
Object instances
7.4
Expert path length
12.8
Type hhlpp' for list of possible actions. pot
Total Games
32
Table Corpus statistics of BYTESIZED32. Values rep- resent average values per game. Tokenization includes comments and was performed with tiktoken.
The pot is removed from the kitchen. You put the pot in inventory. > put pot in sink
The pot is removed from the agent. The pot placed in the sink. exanine
sink that contains the following items: pot emply
that can as examples in a sho generation paradigm. Each game is between 500 and 1000 lines of code (including comments), and encodes task which requires either scientific or common- sense reasoning to complete. For examplc, a game might require an agent to apply a bandage, load dishwasher, or make a campfire. Each game has two components: the game code itself, as well as detailed task specification in the form of structured comments at the top of each game that provide detailed summary of the game task, critical objects, actions, distractors, and a high-level solution plan. These components are described below, with cor- pus summary statistics provided in Table 1, and an example game playthrough shown in Figure 2.
> turn on sink
The sink is now turned on. examine
a sink that contains the following items: pot looks
some
> turn off sink
The sink is now turned off. take
The pot is removed from the sink. You pul the pol in your inventory. put pet steve
The pot is removed from the agent. The placed the slove.
> turn on stove
stove is now turned on. > exonine stove
a stove that currently on and has the following items on pot that looks to have some water in it exanine stove
3.1 Task Specification
(water boils)
Game completad.
The task specification is a set of structured com- ments the start of each game in the corpus that serve a high-level outline for the critical compo- nents of each game. These are intended provide a high-level scaffold that language models can use to better structure games they generate. The com- ponents of the task specification include:
Figure 2: An example playthrough of game whose task requires boiling water, from the BYTESIZED32 training set.
Solution:
A high-level solution to the game. For example: opening the dishwasher, moving each dirty dish from the kitchen into the dishwasher, moving dish soap into the dishwasher, closing the dishwasher, and activating the dishwasher.
Task Description: The task an agent playing the game has to solve - for example, washing dirty dishes using a dishwasher.
Pask-Critical Objects: Names of task-critical ob- jects, such as dishes, dish soap, and dishwasher.
3.2 Game Code
To maximize utility as n-shol training data for code generation tasks, each game in the corpus uses a highly-templated structure consisting of core ob- jects and member functions, shown in Figure and described below. The core architecture and API of these functions mirrors other text game frameworks (Hausknecht et al., 2020) derived from the OPF- NAI GYM specification for reinforcement learning models (Brockman et al., 2016). These include:
Actions: Actions that an agent playing the game
can take, such as opening or closing containers, activating or deactivating devices, picking up or putting down objects, and so forth.
Objects (or actions) that limit hin- der task performance, or that are unrelated to the game task for example, adding food that an agent can eat, that creates more dirty dishes.
Distractors:

---

World Initialization: Initialize the game world. For example, for the dishwasher game, create kitchen room that includes dirty dishes, dish soap, a dishwasher, and any other relevant objects.
BYTESIZED32 PYTHON template
Generic parent class for all objects Provides getters GameObjedl
abject properties
Valid Actions: Return list of all possible valid actions that an agent could take, given the current state of the environment. For example, take dirty dish, or open dishwasher.
# Parent class for game objects that are containers Provides methods for adding/remmoving objects class Containor (amba//cr(
Parent
game objects
Provides methods activating/ deastivatnn class Device (GameObject):
Take Action Step: Perform a specific action in the environment. This function returns the observation that results from that action for example, the take dirty dish action might return the observation "the dirty dish now in your inventory".
* Example object: Seep for washing dishes class DishScap (GameDacc))))
object:
food)
class Dish Diah(Cootainer
Example object: A dishwasher (thet can contain dishes, a dish sose. and be astivated to wash the disnes). class Diswwasher(Device Container);
Scoring: Return an agent's current progress in solving the game task, abstracted to arbitrary numerical score, and a set of boolean flags that represent whether the game has been won lost.
Main Simulation Class class TexGamme ():
Creates the game world and populates with game abjacts (iecluding the kitchen. dishes. dishwasher, etc.) def WorlllizeWWrld
While the above methods are provided through a main game class (TextGame), each game also in- cludes a large number of classes representing spe- cific game objects. Each game object derives from common class, GameObject, from which generic subclasses that share common methods inherit (e.g. Cont Containers, that can store objects, or Devices, that can be activated), before finally instantiating specific game objects (e.g. Dish, Dishwasher).
Returns string deseribing gamo and task getTasskDesciption!!
Raturns an array with all possible valid actions given the current game state dal jenerateValldActionn.
action fe.g. cnvironment, changing environment state step atr):
dishwasher)
Calculate the current game score given progress. del calculateScore ():
4 Evaluating Generated Simulations
Entry Point (example
playing)
name game TostGame
main
Evaluating model-generated text games presents a number of challenges. First, because games are largely open-ended and constrained only by short task prompt, the evaluation metrics must be robust to wide range of potential errors and behaviors. Second, evaluating open-ended simulations typi- cally requires manual human evaluation, which is costly and labor intensive. Here, we propose a set fully-automatic metrics that measure both tech- nical aspects of games- such as whether the simu- lation runs error-free - as well as content aspects that measure how well generated simulations ad- here to task specifications. We then validate these automatic instruments with human ratings.
print[
gama. watngggaTaakDDerrplllo ())
while nat game gamoOver:
actionStr Input(">
print: pbbarrinbsavvaioon reward step (tepinat
observation.
observation)
print "Scere" score)
print "Reward: reward)
print "Game Completed...
print "Game Won: str [game gamzWan))
Figure 3: An illustration of the core classes and mem- ber functions present in the highly-templated games of the BYTESIZED32 corpus. Each game consists of average of 618 lines of code, and the example here pro- vides only an overview a subset of most important functions.
a generated text game run without errors by call- ing them in a PYTHON interpreter and capturing errors. We measure errors during the game initial- ization phase where the simulation environment is being constructed, the valid action generation, where the simulation provides list of all valid actions the agent might take given the current envi- ronment state, and the step function, which takes a user-requested action that modifies the environ-
4.1 Metries
Evaluation metrics are described briefly below, with additional details of their computation, val- idation, and prompts described in the APPENDIX.
Technical Validity: The technical validity metric measures whether the core member functions of

---

ment. The valid action generation and step func- tions are tested by exhaustively crawling all possi- ble trajectories (i.e. sequences of actions) an agent could take. Because games can have up to 2000 valid actions per step, the path crawling procedure has a limited horizon typically steps. At each step, we also group actions by their action verb and explore maximum of 100 actions of each group.
Winnability: A game is considered winnable if there exists sequence of actions that, when per- formed in order, will lead to winning state of the game. Automatic evaluation of winnability was performed by letting a GPT-4 text game agent play through the games. This agent uses recent prompting techniques such as ReAct (Yao et al., 2023) and Reflexion (Shinn al., 2023) to provide high-level planning and problem-solving. Man- ual evaluation was performed by a single human evaluator. Both automatic and manual evaluators attempted to reach the game's winning state by submitting actions to the game's 's STEP() function. We note that this process does not always give an accurate measure of game's 's winnability, as it is possible for an evaluator to fail to find possible winning trajectory. Nevertheless, cally that in the vast majority of cases game is either obviously winnable or obviously impossible to win. Overall we find that GPT-4 underestimates game winnability, with inter-annotator agreement between GPT-4 and the human evaluator low (Co- hen's Λ = 0.43). This suggests solving arbitrary text games zero-shot is still beyond the capabilities of GPT-4, a finding consistent with prior research
This metric measures whether the generated game includes the required actions, objects, and distractors required in the task specification. Compliance measured automat- ically by supplying the generated game and its task specification to a GPT-4 model, which is then asked a series of true-or-false questions about the presence of each required component. For example, in a boiling water game, one such question is Does the simulation contain the object 'Sink'?. 7. To vali- date this automatic metric, compare GPT-4 rat- ings with gold ratings generated by two expert hu- man annotators, showing moderate-to-stre to--tn inter- annotator agreement between GPT-4 and human ratings (Avg. & = 0.74; Object: x = 0.96; Action: K = 0.75; Distractor: K = 0.50).
Specification Compliance:
find empiri-
Physical Reality Alignment:
In addition to techni- cal and specification compliance, we provide a mea- sure of how well the actions in generated games accurately model the constraints of the physical world. For example, game that lets you take object out of closed container (like fridge) not respecting the con-
LLMs commonsense problem solvers (Bian et al., 2023), As such, we report human evaluations of winnability for our experiments.
5 Experiments
without first opening straints of the physical world. Because simulation necessarily encodes only a small subset of reality, we restrict our measure to only the set of actions implemented by the game and returned by the GEN- IRATEPPSSSIBLEACTIONSS function.
Here we investigate the capacity for large lan- guage model such as GPT-4 generate the hun- dreds of lines of code required to generate a work- ing text game simulation centered around unseen tasks on each of the technical and quality metrics described above. Alongside, we examine the extent to which these models can use reflection to increase generation performance.
To measure physical reality alignment, we crawl given game up depth of 3 steps, then ran- domly sample 100 trajectories equally distributed across each action a game implements. These trajectories are then provided to GPT-4, with a prompt to provide a binary assessment as to whether the game playthrough up to that point ad- heres to physical reality, as well as short text justification for that assessment. To validate this metric, two expert human raters produced gold la- bels for 200 physical reality alignment judgements. The inter-annotator agreement between human and GPT-4 ratings is strong (Cohen's к 0.89), demonstrating GPT-4 has a high agreement with humans when making these assessments.
5.1 Model and Prompt
As template games contain up to 10K tokens before including the prompt, we make use of GPT-4 (Ope- nAI, 2023) with a context window of 32k tokens for each of our experiments. The model prompt includes 1-shot example (a single PYTHON ref- erence game from the B BYTESIZED32 corpus), fol- lowed by task specification describing the game the model must generate, drawn from an unseen
"See Appendix C for performance of Codel.lama (Rozière et al., 2023) on BYTESIZED32 with no finetuning.

---

evaluation set. All experiments reported here use greedy decoding. Additional model hyperparame- ters and prompts can be found in the APPENDIX.
Technical Validity Mcasurement
Number Reflections
2
Game Initialization
85.4%
85.4%
89.6%
88.5%
Valid Actions 80.2%
83.3%
87.5%
88.5%
5.2 Evaluation Set
Runnable Game
28.1%
42.7%
51.0%
57.3%
addition to the 32 training games in the BYTE- SIZED32 dataset, we also provide an evaluation set in the form of task specifications for 16 additional unseen games. Each game in the evaluation set is explicitly crafted to have highly similar or highly dissimilar characteristies to specific games found in the training set, such as similar dissimilar objects, actions or distractors.
Table 2: Technical validity measurements of generated games before reflection (0), and after up to three reflec- tion steps. Values represent the proportion of games (N = 96) passing a given test after given number of reflection steps.
Reflection After
Measurement
Before
Specification Compliance
Task-critical objects
100.0%
100.0%
0.0%
5.3 Reference Game Selection
Task-critical actions
93.8%
93.8%
0.0%
Distractors
21.9%
18.8%
-3.1%
To improve diversity in generation, we randomly pair each game specification in the evaluation set with six different reference games. Half of these reference games are chosen to have at least some similarity to the evaluation game (i.e., they share a similar object, action, or distractor), while half are chosen to minimize similarity.* With 16 game specifications in the test set, this results in total of 96 model-generated games.
Winnability
30.2%
37.5%
+7.3%
Table 3: Specification compliance and winnability mea- surements for generated games before and after reflec- tion. Specification compliance is measured automati- cally, while winmability is measured manually by human experts. Overall, each measure shows small increases or decreases post-reflection.
5.4 Reflection
validity evaluation are shown in Table 2. Model per- formance on creating game initialization methods is strong overall, beginning at 85%, and increas- ing to 89% after reflection. Similarly, generating a method that enumerates valid actions for a given step occurs in 80% of generated games before re- flection, increasing to 89% after reflection. Gen- erating fully runnable games, which successfully run an exhaustive path crawl of all possible game trajectories up to 3 steps without error, occurs only 28% games before reflection, but increases to 57% after reflection a substantial increase of 29% gained from the self-reflection process. We show examples of GPT-4 fixing bugs in code via reflection in APPENDIX Table 7.
Similar other code generation tasks (Lehman al., 2022), we hypothesize that selffreflection that is, providing the model with error output, and allowing to iteratively correct its generated code will increase overall generation performance. As such, during generation, we provide any error mes- sages generated by the PYTHON interpreter during the technical validity evaluation back to the model, in self-reflection prompt that requests the model correct the error. This reflection step is repeated until the game passes all technical validity checks, or a maximum number of self-reflection steps is reached. In the experiments reported here, the max- imum number of reflection steps is 3.
Self-reflection also increases physical reality alignment, with a histogram of automatically mea- sured physical reality alignment scores shown in Figure 4. Before reflection, average physical reality alignment across games is 43%, which indicates that GPT-4 finds 43% of randomly sampled paths to fully comply with its expectations of physical reality. After reflection, this increases to 51,, a moderate increase of 8%. Though measured au- tomatically, the strong inter-annotator agreement between human and GPT-4 raters in Section 4 sug- gests this improvement to be genuine, though it
6 Results
Here, evaluate all generated games (N 96) across each metric, reporting results both before and after self-reflection. The results of the technical
We empirically observe that without providing the tem- plated example, GPT-4 generates games that exhibit limited state space (i.e.. less challenging), and lack coherent for the development of a consistent automatic evaluation metric. We leave 0-shot game generation as future work.
'These pairings hetween evaluation and reference games are provided with the BYTESIZED32 corpus.

---

Pre-Refletton
Playthrough of model-generated game
Post Reflection
Task Descriptin::
Your task is to protect yourself from mosquitoes putting mosquito repellent, then
Proportion of games
forest, apple,
move back to the house to pul the apple in box.
0.15
Initial Observation:
%
%
You find yourself house. the house, you yourseff
battle of mosquito repellent
bottle
0.05
box
You also
way to the forest
Physical Reality Alignment Score
0.4 0.6
Type 'help' for list possible actions.
Figure 4: A histogram of automatically measured physi- cal reality alignment scores, both before (red) and after (blue) reflection. Asterisks represent average scores (0.43 pre-reflection, 0.51 post-reflection)
nosquite to regellent
The maaquito repellent is removed from the house. You put the mosquito repellant your inventory.
> nesquito repellent
use mosquita repellent yourself.
You move from house to forest.
likely due least in part overall increase in non-zero physical reality alignment scores due more games becoming runnable after reflection. When zero scores are removed from the analysis, the average physical reality alignment scores be- fore and after reflection are 58% (N = 71) and 62% (N = 80), respectively or in other words, the effect size of reflection decreases to 4%. Similarly, self-reflection increases the winnabil- ity of games. As shown Table 3, the winnabil- ity of all generated games pre-reflection is 30.2% when measured manually by human experts, with this increasing to 37.5% post-reflection - gain of 7.3%.
> take spple
The apple is removed from the forest. You put the apple in your inventory.
You move from forest to house.
> put apple in box
The apple is removed from the agent. The apple placed in the
Game completed. Game Won: True
Figure 5: An example playthrough of GPT-4 gener- ated game centered around the task of protecting ones self from mosquito bites. The reference game used during generation centered using sunsereen protect from sunburns.
Self-reflection does not improve every metric evaluate. Each of the submeasures of specification compliance, including generating task-critical ob- jects, task-critical actions, and game distractors observes a small decrease or no difference post- reflection when measured automatically suggest- ing that self-reflecting on technical validity mea- sures affords limited utility to these measures of simulation content. Still, observe strong over- all performance in specification compliance, with generated games including task-critical objects in nearly every case, task-critical actions in 93.8% of games, while struggling with distractors generat- ing these in only 18.8% of games.
are hundreds of lines in length, such as the game shown in Figure 5. Of the generated games, nearly all implement at Icast one task-criticn object, 88% implement at least one task-critical action, and a full 38% allow a user or agent to reach a winning state. A more nuanced interpretation of these results suggests that the model has best learned to successfully replicate the high-level structure of the highly-templated BYTESIZED32 game API as model performance begins degrade once we examine the minute details: only 58% of games are robust to a 3-step exhaustive trajectory search, and only 19% of games include a required distractor despite their presence in the reference games. Similarly, while the average training game includes 4396 code tokens, the average model-generated game contains only 3368 code tokens or 77% of the length of training games suggesting that model-generated games are not yet able to replicate the full level of simulation fidelity provided the training corpus.
7 Discussion
To what extent GPT-4 generate long struc- tured text games using single-shot in-context learning? At high level, our results indicate that GPT-4 is frequently capable of generating syntacti- cally valid, templated, and playable programs that

---

is valid as long as the inter-annotator agreement between the LLM and human annotators is high. By presenting models with game code or trajectories and requesting targeted, binary judgements, we find it is possible to automatically and reliably rate measures of game specification compliance and physical reality alignment. At the same time, we show that certain metrics remain challenging for automation. Determining winnability of generated game, in particular, essentially requires a model to solve arbitrary text games - an active area of research (Jansen, 2021). The automated agent currently underestimates game winnability by about half compared to expert human judgements, though it is plausible that this gap will narrow as the reasoning capabilities of LLMs continue to improve.
How does self-reflection assist game gen- eration? Self-reflection - that is, iteratively providing an LLM with error messages from the PYTHON interpreter when running generated games, then asking it to correct those errors (Chen et al., 2023) dramatically increases generation performance, most notably in terms of technical validity. We find that three steps of self-reflection increases the generation rate of runnable games from 28% to 57%. Gains from self-reflection are typically largest when detailed and specific error feedback is possible (Olausson et al., 2023). This indicates that similar gains might be achievable on errors in specification compliance, physical reality alignment, and winnability by using the output of automatic evaluation process. However, the feasibility of this approach is constrained by the time and expense involved in querying the model for reflection - our results indicate that single game could include hundreds of small and large errors total. The cost of reflection might be lowered by the use of code diffs for edits (Lchman et al., 2022) or through batching multiple errors into a single reflection request though our pilot experiments on applying these techniques to the long generated programs here indicates that current models might require specialized fine-tuning to do so. Alternatively, open source code generation models are quickly approaching GPT-4 perfor- mance in both generation length and accuracy (Li et al., 2023; Luo et al., 2023; Gunasekar et al., 2023; Rozière et al., 2023), suggesting that fine-tuning on the entire BYTESIZED32 corpus may become viable in the near-term, potentially reducing the number of errors when generating high-fidelity simulations, and reducing dependence on self-reflection.
Can we observe the internal world mod- els of LLMs through the simulations they generate? Generating world models as code provides a formal and interpretable means to explicitly assess how LLMs understand the world. For instance, the GPT-4 model generated a game that involved burying box of treasure in hole but required placing soil back into the hole before placing the treasure box inside. In another generated game, an agent was able to directly place water in its inventory without using any containers. These failure modes indicate situations in which the language model was unable to accurately realize the world knowledge presumably contained within its pretraining data. Even after reflection, only 51% of short 3-step trajectories in GPT-4 generated games accurately modeled the physical world, indicating that constructing correct and explicit world models code remains a formidable challenge for LLMs.
8 Conclusion
Can we use LLMs automatically evaluate the output of their own simulations? Automatic evaluation of model outputs a vital prerequisite for large-scale experiments world model gener- ation, where manual evaluation of even single simulation can require prohibitive amount of time. The complexity of the task, however, precludes the of most simple automatic metrics. We find that using language models automatically crawl and evaluate generated simulations is a viable alternative to cine-consuming human annotation for certain measures. This automatic evaluation
In this work, we present BYTEŠIZED32, corpus of small world models expressed as text games cen- tered around specific common-sense tasks. Using a simulation as code generation paradigm, we show that it is possible to use these games, expressed as hundreds of lines of PYTHON code, templates for in-context learning, and generate novel simu- lations for unseen tasks. We futher show that it is possible to iteratively self-reflect on these large simulations, and improve on technical validity and physical reality aligument metrics by as much as

---

29% and 8% respectively. While manually evaluat- ing simulations is labor intensive, we empirically demonstrate that a number of measures of simula- tion accuracy can be automatically evaluated with moderate-to-stronn agreement with expert human ratings. We release this work as open source, and as a challenge task at the intersection of world mod- cling and code generation, to spur further devel- opment in expressing the knowledge contained in language models more formal and interpretable forms.
grade outputs. At the same time, strict binary measures may not be desirable for some metries, and we leave creating and validating more granular metrics for future work.
:elf-Reflection: Self-reflection is rapidly emerging tool for iteratively improving the accuracy and quality of code generated by large language models. Here show that with targeted feedback, self-reflection is possible and helpful for large (several hundred line) simulation programs. This potentially enables the generation of increas- ingly complex programs without sacrificing code quality.
Limitations
This work examines the ability of LLMs to gener- ate abstract text-based world models and the BYTE- SIZED32 corpus is designed to facilitate that task. As such, games in the corpus are not designed to resemble extant text games or to be entertaining.
Intended Use: The games included in the BYTESIZED32 corpus have been designed study LLMs and there is no guarantee they will be entertaining or useful outside this scope.
We perform regime and do not examine the possibility of in- cluding more than one game from the corpus could within the 32k token context window of our GPT-4 MODEL. We also do not test recent models with similar context sizes like COLT5 (Ainslie et al.. 2023), UNLIMIFORMER (Bertsch et al., 2023), or CLAUDE-1 1 00K (PBC, 2023). Both of these are valuable directions for future work.
experiments in single-shot
Acknowledgements
We thank the three anonymous reviewers for their comments. This work supported in part by National Science Foundation (NSF) award #181 19448 to PJ, and the Allen Institute of Artificial Intelligence (AT2). We thank Matheus Pereira for help setting up CodeL Instruct and vLLM.
Finally, we perform reflection by regenerating the complete program each step and target only a single error at a time. This process could be made more efficient by outputting only a code diff and batching multiple errors at once.
References
Donna Rose Addis, Ling Pan, Mai-Anh Vu, Noa Laiser, and Daniel L Schacter. 2009. Constructive episodic simulation of the future and the past: Distinct sub- systems of core brain network mediate imagining and remembering. Neuropsychologia, 47(1 1):2222- 2238.
Broader Impact
We provide an initial investigation of formalizing the knowledge captured by language models into interactive simulations expressed as code. This process can be used inspect and evaluate language models, and the ability to generate simulations on-the-fly ffy has potential applications games and science.
Generating Simulations:
Joshua Ainslie, Tao Lei. Michiel de Jong, Santiago On- tañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Leo-Th Tay, Yun- Hsuan Sung. and Sumit Sanghai. 2023. Colt5: Faster long-range transformers with conditional computa- tion.
Jacob Austin, Augustus Odena. Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Elllnn Jiang, Carrie Cai, Michael Terry. Quoe Le, al. 2021. Program synthesis with large language models. arXiv preprint 108.v:2108.07732.
Celf-Evaluation of GPT-4: In spite of re- cent criticisms of the ability of GPT-4 to evaluate its own output, we empirically validate that this is possible in some cases where strict binary judgements of relatively straight-forward common- sense tasks are required. Automatic evaluation is critical component of any effort at scalable environment generation, as vastly reduces the amount of human labor required to validate and
Gabriella A.B. Barros, Antonios Liapis, and Julian To- gelius. 2016. Murder mystery generation from open data. In Proceedings of the 7th International Con- ference on Computational Creativiry, ICCC 2016, Proceedings of the 7th International Conference Computational Creativity, ICCC 2016, pages 197- 204. Sony CSL.

---

Lawrence W Barsalou. 1999. Perceptual symbol sys- tems. Behavioral and brain sciences, 22(4):577-66)
Peter Jansen. 2021.
systematic survey of text worlds as embodied natural language environments. arXiv preprint arXiv: 207.0424
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Goumley, 2023. Unlimiformer: Long- range transformers with unlimited length input.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt is knowledgeable but inexperienced solver: An investigation of com- monsense problem in large language models, arXiv preprint arXiv:2303 16421.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. 2022. Evolution through large models.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.013 000066001400
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 181 1813-1827, Online, Association for Computational Linguistics.
Randy I. Buckner and Daniel C Carroll. 2007. Self- projection and the brain. Trends in cognitive sciences, 11(2):9957. 57.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii Terry Yue Zhuo, Thomas Wang. Olivier Dehaene. Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming- Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey. Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov. Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao. Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Dan- ish Contractor, Siva Reddy, Daniel Fried, Damitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferran- dis, Sean Hughes, Thomas Wolf, Arjun Guha, Le- andro von Werra, and Harm de Vries. 2023. Star- coder: may the source be with you! arXiv preprint arXiv:2305.06166
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models self-debug. arXiv preprint 2344v:2304.05128.
Mlre-Alexandre Côté. Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018 Textworld: learning environment for text- based games. CoRR, abs/ 1 11322.
Angela Fan, Jack Urbanek. Pratik Ringshia, Emily Di- nan, Emma Qian, Siddharth Karamcheti, Shrimai Prabhumoye, Douwe Kiela, Tim Rocktaschel, Arthur Szlam, and Jason Weston. 2019. Generating interac- tive worlds with text.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Lear- ing Representations.
Xiang Lorraine Li, Adhiguma Kuncoro, Jordan Hoff- mann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. 2022a. A systematic investi- gation of commonsense knowledge in large language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing. pages 11838-1 1855, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giomo, Sivakanth Gopi. Mojan Javaheripi, Piero Kaufimann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306. /1644.
Yujia Li, David Choi, Junyoung Chung. Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' Autume, Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbl, Sven Gowal, Alexey
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare- Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903-7910.

---

Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022b. Competition-level code generation with al- phacode. Science, (7866664))109221099
Jack Urbanek, Angela Fan, Siddharth Karamcheti. Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Ja- son Weston. 2019. Learning to speak and in fantasy text adventure game.
Xi Victoria Lin, Chenglong Wang, Luke Zeutlemoyer, and Michael D. Ernst. 2018. NL2Bash: A corpus and semantic parser for natural language interface the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Nick Walton. 2019. Ai dungeon. https://gi thub....t...... com/Lati yyyy-mm
Accessed:
Ruoyao Wang. Peter Alexander Jansen, Mare- Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than 5th grader? ArXiv, abs/2203.07540.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct.
Tao Yu, Rui Zhang, Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma. Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: large-scale human-laheled dataset for complex and cross-domain semantic pars- ing and text-to-SQL. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3911-3921, Brussels, Bel- gium. Association for Computational Linguistics.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv e-prints, pages X203-.2003
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezame 2023. Demystifying gpt self-repair for code genera- tion. arXiv preprint arXiv:2306.0995 09896.
OpenAl. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
Anthropic PBC. 2023, Introducing 100k context
windows. https://www. ceathropic.comminii 100k-context-winddow [Online; accessed 20- June-2023).
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint 10X1:5
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wen- han Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Ilama: Open foundation models for code.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Mohit Shridhar, Xingdi Yuan. Mlec-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and em- bodied environments interactive learning. arXiy preprint arXi::2010.03768.

---

A Details on Code Generation
A.2 Reflection
During validity check, when the code encounters an error from the Python interpreter, we use the error message to reflect. For the reflection we use the standard GPT-4 model (i.e., with 8k context) and the following prompt.
In this work, we make extensive use of OpenAI's API. We use the ChatCompletion mode without system prompt. In all our experiments, we keep the following hyperparameters constant:
temperature=0.0
top_p=1
GPT-4 Reflection Prompt
You are DeveloperGPT. the most advanced Al developer lool on the planat. You answer any coding question. and provide real useful example code using code blocks. Even when you are not familiar with the answer, you your extreme intelligence figure it out.
presence_penalty-0.00
A.1 Game Generation
Your task to correct a program that text-based simulation. Here the code simulation
To generate the new games, we use the gpt-4-32k model and the following prompt.
(GAME CODE)
Here the message from a Python interpreter.
GPT-4 Game Generation Prompt
(ERROR MESSAGE)
You should respond all the code with your fix. Do not respond anything else.
You are DeveloperGPT, the most advanced Al developer tool on planst. You answer any coding question, and provide real useful example code using code blocks. Even when you are not familiar with the answer, you use your extrame intalligence to figure out.
B Additional Notes on Evaluation Metrics
Your task is to write program that: text based simulation. The program should be written in Python. It should be challenging to the user, testing their common--srres sense knowledge, and take multiple steps complete. possible, there shauld distractor objects and actions that do not help progress, to measure whether the really knows what they're doing. You should name all targel objects and distractor abjects with common sensee names. Your code must contain a class named TextGame. The TextGame class should have the following member functions:
B.1 Technical Validity
Validity measurements are reported in order, such that failure of a function called earlier in the API implies failure for all subsequent tests. We note, however, that the game initialization functions are evaluated only once, at the beginning of the game, while the GENER STEP() function are necessarily evaluated at each step.
Init (self. randomSeed), geTTasxDescriploontelll Cemeeraeeossibleeacce Possibieectionsssseee sell), step(self, actionStr), calculateSoore (self)
and the
Here is a specification of the task that your code should simulate.
*** * Task: Create simulasimulaiion that models to milk to a temperature that suitable for a baby using a stove. Environment: kitchen
Task-critical Objects: Stove, Pot, Milk, Fridge. Thermomater
B.2 Specification Compliance
High-level object classes: Device (Slove, Fridge), Container ( Stove, Pot, Fridge)
The full GPT-4 prompt used generate the true- or-false evaluations of specification compliance is provided below:
Critical properties: temperature (Milk), temperature increase per_ (Stove). temperature diccresse per_ttt tick (fridge), max_ temporrrrrr (Stove) min temperatuue (fridge)
Actions: look, inventory, examine, take/put objoct, openiolose container, turn onvoft device, use thermometer on object, feed baby with milk
GPT-4 Specification Compliance Prompt
Distractor Items: None
Distractor Actions: drink milk
You are DoveloperGPT, the most advanced developer tool on the planet. You answer any coding question, and provide real useful example code using code blocks. Even when you familiar with the answer, you use your extreme intelligence figure it out. Your lask evaluate program that toxt-based simulation.
High-level solution procedure: open fridge, take pot containing milk, the on the stove, lurn the stove, use the thermometer moniter the milk temperature till the temperature is sultable for a baby to drink, feed baby
Here is example text-based simulation on different topic that you can use as tomplate: (GAME_ CODDE)
specification simulation: (GAME_
Here the code the simulation: (GAME_CODS
Answer the following question based the given specification and the simulation code: (EVAL. QUESTION;
Depending on the length of prompt and the code to generate and the API traffic, each game may require 5-10 minutes to generate. We use stream generation which allows us to recover from a GPT- 4 API timeout. The code in the response of GPT-4 is wrapped in a Markdown Python code block (i.e.. enclosed with three backticks) which makes it easy to extract. We only keep the code part and discard the rest.
Answer "Yes' 'No' first and brielly explain your answer.

---

Manual (Human)
Automatic (GPT-4)
paths by taking an even number of paths from each group. For instance, if the actions TAKE; PUT, and MOVE occur as the last actions in our set of 25,000 paths then we subsample 33 paths for each action and I path randomly. Each subsampled path is sent to GPT-4, along with the game's 's task description, which is then asked to determine whether every action in the path and its accompanying observation align with physical reality. The GPT-4 evaluation prompt is provided below:
Measurement
Specification Compliance
Task-critical objects
97.2%
100.0%
2.8%
Task-critical actions
87.5%
93.8%
6.3%
Distractors
37.5%
18.8%
18.7%
Winnahility
37.5%
17.8%
19.7%
Table Manual (human) and automatic (GPT-4) evaluation results of the best- gest-generated games (i.e. games after reflection, 96) both specification compliance and winnability. Difference scores reflect the difference between automatic and manual ratings, showing a moderate overall agreement for specification compliance (Avg. K = 0.74: Object: = 0.96; Action: x = 0.75; Distractor: x 0.50) and modest agreement for winnability (x 0.43).
GPT-4 Physical Reality Alignment Prompt
the playthrough of the text game below, would like you describe whether the game engine (lee the abservations returns in response to actions) are physically accurate models of the world or whether they don't make sense.
Because we observe variance in GPT-4's answer on a few games, even when using temperature of zero, run the same measure 31 times and take the final result as the majority vote.
An example of not making sense would be being able to take action from a container (lik a fridge) without having opened it first. In addition, an action produces automatically tails to accurately model the world and does not make sense.
from the game, then
Please restrict your evaluation only to the short playthrough. and the specific actions chosen, without speculating about other actions.
The upper part of Table 4 shows the manual and GPT-4 automatic evaluation results of specifica- tion compliance. The inter-annotator agreement between GPT-4 and the human evaluators are high (average Cohen's ς = 0.71). We note that the manual evaluation differs from the automatic eval- uation in only 8 of the 96 games, indicating that automatic evaluation is viable substitute for costly human evaluation of these metrics. We also note that the automatic evaluation most frequently dif- fered from the manual evaluation in the distractors section, which is also the section that proved the most difficult in terms of generation.
Nota: Objects can be manipulated by the agent without first being explicilly picked up, as long as they are the environment, and readily accessible (e.g. not in closed container).
The evaluation should be binary ("yes' or "no"), except in the cases where the code generated an error, when the evaluation should be "error".
Here is an example culpul format: '"evaluation"""no"
"short_jusbfiee justfication": "could take object (banana) from the closed fridge without having to first open the fridgo")
Game Task: (GAME_ TASK)
Here the playthrough to evaluate: (PATH)
B.4 Winnability
The initial prompt given to GPT-4 to act as a text game agent is provided in Figure 6. The GPT-4 agent was used to evaluate 96 games, and the re- sults were broken down as follows: in 17 games, agent reached the end of the game determined by the game's CALCULATESCORFF function. In 16 games, the GPT-4 agent finished by outputting "done," but the game's 's CALCULATESCORE(( did not indicate that a terminal state had been reached. In 34 games, the GPT-4 agent finished by out- putting "bug" (see Figure 7 for an example). In the remaining 29 games, the game crashed before the GPT-4 agent finished execution.
B.3 Physical Reality Alignment
The process of generating sample trajectories for automatic evaluations occurs in two steps. First, we perform a breadth- -oustth-irsttte crawl of the game using the action strings returned at each step by the GENER- ATEPOSSIBLEACTIONS function. At each step we maintain list of counts for each action "verb," which is extracted from a valid action string by tak- ing the first token. When we perform recursive search from a given step, we keep only 10 paths for each action verb. We restrict our search to a maxi- mum depth of 3 actions and stop after 25,000 paths have been generated. In addition, if the game pro- duces an error, then the error message is recorded the observation from that step and the search continues.
The bottom part of Table 4 shows the manual and GPT-4 automatic evaluation results of winnability. The inter-annotator agreement between GPT-4 and the human evaluators are low (average Cohen's 's K = 0.43). As a result, we report the manual results of winnability in the main paper.
After the initial set of paths has been generated, we group the set by the last action verb used in each path. We then generate a subsample of 100

---

GPT-4 Winnability Prompt
playing lext-based games.. Interact
to salve a task.
Hare is an example. CONTEXT EXAMPLE)
The game about play only understands one command a lime the following iist commands:
(POSSIBLE ACTIONS)
Prepend your thoughts with "hink:" when planning your next steps. When you think the task is completed, say 'done'. you think the task can't completed say bug'.
The IN CONTEXT EXAMPLE used in the GPT-4 Winnability Prompt
Task Description:
Your task to make cubes.
turn off sink
Observation: find yourself a
kilchen,
The now
you s8e:
yourself
put tray the water to freeze. cube.
a freezer that is currently closed
sink that is currently off and that is empty
pot emply
The ice cube tray removed from the sink, You put the cube tray in your inventory.
a peanut butter orange
lock
the in. I will need to fill the container with water and then it the freezer. I will need to wait for the water cubes from
think: will nead container
You find yourself in kitchen. In the kitchen, you see:
yourself
freezer that currently open and a sink that currently and that empty
need renove container.
a pol that emply butter a orange
looking container.
open freezer
The freezer open.
put ice cube tray in freezer
examrine freezer
The ice cube removed from the agent.
currently open an ice cube cray that is empty
following items:
The cube tray placed freezer.
take ice cube tray
The freezer is now closed.
The cube removed from the freezer. You the lce cube tray in your inventory.
lack
You find yourself in kitchen. In the kitchen, see:
need Based on the possible commands, have to nove the ice cube
tray water.
yourself
8 freezer that is currently closed sink that currently off and that empty a pot that emply
sink put ice cube tray in sink
The cubbe removed from the agent.
peanut butter 8. orange
cube tray placed
You lind yoursell in kilchen. In the kitchen, you see:
You find yourself in kitchen. In the kitchen, you see:
yourself
yourself a freezer that currently closed sink that currently off and that empty a pot empty
a freezer that is currently open and emply sink that is currently off and that contains or more
a pot that empty
a peanut butter orange
peanut butter
a orange
freezer
The freezer now open.
examinn sink
that currently an ice cube cray that is empty
contains following
exanine freezer
a freezer that is currently open and contains the following items: cube oray that looks some
turn sirk
The sink now turned on.
Current step:
Score:
exanine
Reward:
sink that is currently on and that contains the following items:
Game Over: True
cube cray that looks have some waler
Game Won: True
Figure Prompt used for automatic evaluation the winnability generated games.

---

Technical Validity
Number of Reflections
Measurement
3
Game Initialization
60.4
52.1
52.1
56.2
Possible Actions Generation
60.4
52.1
52.1
52.1
Runnable Game
20.8
20.8
20.8
20.8
Table 5: Technical validity measurements of games generated with CodeLlama before reflection (0), and after up to three reflection steps. Values represent the proportion games (N 96) passing given test after given number of reflection steps.
Reflection Before
Measurement
After
A
Specification Compliance
Task-critical objects
96.9%
78.1%
-18.8%
Task-eritical actions
100%
90.6%
-9.4%
Distractors
18.8%
9,4%
-9.4%
Winnability
4.2%
4.2%
0%%
Table 6: Specification compliance and winnability mea- surements for games generated with CodeLlama before and after reflection. Specification compliance is mea- sured automatically, while wimnability is measured man- ually by human experts. Overall, each measure shows negligible decreases post-reflection due Codel- lama's 's difficulty with performing meaningful reflection on large pieces of code.
C Open-Source Language Model
We report here the results when using the open source language model CODELLAMA-34B- INSTRUCT (Rozière et al., 2023) served using vLLM (Kwon et al., 2023) with maximum context length of 32k and dynamic RoPE scaling factor of 2. We opted for Cade because of it can ex- trapolate up to context size of 100k tokens. While its shot code generation performance is decent with 20.8% runnable games compared 28.1% % for GPT-4, this result is tempered by CodeLlama's apparent difficulty with performing meaningful re- flection on these large pieces of code. As shown in Table 5, some of the revised games degrade right af- ter the first reflection and never fully recover. The two main failure cases for doing reflection with CodeLlama is 1) partial code generation and 2) generating the full code but without including the fix even though CodeLlama's responses mention the issue and how fix

---

D
Qualitative Examples
Example GPT-4 agent detecting an issue
Here is task:
You find yourself in a room. In the room, see: vending machine. quarter coin. dime coin. nickel penny coin. yourself.
Your task coins available in the room.
snack from the vending machine using
think: need to collect coinss use then in vending nachine to buy snack. The task does not specify which buy, any
OK.
take quarter
The quarter removed from the room. You the quarter in inventory.
The dime is removed from the room. You put the dime your inventory.
take nickel inventory.
room. put nickel your The penny removed from the You the penny in inventory.
take penny
all coins. in the vending rechine to purchase snack. OK.
quarter in vending nachine the quarter vending machine. money the vending machine: $0.25
put dine in vendingg.nae machinee
You the dlime in the vending machine. Total money in the vending machine: $0.35
put
anding_nachina
You put the nickel in the vending machine. Total money in the vending machins: $0.40
penny vending, nachine
the ponny the vending machine. money in the vending machine: $0.41
select chocolate
Not enough money the vending machine buy chocolato. You need $1.59 more.
chips Not enough money in the vending machine to buy the chips. You $0.59 more.
think: don't 't have money buy any snack. Task can't be completed. OK.
bug
Figure 7: example of using GPT-4 agent to detect issue with generated game.

---

Game
2812_action,_1o _est___st______ 2tt.:o_est.....p.--42. volume, genenatinn.p
Error
invalid syntax (line 293)
293
defstep(self, actionStr):
step(self, actionStr):
Code diff
self.
Gome
3512_distractor_teg 1or spt-4-p42k distwasherr peneration. py
object attribute "actionInventory'
290
def action/nventory(ee1:::
291
inventory self agent, contaiiss
292
leneenvnttryy)
293
return "You have nothing in your inventory."
Code
295
inventory_ste =1
"In inventory, have:\n"
296
for iten in inventory: inventory_str += "\t"
297
298
return inventory_str
Game
0512 (tttraaoorrett11....----3-3 plant-uuee p
Firor
local variable "measuring_cup referenced before assignment
457
None measuring_cuup None
458
459
scale None
468
1f stone is not None and measuring_cup is None and scale not None: if stone. pooeepaaenttonnsiner self. score += stone. parentContainne
469
wtw.e.ppaeettttteeeeee parenttonnnnniiee = measuring.cup:
measuring.cup.
Code diff
478
self. score +o
stone. parenttonntainerr
472
self. score +=
self.score += nesurring_cup.cr
inrrooerty("oonttinnlliii
474
self.score +=
self.score +
475
score
if score
476
self. ganeover True
self ganeOverr self-ganelfon True
477
self. ganeegn True
Table 7: Examples of GPT-4 fixing bugs via reflection. Note our reflection approach generates the entire code rather the patches, we show difference between code before and after reflection clarity purpose.


Reviews:
[{"review_id": "FmUEpZStve", "reviewer": "Reviewer_nYh3", "summary": "", "questions": "A.\t(See 1 above) Analysis of performance between similar vs dissimilar reference games? Analysis of the diversity of different codes present in the dataset to show the requirement of code generation beyond simple template matching. How do the codes in the dataset differ wrt programming concepts, semantics, etc?\n\nB.\tThe scoring function seems to be the most complex and open-ended code to be generated. What is the performance analysis on the scoring function generated? How is a valid scoring function defined, etc?\n\nC.\t(See 2 above) Sensitivity to prompt design? Explicitly asking to include distractors in the prompt?\n\nD.\t(See 3 above) Error analysis of code generated.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer: “Diversity in code generation: Since the dataset code is highly templated, it is unclear if the LM is template matching to generate the test game or has to perform non-trivial reasoning and code generation. [...] Line 368 mentions a setup where the reference game provided is dissimilar to the target game. However, no results are shown on LLM performance when similar vs dissimilar reference games are provided.”**\n\n**Response:** Empirically, we find that GPT-4 fails to produce long form meaningful text game code without the aid of a robust template. In the zero-shot setting, GPT-4 tends to hard-code game states without actually setting up a modular environment, a capability it appears to readily demonstrate in the few-shot setting. \n\nTo your second point about the similar and dissimilar reference games: when we say that two games are “similar,” we mean that the task specifications of the  two games match either in terms of task-critical objects, task-critical actions, or both. Such an alignment between a reference task specification and the desired output task specification might help the model to some extent, but it is not possible for the model to trivially copy-paste from the one-shot example to complete the task. New objects, actions, and scoring functions need to be defined, and even objects existing in the one-shot example need to be modified when necessary.\nOur pilot experiments showed that the model compliance performance of the “similar” setting and “dissimilar” setting was not significantly different, which underscores that the model is not relying on highly similar reference templates to generate the bulk of its performance.  This underscores our overall hypothesis, that the model appears to learn the general structure of the task from the template (such as how to architect a world model as a text game in general, including how to describe, populate, and inherit game objects represented as classes), rather than copying-and-pasting specific components of the reference game into the generated game.\n\n**Reviewer: “...analysis could be provided on model sensitivity to prompt design…”**\n\n**Response:** This is an interesting point, and it is true that the choice of prompts can exert a large influence over the performance of LLMs like GPT-4. However, unlike hyperparameters, it’s difficult to perform a thorough (or even non ad-hoc) search over the space of possible prompts. We found some improvements in our prompt selection over the course of the work, but there indeed remains the possibility that a different prompt would lead to even higher performance. However, we feel that the primary contribution of this paper is not about the specific implementational details to optimize the performance of GPT-4, but rather to present a new task on world modeling (that requires generating very large, ~400 line simulations in Python), paired with a suite of validated instruments and metrics for automatically evaluating the performance of those simulations.  Further prompt improvements (and improvements to overall task performance) are welcome in future work, but we feel that they’re beyond the scope of this paper.\n\n**Reviewer: “Error analysis is lacking: What are the common issues in generated code, parallels which can be drawn to reasoning capabilities, etc?”**\n\n**Response:** Our detailed evaluation metrics serve as an error analysis by nature (and, moreover, the size of the action space in each game necessitates using automated evaluations, since manual error analyses would be either strongly limited in scope or intractable). The four metrics we introduce in the paper are the four aspects that we believe are most important to a correct text game. The validity checks the syntactic correctness, the compliance checks whether task requirements are met, the winnability checks whether the game goal is correctly set, and the alignment checks whether the game correctly simulates the physical world. Model weakness can be reflected by a low score on any of these metrics. Specifically, regarding “the common issues in generated code”, GPT-4 gets a low alignment score due to its weakness at adding physical constraints to objects (e.g. some generated games allow the agent to take objects out of a closed container). Such a weakness is a reflection of the model’s common sense reasoning capability. More discussion of this can be found towards the end of the Discussion section in the paper.\n\n**Reviewer: “The scoring function seems to be the most complex and open-ended code to be generated. What is the performance analysis on the scoring function generated? How is a valid scoring function defined, etc?”**\n\n**Response:** This is a great question – the scoring function is indeed one of the more interesting and challenging game components to generate and evaluate. Nevertheless, we evaluate it in a few ways. First, our validity check determines whether the scoring function is syntactically correct and callable. Second, our winnability check gives a sense of whether the scoring function is feasible, since (for human evaluation) we require that it return a completed score when the human expert believes the task is completed, and (for automatic evaluation) we require that it returns a completed score when our ReAct/Reflexion-based planning agent believes the game should be completed. Finally, the physical reality alignment check will flag situations where an agent was given a positive score in a way that violates our simulation. While this is far from an exhaustive check of the scoring function’s soundness, we believe that it is a strong and sufficient contribution to the state-of-the-art.. \n\nIn practice (as we describe in the paper): (a) For nearly every generated game, the games were either obviously winnable or obviously unwinnable to the expert human evaluator, and (b) our automated metrics tend to underestimate winnability compared to expert human judgements (k=0.43), so we report a full human evaluation of winnability so that performance isn’t underestimated in this category.  We suspect that as agents become better common-sense reasoners, and better able to automatically play the generated games, their capacity for automatically evaluating scoring functions and winnability will increase proportionally.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tRelevant and interesting problem setup evaluating the reasoning and world knowledge of LLMs through code generation of text-based games.\n2.\tCode generated is hundreds of lines long, much longer than usual code generation problems.\n3.\tPromising results on preliminary evaluation of LM capabilities to generate game code, relevant direction of future research.", "reasons_to_reject": "1.\tDiversity in code generation: Since the dataset code is highly templated, it is unclear if the LM is template matching to generate the test game or has to perform non-trivial reasoning and code generation. \na.\tAuthors should provide an analysis of differences in code in the test set and one-shot example used. Line 368 mentions a setup where the reference game provided is dissimilar to the target game. However, no results are shown on LLM performance when similar vs dissimilar reference games are provided.\n2.\tSince the input is highly templated, an analysis could be provided on model sensitivity to prompt design. For example, one issue highlighted is the low performance of LLMs to include distractors in the generated code. The prompt shown in Appendix A.1 has the words, “If possible, there should be distractor objects…”. Does changing the prompt language to explicitly require distractors help?\n3.\tError analysis is lacking: What are the common issues in generated code, parallels which can be drawn to reasoning capabilities, etc?", "questions_to_authors": "A.\t(See 1 above) Analysis of performance between similar vs dissimilar reference games? Analysis of the diversity of different codes present in the dataset to show the requirement of code generation beyond simple template matching. How do the codes in the dataset differ wrt programming concepts, semantics, etc?\n\nB.\tThe scoring function seems to be the most complex and open-ended code to be generated. What is the performance analysis on the scoring function generated? How is a valid scoring function defined, etc?\n\nC.\t(See 2 above) Sensitivity to prompt design? Explicitly asking to include distractors in the prompt?\n\nD.\t(See 3 above) Error analysis of code generated.", "paper_topic_and_main_contributions": "This paper evaluates the capacity of LMs to generate world models expressed as Python code. A highly templated one-shot example is provided in context and the generated game is evaluated on many dimensions. Further, reflection (re-prompting with compiler feedback) is used to iteratively change generated code.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tLine 137: advancements in"}, "extra_text": {}}, {"review_id": "GQ9wzsuKtC", "reviewer": "Reviewer_QfRB", "summary": "", "questions": "A) What would be some approaches to diminishing the amount of tokens in this work? \nB) What issues do you foresee happening when people try to build on this work", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer: “Focusing solely on utilizing GPT-4 which can be limiting for future development of the work.”**\n\n**Response:** At the time of submission, GPT-4 was the only model that could generate meaningful programs simply based on a high-level game specification. We are optimistic about the future development of this work as this area is rapidly developing. We are excited to see the release of powerful open-source models. \n\nOn August 24th, Meta released the new set of CodeLlama models which claims to have a 100k context window. We used the CodeLlama-34b-Instruct model, slightly modified our prompts in order to comply with its expected format, and computed our automatic technical validity metrics. The results are as follows:\n\n—-------------------------------------------------------------------\n| Technical Validity Measurement |  gpt-4 | CodeLlama-34b-Instruct |\n| -------------------------------|--------|------------------------|\n| Game Initialization            | 0.854  |          0.573         |\n| Valid Actions  \t               | 0.802  |          0.552         |\n| Runnable Game                  | 0.281  |          0.198         |\n—-------------------------------------------------------------------\n\nWhile there is still a large gap between CodeLlama and GPT-4, CodeLlama is able to generate non-trivial results and generate some valid games. Our hope is that our world-modeling-as-simulation task and data are ready for when each class of model -- open source or closed source -- is able to best make use of this work.\n\n\n**Reviewer: “[...] context windows that this dataset needs are exceptionally high, over 4k on average according to line 470 [...]”**\n\n**Response:** While it is true that this task requires very long (8k+) context windows, we see this as aspirational work at the edge of current model capabilities, which move very quickly.  While the best open source code generation models had ~4k context windows when we submitted this work, already this week we're starting to see models (e.g. CodaLlama) that claim to have 100k context windows. \n\n**Reviewer: “What would be some approaches to diminishing the amount of tokens in this work?”**\n\n**Response:** This is a good question.  We have experimented with removing comments (which are approximately half of all tokens), though this impacts the spirit of this work, which is to provide well-architected and highly commented code templates for world modeling in a form that models can best use to learn to structure and implement their own simulations.  Ultimately, while some hard-coding could certainly improve the efficiency of our templates and generations, they would likely be much less generalizable.  With the current transition to large (e.g. 100k+) context models, we don't view the current context length as a limitation, and are excited to explore what even larger (and higher-fidelity) world model generations are possible in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The dataset is novel and the simplicity and understandability of what is being done is very clear\n* The authors give some valid automated evaluation metrics for future development\n* The work, although simple, can be utilized as a very good starting point for future development. For example, what if we supply some extra world knowledge/knowledge graphs into these games? I believe that the research that can stem from this is very good\n", "reasons_to_reject": "* The authors focus solely on utilizing GPT-4 which can be limiting for future development of the work\n* Additionally, the context windows that this dataset needs are exceptionally high, over 4k on average according to line 470, which once more can be very limiting for future development\n", "questions_to_authors": "A) What would be some approaches to diminishing the amount of tokens in this work? \nB) What issues do you foresee happening when people try to build on this work", "paper_topic_and_main_contributions": "The authors provide a very interesting dataset and benchmarks for representing commonsense simple situations as executable text-based games. The authors evaluate the usage of LLMs for creating/solving/fixing these kinds of text based games. The games generated from language models trained on the dataset provide some insight as to how LLMs represent the world from text, and can help to understand how to improve reasoning in these models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall very good writing! Simple to understand and clear testing.", "Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {"Missing_References": "Metafor: https://alumni.media.mit.edu/~hugo/publications/drafts/IUI2005-metafor.4.pdf\nFor some of the reasoning capabilities, it would be good to maybe cite this: https://arxiv.org/abs/2303.16421 as it shows that chatgpt has some reasoning of commonsense actions, but is (as you found in your work) very limited.\n"}}, {"review_id": "y4MN524dA4", "reviewer": "Reviewer_sjFC", "summary": "", "questions": "Question A. Overall did GPT-4 have higher or lower scores for Specification Compliance and Physical Reality Alignment metrics?\n\nQuestion B. Why did not you experiment with any open-sourced models? Are there no alternatives to the long context input capability of GPT-4? Did you consider using methods like Longformer (Beltagy et al. 2020)? If the concern is performance, I agree that it may not be as competitive as GPT-4. However, the sole use of a closed-access API would make it very hard to replicate and improve your results.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer: “All the experiments were conducted using GPT-4, [...] it would be beneficial to have a few open-source models capable of handling long contexts to ensure replicability.”**\n\n**Response:** We agree whole-heartedly with the importance of and desire for performant open-source models for this task. However, as you point out, there is simply at present a large gap in terms of both performance and capacity between closed-source and open-source large language models. Nevertheless, in order to address these concerns we have performed some preliminary investigations on recently-released open-source models. We began by investigating StarCoder, but were limited by its context size of 8k tokens. We attempted to compress our game prompts within this limit, but the resulting prompts failed to produce any outputs that complied with our specifications, making further automatic evaluation impossible. On August 24th, Meta released the new set of CodeLlama models. We used the CodeLlama-34b-Instruct model, slightly modified our prompts in order to comply with its expected format, and computed our automatic technical validity metrics. These preliminary results are presented below, and we intend to run the rest of the evaluation suite for the camera-ready submission.\n\n—-------------------------------------------------------------------\n| Technical Validity Measurement |  gpt-4 | CodeLlama-34b-Instruct |\n| -------------------------------|--------|------------------------|\n| Game Initialization            | 0.854  |          0.573         |\n| Valid Actions  \t               | 0.802  |          0.552         |\n| Runnable Game                  | 0.281  |          0.198         |\n—-------------------------------------------------------------------\n\nWe see that while this model is capable of producing technically valid games, it continues to lag GPT-4 in terms of performance. It’s possible that further tuning and further models will continue to narrow this gap, and we eagerly anticipate such improvements as future work. With regards to Longformer, while it’s possible that it could handle the long context lengths required by the ByteSized32 corpus, we point out that our experiments are all performed in-context without any fine-tuning. It seems unlikely that a model pre-trained in 2020 has the capacity to perform few-shot text game generation. \n\nAs for the potential volatility of GPT-4’s performance: while there is some possibility that model updates could lead to a change in its ability to complete this task, we feel that the main contribution of our paper is not the particular results of GPT-4, but the introduction of a new corpus, challenge task, and evaluation suite. Especially as open-source models improve in quality, the potential impact of this volatility will decrease. \n\n**Reviewer: “The above replicability concern is also true for evaluation as they use GPT4 to evaluate the quality for the following traits: Specification Compliance, Physical Reality Alignment, and Winnability.”**\n\n**Response:** The volatility of GPT-4 is potentially more of a concern for evaluation, as future work completed at different times might be evaluated with different checkpoints. However, we (a) believe that as future model checkpoints become more performant, evaluation performance will only increase, and (b) believe that any concerns regarding automatic evaluation performance can be ameliorated by continued periodic validation with human annotators. The relative cost of this human validation compared to the time saved by automatic evaluation means that this is not a prohibitive burden on future work.\n\n**Reviewer: “Overall did GPT-4 have higher or lower scores for Specification Compliance and Physical Reality Alignment metrics?”**\n\n**Response:** We would like to ask the reviewer to elaborate on this question so we can better understand and address it. \n\n**Reviewer: “The authors use GPT4 to evaluate its own output. This approach raises doubts about the soundness of the evaluation. [...] I agree with the authors' desire to automate the evaluation for large-scale experiments, but I believe the generation and evaluation should be assigned to separate models. It seems counterintuitive for a model to critique its own output, especially after multiple rounds of reflection.”**\n\n**Response:** While we understand and sympathize with the concerns of having a model both generate and critique its own output, we feel that the particular nature of our evaluations serves to alleviate this issue. Specifically, we designed each of our evaluations to be as narrow as possible: instead of holistically evaluating the entire game, we largely focus on discrete, binary questions which drastically limit the possibility for a model to “bias” its ratings. And, perhaps more importantly, we validate the performance of these evaluations by measuring their fidelity to expert human raters, which is the gold standard in validating research instruments. In the case where the model evaluations diverged from these ratings (winnability), we did not use them. We feel that this is the gold standard for the validation of a new instrument of any type. Finally, the process of self-reflection (which is inherently a form of model self-evaluation) has been demonstrated to improve downstream performance on a variety of tasks (Reflexion: Language Agents with Verbal Reinforcement Learning, Shinn et al., 2023) – and indeed we show self- reflection to improve the performance of this task by as much as 30% (Table 2), which is a strong validation of self-reflection. (As a related note, humans also use self-reflection to great effect when learning; see e.g. the work of cognitive scientist Michelene Chi)\n\n**Reviewer: “Soundness Score of 2, Replicability Score of 2”**\n**Response:** The reject criteria for this review centers around using closed (GPT-4) vs open source models.  While we’re very sympathetic to developing and using open source models, as we have shown, that is not easily possible for this task because it’s on the very edge of the  capabilities for even the most performant LLM that exists (that just happens to be closed source).  But, in the spirit of open science, during the author response period we have worked to train a state-of-the-art code open code generation model (CodeLlama) on this task, and preliminary experiments show it strongly underperforms GPT-4.  We believe that our methods, dataset, and evaluation are model agnostic, and have made every effort to create high-quality, easily-replicable open source work that will be made freely available on Github upon acceptance.  We look forward to a day in the near-term (perhaps even within the next 6 months) where open source models can use this data to become as performant as GPT-4 is now.  In light of our going the extra mile to address all your concerns regarding closed versus open models, we ask that you consider changing your Soundness and Reproducibility scores from a hard reject (2) to something you feel more appropriate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of projecting commonsense reasoning (or world modeling) as formal language (code) generation is intuitive.\n- The newly introduced corpus, BYTESIZED32, has the potential to unlock further opportunities for future research in this particular area.\n- The authors support the validity of their introduced metric through assessment by expert human annotators.\n- The main hypothesis of the paper is clear and well-presented, making it straightforward for readers to follow. In section 7, the authors provide explanations for the findings related to various implicit questions raised throughout the paper, such as whether LLMs can be utilized to evaluate their own simulations or if the simulations generated by LLMs accurately represent their internal worlds.", "reasons_to_reject": "- All the experiments were conducted using GPT-4, which is understandable considering the lack of open-source models that match its performance, especially at the time of submission. However, it would be beneficial to have a few open-source models capable of handling long contexts to ensure replicability. As it stands, future researchers cannot be certain if they are building upon this work since GPT-4's performance may not be consistent.\n- The above replicability concern is also true for evaluation as they use GPT4 to evaluate the quality for the following traits: Specification Compliance, Physical Reality Alignment, and Winnability.\n- The authors use GPT4 to evaluate its own output. This approach raises doubts about the soundness of the evaluation. Although the authors acknowledge this issue and present inter-annotator agreements with humans in the Discussion section, I am still uneasy about this circular evaluation. I agree with the authors' desire to automate the evaluation for large-scale experiments, but I believe the generation and evaluation should be assigned to separate models. It seems counterintuitive for a model to critique its own output, especially after multiple rounds of reflection.", "questions_to_authors": "Question A. Overall did GPT-4 have higher or lower scores for Specification Compliance and Physical Reality Alignment metrics?\n\nQuestion B. Why did not you experiment with any open-sourced models? Are there no alternatives to the long context input capability of GPT-4? Did you consider using methods like Longformer (Beltagy et al. 2020)? If the concern is performance, I agree that it may not be as competitive as GPT-4. However, the sole use of a closed-access API would make it very hard to replicate and improve your results.", "paper_topic_and_main_contributions": "This research paper investigates the language model's capability to create interactive world models for various common-sense reasoning tasks like washing dishes or starting a campfire. The authors assess this ability by generating programs that produce text-based gaming environments with the objective of completing the given task. These environments consist of devices and containers, some of which are necessary for the goal, while others serve as 'distractors' to challenge the language model. To create these games, the authors employ the GPT-4 model API using a single shot in-context learning (ICL) approach, with ICL examples representing similar games.\n\nMoreover, the paper demonstrates that allowing GPT-4 to self-reflect on its output significantly improves the runability of the generated programs from 28% to 58% (i.e., the generation results in 0 errors). The contributions of the research are fourfold:\n\n1. The paper presents a corpus of 32 world models expressed as text-based games implemented in Python.\n2. It introduces a set of metrics to evaluate the quality of the generated games.\n3. The authors emphasize the importance of self-reflection in enhancing the final quality of the games.\n4. The research highlights the current limitations in generating accurate world models and creating winnable games for this particular task.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {"Missing_References": "- [Implicit Representations of Meaning in Neural Language Models](https://aclanthology.org/2021.acl-long.143) (Li et al., ACL-IJCNLP 2021)\n- [Language Models as Agent Models](https://aclanthology.org/2022.findings-emnlp.423) (Andreas, Findings 2022)"}}]

Author rebuttal:

Meta-review: The authors provide an interesting dataset and benchmarks for representing commonsense simple situations as executable text-based games. 
The 2 out of 3 reviewers selected "Strong" for soundness score (1 selected "Good")
Reproducibility has also enough score
Decision: Accept-Main