Remember what you did so you know what to do next
Authors: Manuel Rafael Ciosici
Alex Hedges
Yash Kankanampati
Justin Andrew Martin
Marjorie Freedman
Ralph M. Weischedel
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
text games
TL;DR: We present a new ScienceWorld result that is 3.5x higher than that of reinforcement learning and matches some of the results of GPT-3-based architectures.

Abstract: We explore using the 6B parameter GPT-J language model to create a plan for a simulated robot to achieve 30 classes of goals in ScienceWorld, a text game simulator for elementary science experiments and for which previously published empirical work has shown large language models (LLM)s to be a poor fit (Wang et al., 2022). Using the Markov assumption, the LLM outperforms the state-of-the-art based on reinforcement learning by a factor of 1.4. When we fill the LLM’s input buffer with as many prior steps as will fit, improvement rises to 3.3x. Even when training on only 6.5% of the training data, we observe a 2.3x improvement over the state-of-the-art. Our experiments show that performance varies widely across the 30 classes of actions, indicating that averaging over tasks can hide significant performance issues.

Remember what you did so you know what to do next
Manuel R. Ciosici, Alex Hedges, Yash Kankanampati, Justin Martin, Marjorie Freedman, Ralph Weischedel manuelc@isi.c edu, mrfeisi. edu, weisched@isi.edd Information Sciences Institute, University of Southern California
Abstract
Wang al. (2022)). We selected ScienceWorld for this study since (1) supports several types of primitive actions, many types entitics, and 30 classes of tasks representing significantly differing challenges; (2) provides a simulator for the envi- ronment, including natural state changes (e.g., ice melting, a butterfly emerging from chrysalis); (3) may benefit from applying common edge; (4) might be improved by external knowledge from knowledge graph or from text; and (5) had substantial amount of training data so that we could explore the effect of training set size. Our empirical results contrast with those of Wang et al. (2022), who evaluated five architectures via individually training 30 different models, one for each of the 30 classes of tasks. Those models chose the next step given the current text feedback from the simulator and the previous action (i.e., Markov assumption). By contrast, we train a single GPT-J model to cover all 30 classes of tasks and explore filling the LLM's input buffer with as much game history as fits. A contemporancous study (Lin et al., 2023) uses two models: T5-large makes routine decisions; when needed for some decisions, the architecture calls GPT-4. Though performance relying GPT-4 ex- ceeds GPT-J, when Lin al. (2023) used GPT-3.5 Turbo, rather than GPT-4, performance is compa- rable to our results with GPT-J which uses 29-fold fewer parameters than GPT-3.5 Turbo.
We explore using moderately sized large lan- guage model (GPT-J 6B parameters) to create plan for a simulated robot to achieve 30 classes of goals in ScienceWorld, text game simulator for elementary science experiments. Previously published empirical work claimed that large language models (LLMs) poor fit (Wang et al., 2022) compared to reinforcement learn- ing. Using the Markov assumption (a single previous step), the LLM outperforms the rein- forcement lcarning-bast based approach by a factor of 1.4. When fill the LLM's input buffer with as many prior steps as possible, improve- ment rises 3.5x. Even when training on only 6.5% of the training data, observe 2.2x improvement over the reinforcement-caamiee based approach. Our experiments show that performance varies widely across the 30 classes actions, indicating that averaging over tasks hide significant performance issues.
knowl-
In work contemporaneous with ours, Lin et al. (2023) demonstrated a two-part approach (SwiftSage) that uses a small LLM (TS-large) complemented by OpenAl's massive L.LMs to achieve outstanding results in Science Our 6-B parameter, single-stage GPT-J matches the performance of SwiftSage's two-stage ar- chitecture when it incorporates GPT-3.5 turbo which has 29-times more parameters than GPT- J.
1 Introduction
What we have lcarned is:
Our research interest is using a midest-sized, self-hosted large language model (LLM) for plan- ning so that a robot can execute the plan(s) to achieve goals. Google's InnerMonologue project (Huang et al., 2023) uses PALM (540B parameters) and real robot; Lin et al. (2023) use small LLM (T5-large) but for many decisions relies the massive model GPT-4 (OpenAI, 2023) achieve goals in Sornd Our study uses GPT-J (6B parameters, Wang (2021)) in an open- source, simulated environment, ScienceWorld (SW,
By including a sequence previous actions, not just the most recent one, the average score triples compared to DRRN.
2. Performance of the LLM model degrades gracefully with much less training data. With only 6.5% of the training data, the score dou- bles compared DRRN.
3. If one uses all the training data provided, GPT- learns enough that adding a pre-condition system text does not help (i.c., overall perfor- mance stays the same or drops slightly).

---

4. Despite previous findings that LLMs perform poorly in ScrlnceWorld, we find they outper- form the DRRN by a factor of 3.5.
that require prior knowledge, like ScienceWorld Some recent work has shown how extremely large language models (over 100B parameters) can plan in simulated and real-world environments (Huang et al., 2022; Ahn et al., 2022; Huang et al., 2023). SayCan (Ahn et al., 2022) and Inner Mono- logue (Huang et al., 2023) use a huge language model (e.g., a 540B parameter PaLM, Chowdhery et al. (2022)) to create a plan for robot convert into actions. In contrast to InnerMonologue, the system in this paper uses a language model that is 90 times smaller (6B parameter GPT-J, Wang (2021)), which both plans and turns its plan into low-level actions, though in a simplified, simulated environment.
5. The 6B parameter GPT-J can match the per- formance of complex dual-LLM architectures that rely on LLMs 29 times larger than GPT-J.
2 Related Work
Reinforcement Learning (RL). Jansen (2022) sys- lematically surveyed the available text game envi- ronments, RL techniques, and some of the field's challenges, such as low-complexity text environ- ments and the limitations of classic planning tech- niques such as PDDL (Ghallab et al., 1998) when it comes to text game enviromments. Two of the sur- veyed RL methods (DRRN and CALM) achieved the highest scores in the original ScienceWorld evaluation (Wang et al., 2022). DRRN (He al., 2016) one of the most successful RL methods for text games. When playing Science World, DRRN requires assistance from the game in the form of a list of valid actions to choose, as Science World's action space is too large for the model ( (Wang et al., 2022). CALM (Yao et al., 2020) attempts to ad- dress DRRN's need for a list of valid actions by using a language model (GPT-2) to generate set of actions for its RL component to choose from. Despite succeeding in some text games, RL strug- gles to play games with large action spaces or those that require background knowledge not learnable from the game.
The task
3.1 ScienceWorld Description
ScienceWorld (Wang et al., 2022) is a multiroom text game that tests scientific reasoning abilities at the level of a standard elementary school sci- curriculum. It contains 30 different classes of tasks, each with many variations that challenge the player to adapt to environmental changes. The variations challenge players to demonstrate work- ing knowledge of scientific concepts and experi- ments rather than the declarative knowledge usually demonstrated by question question-answent answering LLMs. For example, some task variations prevent memoriza- tion (e.g., g.. by randomizing object locations). Others require different experimental path by changing the nature of the task (c.g., melting aluminum in- stead of ice) or by adding obstacles (e.g.. a broken stove requires players to find another heat source). Science World also tests players' ability to gener- alize by including tasks where the target object is known and can, therefore be solved using prior knowledge (e.g.. determining the boiling point of water) and tasks where the target object is unknown, and the player must perform scientific experiment (e.g., determining the boiling point of an unknown liquid substance). There are 7 207 total task vari- ations across the 30 tasks, split unevenly among the tasks with a minimum of 10 variations per task and a maximum of 1 386. The game allocates 50% of all variations for training, 25% for development, and 25% for testing.
Language models have recently surpassed RL methods in traditional RL benchmarks, including playing Atari games (Chen et al., 2021; Janner et al., 2021). Kim et al. (2022) used DistilBERT (Sanh et al., 2020) for its common sense to improve on DRRN in Zork1, Zork3, and 12 other text games that pre-date Science World. Singh et al. (2021) ex- plored building on BART (Lewis et al., 2020) as world model for playing Zorkl and eight other text games that pre-date ScienceWorld Most recently, Lin et al. (2023) employed T5 (Raffel et al., 2020) for an initial plan to achieve Science goals and GFT-4 (OpenAI, 2023) to re-plan when the T5 model was challenged.
Language models generally approach the prob- lem as sequence modeling task, performing the equivalent of offline reinforcement learning (Levine et al., 2020). L.MSs can also store large amounts of world knowledge, giving them a potential ad- vantage over pure RL. methods in environments
ScienceWorld has over 100 possible actions, of which few tens or hundreds are valid at any point. Players receive points when they achieve task-specific milestones and a score of O if they do

---

not perform the required task or perform it incor- rectly, e.g., focusing on an animal when the task is to find a plant. Furthermore, the design of the train/dev/test split is such that players do not en- counter test tasks during training. For example, melting ice is a training task, test and dev include variations of the melting task that involve different materials (e.g.. aluminum), requiring different tem- peratures, and even alternative heat sources (e.g., forge instead of a stove). The train/dev/test setup makes the game difficult for traditional RL systems as requires agents to generalize from the train set based on background world knowledge, which RL agents typically do not have.
transformer-based LLMs. Except for one experi- ment condition, we fill GPT-J's imput buffer with as much prior history as possible.
We train GPT-J-based models for the follow- ing conditions: All train Markov uses the entire training data of 359 games, but only gives GPT-J the prior action and game observation (i.e., uses a Markov assumption, like the agents tested by Wang et al.): All train uses the entire training data set and fills GPT-J's input buffer; No variations uses roughly half of the training data (as described in the previous section) and fills GPT-J's input buffer. Finally, Up to 18 games is trained only the small subset of 18 task variations per task (480 games, approximately 6.5% of the entire training data) but fills GPT-J's input buffer. We include a complete description of the training details and hyper-parameten in Appendix B.
3.2 Data generation
For any task variation, ScienceWorld can automati- cally generate one or more unique gold paths con- sisting of actions that will achieve the required goals. For training data, we generate up to 3 unique gold paths for variations in the train set, yielding 7 359 unique gameplays. From this training data set, we sample several smaller train sets. We cre- ate one training set containing only one unique gold path per task variation, yielding 3 589 unique gameplays. We also create task-balanced train- ing set by sampling a maximum of 18 gameplays per task from the half-sized training set', result- ing in 480 unique gameplays, almost evenly dis- tributed across the 30 ScienceWorke tasks but in- cluding only a small subset of each task's varia- tions, thus challenging agents to generalize from much less varied data. All the training data we used is available https: //gi thub. wwt. com/isi-vistal. vista/ edience_world__ _dta..
To evaluate potential improvements to the LLM from external components, we designed a precondi- tions checker to assist the LLM. The preconditions system parses Science keeps track of the state of doors and drawers (open or closed). If GPT-J attempts to reopen an object already in an open state, the preconditions system intercepts GPT-J's output preventing it from reach- ing Science World. The preconditions system then replies to GPT-J pretending to be the game and have performed the requested action. By prevent- ing redundant actions from reaching the game, the preconditions system prevents GPT-J from wasting game turns on needless close or open actions. The preconditions system aims to add programmatic support for common sense and alleviate LLM ten- dencies occasionally emit redundant actions.
text descriptions and
4 Results
4.2 Evaluation
4.1
Training
Unlike Wang al. (2022), who evaluated using sample of 300 dev games (16.5% of dev), or Lin et al. (2023) who evaluated on sample of only 270 test games, we evaluate over the full test set of 1 819 games. The prior literature is also incon- sistent about computing scores. As in this work, Wang et al. (2022) assume that failed games have score of O while Lin et al. (2023) use the score just prior to the agent failing. We include de- tailed comparison of our testing setup with that of Wang et al. (2022) and Lin al. (2023) in Ap- pendix C. Like Wang et al. (2022), we report results after evaluations with an environment limit of 100 actions/steps, meaning that during the evaluation,
We fine-tuned GPT-J on Science games tran- scribed as a natural language dialog between an agent and the game. The agent issues actions; the game replies with the current observation but does not mention the score. This plain text sequence- modeling format differs from prior approaches to framing the RL problem sequence modeling task (see Appendix A for details). Formulating games as dialog transcripts allows us to use any autoregressive pretrained LLM and take advantage of the vast prior knowledge encoded in modern
'For tasks with fewer than 18 variations, we take as many gold paths are there are task variations.

---

Games
Actions (% total)
Std. Dev.
Train
Score
Improv.
Won
Lost
Valid
AVs
IOs
IS
RAs
Other
DRRN
N/A
17.95
1.0x
GPT-J
2 All train Markov
7:3
24.74
1.05
1.4x
117
74
61.13
3.51
31.14
2.43
1.77
0.02
3
All train
7:359
62.57
4.32
3.5x
1012
383
90.51
0.06
4.68
2.51
2.07
0.17
4 No variations
388
63.35
6.94
3.5x
037
372
90.39
0.07
4.28
3.08
2.00
0.19
5 Up 18 games
480
39.78
2.35
2.2x
479
682
83.59
0.39
11.46
2.81
1.46
0.30
Table I: ScienceWorld scores over the 819 games that comprise the test set. The train column shows the number of train games available during training. Improv. relative improvement over DRRN. AVs Affordance Violations; IOs = Invalid Objects; IS Invalid Syntax; RAs Redundant Actions. Note that the sum of won and lost games does not equal the total number test games; as in many games, agents obtain a score that is neither (lose) nor 100 (win).
games end if an agent fails, wins, or reaches the action/step limit. We discuss the effect of the 100 action limit on evaluation in Appendix E.
tained 8 points on the entire test set, 3.1 times less than our Markov-instructed GPT-J (our weakest model). We postulate that our Markov-instructed GPT-J agent outperforms T5 due to our more nat- ural formulation of the task and GPT-J's longer maximum input length (2 048 word pieces vs. 512).
Out of the box, ScienceWorld only reports a player's score, no other performance metrics. To better understand a model's behavior, we analyzed our game transcripts to count the numbers of games lost and won (0 or 100 points) and to classify each action emitted by GPT-J either valid one of five categories of invalid actions: Affordance Vio- lations (AVs, e.g., the agent tried pour chair), Invalid Objects (IOs, i.e., the agent tries interact with a non-existent, hallucinated object), Invalid Syntax (IS, i.e., the action is valid English, but is not a valid ScienceWorld command), Redundant Actions (RAs, e.g., the agent tries to open an al- ready open door), and Other. We present our re- sults in Table 1, where each score is the mean five training runs with different random seeds. All our SPPT-J-bsed agents outperform the DRRN, the best- persopeefori agent in the original ScilnceWorld evaluation (Wang et al., 2022).
All train, the model trained on the entire set of 359 games, and which conditioned its actions on as much history as possible (on average the previ- ous 43.42 actions, see Appendix A), outperformed DRRN by factor of 3.5 (row 3). The model won 012 games (55% of the 819 test games) and rarely emitted invalid actions (90.51 % action valid- ity).
Adjusting the training data such that each vari- ation only appears with a single solution reduces the training data to slightly less than half. However, the resulting model (No variations, score almost one point higher than using all train- ing (row 3). The model still wins, on average, I 037 games (57%) with an action validity of 90.39%.
4) obtains
If we evaluate these two non-Markov models (All train training and No variations using the eval- uation methodology of Lin et al. (2023), the scores change little (62.59 vs. 62.57 for All train; 61.24 vs. 63.35 for No variations). While these scores are far from the top performing SwiftSage system (TS + GPT-4), they are similar to the 62.22 score obtained by the 75 GPT-3.5-turre SwiftSage sys- tem (Lin et al., 2023, Table 3). Thus, the single model 6B-parameter GPT-J model closes the gap to the approximately 29 times larger dual model which incorporated GPT-3.5-turbo.
All train Markov, the GPT-J model trained on the entire training data using the Markov assump- tion (i.e., conditioning actions only on the prior action and the game feedback) outperforms DRRN by a factor of 1.4 (row 2). Only 61.13% of the actions emitted by this model are valid, and almost a third (31.14%) involve non-existing objects. Our result starkly contrasts those of Wang et al. (2022) who evaluated an LLM trained with the Markov assumption T5 architecture (Raffel et al., 2020) initialized with the Maccaw weights (Tafjord and Clark, 2021)) and found performed poorly. De- spite T5 being twice the size of the GPT-J in our experiments (1 IIIB parameters vs. 6B), it only ob-
Interestingly, even when only 480 games are available for training (a mere 6.5% of the train- ing data), GPT-J learns to play ScienceWorke and

---

this work can help alleviate some of these behav- iors. Other external components could assist LLMs with structured knowledge. Such knowledge could be quickly inspected and updated by humans. Our team has yet to succeed in significantly assisting the LLM using external knowledge sources. While we have managed to teach the LLM to interact with Q&A system that could answer Is-A questions for the four classification tasks, such knowledge already present in the LLM, as evidenced by the high scores on classification tasks in Appendix D. RAs (56)
+P. RAs (%)
Score
All train Markov
1.77
0.95
+0.03
All train
2.07
0.99
-0.06
No variations
2.00
0.74
-0.14
Up to 18 games
1.46
0.67
+0.07
Table Percentage of Redundant Actions (RAs) emit- ted by our models before and after adding the precondi- tions system (+P) and the change in SW scores.
still achieves 2.2x improvement over DRRN (Up to 18 games, row 5). This model wins far fewer games (26.3%), emits fewer valid actions (83.59%), and has a tendency to try to interact with halluci- nated objects (11.46% of its actions involve non- existing objects). Despite the lower improvement over DRRN, this result indicates that GPT-J can generalize from even small amounts of training data. We hypothesize that the background knowl- edge GPT-J gained during its pre-trate tre-training drives its ability to generalize from such little training data.
Despite Wang et al. (2022)'s 's findings that LLMs perform poorly in Science we find that with careful formulation of the prompt and access to prior interactions with the game, even single rea- sonably sized LLM achieves 3.5x improvement over DRRN. The 6B-parameter GPT-J models can match the performance of SwiftSage 75 + GPT- 3.5 3.5-turbo, a more complex architecture that uses a model 29 times larger than GPT-J.
Even when learning from just few hundred games (6.5% of the training data), GPT-J achieves 2.2x improvement over DRRN, Despite the size- able overall score improvement, the LLM performs poorly on some ScienceWorle tasks. The strong correlation between the performance of our GPT- J- J-based models suggests that some tasks are gen- uinely more difficult for the LLM.
All our results have standard deviation of only few points, indicating good stability over the five training runs with different seeds. However, like Wang et al., we find that our models' performance varies greatly across the 30 tasks. We discuss the per-task performance in Appendix D.
We show the effect of the preconditions system in Table 2. The preconditions system almost halves the percentage of redundant actions passed from our models to Science World. However, the Sci- enceWorld score changes only a fraction of a point, inconsistently, up or down. The lack of change in the WofenceWWord score is not surprising. We eval- uate all models with an environment step/actions limit of 100, meaning we can interpret the percent- ages in the RAs column the average count of redundant actions per game. Even if the precondi- tions system prevented all redundant actions, that would only give each agent about two extra ac- tions per game, not enough to influence the final score meaningfully. Despite the lack of increase in the Science score, the preconditions sys- tem achieves the goal of removing non-common sensical actions from the LLM-game interaction.
Limitations
This paper shows that an agent based on GPT-J, a Large Language Model (LLM), can perform ele- mentary science experiments in a simulated text environment. Our GPT-J agent outperforms the state-of-the-art reinforcement learning models by factor of 3.5 using a single model rather than DRRN's 30 task-specific models. While GPT-J's performance is impressive, readers should remem- ber that Large Language Models like GPT-J have unforeseen gaps in knowledge and their behavior is often unpredictable. is appealing to look at our experience with GPT-J and assume that one can fine-tune LLMS and then task them with operating machinery or robots in the real world. However, due to LLMs' unpredictable behavior, such an ap- proach could result in material damage or even injure humans, animals, or the environment. This warning also applies cases where LLMs are al- lowed to operate computer APIs which, despite their virtual nature, can have undesired effects the real world. Should LLMs be allowed to oper-
5 Discussion and Conclusion
LLMs still exhibit non-common-sensicee behav- iors, such as emitting redundant actions or trying to interact with nonexistent objects. External com- ponents such as the precondition system shown in

---

ate APIs or real-world machinery, they should not be given complete control over the tools but oper- ate within carefully chosen boundaries strictly en- forced through software or through physical means.
Acknowledgements
This material is based on research supported by DARPA under agreement number N66001-19. 24032. The U.S. Government is authorized to reproduce and distribute reprints for Governmen- tal purposes, notwithstanding any copyright nota- tion thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or im- plied, of DARPA or the U.S. Government.
When considering using LLMs to plan and exe- cute in new domains, one should remember that the LLM used in this paper benefited not just from the opportunity to learn from example experiments but also from its background knowledge. By its nature, knowledge of elementary science is widely spread in texts on the web, such as those that comprised GPT-J's pre-training data. For detailed presen- tation of GPT-J's pre-training corpus, we direct readers to Gao et al. (2020).
References
Finally, when interacting in ScienceWorld, LLMs have a text-only view of the environment. The environment is described via text with just the detail necessary for accomplishing the tasks, akin to having access to perfect Computer Vision and system that can filter out trifling information, allow- ing the LLM focus planning. Access per- feet Computer Vision and relevance filters is a sub- stantial limitation common to all approaches that operate in text-only environments. More research is necessary to understand how to teach I.MSs to in- corporate information from other modalities, such as sight and sound. Approaches such as PaL.- E (Driess et al., 2023) have shown that it is possi- ble to integrate multiple modalities into Extremely Large Language Models. But, 526B parameters, these models' compute and energy requirements seem to make them impractical for onboard pro- cessing of mobile robotics platforms.
Michael Ahn, Authony Brohan, Noah Brown, Yev- gen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Pe- ter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes. Pierre Sermanet, Nico- las Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022. Do As I Can and Not As Say: Grounding Language in Robotic Affordances. arXiv:2204.0169 (cs].
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision trans- former: Reinforcement learning via sequence model- ing. In Advances in Neural Information Processing Systems, volume 34, pages 15084-15097. Curran As- sociates, Inc.
Ethics Statement
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal. Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz. Erica Moreira. Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slay Petrov,
Environmental impact questions arise for any sci- entific work that involves Large Language Mod- els (LLMs). Most of the energy consumption of LLMs occurs during pre-training (Patterson et al., 2021). This work relies on already pre-trained lan- guage models, which we only fine-tune. GPT-J, an LLM with 6B parameters, is small enough to be fine-tuned on single modern compute node, contrasted with LLMs using hundreds of billions of parameters. For fine-tuning, we use nodes with 4x NVIDIA A6000 GPUs, and we further increase the training efficiency by offloading the optimizer from GPU memory via DeepSpeed (Ren al., 2021; Ra- jbhandari et al., 2020), thus climinating the need for the newest, most power-hungry GPUs.

---

and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv:220400231 [cs].
Minsoo Kim, Yeonjoon Jung, Dohyeon Lee, and Seung- won Hwang. 2022. PLM-based world models for text-based games. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1324-1341, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Danny Driess. Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter. Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, igor Mordatch, and Pete Florence. 2023. M-LM-E:: An Embodied Multimodal Language Model. 1.Xiv22303.03378 [es].
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline Reinforcement Learning: Tuto- rial, Review, and Perspectives on Open Problems. arXiv:2005.01643 [cs].
Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He. Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Model- ing. arXiv:2101.00 00027 [cs].
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene for natural language generation, translation, and com- prehension, In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online, Association for Computa- tional Linguistics.
pre-training
Malik Ghallab, Adele Howe, Craig Knoblock, Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. 1998. PDDL-the the plan- ning domain definition language. Technical Report CVC TR-98 003/D003DCSSSS TR- TR-1165, Yale Center for Com- pulational Vision and Control.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chan- dra Bhagavatula, Yejin Choi, and Xiang Ren. 2023. SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Li- hong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL).
OpenAL 2023. Gpt-4 technical report.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang. Alquue Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Car- bon emissions and large neural network training. arXiv:2104.10355 [cs].
ACL- Association for Computational Linguisties.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9118-918 91477 PMLR.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Wenlong Huang. Fei Xia. Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu. Sergey Levine, Karol Hausman, and brian ichter. 2023. In- ner monologue: Embodied reasoning through plan- ning with language models. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1769-1782 PMLR.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase. and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting, Networking, Storage and Analysis, pages I- 16.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Am- inabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO- Offload: Democratizing Billion-Scale model train- ing. In 2021 USENIX Annual Technical Conference (USENIX ATCENIXATCC21 21), pages 551-564 USENIX Associ- ation.
Michael Janner, Qiyang Li, and Sergey Levine. 2021. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Informa- tion Processing Systems, volume 34, pages 1273- 1286. Curran Associates, Inc
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Peter Jansen. 2022. A systematic survey of text worlds as embodied natural language environments. Pro- ceedings of the 3rd Wordplay: When Langugge Meets Games Workshop (Wordplay 2022), pages 1-15, Seat- tle. United States. Association for Computational Linguistics.
Ishika Singh, Gargi Singh, and Ashutosh Modi. 2021. Pre-trained language models as prior knowledge for playing text-based games.

---

Oyvind Tafjord and Peter Clark. 2021. General- Purpose Question-Answente with Macaw. arXiv:2109.02593 [cs].
Number actions GPT-J's prompt during training (AVG: 43.42)
3500
Ben Wang- 2021.
JasstTransforme-JAA Model- Parallel Implementation of Transformer Lan- guage Model with JAX. https::/githhsi com/ wta.r/oll//mees-ttrsfffrrrrr
3000
Number of bulfors
2500
2000
Ruoyao Wang, Peter Jansen, Mare-Alexante Côté, and Prithviraj Ammanabrolu. 2022. ScienceWorld: your agent smarter than 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279-11211 177991 Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
1000
900
50
Number
Shunyu Yao. Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and ex- plore: Language models for action generation in text- based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 8736-8754, Online. Association for Computational Linguistics.
Figure I: Histogram illustrating the number of actions that fit in GPT- input for the All train training set.
Sample ScienceWorle dialog
We fine-tune GPT-J on formulation of Science- World games transcribed as a natural language di- alog between an agent (A) and the game (G). The agent issues actions, and the game replies with the current observation. Unlike prior work that mod- eled RL algorithms using language models, we provide no information about the current score yet to be achieved. Listing 1 illustrates the dialog format. Training GPT-J becomes autore- gressive text modeling task over the dialog format. During inference, we fill GPT-J's input buffer with as much history as possible in the form of prior (action, observalion) tuples in the dialog format and expect GPT-J to generate an action row (i.e., the text after A :). GPT-J's input buffer, limited to maximum of 2048 word pieces, accommodate, average, 43 prior actions using the dialog format (Figure 1).
Our plain text, natural language sequence model- ing format differs from prior approaches to framing Reinforcement Learning as sequence modeling task. Instead of natural language input, Janner al. (2021) use special tokens to represent a discretized version of the previous states and actions as input to a transformer decoder that predicts the next action. Notably, the transformer is bespoke model, not pre-trained Language Model. For The Decision Transformer (Chen et al., 2021), projected into continuous space using learned linear projection, and the projections become The Decision Transformer's input. Another difference between our simplified dialog modeling technique and The Decision Transformer that our input
inputs are

---

does not contain information about the game score of the yet unrealized score. This frees up space in the LLMs input buffer to allow for more prior history.
ways to compute mean performance in Science- World. In Table 1, we report scores averaged over the 1 819 games in the test set. This mean gives equal weight to each game in the set. But, since variations are nonuniformly distributed over tasks, the contribution of each task to the final score is determined by the number task of variations in the test set (e.g., the Changes of State (Boiling) task only has 9 variations in test, while Friction (known surfaces) has 348 variations). Wang et (2022) computed the mean score by calculating the mean score per task and then the mean score over the 30 tasks. This method gives equal weight to each task while potentially obscuring failures in tasks with many variations. The difference between the two is similar to that between micro- and macro-precision. and both methods are valid. In Table 3, com- pare the micro- and macro- averaged scores. The macro score is lower than the micro score for GPT- J-based models. Nonetheless, the best-performing GPT-J-based model still outperforms DRNN by a factor 2.7.
B Training details
We trained GPT-J with b representations us- ing DeepSpeed stage 2 model parallelism over 4 NVIDIA A6000 GPUs. We used the Adam W opti- mizer with a weight decay of 0.01 and offloaded it system memory via DeepSpeed (Ren et al., 2021; Rajbhandari et al., 2020). We train each model for two epochs over its corresponding train data and a batch of 16 (one input per batch, four gradient accumulation steps, GPUs) and a learning rate of le-4.
C Evaluation setup
After training on each model's respective training data, we evaluated each agent on the task variations provided in ScienceWorld test set. The evaluation required agents to play total of 819 games split unevenly over the 30 tasks. The distribution of games to tasks appears in the Games column of Table 5.
Micro Score
Improv.
Macro Score
Improv.
DRRN
17.95
1.0x
18.75
1.0x
By contrast, Wang et al. (2022) measured DRRN's performance against a random sample of the dev set every 500 steps during DRRN's 's train- ing and reported the average score for the last 10% of training steps. For a fair comparison between DRRN and models, trained DRRN using the published code and parameters, evaluated against the entire ScienceWorle test set, and re- ported DRRN's performance Table 1.
All train Markov
25.19
1.4x
22.73
1.2x
All train
62.57
3.5x
50.60
2.7x
No variations
63.35
3.5x
50.61
2.7x
Up 18 games
39.78
34.29
1.8x
Table 3: Comparison of micro- and macro- scores.
D Scores per Task
Before we tested our trained DRRN model on the test set, we confirmed that our training was successful by performing the same evaluation as in the original Science World paper. In this evaluation, trained DRRN obtained 18.92 points, similar the published 17 points. Since these results are close, we conclude that our retrained DRRN model matches the capabilitics of the DRRN model from the original Science World paper. For completeness, we also evaluated our trained DRRN model on the entire dev set, where it obtained 18.1 points com- pared to the 17.95 that it obtained when evaluated on the entire test set (i.e., the result we include in Table 1). The difference in performance between DRRN on dev and test similar what we ob- served for our GPT---based models.
Table shows that our GPT-J-based agent can achieve up to 3.5x the score of DRRN. Looking into the average scores separated by task provides further insight into the successes and shortcomings of the LLM approach to playing ScienceWorke Table 5 shows results split by task and by task cate- gory. Each row in the table is based on five training and evaluation runs with different seeds. In each cell, we present the mean score and the standard deviation.
Like Wang et al. (2022), we find that perfor- mance varies considcrably from task to task. Even for our best model, All train, the scores range from points (on Mixing paints (tertiary colors)) to 100 points on several tasks, e.g., Identify longest-lived animal. However, is not just the score that varies from task to task. We observe many variations
One final thing to note is that there are several

---

Listing 1: An example transcript used to train GPT-J.
Your task is to measure the melting point of chocolate, which is located around the kitchen. First, focus on the thermometer. Next, focus on the chocolate. If the melting point of chocolate is above -10.0 8 degrees celsius focus on the blue box. If the melting point of chocolate is below -10.0 degrees celsius, focus on the orange box. The boxes are located around the kitchen.
A: look around
G: This room is called the hallway. In it, see: the agent; substance called air; picture. You also see: A door to the art studio (that is closed); A door to the bedroom (that is closed); A door to the greenhouse that is closed); door to the kitchen (that is closed); door to the living room (that is closed); door to the workshop (that is closed)
A: inventory
G: In your inventory, you see: an orange
A: open door kitchen
G: The door is now
A: go to kitchen
G: You
to the kitchen. A: look around
G: This
called the kitchen. In it, you the agent; substance called air; blue box (containing nothing); chair. the chair is: nothing.; counter. On the counter is: bowl (containing red apple, banana, orange, potato), drawer.; cupboard. The cupboard door is closed. freezer. The freezer doar closed. a fridge. The fridge is closed.
glass jar (containing substance called sodium chloride); lighter: a orange box (containing nothing); oven, which The oven door closed.
turned off. painting; sink, which is turned off. the sink is: nothing.; substance called soap; stopwatch, which deactivated.
stove, which turned off, On the stove is: nothing.: a table. On table is: a glass cup (containing nothing).; a thermometer currently reading a temperature
degrees celsius. You also see: door to the bathroom (that is closed); A door to the hallway (that is open); ; door to the outside (that is closed)
A: pick up thermometer
G: You move the thermometer to the inventory.
A: focus on thermometer in inventory
G: You focus on the thermometer.
A: look around
G: This room is called the kitchen. In it, you see: the agent; substance called air; a blue box (containing nothing); a chair. On the chair is: nothing.; a counter. On the counter is: a bowl (containing a red apple, banana, an orange, potato), drawer cupboard. The cupboard door is closed. freezer. The freezer door is closed. is closed.
fridge. The fridge door glass jar (containing a substance called sodium chloride); lighter; a orange box (containing nothing); oven, which is turned off The oven door is closed. sink is: nothing.; deactivated.
painting; sink, which is turned off. In the substance called soap; stopwatch, which is stove, which is turned off On the stove is: nothing.; table. On the table is: glass door to the bathroom (that is closed); A door to the hallway (that is open); A door to the outside (that is closed)
(containing nothing).. You also see: A
A: open freezer
G: The freezer is open.
A: look in freezer
G: Inside the freezer is: nothing

---

the values for standard deviation (shown in paren- theses next each score), with many tasks having a standard deviation multiple times larger than the ones Table and included at the bottom of Ta- ble 5.
Number actions in training games (AVG: 56.90)
200
500
Number of Games
All four models' performance varies lot with the task. A strong Pearson correlation between the per-task results of all four models (Table hints that some tasks are genuinely more difficult for our GPT-J models.
400
200
All train
All
vari- ations
Up to 18 games
train Markov
Number
Figure Histogram of the number of actions in games from the No variations train set.
All train
All train Markov
0.83
No variations
0.99
0.84
Up to 18 games
0.89
0.73
0.91
turns/actions, the other GPT-J-based models con- tinue meaningfully accumulate points up the evaluation limit of 100 turns/actions. Past the 100 actions limit, the increase in ScienceWorld scores is marginal for all our models.
Table 4: Pearson correlation between the results in the
corresponding columns of Table 5.
E Environment step limits
Evaluating the performance of players in turn- based game raises the question of after how many steps should report and compare performance. Wang al. (2022) and Lin al. (2023) report the performance models allowed to play up to 100 actions. This paper also reports performance up to a maximum of 100 actions.
However, the training games that Science World generates suggest that evaluations with than 100 steps might be necessary. Figure shows that, while the average length of training games is 57 actions and the great majority of games in the train- ing set can be completed in less than 100 steps, a considerable number of games require more than 100 actions to complete. (Lin et al.. 2023) cited computational costs argument for evaluating up only 100 actions
Computational costs are less important for the GPT-J models we evaluate since the models are small enough to run on a modern workstation. However, we still report results after maximum of 100 actions because (1) it makes our results easier to compare to prior literature and (2) because see diminishing returns from running evaluation past 100 actions. Figure 3 shows our models' Sci- enceWorld test score as a function of the number of game turns/actioos. While the Markov assumption GPT-J (All train Markov) flat-lines after about 50

---

Task
Games
All train
All train Markov
No variations
Up to 18 games
Matter
Changes of State (Boiling)
9
1.13 (2.29)
0.18 (0.10)
2.51 (3.77)
7,20 (6.93)
Changes of State (Melting)
9
0.31 (0.37)
0,13 (0.12)
2,44 (3.74)
14.42 (11,18)
Changes of Stare (Freezing)
0.00 (0.00)
0.00 (0.00)
0,96 (2.14)
8.84 (8.63)
Changes State (Any)
0.24 (0.18)
0.04 (0.06)
1.91 (3.47)
19.00 (14.80)
Measurement
Use T'errmometer
135
63.39 (15.55)
20.23 (7.18)
73.59 (21.92)
29.30 (6.47)
Measuring Boiling Point (known)
109
17.76 (8.27)
13,90 (11.68)
11.28 (10,98)
22.88 (13,18)
Measuring Boiling Point (unknown)
52.03 (17.23)
53.58 (32.09)
59.19 (7.20)
52.60 (21.57)
Electricity
Create circuit
5
82.08 (11.26)
25.00 (9.10)
92.36 (4.80)
77.80 (8.27)
Renewable vs Non-renewable Energy
5
59.36 (15.15)
12.72 (4.20)
58.72 (11.50)
45.28 (11.29)
Test Conductivity (known)
225
40.02 (21,31)
16,35 (3.90)
42.58 (16,30)
30.57 (14,14)
Test Conductivity (unknown)
150
52.20 (2.56)
37.66 (0.34)
60.67 (21.67)
46. (3.74)
Classification
Find living thing
96.62 (7.55)
50.10
99.84 (0.35)
72.18 (12.33)
Find non-living thing
(0.00)
(5.61)
99.73 (0.60)
(19.29)
Find a plant
75
99.47 (1.19)
43.71 (6.29)
99.40 (0.68)
46.65 (11.72)
Find an animal
75
96.87 (7.01)
48.68 (6.54)
99.91 (0.20)
71.79 (26.36)
Biology
Grow plant
33
11.92 (0.45)
5,95 (0.14)
12.47 (0.99)
9.09 (1.47)
Grow fruit
46.72 (4.93)
11.60 (0.92)
44.78 (9.55)
25.08 (11.05)
Chemistry
Mixing (generic)
41.13 (8.86)
10.13 (3.38)
35.63 (8.84)
28.45 (5.23)
Mixing paints (secondary colours)
9
0.00 (0.00)
10.67 (3.65)
0.00 (0.00)
0.67 (1.49)
Mixing paints (tertiary colours)
9
0.00 (0.00)
5.27 (1.88)
0.00 (0.00)
0.58 (1.29)
Biology
kdentify longest-lived animal
32
100.00 (0.00)
71.25 (2.61)
98.75 (1.71)
60.63 (22.30)
ldletify shortest-lived animal
(0.00)
66.25 (4.76)
(4.
(17.48)
Identify longest-then- thenssttt--- seeem-shotest-livee lived animal
32
100.00 (0.00)
58.75 (7.36)
98.75 (1.71)
50.41 (21.90)
Identify life stages (plant)
5
45.7 73.72 (6.58)
19,40 (6.47)
46.56 (19.76)
29.20 (10.53)
kdentify life stages (animal)
4
6.00 (10.84)
8.60 (2.61)
12.00 (11.51)
7.00 (5.70)
Forces
Inclined Planes (determine angle)
76.67 (6.84)
12.48 (1.05)
59.83 (8.07)
39.86 (4.55)
Friction (known surfaces)
348
81.86 (4.19)
12.46 (0.61)
82.18 (4.73)
43.92 (2.00)
Friction (unknown surfaces)
42
74.12 (8.15)
13:48 (1.02)
73.52 (8.23)
42.00 (6.72)
Biology
Mendelian Genetics (known plants)
30
35.79 (17.24)
8.57 (0.00)
24.64 (15.14)
22.85 (6.16)
Mendelian Genetics (unknown plants)
120
36.49 (12.83)
7.59 (0.02)
29.04 (7.00)
23.44 (6.34)
Overall
819
62.57 (4.32)
24.74 (1.05)
63.35 (6.94)
39.78 (2.35)
Table Results by task GPT-J-baser agents

---

001
85
100 T0s
140 145 150 155 105 120 180 195 120 125 130
Al train
All crain Markow
No voriations
Up to 1. gomes
Figure 3: Mean ScienceWorld score as function of game actions (steps). Each line the average evaluating five trained with different seeds. We stopped running All train Markov early due to its consistent flat line.


Reviews:
[{"review_id": "laeagteQOY", "reviewer": "Reviewer_4gZk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we particularly appreciate those changes that are specific. On some other points, there seems to be a difference of opinion with us and possibly with the other reviewers; we address those first.\n\n* You made the following point: \n\n> It's trivial to see that including more context for GPT-J will improve the result compared to only giving it the previous action and observation, and including less training data will reduce that result.\n\nThose two points may be obvious to some. However, the context for our study appeared in EMNLP 2022, Ruoyao et al. [1]. The authors claim, \n\n> \"We evaluate the performance of 5 state-of-the- art reinforcement learning and language model agents on these benchmarks, empirically demonstrating that existing agents for text-based games as well as novel models adapted from transformer-based scientific question-answering solvers perform poorly on tasks (such as melting ice) that 5th grade science students can perform with ease.\"\n\nWe feel that negating the published claim that transformers perform poorly is worth noting and worth showing that providing adequate context to the transformer is what the original authors missed.\n\n_____________\n\n* You judge that \"The paper's contribution is minimal.\"  Below is an extract from the ACL’s guidelines for short papers \n\n> \"Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages. Some kinds of short papers are:\n>\n>        ◦ \"A small, focused contribution\n>        ◦ \"A negative result\n>        ◦ \"An opinion piece\n>        ◦ \"An interesting application nugget\"\n\nWe feel that correcting incorrect claims in a recent paper, showing remarkably improved (>3x) performance on the 30 classes of tasks in ScienceWorld, estimating confidence intervals on measured performance, and showing that far less training data is needed than employed in the earlier paper while simultaneously doubling scores with ~7% of the training data, together qualify as a small, focused contribution.\n\n_____________\n\nOn the following points, we are glad to make concrete changes:\n\n* We agree when you point out that \"The authors do not test their model on novel unseen task, just a variation of a task seen during training.\" In fact, we have tested training on most of the classes of tasks and tested on classes of tasks that were not seen during training. We did not include those results in the draft due to the four-page limitation. In short, for some classes of tasks, performance is poor for some classes of task; for other new classes of tasks, performance is good.  If this short paper were accepted, we could use the extra page to report those results or would gladly add them to the appendix.\n* You state that \"explore stuffing the LLM’s input buffer\" and \"It is seductive to look at our experience with GPT-J\" are not suitable for an academic paper and later mention stuffing and seductive again.  We will replace *stuffing* with *filling* and *seductive* with *attractive*. \n* We will add the two references you mention to the discussion of related work and distinguish our work from theirs:  https://arxiv.org/abs/2107.08408 and https://aclanthology.org/2022.emnlp-main.86/ \n* We will adopt the table style you recommend in https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf for our tables 1 and 2, though those guidelines do not seem to be specified by the ACL.\n* We can make the deletions from the Ethics Statement per your request. However, please note the ACL’s guidance on the Ethics Statement; The EMNLP 2023 call for papers [2] points authors to the EMNLP 2021 FAQ [3], where authors are urged to address misuse potential and the environmental impact of computing used for experiments. Thus, what you want us to delete seemed pertinent given the ACL’s FAQ.\n* You scored the reproducibility of our paper using the second lowest possible grade. We will amend the draft to state where the original ScienceWorld training data may be obtained:\n    * The training and evaluation data we used are generated by ScienceWorld (Ruoyao et al., [1]), an Apache-licensed open-source environment. To improve the reproducibility of our work beyond the details in the paper, we will include a link for readers to download the exact training data sets for all evaluations.\n    * The LM model we fine-tune is GPT-J, an Apache-licensed open-source model, trained on The Pile, an open-source corpus.\n    * We followed the EMNLP guidelines on Optional Supplementary Materials [4] and put in the appendix sample inputs (Appendix A), hyperparameter details (Appendix B), and evaluation setup (Appendix C). All these are also pointed to from the paper body (L174, L195, L219).\n\n## References\n\n[1] Wang, Ruoyao, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. \"ScienceWorld: Is Your Agent Smarter than a 5th Grader?\" In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11279–98. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. https://aclanthology.org/2022.emnlp-main.775\n\n[2] https://2023.emnlp.org/calls/main_conference_papers/#ethics-policy\n\n[3] https://2021.emnlp.org/call-for-papers/ethics-faq \n\n[4] https://2023.emnlp.org/calls/main_conference_papers/#optional-supplementary-materials", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows that LLMs can achieve decent results on the ScienceWorld environment by making use of its world and commonsense knowledge.", "reasons_to_reject": "There are many reasons to reject this paper:\n- The paper is poorly written and structured. It contains words such as \"explore *stuffing* the LLM’s input buffer\" and \"It is *seductive* to look at our experience with GPT-J\" that are not suitable for an academic paper.\n- The paper's contribution is minimal. It's trivial to see that including more context for GPT-J will improve the result compared to only giving it the previous action and observation, and including less training data will reduce that result.\n- The experiments design are not clearly motivated. Specifically, the **no variations** and the **Up to 18 games** finetuning methods.\n- The authors do not test their model on novel unseen task, just a variation of a task seen during training. \n\nIn general, I think the paper requires serious revisions and would then might be considered a good workshop submission, but not a conference submission.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper finetunes GPT-J on all 30 classes of tasks of the ScienceWorld environment using 4 different methods:\n\n1. **All train Markov** uses the entire training data but only provide the prior action and game observation as input (Markov assumption). \n2.  **All train** uses the entire training set but \"fill's GPT-J's input buffer\" by previous actions and observation.\n3. **No variations** Like (2) but using half of the training data.\n4. **Up to 18 games** Like (2) but trained on a small subset of 18 task variations per task.\n\nThe model interacts with the environment through a dialog where the agent (GPT-J) issues actions and the game replies with the current observation and score. The authors also use a *preconditions checker* to assist the LLM. It intercepts GPT-J's output in case the model is providing redundant actions to the environment (opening a door that is already open). \n\nThe authors show the training using method #2 triples the average score compared to method #1, while method #4 only doubles the result as compared to previous SoTA, and that the preconditions system is unnecessary if all of the training data is used.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "There are several works that use LMs to interact with text-based environments. Here are a few examples:\n- https://arxiv.org/abs/2107.08408\n- https://aclanthology.org/2022.emnlp-main.86/"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a lot of improvements that can be made:\n\n1. Tables 1 and 2 needs proper styling. Please see for reference: https://people.inf.ethz.ch/markusp/teaching/guides/guide-tables.pdf\n2. Please avoid using words such as \"stuffing\" and \"seductive\"\n3. The Limitations and Ethics sections need to be rewritten entirely. Saying that your finetuned LLM shouldn't interact in the real world is a trivial point. For example, the paragraph below shouldn't be there.\n\n> LLMs' unpredictable behavior, such an approach could result in material damage or even injure humans, animals, or the environment. This warning also applies to cases where LLMs are allowed to operate computer APIs which, despite their virtual nature, can have undesired effects in the real world. Should LLMs be allowed to operate APIs or real-world machinery, they should not be given complete control over the tools but operate within carefully chosen boundaries strictly enforced through software or physical means.\n\n4. Saying that you are based in California and that the:\n> California grid was 59% powered by energy from zero-carbon sources \n\nshould not be an Ethics statement."}}, {"review_id": "L28TjQ1ugP", "reviewer": "Reviewer_4kDw", "summary": "", "questions": "1. Are there ways to disentangle the effect of GPT-J’s pre-trained knowledge from its reasoning ability? Perhaps by exploring how well a non-pre-trained GPT-J performs on these tasks, when simply fine-tuned to perform them?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The reviewer asked**:\n\n> Are there ways to disentangle the effect of GPT-J’s pre-trained knowledge from its reasoning ability? Perhaps by exploring how well a non-pre-trained GPT-J performs on these tasks, when simply fine-tuned to perform them?\n\n**Reply:**\n\nExcellent question. Yes. One way could be to take an out-of-the-box language model and only use in-context learning to inform the LM how to play the game and what actions are possible. Unfortunately, GPT-J does not have a good enough in-context learning ability to play ScienceWorld. This was one of the first experiments we tried. GPT-J could not learn the ScienceWorld action syntax. We tried this experiment with GPT-3, the first LM that exhibited good in-context learning abilities. While GPT-3 learned the syntax of ScienceWorld commands, it did not play more than a handful of actions before losing the game by issuing an incorrect command or getting stuck in a loop.\n\nThe ScienceWorld game as originally published by Ruoyao et al. (2022) incorporates 4 pairs of known vs. unknown tasks designed to probe the difference between knowledge the system has and knowledge it learns through game play. In the known condition, the object is assigned a real-world name (e.g., *metal fork*) and players can rely on prior knowledge about that object. In the unknown condition, the object is an English language variable name (e.g., *unknown substance B*) forcing players to perform experiments and apply what they learn to complete the task. Table 5 in the appendix presents results broken by task, thereby allowing readers to see each of the four comparisons directly (e.g., comparing *Measuring Boiling Point (Known)* with *Measuring Boiling Point (unknown)*. At least at the task level, the results here are difficult to interpret. For two of the pairings, results on unknown objects are much higher in absolute for all four of the conditions we explore. For the other two tasks, there is less of a pattern. In some training conditions, known is higher than unknown; in others, the scores vary in absolute value. There are possible explanations, for example: (1) prior knowledge about an object could be harder for the model to overcome in the case of unexpected situations (e.g., encountering a broken stove); or (2) similarity in the naming of the unknown objects could simplify the transformer’s ability to transfer information across tasks, e.g., making it easier to share knowledge between *Unknown Substance 1* and *Unknown Substance 2* than it is to transfer across, e.g., *Metal Fork* and *Glass Jar*.\n    \nGiven the scope of a short paper, we have not performed sufficient analysis to explore why, but if accepted we appreciate the insight that this would be good fodder to include as a part of future questions raised by this work.\n\n_________________\n\n**The reviewer wrote**\n\n> “Acknowledging space limitations, some more analysis into why the LM likes to hallucinate invalid objects (even in the full-context settings), given that’s such a massive proportion of the errors, would be useful. This being the predominant source of error seems counter-intuitive to me because the LM should technically be able to refer to its context as to what objects are present, and be able to easily learn not to generate objects that weren’t previously referred to in context.”\n\n**Reply**: \n\nIn the full-context setting, 4.28% of emitted actions include invalid objects (row 3 in Table 1). This percentage increases as the training data decreases. It also increases drastically for the *All Train Markov* GPT-J. We agree that hallucinations are interesting, but we find it difficult to properly address them in a short 4-page paper alongside our other analyses. Please note that even far larger models like GPT-3 still hallucinate entities and whole facts.  \n\n_________________\n\n**The reviewer wrote**: \n\n> The way things are presented in the paper, it’s still a little bit unclear what type of generalization each condition tests for. Esp. for the “no variation” and “up to 18 games” conditions – these could be articulated a bit clearer either through a better name or a better explanation somewhere. E.g. L156-158 tells us at a very low level that the “no variation” condition is created by including only one unique gold path per task variation. However the implication of this condition is not clearly articulated: why is this interesting to explore? (e.g. is this correct?: “no variation” tests for the ability of generalize within task variants while “up to 18 games” tests for the ability to generalize across task variants). This will make it clearer the implications of the results as to what types of generalization LMs succeed and fail at.\n\n**Reply**: \n\nWe agree that the paper needs a better explanation of the three evaluation settings, including a better name for the settings. We will revise the explanation. In the meantime, let us try to explain what each condition tests for.\n\nThe *All train* set aims to answer the question: *What is the highest SW Score GPT-J can achieve if it has access to a large set of training data?* The *All train* set contains 7359 games covering the 30 classes of tasks contained in ScienceWorld and listed in Table 5. Each task category is represented by several specific task instantiations contained in ScienceWorld. For example, if the category is *Changes of State (Melting)*, some actual instantiations are: boil water, boil water but the stove is broken, boil water but the sink is broken, boil chocolate, boil lead, etc. For each instantiation, we collect from ScienceWorld up to three solution variations, i.e., unique sequences of actions that solve the task.\n\nThe *No variations* set aims to answer the question: *How much does GPT-J’s performance drop if it does not have access to a large amount of training data?* The *All train* set frequently includes more than one correct sequence of actions to achieve the goal. If LMs learn a lot during pretraining, they shouldn’t require that much training to play a game that tests for knowledge that should have been gained in pre-training. *No variations* covers the same tasks as *All train*, the same instantiations as *All train*, but it only includes a single gold path per task instantiation (i.e., no variations).\n\nThe final training set, *Up to 18 games* acknowledges that in the prior two train sets, some tasks are overrepresented and tries to answer: *Can GPT-J learn to play SW from only 480 games but which are balanced over the tasks?* To get an idea of the imbalance in the *No variations* training set, multiply the *Games* column in Table 5 by 2 to get the number of games for each task in the *No variations* set. For example, *No variations* contains 250 games where the player must measure the conductivity of a known material, but only 18 games where the player must mix paints to obtain tertiary colors.\n\nEven with a training set of only 480 games, GPT-J still outperforms the RL system (DRRN). Interestingly, Table 5 in our paper shows that for some underrepresented tasks *Up to 18 games*-trained GPT-J improves its performance (e.g., Changes of state), but for many others, it does not (e.g., Mixing paints).\n\n_________________\n\n**The reviewer wrote**: \n\n> Looking at the types of generalization tested, it seems there is one condition that is still missing: that is to say, generalizing /across task classes/. For example, during training LMs do not see any melting tasks, but must do melting tasks during dev and test. I imagine this would be a very hard split and it would be interesting to see whether modern LMs succeed at this already, or whether there is room for improvement.\n\n**Reply**:\n\nWe have tested training on most of the classes of tasks and tested on classes of tasks that were not seen during training. We did not include those results in the draft due to the four-page limitation. In short, for some classes of tasks, performance is poor for some classes of task; for other new classes of tasks, performance is good.  If this short paper were accepted, we could use the extra page to report those results or would gladly add them to the appendix.\n\n_________________\n\n## References\n\nWang, Ruoyao, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. “ScienceWorld: Is Your Agent Smarter than a 5th Grader?” In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11279–98. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. https://aclanthology.org/2022.emnlp-main.775.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach introduced in this paper works empirically well, outperforming an RL-based approach by 3.3x. Moreover, it’s clear this is largely due to increasing context length and conditioning on the full action sequence rather than just the previous action.\n2. Perhaps the standout strength of this paper is in its analysis: this paper presented a detailed evaluation of GPT-J in multiple conditions (varying the amount and variation in the training data, amount of context seen by the language model, and presenting a detailed breakdown of error types). This analysis gives solid insight into where benefits come from: in this case likely the additional context and world knowledge in the pre-trained language model.\n", "reasons_to_reject": "1. Acknowledging space limitations, some more analysis into why the LM likes to hallucinate invalid objects (even in the full-context settings), given that’s such a massive proportion of the errors, would be useful. This being the predominant source of error seems counter-intuitive to me because the LM should technically be able to refer to its context as to what objects are present, and be able to easily learn not to generate objects that weren’t previously referred to in context.\n2. The way things are presented in the paper, it’s still a little bit unclear what type of generalization each condition tests for. Esp. for the “no variation” and “up to 18 games” conditions – these could be articulated a bit clearer either through a better name or a better explanation somewhere. E.g. L156-158 tells us at a very low level that the “no variation” condition is created by including only one unique gold path per task variation. However the implication of this condition is not clearly articulated: why is this interesting to explore? (e.g. is this correct?: “no variation” tests for the ability of generalize within task variants while “up to 18 games” tests for the ability to generalize across task variants). This will make it clearer the implications of the results as to what types of generalization LMs succeed and fail at.\n3. Looking at the types of generalization tested, it seems there is one condition that is still missing: that is to say, generalizing /across task classes/. For example, during training LMs do not see any melting tasks, but must do melting tasks during dev and test. I imagine this would be a very hard split and it would be interesting to see whether modern LMs succeed at this already, or whether there is room for improvement.", "questions_to_authors": "1. Are there ways to disentangle the effect of GPT-J’s pre-trained knowledge from its reasoning ability? Perhaps by exploring how well a non-pre-trained GPT-J performs on these tasks, when simply fine-tuned to perform them?\n", "paper_topic_and_main_contributions": "This paper fine-tuned GPT-J for ScienceWorld, and demonstrated a 3.3x improvement over DRRN, a RL-based method. Furthermore, this paper presented a detailed analysis of how various amounts of training data, context length, etc. affected the performance of the model, including a breakdown of what types of errors were most common.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0LFLfDyK9D", "reviewer": "Reviewer_bA1q", "summary": "", "questions": "Q1: For Table 1, line 1, it is better to include the training set size for DRRN too, so that readers can see a direct comparison of if data size influences performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The reviewer wrote:**\n\n> W1: Since the pertaining data used for GPT-J and T-5 is different (GPT-J uses The Pile and T5 uses C4 + Macaw QA), it is hard to tell how important of a role this difference in pertaining data contribute the final result difference. There should be some experiments controlling this variable - for example, what if the training task formulation and increased context length are used on T5? Would that version of T5 also give us comparable performance similar to the one from GPT-J?”\n\n\n**Reply:**\n\nWe agree that controlling for the pretraining data of GPT-J and T5 could provide insights. However, there are factors beyond the training sources. For instance, T5 has an input buffer length of 512 word pieces while GPT-J has an input buffer length of 2048 word pieces. It is, therefore, not possible to train a T5 model with the same historical context size that we used for GPT-J. \n  \n_________________\n\n**The reviewer asked**: \n\n> Q1: For Table 1, line 1, it is better to include the training set size for DRRN too, so that readers can see a direct comparison of if data size influences performance.”\n\n**Reply:**\n\nWe agree that it would be helpful to illustrate DRRN’s training size, but we have not found a way to express the comparison fairly. DRRN trains with online Reinforcement Learning, meaning that, from the first moment of training, DRRN interacts live with ScienceWorld. For DRRN, we reproduced Ruoyao et al. (2022) training, a process we described in detail in Appendix C of our paper. Ruoyao et al. (2022) trained DRRN with 8 parallel threads each going up to 100k steps in the live ScienceWorld environment. The SW environment was reset after success/failure or after reaching 100 steps. It is difficult to express this amount of training in “games” as DRRN does not observe gold games but learns from its own mistakes. Its early games are only a few actions long, while its latter games are longer. It would be misleading to readers if we estimated DRRN’s training data 8 * 100000 / 100 = 8000 games since this would underestimate the amount of games DRRN plays (remember that early games are shorter due to DRRN’s mistakes).\n\n_________________\n\nThank you for catching the grammar issue on L60. We will correct it.\n\n_________________\n\n## References\n\nWang, Ruoyao, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. “ScienceWorld: Is Your Agent Smarter than a 5th Grader?” In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 11279–98. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. https://aclanthology.org/2022.emnlp-main.775.\n\nLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. arXiv:2106.01345 [cs].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: Simple, but convincing and important message. The revelation that LLM transfer learning, when formulated right, can perform well on text-based science games can provide guidance for future work, especially under the circumstance that previous work suggested otherwise.\n\nS2: The method itself (LLM transfer learning with game trajectory converted to text) is simple, which is important: simple methods can usually enjoy wider adaptation.\n\nS3: Straightforward experiments and good analysis support the claims well.", "reasons_to_reject": "W1: Since the pertaining data used for GPT-J and T-5 is different  (GPT-J uses The Pile and T5 uses C4 + Macaw QA), it is hard to tell how important of a role this difference in pertaining data contribute the final result difference. There should be some experiments controlling this variable - for example, what if the training task formulation and increased context length are used on T5? Would that version of T5 also give us comparable performance similar to the one from GPT-J?", "questions_to_authors": "Q1: For Table 1, line 1, it is better to include the training set size for DRRN too, so that readers can see a direct comparison of if data size influences performance.", "paper_topic_and_main_contributions": "This short paper studies the effectiveness of a pre-trained LLM (GPT-J 6B) to tackle ScienceWorld, a simulated game benchmark for elementary science experiments. The empirical results obtained in this paper suggest that an LLM-based transfer learning scheme, when formulated right, can significantly outperform (by a factor of 1.4x) previously built RL-based agents. Importantly, this finding contrasts findings in previous publications which showed LLM could hardly fit this task. Interestingly, the LLM used in this paper (GPT-J 6B) is smaller in model size compared to the LLM used in the previous work that performed poorly (T5 11B), suggesting that diverse pre-training, training task formulation and input context length, three of the biggest differences between the GPT-J and T-5 used for comparison, could be important to make LLM transfer learning work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L60-61: \"a preconditions\" -> \"a pre-condition\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This short paper offers a focused exploration of the effectiveness of LLMs, specifically GPT-J 6B, when applied to the ScienceWorld domain. The study involves fine-tuning GPT-J for ScienceWorld and highlights an 3.3x enhancement in performance compared to DRRN, which is an RL-based method. 
Additionally, the paper presents various analyses of factors influencing performance, including the effects of training data volume and context length.  It also offers a breakdown of the most common error types encountered during experimentation. However, two reviewers find the paper not exciting enough. Therefore, I recommend accepting the paper to findings.
Decision: Accept-Findings