Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games
Authors: Ruoyao Wang
Peter Jansen
Venue: EMNLP 2023 Findings (2023)
Keywords: text games
reinforcement learning
behavior cloning
self-supervision
TL;DR: We demonstrate that behavior cloned transformers can self-supervise, using generalizability as a self-supervision signal, and achieve 90% of the performance of a supervised model on interactive text games.

Abstract: In this work, we introduce a self-supervised behavior cloning transformer for text games, which are challenging benchmarks for multi-step reasoning in virtual environments. Traditionally, Behavior Cloning Transformers excel in such tasks but rely on supervised training data. Our approach auto-generates training data by exploring trajectories (defined by common macro-action sequences) that lead to reward within the games, while determining the generality and utility of these trajectories by rapidly training small models then evalauating their performance on unseen development games. Through empirical analysis, we show our method consistently uncovers generalizable training data, achieving about 90\% performance of supervised systems across three benchmark text games.

Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games
Ruoyao Wang and Peter Jansen University of Arizona, USA (ruoyaowang, paj Laajanse year edu
Abstract
PathCrawling
Evaluate Generality
Train agents on groups of paths. Evaluate agents text games to test path generality.
Crawl all paths that lead to reward
this work, we introduce a self-supervised behavior cloning transformer for text games, which are challenging benchmarks for multi- step reasoning virtual environments. Tra- ditionally, Behavior Cloning Transformers ex- cel in such tasks but rely on supervised train- ing data. Our approach auto-generates train- ing data by exploring trajectories (defined by common macro-action sequences) that lead to reward within the games, while determining the generality and utility of these trajectories by rapidly training small models then evaluat- ing their performance on unseen development games. Through empirical analysis, we show our method consistently uncovers generaliz- able training data, achieving about 90% per- formance of supervised systems across three benchmark text games.!
15% Not Generalbatle
T5
90% Generalzable
Path Group
20% Generalbable
Path
Ittrate
Continue crawiling most general paths caspccomllttio..
[
90% Generolizbble
Path Group
T5 Path Group 2.2
30% Nat Geneealbable
state
[w/heward]
actions
An overview our approach. Initially, game trajec- tories from the current state are extracted up to a specified hori- zon, which extends to the first reward. The generality of these paths is assessed by training a compact TS model and evalu- ating its performance on unseen development games. High- performing trajectories are subsequently extended through further exploration in an iterative process until the game is ultimately completed.
Figure I:
1 Introduction
Complex tasks often involve decomposing prob- lems into sequential steps to accomplish goal. This is particularly the case for text games, which are interactive virtual environments where an agent progressively selects actions until a task is com- plete. Examples of such tasks range from cook- ing a recipe (Côté et al., 2018), finding a treasure (Yuan et al., 2018), executing a science experiment (Tamari al., 2021; Wang et al., 2022), evaluat- ing common sense reasoning (Shridhar al., 2021; Murugesan et al., 2021; Gelhausen al., 2022). Text game agents are frequently modeled using reinforcement learning (e.g. He et al., 2016; Am- manabrolu and Hausknecht, 2020; Yao et al., 2020; Singh al., 2021; Tuyls al., 2022), wherein agent explores environment with the goal of learning policy that chooses actions leading reward and task completion. More recently, text game agents recast reinforcement learning as
sequence- sequence-to-sequeee to-sequence problems using architectures like the Behavior Cloning Transformer (Torabi et al., 2018) to leverage the inference capability and robustness of modern transformers (Wang et al., 2023; Li et al., 2022; Lin et al., 2023). In this setup,
the transformer is provided with an environment ob- servation input, and produces a string indicating the next action take as output. consequence of this reformulation is that, rather than exploring their environments, behavior cloning transformers require supervised training data, necessitating gen- erating human gold playthroughs.
In Figure 1, present our approach to over- come this data supervision limitation by automati- cally crawling, grouping, and evaluating candidate paths that can serve as useful and generalizable training data. By rapidly evaluating these candi-
'Relcased as open souree: https://github.cr com/ cogni www.tilabebbaeeeeea www.ttteaaabaaaahiinnniee iand//iigatidd

---

Arithmetic Game: Your task to solve the math problem Then, pirk up the frrm with the same quanity as the math problem arswen, and place in the box.
Generalizable Micro- Action Sequence
You are In the kitchen.
Taok-critical information
You see a math problem, pears, bananas,
You take the math problem.
The math problem says: multiply times
You take 24 bananas.
You put 24 bananas answer box.
take math problem
read math problem
take bananas
put 24 bananas
answer
Path Group: take(X), read(X), take(y)), put (Y, 2)
Game Variation 1:
math problem, read math problem, 24 bananas, put 24 bananas answer
Game Variation 2: take math problem, read math problem, take 12 apples, put 12 apples in answer box
Game Variation 3: take math problem, read math problem, take 16 pears, put 16
box
LLM FACAN
Festarn LLM
T5
90% urnsen girees
take(X), read[X), take(Y)), put (Y, 2) nert oction sequence gencraliabble solution
Ungeneral wonerrrllaallllo Micro- Actio Sequence
You are in the kitchen. You see a math problem. pears, 24 bananas,..
You take 24 bananas.
You put 24 bananas the box,
Though action sequences such as this appear valid solutions lask), they
short
generalizable solutions because they require knowladge they not yet have (ie. the math problem and its answer). By training an LLM on specific path groups, then testing the LLM unseen games, the behavior cloning transformer automatically discover which micro-action sequencaa are generalizable solutions.
take
put 24 bananas answer box
24 bananas
Path Group: take/X)), put [X, Y)
Variation 24 bananas, put 24 bananas answer Game Variation 2: take 12 apples, put 12 apples In answer box
Game Variation 3: take 16 pears, put 16 pears in answer box
with Drese paller
T5
SAI LESAMIT (AITNON
15% OR unussn
take put Y)
Inter www.:nnitiiionnnoooco.ooo sequeece is not generaliattl solution
Figure 2: Two example path groups from the Arithmetic game, emphasizing that not all high-scoring paths serve as useful training data. Paths having the same macro-action sequence from parametric game variations are grouped together, as depicted ahove. These examples underscore the concept that shorter, quicker paths, like the one at the bottom, may lack generalizahility. leading to poor model performance.
date sets of training data by training small models, we show how the performance of those models on unseen development games can be used as self- supervision signal to help guide further path crawl- ing, and discover useful training data automatically, This approach is validated on three benchmark games from the TEXTW SEXTWORLDEXPRESS simulator (Jansen and Cote, 2023). Our contributions include: (I) demonstrating generating self-supervised train- ing data through crawling, grouping, and evaluat- ing game trajectories, (2) using the performance of small rapidly-traine models as self-supervision signals, and (3) developing heuristic methods to align and merge training data, reducing the task search space and ultimate training costs.
alizable (see Figure 2), so such method would yield poor training data, while also being compu- tationally expensive. Here, we show that by train- ing small models on candidate training data, and evaluating performance on unseen development games, we can quickly evaluate the generalizability of that data. In terms of tractability, by iteratively patherawling only up to the next reward, and then continuing pathcrawling only from the most gener- alizable paths, we can limit the size of the search to only paths most likely be generalizable and form quality training data.
In this work, we use text games that have para- metric variations of each task, such as different math problems to solve for an arithmetic game, which is standard practice for evaluating a model's generalization ability (e.g., Côté et al., 2018; Mu- rugesan et al., 2021; Wang et al., 2022).
2 Approach
Figure/ 1 provides an overview of our approach. The goal of our work is to find set of generaliz- able training data that can be substitute for hu- man gold playthroughs of text games, and suitable for training& a behavior cloning transformer agent, While simply exhaustively patherawling all possi- ble trajectories in a game offers a method to find winning paths, not all winning paths are gener-
Path Crawling: crawler exploring all potential trajectories in a game from the start state until achieving non-zero reward or reaching a limited horizon. This crawler also navigates multiple parametric game variations, each with distinct objectives, such as making a stirfry versus baking a cake in a cooking game.
Our process begins with path

---

Path Grouping into Parameterized Macro- Action Sequences: Text game actions consist of a command verb (e.g.. take, put, open) with one (e.g. open closet) or two (e.g. put apple in cup- board) arguments, where a sequence of actions (i.e. a full or partial playthrough) is called trajectory. When creating candidate training sets for the be- havior cloning transformer, we attempt to group together trajectories that best as can tell appeaa to solve different parametric variations of game using roughly the same sequence of ac- tions. We do this by abstracting the trajectories into variabilized action sequences. For example, in the TEXT WORLD COMMON SENSE game, trajectory such as "take hat", "put hat on hat rack" from one variation, and "take dirty shirt", "put dirty shirt in washer" from another variation, could both be abstracted into the same parameterized macro- action sequence "TAKE(X), PuT(X, Y)". We call a set of trajectories from different game variations that can all be described using the same parameter- ized macro-action sequence a path group. In the experiments reported below, path groups are typi- cally created by grouping paths from 100 training variations of a game.
alizable performance to this point, the model con- tinues path crawling, commencing with this macro- action sequence and assessing generality until the next reward signal. This cycle persists until reach- ing a winning state, effectively segmenting the task into subtasks demarcated by non-zero rewards.
Benchmarks:
Our exploration of using path crawl- ing for generating self-supervised training data spans three benchmark text games. TEXTWORLD COMMON SENSE (TWC) (Murugesan et al., 2021) tasks agents with assigning household items (e.g. dirty shirt) their canonical locations (e.g. a washing machine). ARITHMETIC engages agents in a math problem, followed by a sack-and-plase task using the calculated result. SORTING requires agents to arrange objects by quantity. Each game presents parametric variations altering the envi- ronment and task-specific objects across episodes, For all experiments, we generate distinct training, development, and test sets, each with 100 varia- tions of each game. Objects crucial tasks are unique across sets, Refer to the APPENDIX for game specifics and playthrough examples.
Models
Evaluating Path Groups: The above process yields N path groups, each representing a unique parameterized macro-action sequence. These groups provide the training data for behavior cloning model. Although N is typically large, we've 'e've observed that generalizable sequences are often among the shorter paths (though not the short- est - Figure 2 provides a counterexample). For efficiency, we select the K' shortest path groups and train a T5-based behavior cloning transformer individually on each, assessing the model perfor- mance on 100 unseen game variations from the development 2 The process continues until identify group surpassing a predefined perfor- mance threshold T. If no single group meets this early stopping criterion, we attempt to merge path groups to discover higher-scoring combinations, detailed further in APPENDIX C.2.1.
We assess four models to showcase the effective- ness of the self-supervised behavior cloning trans- former. Further model details and hyperparameters can be found in the APPENDIX.
Supervised Behavior Cloning Transformer: This supervised baseline employs sequence-to- sequence model that packs the game task descrip- tion, current and previous observations, and prior action into a prompt, while training it to generate the agent's 's next action. Gold paths for training are generated by gold agents. At inference time candi- date next-actions are generated using beam search, and the first valid action is used. For efficiency, all reported experiments employ the T5-BASE (220M parameter) model (Raffel et al., 2020).
Self- cupfrsuper Behavior Cloning Transformer:
This model, the focus of our work, parallels the supervised transformer, with the exception being its training data is generated via the path crawling method described in Section 2.
Incremental Path Crawling: Reinforcement learning problems can offer sparse or dense re- wards. To minimize the search space and the total number of path groups to evaluate, we initially crawl and assess paths up to the first reward in- stance. Upon identifying a path group with gener-
Additional comparisons: Although our aim is showcase behavior cloning self-supervision, we offer additional baselines to contextualize the re- sults. The DRRN (He et al., 2016) is a strong rein- forcement learning baseline that separately encodes
An analysis of performance versus numher of parametric variations included in training is provided in the APPENDIX.

---

DRRN
GPT-4
Behavior Cloned Transformer
Baseline
Baseline
Supervised
Self-Supervised
Benchmark
Score
Steps
Score
Steps
Score
Steps
Score
Steps
Arithmetic
0.17
10
0,76
15
0.54
5
0.44
5
Sorting
0,03
21
0.04
7
0.72
7
0.59
6
TWC
0.57
27
0,90
6
0.90
6
0.89
15
Average
0.26
19
0.57
9
0.72
6
0.64
9
Table I: Model performance across three text game henchmarks, assessed on 100 unseen parametric test set variations. The state-lower
score (0-1) represents game progress, while steps denote the average game steps taken to reach final state -lower is better. The self-supervised transformer closely mirrors the supervised model's performance, marginally surpasses the GPT-4 haseline, and significantly outperforms the DRRN reinforcement learning baseline.
observations and actions into different embedding spaces, then learns a policy that selects actions at each step that maximize reward. Despite its age, the DRRN maintains near state-of-the-arr perfor- mance across many text game benchmarks (e.g. Xu et al., 2020; Yao ct al., 2020; Wang et al., 2022) see Jansen (2022) for review. For further context, we include zero-shot GPT-4 baseline (OpenAI, 2023), state-of-the-art model with undisclosed parameter size and training data. More details ure provided in the APPENDIX.
reasoning - for which smaller models tend to per- form poorly (Razeghi et al., 2022).
How do the trajectories identified by self- supervision compare to gold trajectories? As shown in Table 2. self-supervised paths closely match gold paths, although they occasionally in- clude task-irrelevant actions (e.g. look around) which diminish their efficiency. Given that the be- havior cloning method models games as a Markov decision process (limited by imput length depend only on the previous state), these irrelevant actions may hinder the agent's 's recall vital action history information, contributing to the slight performance drop compared to the supervised model.
4 Results and Discussion
Table shows the overall performance for all mod- els. The DRRN reinforcement learning baseline yields modest overall performance, scoring av- erage of 0.26 across all benchmark text games and taking average of 19 steps per game episode be- fore completion. The supervised behavior cloning transformer significantly outperforms this, achiev- ing a score of 0.72 across all games, effectively tripling task performance while also reducing the average steps required per episode to though at the expense of requiring gold training data. The GPT-4 baseline underperforms the sorting game but excels in the other games, resulting in an erage score of 0.57 and requiring 9 steps finish. The self-supervised behavior cloning transformer achieves a score of 0.65, equivalent to 90% of the supervised system's performance, but uses entirely self-generated training data. It substantially outper- forms the reinforcement learning baseline (which also lacks supervision), improving performance by a factor of 2.5 while generating solutions that are twice as efficient. The self-supervised behavior cloning transformer meets or exceeds the GPT-4 baseline on two of three games, despite the signifi- cant difference in model size, while only underper- forming on the benchmark requiring mathematical
How can the performance of this method ver- sus the DRRN and GPT-4 be contextualized? Our goal in this work is to demonstrate self- supervision of behavior cloning transformer us- ing the pathcrawling method described Section 2. Nevertheless, other baselines offer valuable context. The poor performance of the DRRN, a robust rein- forcement learning baseline for text games, under- scores the difficulty of exploring expansive search spaces without guiding context one of the moti- vating factors towards the application of pretrained language models guide exploration more effec- tively. On the other hand, while the GPT-4 model performs comparably to the self-supervised behav- ior cloning model, it is likely to be at least three orders of magnitude larger. This highlights the potential for small models to perform well given suitable exploration mechanisms.
How many training examples are required to learn task? Figure 3 shows the performance of the T5-agent trained with varying numbers of gold trajectories. Generally, task performance in- creases roughly linearly with the number of train- ing examples provided. An exception to this is the A ARITHMETIC game, where nearly all 100 training

---

Arithmetic
Arihhmelic Sorting
S5
TAKE(X) READ(X), LOOK-AROUND, TAKE(Y), PUTOY, TAKE(X), READ(X), TaKe(Y), POT(Y, 2)
Gold
0.8
TWC
Sorting
SS
TAKE(X), PUT(X. Y), TAKEZZ,, PUTÚZ. YI, TAKI(A) TAKB(X), PUT(X, Y), TAKE/Z),
Score
Gold
TAKE(A)...
Text World Common Sense
(TAKR(X), LOOK-AROUND, PUT(X, Y)) S8
(TAKB(X), INVENTORY, OPEN(Y), PUT(X, Y))
Gold
(TAKB(X), PUT(X, Y))
(TAXB(X), OPEN(Y), PUTXX. Y)
40
Table 2: Paths identified by our self-supervised method (SS) compared to gold paths for each game, showing that self- supervised paths nearly identical to gold paths. Paths in parenthesis represent merged path groups.
Training Examples
Figure 3: Performance of supervised behavior cloning agent on each three benchmark games, when trained with varying numbers training examples. Performance reported on the development set.
ries? How easily can we find generalizable train- ing data for given game, amongst all its winning trajectories? To investigate this, we trained models with varying amounts of gold (generalizable) trajec- tories, versus other randomly sampled winning (but typically not generalizable) trajectories. At one ex- treme, model is trained using entirely randomly sampled winning trajectories, while at the other extreme, the model is trained entirely using gold trajectories. The results, shown in Figure 4, show that the self-supervision method provides large benefit, increasing performance between 15% and 43% across all games highlighting the benefit of our self-supervision method.
1
Arithmetic
Sorting
0.8
TWC
Score
0.6
0.4
0.2
0
20 40
60
80
100
Proportion of Training Examples that are Gold(%))
Figure 4: Performance of the supervised behavior cloning agent each of the three benchmark games, when trained with varying proportion of gold trajectories interspersed with randomly sampled non-gold trajectories. Error bars represent the standard deviation 5 runs per data point the develop- ment set.
How is this method affected by training hyper- parameters? We have observed substantial differ- ences performance across various training hy- perparameters (such as number of training epochs, or training random seed), and suspect part of the difference between supervised and self euff-supeeiied performance may be due to these fluctuations.
examples needed to gain moderate performance. We hypothesize that this is result of the pretrained model generally performing poorly at mathemat- ical tasks. These results suggest that, pragmati- cally, using the self-super behavior cloning transformer may be possible with less training data on some environments, while other environments (such as ARITHMETIC) may benefit from more training examples. When using this self-supervised system in practice (i.e. on environments without gold training data), the end user may wish the number of training examples they use as a hy- perparameter in their model - though generally, larger the number available, the more accurately the path crawler will be able to estimate performance.
Conclusion
We present self-supervised behavior cloning trans- former for text games, that leverages a process of in- cremental path crawling in action spaces leading rewards, while evaluating the generality of groups of crawled paths on unseen games, to generate high- quality training data. Our method achieves compa- rable performance to models trained on gold trajec- tories, efficiently exploring the environment with- out needing human playthroughs gold agents. Our model excels over a robust reinforcement learn- ing baseline and attains about 90% of supervised models' performance, emphasizing the potential of efficient exploration mechanisms, even for smaller models.
How effective is the data grouping method com- pared to randomly sampled winning trajecto-

---

Limitations
can be solved by merging small set of groups (for example, for TEXTWORLD COMMON SENSE, two groups are required: (1) TAKE(X) OPEN(Y) PuT(X, Y), which picks up an object, opens its container, then places it in the container, and (2) TAKE(X) PUT(X, Y), for cases where a container doesn't 't need to be opened, like table). For more complex tasks that require a large number of dif- ferent actions to solve across game episodes, this technique would require increasing the amount of training data provided, proportional to the number of groups estimated to be required to solve the task.
We present a method for generating self-supervised training data for behavior cloning transformer mod- cls, presented in the context of text games. Be- havior cloning transformers (Torabi et al., 2018) and other frameworks for modeling reinforcement learning problems with transformers (including De- cision Transformers (Chen et al., 2021) and Trajec- tory Transformers (Janner et al., 2021)) can outper- form other reinforcement learning models in spe- cific contexts, but still face a number of central chal- lenges and limitations that depend upon the com- plexity of the problem space. The self-supervised behavior cloning transformer work has similar limitations, as well as other limita- tions that result from the computational complexity of path crawling.
Ethics Statement
present in this
Large language models typi- cally perform complex multi-step inference in a manner that isn't easily possible to inspect or eval- uate. Framing tasks text games in embodied vir- tual environments helps make the inference steps explicit, such that the choices model makes each step are auditable and interpretable. This helps expose potential issues in reasoning in mod- cls. For example, language models can correctly answer more than 90% of multiple choice science exam questions, suggesting they can perform com- plex reasoning, but fail to answer most of those same questions when they are reframed as text games that require explicit multi-step reasoning (Wang et al., 2022).
Broader Impacts:
Large Action Spaces:
At each time step, the envi- ronment simulator provides a set of valid actions that are possible to take. In text games, this action space frequently contains ten or more actions, each of which typically takes one or two arguments that represent objects in the environment - while ex- treme cases can contain 70 or more actions, and up 300 action templates (Hausknecht al., 2020). As such, action spaces can quickly become large, containing thousands (or hundreds of thousands) of possible valid action-object combinations per step. While reinforcement models typically struggle with exploring these large action spaces resulting in low model performance, the path crawling model pre-
Intended Use: This work presents self- supervised system that generates its own training data based on paths that a language model finds generalizable. If applicd more broadly to general reinforcement learning problems apart from text games, the model has the potential to find shortcuts through the action space that lead to task solutions based on language model's 's existing competen- cies, but do so through biased or harmful means. As such, for tasks that have the possibility of pro- ducing harm to specific groups, the training data produced by self-supervision should be manually inspected by a human to evaluate for bias or other potential harms. Because this work groups solu- tions into abstracted sequences of actions (i.e. path groups with similar solution methods, instantiated with different specific arguments, it may offer means of inexpensively manually evaluating self- supervised training data helping speed the identi- fication of bias by streamlining the review of self- supervised training data for tasks modelled using self-supervised behavior cloning models.
sented in this work would struggle to crawl very large action spaces, and may fail to find any win- ning paths within a given time budget. In practice, with the current model, we have found that action spaces containing approximately 500 valid actions per step are intractable to crawl beyond 3 steps (i.e. above 108 states per episode).
Tasks with sparse rewards are frequently challenging for reinforcement learning models. This work separates path crawling and next-action generation, making problems with mod- erately sparse rewards (i.e. within 4-5 steps) gen- erally quick to find. However, for tasks with large action spaces and extremely sparse rewards, the path crawling procedure would become intractable.
Sparse Rewards:
Complex, Varied Trajectories: To evaluate path generality, this work groups paths by abstracted pa- rameterized macro-action sequences. In the three benchmark tasks investigated in this work, the tasks

---

Acknowledgements
Games Workshop (Wordplay 2022), pages 1-15, Seat- tle, United States. Association for Computational Linguisties.
Thanks to Mare-Alexandee Côté, Raj Am- manabrolu, and the anonymous reviewers for help- ful comments. Thanks to Dan Yang for drawing the little robot in Figure 1. This work supported in part by National Science Foundation (NSF) award #1815948 to PJ, and gift from the Allen Institute for Artificial Intelligence (AI2).
Peter Jansen and Marc-alexandre Cote. 2023. TextWorldExpress: Simulating text games at one million steps per second. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 169-177, Dubrovnik, Croatia. Association for Computational Linguisties.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clin- ton Wang, Linxi Fan, Tao Chen, De-An Huang. Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu, 2022. Pre-trained language models for interactive decision- making. In Advances in Neural Information Process- ing Systems, volume 35, pages 11211.. Curran Associates, Inc.
References
Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph constrained reinforcement learning for natural language action spaces. International Con- ference on Learning Representations.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lec, Aditya Grover, Misha Laskin, Picter Abbeel. Aravind Srinivas, and Igor Mordatch. 2021. Decision trans- former: Reinforcement learning via sequence mod- eling. Advances in neural information processing systems, 34:15084-15000 15097.
Bill Yuchen Lin, Yicheng Fu, Karina Yang. Prithvi- raj Ammanabrolu, Faeze Brahman, Shiyu Huang. Chandra Bhagavatula, Yejin Choi, and Xiang Ren, 2023. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks, arXiv preprint arXiv:2305./7390
Maee-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla EI Asti, Mahmoud Adada, et al. 2018. Textworld: A learning environ- ment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial In- telligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41-75. Springer.
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapani- pathi, Pushkar Shukla, Sadhana Kumaravel, Gerald Tesauro, Kartik Talamadupula. Mrinmaya Sachan, and Murray Camphell. 2021. Text-based RI. Agents with Commonsense Knowledge: New Challenges, Environments and Baselines. In Thirty Fifth AAAI Conference on Artificial Intelligecce
OpenAI. 2023. GPT-4 technical report. arXiv preprint 2:Xiv:2303.08744
Paul Gelhausen, Marion Fischer, and George Peters. 2022. Affordance extraction with an external knowl- edge database for text-based simulated environments. ArXiv, abs/2207.0022
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare- Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7903-7910.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies few-shot reasoning. arXiv preprint arXiv:2202.07206
Ji He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng. 2016. Deep reinforcement learning with combinatorial action space for predicting popular Reddit threads. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1838- 1848, Austin, Texas. Association for Computational Linguistics.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR).
Michael Janner, Qiyang Li, and Sergey Levine. 2021. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Informa- tion Processing Systems.
Ishika Singh, Gargi Singh, and Ashutosh Modi. 2021. Pre-trained language models as prior knowledge for playing text-hased games. ArXiv, abs/2107.0842
Ronen Tamari, Fan Bai, Alan Ritter, and Gabriel Stanovsky. 2021. Process-level representation of scientific protocols with interactive annotation. Proceedings of the 16th Conference of the European
Peter Jansen. 2022. A systematic survey of text worlds as embodied natural language environments. Pro- ceedings of the 3rd Wordplay: When Language Meets

---

Chapter of the Association for Computational Lin- guistics: Main Volume, pages
Common (TWC)
Online. Association for Computational L Linguistics.
Tusk Description: Your lask is to pick up objects, then place them in their usual locations the environment.
You are in the bedroom. In one part of the room you see wardrobe that closed. There also desk chair, that has nothing a dressing table, that has nothing on it. In another part of the room you see chest drawens that is closed. In
Faraz Torabi, Garrett Warnell, and Peter Stone. 2018. Behavioral cloning from observation. In Proceed- ings of the Sveent--eventh International Joint Con- ference on Artificial Intelligence, IJCAI-18, pages 4950-4957. International Joint Conferences on Arti- ficial Intelligence Organization.
You also
of the
you
hed, that
has nothing on it. There is also a desk, that hus nothing on it. You also see night stand, that has nothing on it. In another part of the room you see a clean brown shirt.
Inventory:
Your inventory currenily empty. >e fake clean brown shirt
Jens Tuyls, Shunyu Yao, Sham M. Kakade, and Karthik Narasimhan. 2022. Multi-stage episodic control for strategic exploration in text games. In The Tenth International Conference Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
You take the clean brown shirl. Inventory:
a clean brown shirt
open wardrobe
You open the wardrobe. It's empty inside.
Inventory:
a clean brown shirt
put clean brown wardrobe
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. ScienceWorld. Is your agent smarter than 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in
Game completed.
Table 3: An example of TEXTWORLD COMMON SENSE (TWC) game. Actions for the solution are itali- cized. Relevant task objects and target locations are bolded, while distractor locations are underlined.
Natural Language Processing, pages 1 11279-11 11 2988, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Ruoyao Wang, Peter Jansen, Marc-Alexandee Côté, and Prithviraj Ammanabrolu. 2023. Behavior cloned transformers are neurosymbolic reasoners. In Pro- ceedings of the 17th Conference of the European
B Environments
Chapter of the Association for Computational Lin- guistics, pages 2777-2788, Dubrovník, Croatia. As- sociation for Computational Linguistics.
B.1 Text World Common Sense (TWC)
WEXLD COMMON SENSE (Murugesan et al., 2021) is a common sense reasoning benchmark in which the agent needs collect items in house- hold environment and put them in their cannon- ical ("common-sense") locations. In the experi- ments reported in this paper, we generate paramet- ric episodes containing a single room with one tar- get item in it, roughly corresponding to the "easy" setting in Murugesan et al. (2021). The agent re- ceives a score of 0.5 for picking up the target item, then additional 0.5 score for putting that object in the correct location. An example playthrough of TEXTWORLD COMMON SENSE is shown in Table 3.
Yunqiu Xu. Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang. 2020. Deep reinforcement learning with stacked hierarchical at- tention for text-based games. Advances in Neural Information Processing Systems, 13:16495-16507.
Shunyu Yao, Rohan Rao, Mathew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and ex- plore: Language models for action generation in text- based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Procexs- ing (EMNLP), pages 8736 8754, Online. Association for Computational Linguistics.
Xingdi Yuan, Marc-Alexandre Côté. Alessandro Sor- doni, Romain Laroche, Rémi Tachet des Combes, Matthew J. Hausknecht, and Adam Trischler. 2018. Counting to explore and generalize in text-based games. ArXiv, abs/1 806. 1525.
B.2 Arithmetic
The ARITHMETIC game requires the agent to read then answer an arithmetic question (e.g. add 42 and 9), then put an item from the environ- ment whose quantity matches the answer (e.g. 51 pineapples) into container called the answer box. Each episode contains a number of distractor items whose quantity corresponds performing the arith- metic incorrectly (e.g. 33 peaches, corresponds subtracting 9 from 42). The agent receives a score of 0.5 for first reading the math problem, and will
A Additional Experiment Information
The heuristic method employ investigates shorter paths (i.e. macro-action sequences that take fewer actions reach reward states) first. After grouping paths, the number of initial groups to explore for each game is 112 for TWC, 220 for ARITHMETIC, and for SORTING.

---

Arithmetic Game
Sorting
Task Description: Your first usk to solve the math problem. Then. pick up the item with the same quantity the mith problem answer, and place box.
Tusk Description: Your lask son objects by quantity First, place the object with the smallest quantity the box. Then, place the objects with the next smallest quantity in box, and repeat until objects have heen placed in the bux.
In one part of the room you hench that has 578 apples on it. There is also laundry basket that has 51 pineapples. work table that
You the laundry
You are the living room. In one part of the room you see a TV stand that has 23mg of nal on There also cuffee table, that has nothing
peaches. and 26 struwberries on It. moother part of the room you see a washing machine that closed. In one part of the room see hox, that empty. also math problem. Inventory:
hunanas, and 16 oranges it. You also
You also see book case, that bas nothing on it another purt the room you see a box, that is empey. In one part of the room you see a sofa that has 2g of marble
see clothes drier that closed.
You also see a arm chair that has 39mg of wood on it. In another part of the room you wastepaper hasket that closed. In one part of the you side table that has 9g of oak on Inventory:
table that
38mg marble
Your inventory is currently empty. take math problem
You take the math problem.
Your inventory currently empty. >e take 23mg of ook
Inventory:
math problem
read math problew
You take the 25mg of oak.
Inventory:
Your task solve the following math problem: add 42 and 9, Then, pick up the item with the same quantit the answer, and place the box.
23mg of oak
23mg of ouk box
Inventory:
math problem
You put the 25mg of oak in the box. Inventory:
pineapples
Your inventory currently empty.
You take the 51 pincapples.
tuke 38mg marble
Inventory:
math problem
You take the 38mg of marble.
51 pincapples
Inventory:
put 51 pineupples in box
38mg of marble
put 38mgg marble
Game completed.
You the 38mg of marble the box. Inventory:
Table 4: An example ARITHMETIC game. Actions for the solution italicized. Relevant task objects bolded, while distractor objects are underlined.
Your inventory currently empty.
>
39mg of wood
You take the 39mg of wood.
Inventory:
39mg of wood
> put 39mgg of in hox
You put the 39mg of wood bux. Inventory:
receive a full score 1.0 for placing the correct
Your inventory currently empty.
item into the answer box, regardless of whether it first reads the math problem or not. Placing incor-
> take 2x of marble
You take
of marble.
rect items into the answer box will immediately result in task failure. An example playthrough of ARITHMETIC is shown in Table 4.
Inventory:
of marblle
> put 2g of marble in box
Game completed.
Table 5: example SORTING game. Actions for the solution italicized. Relevant task objects are bolded.
B.3 Sorting
In the SORTING game, the agent needs to place items of different quantities into a specical con- tainer (the answer box) in ascending order. For most parametric episodes, the item quantities in- clude the additional challenge of containing differ- ent units (e.g. 23mg of oak vs 2g of marble) that the agent needs to interpret and normalize to cor- rectly solve the task. Episodes randomly contain between 3 and 5 items that need to be sorted. The agent receives a partial reward (1 ittes)) each time it places the correct object (i.e. next-largest quantity) into the answer box. The task fails any time it places items in the answer box in an incor- rect order. An example playthrough of SORTING is shown in Table 5.
C
Baselines
C.1
Supervised Behavior Cloning Transformer
The input (prompt) of the T5 model is formatted as:

 OBS Or 
 INV oiT 
 LOOK ooook
</ cextra id Ib 
 PACT 04 </s POBS Or 

where 
 is the special token for separator and <extra_id_0 the special token for the mask of text to generate used by the T5 model. is the task description. 0z, of inv of oooo 0t 1. and or rep- resent the current observation, current inventory information, current room description obtained by

---

the action "look around", last action, and last ob- servation respectively. The separators OBS, INV, LOOK, PACT, and POBS are their corresponding special tokens.
container be opened before they can be placed in- side (i.e. TAKE(X) OPEN(Y) PuT(X, Y)), while others can be directly placed in their winning loca- tions (e.g. TAKE(X) PUT(X, Y)). Where sin- gle path group does not solve all development variations, we take training data from the highest- performing variations of multiple path groups. and use these to assemble final higher-performe training set.
During development, we train the behavior cloning T5-BASE model from 2 to 20 epochs, then choose the model that achieves the highest devel- opment score for testing. The models are tested on 100 unseen test variations to get the final perfor- mance. All three game environments have step limit of 50 steps, which means if an agent does not complete within 50 steps, the score at the last step will be considered as the final score and the environment will be reset.
We take the top 5 performing groups based on their development scores and combine each two of them to get 10 merged groups (i.e. (9)) Triv- ially combining the training data from multiple groups does not work well, because each of these groups may contain both generalizable paths, well as paths that fail to generalize on episodes where another group of data is effective. Instead, we combine groups by choosing individual trajec- tories from each group that perform well on their respective development episodes. If two groups have paths that perform well for a given develop- ment episode, we choose the trajectory that has the highest development score. While this method has the benefit of being pragmatic (i.e. it chooses to combine paths that perform well empirically). presents the challenge of potentially overfitting on the unseen development set, which may limit ultimate generality when the model is eventually evaluated on the test set. To mitigate this, when groups need to be merged, we flip the training and development sets, so that the path crawler is always generating hypotheses one set, and evaluating them on a different, unseen set. We ultimately choose the merged group with the highest score on the development set as the final training set, and use this to generate final performance figures on the unseen training set.
The T5 model is trained on gold paths generated by gold agents provided by the WOI.D.X PRESS simulator. At inference time, the prompt is encoded from the observation of the current game state, and possible candidate next-actions are gen- crated using beam search. We use a beam size of 8 and select the first valid action. If no valid actions are present, the model picks the valid action from the game environment with the highest non-zero unigram overlap the candidate actions generated during beam search.
C.2 Self-supervised Behavior Cloning Transformer
During the group evaluation stage, we take the top 10 shortest paths to evaluate for all three games. To vastly reduce computation time, the behavior cloning T5-models are not tuned but rather trained statically for 20 epochs for each path group. Per- formance evaluated on 100 unseen development variations. The input strings are formatted the same as the supervised behavior cloning transformer. A group that achieves a development score higher than threshold T will be accepted. In our exper- iments, we use T = 0.95 for the TEXTWORL.D COMMONSENSE, T = 0.4 for the ARITHMETIC game, and T 0.5 for the SORTING game. In practice, one cycle of training a behavior cloning agent on candidate set of paths then evaluating on 100 unseen development variations takes ap- proximately 12 minutes on desktop hardware (i.e. RTX 4090), allowing experiments to complete in approximately 2 to 4 hours.
C.3
Deep Reinforcement Relevance Network (DRRN)
We make use of an existing benchmark implemen- tation of the DRRN model?. At each time step, the task description, observation, agent inventory information, and room description are appended into single string (as in the case of the behavior cloning models) and encoded by a GRU. Valid ac- tions at each step are encoded by a separate GRU. A Q-network, which consists of two linear layers, estimates the Q-value of each action using the en- coded observation-action pair. The action with the
C.2.1 Merging Multiple Solution Paths
Frequently, games may not solvable with paths from a single group. For example, in TEXTWORLD COMMON SENSE, some items require that their
Ptt thub. com/mi wom/microsoof//dmn

---

highest estimated Q-value will be chosen the next action. We trained the DRRN for 100K steps. 16 environments are trained parallel. The fi- nal results represent the average performance of 5 different models trained using different starting random seeds.
the task.
Why does the self-supervised model need much more steps to win in the TWC game? In the TWC games, agents will not fail immediately if puts object at a wrong location. While the self- supervised achieved similar score on the TWC task, makes more wrong decisions during the pro- ocss, resulting in a larger number steps required to complete the task.
C.4 GPT-4
The prompt of the GPT-4 agent includes the all the information in the behavior cloning transformer's prompt, which are the task description, the current observation, current inventory information, current room description obtained by the "look around" action, the last action, and the last observation. We also offer a full list of valid actions at each step and ask GPT-4 to choose one action from the list. An example of the prompt used for the GPT-4 agent is found below:
GPT-4 Agent Prompt
playing game. Choose proper action list valid actions to win the game.
Task description:
Your first task is to solve math problem. Then, pick the item
with same quantity as the problem answer, and place in the box.
Observation:
You take the math problem.
You are the supermarket. one part of the room you see box, that empty. There also showcase that has 29 blueberries. avocados, cabbages, eggplants, peas, and peppers on it.
Inventory:
math problam
Previous observation:
supermarket. that is empty. There is also a math problem. You also see showcase that has 29 bluaberries, 42 avocados, 22 cabbages, 88 eggolants, peas, and peppers it.
part
you
Previous
take math problem
Here are all the valid actions. Your response should be one of them:
Inventory take 88 eggplants, put math problem box, take bluoberris.taae 46 peas, look take 22 cabbages, take 17 problem.put problem
peppers,tske avocados, showcase
D Additional Error Analyses
Why does GPT-4 perform poorly in the SORT- ING game? In the SORTING game, an agent needs to put 3-5 objects into the answer box in order. In most of its error cases, the GPT-4 agent puts the first one correctly, but forgets what it did later and takes objects from the answer box out and put them back in wrong order. This suggests that this GPT-4 agent does not have a good long-tem term memory of
objects into the answer box


Reviews:
[{"review_id": "XU6cyvoBVU", "reviewer": "Reviewer_U234", "summary": "", "questions": "Based on my understanding, the sorting task is extremely simple, and it doesn’t rely on any information about the item types. Do you have a guess as to why GPT-4 performs so poorly at this task? Based on the results in Table 1, it seems to be performing roughly at chance, which is surprising to me. Could you include the prompts used in future versions of this paper?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reasons to Reject:\n\n### Reviewer:\n\n“The paper’s primary contribution seems unlikely to generalize beyond text games: in particular, searching over all possible actions (up to the initial reward) is only possible because language in text games is highly restricted (and we assume access to the set of valid actions). Additionally, path grouping based on generalization seems to work only because the tasks evaluated in this paper are limited in their complexity, such that many instances of a task can be solved with the same sequence of macro-level actions. Furthermore, intermediate rewards will not exist in most real-world tasks, limiting the usefulness of incremental path crawling.”\n\n### Response:\n\n**Text games as niche area:** One of the central criticisms of this review is that the work is only applicable to the niche of text games/interactive environments, which violates the ACL reviewing policy (https://2023.aclweb.org/blog/review-acl23/, Section 2, “Topic is too niche – A main track paper may well make a big contribution to a narrow subfield.”).  While text games and interactive environments are a relatively new subfield to NLP conferences, with papers starting to appear only about 5 years ago, they have rapidly grown into a rich subfield with approximately 200 papers published in only a few years (see “A Survey of Text Games for Reinforcement Learning Informed by Natural Language”, Osborne et al. 2022; “A Systematic Survey of Text Worlds as Embodied Natural Language Environments”, Jansen 2022; for review). We feel that the excitement score is not indicative of the impact of this work to this rapidly growing subfield. \n\n**Negative comments on parametric variations, though these are standard research methods:** The reviewer critiques the use of parametric variations of text games (“many instances of a task can be solved with the same sequence of macro-level actions”), even though this is standard research methods in the subfield.  Earlier on in the subfield, agent evaluation was primarily conducted on stand-alone interactive fiction games (e.g. Zork), or on benchmarks (e.g. “Interactive Fiction Games: A Colossal Adventure”, Hausknecht et al.) that combine a large number of interactive fiction games split into train, development, and test sets. However, text games and their respective action spaces can be so different that agents can hardly generalize from one game to another, and task performance was (and still is, to a degree) very low. **Recent text game benchmarks** (e.g. “TextWorld: A Learning Environment for Text-based Games”, Cote 2018; “Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines”, Murugesan et al. 2021, “ScienceWorld: Is your Agent Smarter than a 5th Grader?”, Wang et al. 2022, “TextWorldExpress”, Jansen et al. 2023) **all control for this difficulty by using the idea of parametric variations of a given task.**  With parametric variations, a simulator may generate 300 variations of a task (split between train, development, and test sets), with each variation varying both task-critical objects (e.g. the item of clothing that has to be picked up in TWC) as well as environment details (e.g. the environment might have a kitchen and pantry in one variation, and a garage and driveway in another). \n \n**Unlikely to generalize to sparse-reward conditions:** While it is true that this method would likely perform poorly on long-horizon sparse-reward tasks, that is not a criticism of this method, but rather the entire field of reinforcement learning.  Indeed, there is no known reinforcement learning method that functions in such cases.  That being said, we note that our method has been empirically shown to generalized to game variations with different number of intermediate rewards (in the the sorting game, each variation has 3-5 objects to sorting) and different solving paths (in the TWC game, some variations need to open a container first while the others do not).  Classical strong deep reinforcement learning models (e.g. DRRN) – that, shockingly, still outperform most fancy unsupervised (e.g. non-behavior cloning) models for text games, still substantially underperform the method proposed in this work.\n\n## Response to Questions for Authors:\n\n**Question A:** Based on my understanding, the sorting task is extremely simple, and it doesn’t rely on any information about the item types. Do you have a guess as to why GPT-4 performs so poorly at this task? Based on the results in Table 1, it seems to be performing roughly at chance, which is surprising to me. Could you include the prompts used in future versions of this paper?\nResponse: This is an interesting question to answer. We conducted an error analysis on the playthroughs of the GPT-4 to discover the answer. \n\nIn the sorting game, the agent must sort 3-5 objects of random quantity in the environment (e.g. 5 apples, 10 pears, 7 oranges) by picking them up, then placing them in an “answer box”. A correct action sequence might be “take 5 apples”, “place 5 apples in answer box”, “take 7 oranges”, “put 7 oranges in answer box”, etc. In our error analysis, GPT4 generally appears to correctly pick the first one or two items and place them in the answer box correctly.  But after that, it frequently tries to take items back out of the answer box, rather than put the next-ordered item in the box. This suggests that it has challenges in keeping track of its task progress. The Sorting game in the benchmark we use (TextWorldExpress) has a draconian scoring function where making an error typically results in a very low or zero score – so if an agent can’t solve the full task, the scoring function prefers it to make partial high-confidence progress rather than make guesses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and the method clear. Despite not performing as well as GPT-4 on some tasks, the T5-based model is substantially smaller and approaches the performance ceiling defined by supervised behavior cloning approaches. To the best of my knowledge, the paper uses reasonable baselines and provides a clear description of tradeoffs between models.", "reasons_to_reject": "The paper’s primary contribution seems unlikely to generalize beyond text games: in particular, searching over all possible actions (up to the initial reward) is only possible because language in text games is highly restricted (and we assume access to the set of valid actions). Additionally, path grouping based on generalization seems to work only because the tasks evaluated in this paper are limited in their complexity, such that many instances of a task can be solved with the same sequence of macro-level actions. Furthermore, intermediate rewards will not exist in most real-world tasks, limiting the usefulness of incremental path crawling. \n\n**In response to the rebuttal**: I do not believe that these criticisms violate the *CL reviewing guidelines as the authors state. The motivation behind text games is to \"simulate complex natural language problems in controllable settings\" (Osborne, et al. 2022, cited in the author response). As a result, it is not at all unreasonable to suggest that methods for text games should (at least attempt to) generalize to the real-world domains which they are intended to simulate. After reading the author response, I have chosen to keep my score unchanged.", "questions_to_authors": "Based on my understanding, the sorting task is extremely simple, and it doesn’t rely on any information about the item types. Do you have a guess as to why GPT-4 performs so poorly at this task? Based on the results in Table 1, it seems to be performing roughly at chance, which is surprising to me. Could you include the prompts used in future versions of this paper?", "paper_topic_and_main_contributions": "- This paper proposes a behavior cloning-based approach for text games which simulates supervised approaches without the need for ground truth demonstrations. The approach involves (1) generating a tree of possible action sequences in the text game, (2) merging related sequences into groups based on macro-action predicates, and (3) training small language models on these merged groups and evaluating which models generalize to unseen examples in the text game.\n- This paper evaluates its method on three benchmarks: (a) Arithmetic (which involves a simple math problem and a pick-and-place task); (b) Sorting (which requires agents to rearrange objects by quantity and reason over different units of measurement); and (c) the TextWorld Common Sense task.\n- The proposed model outperforms a DRRN baseline on all three tasks; it does not perform as well as GPT-4 on the arithmetic and commonsense tasks, but outperforms GPT-4 on the sorting task. The model approaches supervised approaches but does not quite close the gap between supervised and self-supervised models for these domains.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AA1TuT43NQ", "reviewer": "Reviewer_AHeS", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reasons to Reject:\n\n### Reviewer: \n“The generalization of this method might be poor as it heavily relies on the forms of games. Specifically, the authors set the grouping heuristic as the actions in each text game. However, this practice is not generalizable as it is not likely to expect all text games to fit this heuristic. Therefore, the proposed method might be limited to a small set of tasks.”\n\n**Response:**\nThis is incorrect. The only assumption we have about the form of the text games is that each text game has a set of parametric variations, which is a common setting in most recent text game benchmarks (e.g. “TextWorld: A Learning Environment for Text-based Games”, Cote 2018; “Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines”, Murugesan et al. 2021, “ScienceWorld: Is your Agent Smarter than a 5th Grader?”, Wang et al. 2022, “TextWorldExpress”, Jansen et al. 2023). Otherwise, all games are formulated as the standard formulation of reinforcement learning problems for text games (e.g. a partially-observable Markov decision process (POMPD) with a known action space and that return an observation and a reward after each action).  \n\nWhile it is a valid criticism that the proposed self-supervised model may not perform well if the action space is huge and the reward is sparse, this is true of /all/ known reinforcement learning methods. We feel that the excitement score is not indicative of the impact of this work – i.e. the performance of a fully-supervised behavior cloned model purely through self-supervision.\n\n### Reviewer: \n“The description of the method is not clear or formal. In Section 2, the authors illustrate their method mainly by figures and examples. Although it might suffice to describe their implementation, it is unclear how readers should adapt this method to their test cases in a principled way.”\n\n**Response:** Space is limited in the main body of the paper, if accepted, we will use the extra page to include additional implementation details including an algorithm listing.  Adapting this method to other test cases is not difficult as the code of this work is released as open source and our trained models can be used as-is or adapted by others. \n\n### Reviewer: \nLack of important baselines. The GPT-4 baseline is only evaluated in a zero-shot fashion. However, in recent years, the community has developed multiple ways to properly prompt the model. The authors should evaluate these recent techniques to make sure the baseline is proper.\n\n**Response:** We (respectfully) strongly push back on this comment.  The purpose of this paper is to show a novel method that archives near-supervised performance from a self-supervised behavior cloning model.  GPT-4 performance (and indeed any other model performance) is unrelated to this, and provided only for context. \nFurther, even though the GPT-4 model is orders of magnitude larger and trained on vastly more data than the T5-base model, it still finds some of our benchmarks challenging, indicating that our benchmarks are non-trivial to solve. \nBased on the review policies (https://2023.aclweb.org/blog/review-acl23/, Section 2, “The authors could also do [extra experiment X]”), performing a study on different prompting strategies for GPT-4 would appear tangential to the present study on self-supervision, and is not an appropriate reason for rejection.  We have offered strong empirical evidence to support the primary claims of the paper (supervised performance in a self-supervised behavior cloning model), and the benchmarks are standard benchmarks from TextWorldExpress.  Building a better GPT-4 agent is beyond the scope of this paper, and, even if the GPT-4 agent achieved perfect performance, would not detract from the narrative of the paper – that a self-supervised behavior cloning method can achieve supervised performance.  We believe the current soundness (2) and excitement (2) scores do not reflect the empirical quality of this work, or the potential impact – a small, self-supervised model that achieves near fully-supervised performance is a truly remarkable discovery, even if scoped exclusively to the field of text games.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The text games are interesting benchmarks that mark the capability of language models solving real-world problems.\n* The proposed method is intuitive and easy to understand. This might potentially increase the impact of this work.", "reasons_to_reject": "* The generalization of this method might be poor as it heavily relies on the forms of games. Specifically, the authors set the grouping heuristic as the actions in each text game. However, this practice is not generalizable as it is not likely to expect all text games to fit this heuristic. Therefore, the proposed method might be limited to a small set of tasks.\n* The description of the method is not clear or formal. In Section 2, the authors illustrate their method mainly by figures and examples. Although it might suffice to describe their implementation, it is unclear how readers should adapt this method to their test cases in a principled way.\n* Lack of important baselines. The GPT-4 baseline is only evaluated in a zero-shot fashion. However, in recent years, the community has developed multiple ways to properly prompt the model. The authors should evaluate these recent techniques to make sure the baseline is proper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors aim to reduce the data dependency of Behavior Cloning Transformers. The authors propose an approach that automatically generates data to train models that solve a class of text games. Specifically, the authors argue that those models can be trained in a self-supervised manner, where the data is generated through crawling, grouping, and evaluating generality. They conduct experiments on three benchmarks: Textworld Common Sense (TWC), Arithmetic, and Sorting. The results show that the proposed method is close to the supervised model while surpassing two other baselines.\n\nThe highlights include:\n* The text games are interesting benchmarks that mark the capability of language models to solve real-world problems.\n* The proposed method is intuitive and easy to understand. This might potentially increase the impact of this work.\n\nThe weaknesses are:\n* The generalization of this method might be poor as it heavily relies on the forms of games.\n* The description of the method is not clear.\n* Lack of important baselines.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SkyUGCfgTw", "reviewer": "Reviewer_TBi2", "summary": "", "questions": "1. The proposed method provides a way to create synthetic trajectories quickly. Of course, faster is better but in C.2, the authors say that the entire training process finishes in 2-4 hours. Since the synthetic data creation is offline, why is time the variable to be minimized? \n\n\n2. In Table 1, why does the self-supervised model take a significantly larger number of steps on TWC? Additionally, including standard error bars on Table 1 would be very informative since the test set is only 100 examples\n\n\n3. By building upon trajectories that were selected in the previous iteration, the proposed technique seems to sacrifice some notion of diversity. Did you have any experiments or ablations varying group size etc. to see if this effect occurs? This would be exacerbated at larger action spaces.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reasons to Reject:\n### Reviewer: \n“On concern with Table 1 is that the self-supervised method is only compared to an oracle i.e. the supervised model, and two baselines to which it is not directly comparable since the way these are trained are so different i.e. DRRN or GPT4. It would be significantly improved by including a baseline that ablates the effect of the data creation using the iterative method i.e. a random path crawler that generates an equal number of synthetic paths which is then used to train the agent. This would show the effect of how helpful it is to perform the iterative selection step.”\n\n### Response: \nThank you for your suggestion. The primary narrative is that we have developed a self-supervised behavior cloning model that achieves nearly the same performance as a fully-supervised behavior cloning model. The other baselines (DRRN, GPT-4) are provided for context only, and (as you say) use somewhat different solution methods and as such are not directly comparable.  The primary experimental comparison is between supervised vs self-supervised. \n\nWith respect to your suggested ablation study (random path crawler), we think that we have already conducted a similar study that we removed from the paper for space.  In that study (which we think of as a noise sensitivity study), we inject a controlled percentage of random paths into each path group, to empirically measure how robust the self-supervision method is to finding paths within noisy groups.  (But note, in this case the paths aren’t completely random: they still arrive at a winning state to the task, they just likely take a solution path that isn’t generalizable).  At the extreme, in this noise analysis we train the T5-base model on paths that are entirely from these random groups.  In these cases, one might hypothesize that the model is being fine tuned on the general protocol of text games in particular (e.g. input: observation, history; output: next action), and may distill some aspects of the task from whatever subparts of the paths are good, but that must ultimately also rely on its internal pre-trained knowledge to do well at the task.  In this case, the model performance decreases substantially (0.21 vs 0.54 on Arithmetic, 0.37 vs 0.72 on Sorting, and 0.68 vs 0.90 on TWC).  This likely speaks to your question, and empirically demonstrates the utility of the proposed method in its impact on performance.  If accepted, we can include this additional analysis (or a similar analysis with true randomly-selected paths, for noise) in the extra page, or in the Appendix, as space allows.\n\n## Response to Questions for Authors:\n### Question 1: \n*The proposed method provides a way to create synthetic trajectories quickly. Of course, faster is better but in C.2, the authors say that the entire training process finishes in 2-4 hours. Since the synthetic data creation is offline, why is time the variable to be minimized?*\n\n**Response:** We will clarify this point in the paper. The proposed method is not entirely offline (i.e. the pathcrawler identifies candidate paths, then the generality of those paths is empirically evaluated by training T5 models using the found paths and evaluating their performance on the development set) – so generally you still do want to minimize time. In the worst case, if you had to explore all possible groups of paths, one may have to train an intractibly large number of T5 models. Indeed, one of the major contributions of this paper is the heuristic described in Section 2 (Evaluating Path Groups, and Figure 2) that allows pathfinding to become tractable for self-supervision.\n\n### Question 2: \n*In Table 1, why does the self-supervised model take a significantly larger number of steps on TWC? Additionally, including standard error bars on Table 1 would be very informative since the test set is only 100 examples*\n\n**Response:** Thank you for this interesting question.  To answer this, we conducted an error analysis on the playthroughs of our agent in the TWC game. It turns out that in the TWC game, agents can try to put objects in the wrong place without any punishments. The self-supervised model usually tries several different places before it finds the correct location, resulting in a larger number of steps to complete.  If accepted, we can include this additional error analysis in the body of the paper or Appendix.\n\n### Question 3: \n*By building upon trajectories that were selected in the previous iteration, the proposed technique seems to sacrifice some notion of diversity. Did you have any experiments or ablations varying group size etc. to see if this effect occurs? This would be exacerbated at larger action spaces.*\n\n**Response:** This is a valid criticism of both this reinforcement learning model, but also essentially all reinforcement learning models (which have to balance exploration versus exploitation). One could find or build problems where an incorrect action at the start of a game causes failure many steps later. (Take Zork, a text dungeon adventure game developed in the 1970s as an example. Forgetting a particular game-critical item at the start of Zork that you need later in the game can result in game failure). But, to the best of our knowledge, there are no solutions for this – indeed, the very best models solve only 10-12% of more complex games like Zork. Here, we demonstrate that for a selection of challenging benchmarks, we’re able to get near supervised performance with this self-supervised model as formulated. Follow-on work could certainly address more complex reinforcement learning problems (like long-distance dependencies) and compare this algorithm to others. With respect to varying hyperparameters (like group size, etc.), we did vary these hyperparameters during development, but they appear to only affect convergence time, not ultimate performance (except at extremes).  Ultimately, while this work doesn’t solve every problem in reinforcement learning, it does empirically demonstrate that a class of exciting new models (behavior cloning) can be converted from fully-supervised to fully-self-supervised models for many text games that existing agents struggle with.  While that is incremental progress, we feel that it is a relatively large, and potentially impactful step to this subfield that is both exciting and sound.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is simple and intuitive, almost like a beam search of trajectories and shows solid empirical results on all three games. The writing is clear and easy to follow. \n\n\nThe proposed technique is also general enough to be applied to more complicated game settings with tweaks.", "reasons_to_reject": "On concern with Table 1 is that the self-supervised method is only compared to an oracle i.e. the supervised model, and two baselines to which it is not directly comparable since the way these are trained are so different i.e. DRRN or GPT4. It would be significantly improved by including a baseline that ablates the effect of the data creation using the iterative method i.e. a random path crawler that generates an equal number of synthetic paths which is then used to train the agent. This would show the effect of how helpful it is to perform the iterative selection step.", "questions_to_authors": "1. The proposed method provides a way to create synthetic trajectories quickly. Of course, faster is better but in C.2, the authors say that the entire training process finishes in 2-4 hours. Since the synthetic data creation is offline, why is time the variable to be minimized? \n\n\n2. In Table 1, why does the self-supervised model take a significantly larger number of steps on TWC? Additionally, including standard error bars on Table 1 would be very informative since the test set is only 100 examples\n\n\n3. By building upon trajectories that were selected in the previous iteration, the proposed technique seems to sacrifice some notion of diversity. Did you have any experiments or ablations varying group size etc. to see if this effect occurs? This would be exacerbated at larger action spaces.", "paper_topic_and_main_contributions": "This paper explores the viability of generating self-supervised trajectories to train a decision-transformer style agent for text-based games. The authors present a simple iterative expansion technique to create training data where a path crawler is first employed to explore the game initially for a set of short paths which are then grouped and evaluated by training small T5 style agents on the created path groups and evaluating them against a held out set. The best-performing path groups are then expanded upon iteratively until a desired target is reached. Experiments comparing the proposed data creation technique on 3 separate games show that this self-supervised method outperforms a DRRN baseline on all games and approaches the score reached by a comparable agent trained on gold trajectories.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposes a behavior cloning-based approach for text games which simulates supervised approaches without the need for ground truth demonstrations.
The 2 out of 3 reviewers selected "Good" for soundness (1 selected "Strong")
Decision: Accept-Findings