Exploration Based Language Learning for Text-Based Games
Authors: Andrea Madotto
Mahdi Namazifar
Joost Huizinga
Piero Molino
Adrien Ecoffet
Huaixiu Zheng
Alexandros Papangelis
Dian Yu
Chandra Khatri
Gokhan Tur
Venue:  (2020)
Keywords: Text-Based Games
Exploration
Language Learning

Abstract: This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents. Moreover, they provide a learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. 
One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space.
Existing methods for solving text-based games are limited to games that are either very simple or have an action space restricted to a predetermined set of admissible actions. In this work, we propose to use the exploration approach of Go-Explore (Ecoffet et al., 2019) for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories.
Our experiments show that this approach outperforms existing solutions in solving text-based games, and it is more sample efficient in terms of the number of interactions with the environment. Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.

Under review as a conference paper at ICLR 2020
EXPLORATION BASED LANGUAGE LEARNING FOR TEXT-BASES GAMES
Anonymous authors Paper under double-blind review
ABSTRACT
This work presents an exploration and imiration-learning-aaa agent capable of state-of-the-arr art performance in playing text-based computer games. Text-based based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of inter- est as they can be seen as testbed for language understanding, problem-solving. and language generation by artificial agents. Moreover, they provide learning environment in which these skills can be acquired through interactions with an environment rather than using fixed corpora. One aspect that makes these games particularly challenging for learning agents is the combinatorially large action space. Existing methods for solving text-based games are limited to games that are either very simple or have action space restricted to predetermined of admissible actions. In this work, we propose to use the exploration approach of Go- Go-Explore (Ecoffet al. 2019) for solving text-based games. More specifically, in an initial exploration phase, we first extract trajectories with high rewards, after which we train a policy to solve the game by imitating these trajectories. Our ex- periments show that this approach outperforms existing solutions in solving text- based games, and it is more sample efficient in terms of the number of interactions with the environment, Moreover, we show that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.
INTRODUCTION
Text-based games became popular in the mid 80s with the game series Zork (Anderson & Galley 1985) resulting in many different text-based games being produced and published (Spaceman 2019)
These games use plain text description of the environment and the player has to interact with them by writing natural-languase
commands. Recently, there has been a growing interest in developing agents that can automatically solve text-based games (Côté et al. 2018) by interacting with them. These settings challenge the ability of an artificial agent to understand natural language, common sense knowledge, and develop the ability interact with environments using language (Luketinal et al. 2019 Branavan et al. 2012).
Since the actions in these games are commands that are in natural language form, the major obstacle is the extremely large action space of the agent, which leads to combinatorially large exploration problem. In fact, with vocabulary of N words (e.g. 20K) and the possibility of producing sentences with at most m words (e.g. 7 words), the total number of actions is O(N") (e.g. 20K7 ~ 1.28e To avoid this large action space, several existing solutions focus on simpler text-based games with very small vocabularies where the action space is constrained to verb-object pairs (DePristo & Zubek [2001: Narasimhan et al. 2015 Yuan et al. 2018 Zelinka| 2018). Moreover, many existing works rely using predetermined sets of admissible actions (He et al. 2015 Tessler et al.) 2019 Zahavy et al., 2018). However, a more ideal, and still under explored, alternative would be an agent that operate in the full, unconstrained action space of natural language that can systematically generalize to new bext- games with no or few interactions with the environment.
To address this challenge, Explore (Ecoffet et al. 2019) algorithm. Specifically, we propose to first extract high reward trajec- tories of states and actions in the game using the exploration methodology proposed in Go-Explore
propose to the idea behind the recently proposed Go-

---

Under review as a conference paper at ICLR 2020
and then train policy using a Seq2Seq (Sutskever et al., 2014) model that maps observations to actions, in an imitation learning fashion. To show the effectiveness of our proposed methodol- ogy, we first benchmark the exploration ability of Go-Explore on the family of text-based games called CoinCollector (Yuan et al. 2018). Then we use the 4,440 games of "First TextWorld Prob- lems" (Côté 2018), which are generated using the machinery introduced by Cooé et al. (2018), to show the generalization ability of our proposed methodology. In the former experiment we show that Go-Explore finds winning trajectories faster than existing solutions, and in the latter, we show that training Seq2Seq model on the trajectories found by Go-Explore results in stronger generaliza- tion, as suggested by the stronger performance on unseen games, compared to existing competitive baselines (He et al. 2015 Narasimhan et al. 2015).
Reinforcement Learning Based Approaches for Text-Based Games Among reinforcement learning based efforts to solve text-based games two approaches are prominent. The first ap- proach assumes an action as a sentence of a fixed number of words, and associates separate Q- function (Watkins, 1989, Mnih al. 2013) with each word position in this sentence. This method was demonstrated with two-word sentences consisting of verb-object pair (e.g. take apple) (De- Pristo & Zubek, 200T Narasimhan et al. 2013 Yuan et al.) 2018 Zelinka 2018 Fulda al.) 2017). In the second approach, one Q-function that scores all possible actions (i.e. sentences) is learned and used to play the game (He et al. 2015: Tessler et al., 2019 Zahavy et al. 2018). The first ap- proach is quite limiting since fixed number of words must be selected in advance and no temporal dependency is enforced between words (e.g. lack of language modelling). In the second approach, on the other hand, the number of possible actions actions (a predetermined low cardinality set of actions that the agent can take) are not provided to the agent. A possible solution this issue has been proposed by Tao et al. 2018), where hierar- chical pointer-generator is used to first produce the set of admissible actions given the observation, and subsequently one element of this set is chosen as the action for that observation. However, in our experiments we show that even in settings where the true set of admissible actions is provided by the environment, Q-scorer (He et al. 2015) does not generalize well in our setting (Section 5.2 Zero-Shot) and we would expect performance to degrade even further if the admissible actions were generated by a separate model. Less common are models that either learn to reduce a large set of actions into smaller set of admissible actions by eliminating actions (Zahavy et al., 2018 or by compressing them in a latent space (Tessler et al. 2019).
become exponentially large if the admissible
Exploration in Reinforcement Learning In most text-based games rewards are sparse, since the size of the action space makes the probability of observing a reward extremely low when taking only random actions. Sparse reward environments are particularly challenging for reinforcement learning as they require longer term planning. Many exploration based solutions have been proposed
to address the challenges associated with reward sparsity. Among these exploration approaches
are novelty search (Lehman & Stanley 2008 2011 Conti et al., 2018 Achiam & Sastry, 2017
Burda et al., 2018), intrinsic motivation (Schmidhuber 1991b] Oudeyer & Kaplan| [2009] Barto
2013), and curiosity based rewards (Schmidhuber) 20066; 1991a: Pathak et al. 2017). For text based games exploration methods have been studied by Yuan et al. (2018). where the authors showed
the effectiveness of the episodic discovery bonus (Gershman & Daw 2017) in environments with sparse rewards. This exploration method can only be applied in games with very small action and state spaces, since their counting methods rely the state in its explicit raw form.
METHODOLOGY
Go-Explore (Ecoffet et al. 2019) differs from the exploration- based algorithms discussed above in that it explicitly keeps track of under-explored areas of the state space and in that it utilizes the determinism of the simulator in order to return to those states, allowing it to explore sparse-reward environments in a sample efficient way (see Ecoffet et al.) (2019) as well as section 4.1). For the experiments in this paper we mainly focus on the final performance of our policy, not how that policy is trained, thus making Go-Explore a suitable algorithm for our experiments. Go-Explore is composed of two phases. In phase I (also referred to as the "exploration" phase) the algorithm explores the state space through keeping track of previously visited states by maintaining an archive. During this phase, instead of resuming the exploration from scratch, the algorithm starts exploring from promising states in the archive find high performing trajectories. In phase 2 (also referred
2

---

In phase 1, Go-Explore builds an archive of cells, where cell is defined as set of observations that are mapped to the same, discrete representation by some mapping function f(z). Each cell is asso- ciated with meta-data including the trajectory towards that cell, the length of that trajectory, and the Under review as a conference paper at ICLR 2020
Table 1: Example of the observations provided by the CookingWorld environment.
Description
garden well, here we are garden. There
Admissible commands
roasted red apple and a rod onion on the floor
Inventory
You are carrying: a black pepper doer
-drop black pepper
Prev. Action
Open
black pepper
Gather all tollowing ingredients and lollow the
examine red apple
directions to prepare this tasty meal.
examine rod onion
Quest
Ingredients: black pepper, red apple salt. Directions: chop the red apple, roust the red upple. prepare meal
north
look
take red apple
Feedback
That s already open
take red onion
to as the "robustification" phase, while in our variant we will call it "generalization") the algorithm trains policy using the trajectories found in phase 1. Following this framework, which is also shown in Figure 3 (Appendix A.2), we define the Go-Explore phases for text-based games.
Let us first define text-based games using the same notation as Yuan al. (018)) A text-based game can be framed as a discrete-time Partially Observable Markov Decision Process (POMDP) (Kael-) bling al. T998) defined by (S, T, A, S22 O, R), where: S is the set of the environment states, T is the state transition function that defines the next state probability, i.e. T(se+1 8)) E S. A is the set of actions, which in our case is all the possible sequences of tokens, 52 is the set of observations, i.e. text observed by the agent every time has it to take an action in the game (i.e. dialogue turn) which is controlled by the conditional observation probability i.e. ( at-1), and, finally, R is the reward function i.e. r = R(s,a).
Let us also define the observation or E S2 and the action a E A. Text-based games provide some information in plain text at each turn and, without loss of generality, we define an observation as the sequence of tokens {of.-. ... of) that form that text. Similarly, we define the tokens of an action a, as the sequence {aj,... aa,") Furthermore, we define the set of admissible actions Ar E A as Ar = {ao.... ag J. where each ag which is a sequence of tokens, is grammatically correct and admissible with reference to the observation Oz.
2.1 PHASE 1: EXPLORATION
cumulative reward of that trajectory. New cells are added to the archive when they are encountered in the environment, and existing cells are updated with new meta-data when the trajectory towards that cells is higher scoring or equal scoring but shorter.
At each iteration the algorithm selects a cell from this archive based on meta-data of the cell (e.g. the accumulated reward, etc.) and starts to randomly explore from the end of the trajectory asso- ciated with the selected cell. Phase 1 requires three components: the way that observations are embedded into cell representations, the cell selection, and the way actions are randomly selected when exploring from a selected cell. In our variant of the algorithm, f'(x) is defined as follows: given an observation, we compute the word embedding for each token in this observation, sum these embeddings, and then concatenate this sum with the current cumulative reward to construct the cell representation. The resulting vectors are subsequently compressed and discretized by binning them in order to map similar observations to the same cell. This way, the cell representation, which is the key of the archive, incorporates information about the current observation of the game. Adding the current cumulative reward to the cell representation is new to our Go-Explore variant as in the orig- inal algorithm only down-scaled image pixels were used. It turned out to be very very effective to increase the speed at which high reward trajectories are discovered. In phase 1, we restrict the action space to the set of admissible actions A, that are provided by the game every step of the game This too is particularly important for the random search to find a high reward trajectory faster. Finally, we denote the trajectory found in phase 1 for game g as T [(oo: ao ro), .... (ot, de, r.)]-
'Note that the final goal is to generalize to test environments where admissible actions are not available. The assumption that admissible actions are available at training time holds in cases where we build the training environment for an RL agent (e.g. -rafteed dialogue system). and system trained in such an environment can be practically applied as long as the system does not rely on such information test time. Thus, we assumed that these admissible commands not available test time.
3

---

Under review as a conference paper at ICLR 2020
Table 2: Statistics of the families of text-based games used in the experiments. The average is among the different games in CookingWorld and among different instance of the same game for CoinCollector.
CoinCullector
CookingWorid
Vocubulary size VVD
1,250
20,000
Action Space
#Level
(Hard)
222
Max Room
90
12
Avg. &lok. Description
64
97
Avg. #Adm, Actions
14
2.2 PHASE 2: GENERALIZATION
Phase 2 of Go-Explore uses the trajectories found in phase 1 and trains policy based on those trajectories. The goal of this phase in the original Go-Explore algorithm is to turn the fragile policy of playing trajectory of actions in sequence into a more robust, state-conditioned policy that can thus deal with environmental stochasticity. In our variant of the algorithm the purpose of the second phase is generalization: although in our environment there no stochasticity, our goal general policy that can be applied across different games and generalize unseen games. In the original Go-Explore implementation, the authors used the backward Proximal Policy Optimization algorithm (PPO) (Salimans & Chen 2018 Schulman et al., 2017) to train this policy. In this work we opt for simple but effective Seq2Seq imitation learning approach that does not use the reward directly in the loss. More specifically, given the trajectory 7. we train Seq2Seq model to mini- mize the negative log- ligeliild the action a, given the observation or. In other words, consider a word embedding matrix E' gd where is the embedding size and |V] is the cardinality of the vocabulary, which maps the input token to embedded vector. Then, we define encoder LSTM and a decoder LSTMdee. Every token og from the trajectory Ty converted to its embed- ded representation using the embedding matrix E and the sequence of these embedding vectors is passed through LSTM
learn
h hene LSTM (E(o))) (E(o)),
The last hidden state Maal is used as the initial hidden state of the decoder which generates the action a token by token. Specifically, given the sequence of hidden states H E Rdx[(s] of the encoder, tokens of are generated as follows:
hgee LSTMacc ((
Cj = Softmax (nd H)
(3)
dist Softmax(W[(h)
where W E 2dx is matrix that maps the decoder hidden state, concatenated with the context vector, into a vocabulary size vector. During training, the parameters of the model are trained by minimizing:
Jasl
Lp(ad).)) = -- log(dist(((a))
(5)
which is the sum of the negative log likelihood of each token in a (using teacher forcing (Williams & Zipser 1989)). However, at test time the model produces the sequence in an auto-regressive
way (Graves 2013) using greedy search.
EXPERIMENTS
3.1 GAMES AND EXPERIMENTS SETUP
A set of commonly used standard benchmarks Yuuan et al.) 2018 Côté et al.) 2018 Narasimhan ct al. 2015) for agents that play text-based games are simple games which require no more than two words in each step solve the game, and have a very limited number of admissible actions
Or the of trajectories, depending the training setting
4

---

Under review as a conference paper at ICLR 2020
Figure 1: CoinCollector results of DQN++ and DRQN++ (Yuan et al. 2018 versus Go-Explore
Phasel, 1, i... just exploration.
Celector Store
Model
Frames
Steps
DQN++
546.74K + 55.75 7
38.8 = 0.2 42.0 + 0,8
+RON++
508,96K +: 23.61K
Go-Phasel
205,00K + 29. 29.41K
30 + 0.0
per observation. While simplifying, this setting limits the agent's ability to fully express natural language and learn more complex ways to speak. In this paper, we embrace more challenging environments where multiple words are needed at each step solve the games and the reward is particularly sparse. Hence, we have selected the following environments:
CoinCollector (Yuan et al. 2018) is a class of text-based games where the objective is to find and collect coin from a specific location in a given set connected rooms The agent wins the game after it collects the coin, at which point (for the first and only time) reward of +1 is received by the agent. The environment parses only five admissible commands (go north, go east, go south, go west, and take coin) made by two worlds;
CookingWorkee CCôté 2018) in this challenge, there are 4,440 games with 222 different levels of difficulty, with 20 games per level of difficulty, each with different entities and maps. The goal of each game is to cook and eat food from a given recipe, which includes the task of collecting ingredients (e.g. tomato, potato, etc.), objects (e.g. knife), and processing them according to the recipe (e.g. cook potato, slice tomato, etc.). The parser of each game accepts 18 verbs and 51 entities with predefined grammar, but the overall size of the vocabulary of the observations is 20,000. In Appendix A.I we provide more details about the levels and the games' grammar.
In our experiments, we try to address two major research questions. First, we want benchmark the exploration power of phase 1 of Go-Fxplore in comparison to existing exploration approaches used text-based games. For this purpose, we generate 10 CoinCollector games with the hardest setting used by Yuan et al. (2018). i.c. hard-level 30 (refer to the Appendix A.] for more information) and use them as benchmark. In fact, CoinCollector requires many actions (at least on hard games) to find a reward, which makes it suitable for testing the exploration capabilities of different algorithms. Secondly, we want to verify the generalization ability of our model creating complex strategies using natural language. CoinCollector has a very limited action space, and is mainly designed to benchmark models on their capability of dealing with sparse rewards. Therefore we use the more complex CookingWort World games to evaluate the generalization capabilities of our proposed approach. We design three different settings for Cooking World: 1) Single: treat each game independently. which means we train and test one agent for each game to evaluate how robust different models across different games.; 2) Joint: training and testing a single policy on all the 4,440 Cooking World games at the same time to verify that models can learn play multiple games at the same time; 3) Zero-Shot: split the games into training, validation, and test sets, and then train our policy on the training games and test it on the unseen test games. This setting is the hardest among all, since requires generalization to unseen games.
In both CoinCollector and Cooking games an observation on provided by the environment consists of room description D, inventory information I, quest Q, previous action P and feedback F provided the previous turn. Table [ shows an example for each of these components. In our experiments for phase of Go-Explore we only use D as the observation.
i VVisual example provided at: http://gihuub.co//xiiiiiiiii hhapagiiiiriiiyuaaaaeeltttttddd TexiWorlddCoinnollleee
WWe gave this name this family of games, following the naming convention of Worl environments
5

---

Under review as a conference paper at ICLR 2020
3.2 BASELINES
For the CoinCollector games, we compared Go-Explore with the episodic discovery bonus (Gersh- man & Daw 2017) that was used by Yuan et al. (2018) improve two Q-learning-based baselines: + and DRON+ We used the code provided by the authors and the same hyper-parameter
For the CookingWorke games, we implemented three different treatments based on two existing methods:
LSTM-DON (Narasimhan al. 2015 Yuan et al. 2018): An LSTM based state encoder with a separate Q-functions for each component (word) of fixed pattern of Verb, Adjectivel, Nounl, Adjective2, and Noun2. In this approach, given the observation or, the tokens are first converted into embeddings, then an LSTM is used to extract sequence of hidden states Hagn E Rd A mean-pool layer is applied to Hdgn to produce single vector hat that represents the whole sequence. Next, linear transformation Wype E Ropl is used to generate each of the Q values, where Vyypel <
[V] is the subset of the original vocabulary restricted to the word type of partieular game (e.g for Verb type: take, drop, etc.). Formally, have:
(6) Next, all the Q-functions are jointly trained using the DQN algorithm with c-greedy explo- ration (Watkins, 1989 Mnih et al., (015). At evaluation time, the argmax of each Q-function is concatenated to produce des Importantly, in Vype special token 
 is used denote the absence of a word, so the model can produce actions with different lengths. Figure in Ap-
(Ox, Auyee) = h Wype
where, type E {Verb, Obj, Noun, Obj2.Noun2).
pendix A.. A..2 shows depiction of this model.
LSTM-DON+ADM: It is the same model as LSTM-DQN, except that the random actions for e-greedy exploration are sampled from the set of admissible actions instead of creating them by sampling each word separately.
DRRN (He al. 2015): In this approach model learns how to score admissible actions instead of directly generating the action token by token. The policy uses an LSTM for encoding the observation and actions are represented as the contain. Then, the Q value is defined as the dot product between the embedded representations of the observation and the action. Following the aforementioned notation, hat is generated as in the
of the embedding of the word tokens they
LSTM-DQN baseline. Next, we define its embedded representation as G; = =xe E(at). where E is embedding matrix in Equation 1. Thus. the Q-function is defined as:
Q(or, a,) = ho
At testing time the action with the highest Q value is chosen. Figure X in Appendix A.2 shows depiction of this model.
3.3 HYPER-PARAMETEER
In all the games the maximum number of steps has been set to 50. As mentioned earlier, the cell representation used in the Go-Explore archive is computed by the sum of embeddings of the room description tokens concatenated with the current cumulative reward. The sum of embeddings is computed using 50 dimensional pre-trained GloVe (Pennington et al.) (014)) vectors. In the Coin- Collector baselines we use the same hyper-parameters as original paper. In Cooking World all the baselines use pre-trained GloVe of dimension 100 for the single setting and 300 for the joint one.
The LSTM hidden state has been set to 300 for all the models.
RESULTS
4.1
COINCOLLECTOR
In this setting, compare the number of actions played in the environment (frames) and the score achieved by the agent (i.e. +1 reward if the coin is collected). In Go-Explore we also count the ac- tions used to restore the environment
selected cell, i.e. to bring the agent to the state represented
TThe authors released their code ehtt:///dhhlooooooooierrrrrrrrre

---

Under review as a conference paper at ICLR 2020
Table 3: Cooking World results on the three evaluated settings single, joint and zero-shot.
Single
Joint
Zero-Shot
Model
Score
Steps
Win
Score
Steps
Win
Score
Steps
Win
LSTM-DON
2206
201832
412
473
213618
172
31
21480
14
+ADM
10360
140940
1770
623
210283
200
37
21530
14
DRRN
16075
78856
3195
4560
184888
216
451
17243
37
Gx-Explore Seq2Seq
17507
68571
3757
11167
85967
2326
1038
10805
207
Go-Explore Phuse I
19437
49011
4279
Max Possible
19882
4440
19882
4440
2034
444
0.
good
reciped car
recsped
recipna resided
rerd saal eooet droping gid
recised ecoook
meseen Hare coull cutt seseoooo-ggad
resiout ta cosod oparri sdropt yood
recigeed anskent nak crikk cone oppee vitria +jo22
oppen dragp
sload
recain sugert
medibc recpped takes
recavve
Figure 2: Breakdown of results for the Cooking World games in the Joint setting, The results are normalized and sorted by increasing difficulty level from left to right, averaged among the 20 games of each level.
in the selected cell. This allows a one-to-one comparison of the exploration efficiency between Go- Explore and algorithms that use a count-based reward in text-based games. Importantly, Yuan al. (2018) showed that DQN and DRQN, without such counting rewards, could never find successful trajectory in hard games such as the ones used our experiments. Figure [ shows the number of interactions with the environment (frames) versus the maximum score obtained, averaged over 10 games of the same difficulty. As shown by Yuan et al. (2018), DRQN++ finds a trajectory with the maximum score faster than to DQN++. On the other hand, phase I of Go-Explore finds an optimal trajectory with approximately half the interactions with the environment. Morcover, the trajectory length found by Go- fo-Explore is always optimal (i.e. 30 steps) whereas both DQN++ and DRON++ have an average length of 38 and 42 respectively.
4.2 COOKINGWORLD
In CookingWorld we compared models the three settings mentioned earlier, namely, single, joint, and zero-shot. In all experiments, we measured the sum of the final scores of all the games and their trajectory length (number of steps). Table summarizes the results these three settings. Phase of Go-Explore on single games achieves total score of 19,530 (sum over all games), which is very close to the maximum possible points (i.e. 19,882), with 47,562 steps. A winning trajectory was found in 4,279 out of the total of 4,440 games. This result confirms again that the exploration strategy of Go-Explore is effective in text-based games. Next, we evaluate the effectiveness and the generalization ability of the simple imitation learning policy trained using the extracted trajectories in phase 1 of Go-Explore in the three settings mentioned above.
Single In this setting, each model trained from scratch in each of the 4,440 games based on the trajectory found in phase of Go- Go-Explore (previous step). As shown in Table B the LSTM- DQN (Narasimhan et al., 2015) Yuan et al. 2018) approach without the use of admissible actions performs poorly. One explanation for this could be that it is difficult for this model to explore both language and game strategy at the same time; it is hard for the model to find a reward signal before it has learned to model language, since almost none of its actions will be admissible, and those reward signals are what is necessary in order to learn the language model. As we see in Table 3 however, by using the admissible actions in the e-greedy step the score achieved by the LSTM-DO DN increases dramatically (+ADM row in Table DRRN (He et al.) (015)) achieves a very high score, since it explicitly learns how rank admissible actions (i.e. much simpler task than generating
7

---

Under review as a conference paper at ICLR 2020
text). Finally, our approach of using Seq2Seq model trained the single trajectory provided by phase of Go-Explore achieves the highest score among all the methods, even though we do not use admissible actions in this phase. However, in this experiment the Seq2Seq model cannot perfectly replicate the provided trajectory and the total score that it achieves is in fact 9.4% lower compared to the total score achieved by phase of Go-Explore. Figure 7 (in Appendix [A.3] shows the score breakdown for each level and model, where we can see that the gap between our model and other methods increases as the games become harder in terms of skills needed.
Joint In this setting, a single model trained on all the games at the same time, to test whether one agent can learn to play multiple games. Overall, as expected, all the evaluated models achieved a lower performance compared to the single game setting. One reason for this could be that learning multiple games at the same time leads to situation where the agent encounters similar observations in different games, and the correct action to take in different games may be different. Furthermore, presented greatly affects the performance of
it is important to note that the order in which games LSTM-DQN and DRRN. In our experiments, we tried both an easy-to-hard curriculum (i.e. sorting the games by increasing level of difficulty) and shuffled curriculum. Shuffling the games each epoch resulted in far better performance, thus we only report the latter. In Figure 2 we show the score breakdown, and we can see that all the baselines quickly fail, even for easier games.
Zero-Shot In this setting the 4,440 games are split into training, validation, and test games. The split is done randomly but in a way that different difficulty levels (recipes 1, 2 and 3), are represented with equal ratios in all the splits, i.c. stratified by difficulty. As shown in Table 3 the zero-shot performance of the RL baselines is poor, which could be attributed to the same reasons why RL baselines under- peaderr in the Joint case. Especially interesting is that the performance of DRRN is substantially lower than that of the Go-Explore Seq2Seq model, even though the DRRN model has access to the admissible actions at test time, while the Seq2Seq model (as well as the LSTM- DQN model) has to construct actions token-by-token from the entire vocabulary of 20,000 tokens. On the other hand, Go-Explore Seq2Seq shows promising results by solving almost half of the unseen games. Figure 8 (in Appendix (A.3) shows that most of the lost games are in the hardest set, where a very long sequence of actions is required for winning the game. These results demonstrate both the relative effectivvness of training a Seq2Seq model on Go-Explore trajectories, but they also indicate that additional effort needed for designing reinforcement learning algorithms that effectively generalize to unseen games.
5 DISCUSSION
Experimental results show that our proposed Go-Explore exploration strategy is a viable methodol- ogy for extracting nigh-performing trajectories in text-based games. This method allows us to train supervised models that can outperform existing models in the experimental settings that we study. Finally, there are still several challenges and limitations that both our methodology and previous solutions do not fully address yet. For instance:
State Representation The state representation is the main limitation of our proposed imitation learning model. In fact, by examining the observations provided in different games, we notice large overlap in the descriptions (D) of the games. This overlap leads to a situation where the policy receives very similar observations, but is expected to imitate two different actions. This show especially in the joint setting of Cooking World, where the 222 games are repeated 20 times with different entities and room maps. In this work, we opted for simple Seq2Seq model for our policy, since our goal is to show the effecctivnees of our proposed exploration methods. However, a more complex Swwarrchcal-SeeqSSS. model (Sordoni et al. (015)) or a better encoder representation based on knowledge graphs (Ammanabrolu & Riedl 2019a b) would likely improve the of performance this approach.
Language Based Exploration In Go-Explore, the given admissible actions are used during ran- dom exploration. However, in more complex games, e.g. Zork I and in general the Z-Machine games, these admissible actions are not provided. In such settings, the action space would explode in size, and thus Go-Explore, even with appropriate cell representation, would have hard time
8

---

Under review as a conference paper at ICLR 2020
finding good trajectories. To address this issue one could leverage general language models to pro- duce a set of grammatically correct actions. Alternatively one could iteratively learn a policy to sample actions, while exploring with Go-Explore. Both strategies viable, and comparison is left to future work.
It is worth noting that hand-tailored solution for the CookingWorle games has been proposed in the "First World Problems" competition (Côté et al. 2018). This solution managed to obtain up to 91.9% of the maximum possible score across the 514 test games on an unpublished dataset. However, this solution relies on entity extraction and template filling, which we believe limits its potential for generalization. Therefore, this approach should be viewed as complementary rather than competitor to our approach as it could potentially be used as an alternative way of getting promising trajectories.
6 CONCLUSION
In this paper we presented a novel methodology for solving text-based games which first extracts high-performing trajectories using phase I of Go-Explore and then trains a simple Seq2Seq model that maps observations to actions using the extracted trajectories. Our experiments show promising results in three settings, with improved generalization and sample efficiency compared to existing methods. Finally, we discussed the limitations and possible improvements of our methodology, which leads to new research challenges in text-based games.
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint 1703vv:703.017200 2017.
Prithviraj Ammanabrolu and Mark Riedl. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3557-3565, 2019a.
Prithviraj Ammanabrolu and Mark O Riedl. Transfer in deep reinforcement learning using knowl- edge graphs. arXiv preprint arXiv:1908.06555 2019b.
Tim Anderson and Stu Galley. The history of zork. The New Zork Times, 4(1-3), 1985.
Andrew G Barto. Intrinsic motivation and reinforcement learning. In Intrinsccally motivated learn- ing in natural and artificial systems, pp. 17-47. Springer, 2013.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12844, 2018
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff' Clune. Improving exploration in evolution strategies for deep reinforcement learning via a pop- ulation of movelty-seeking agents. In Advances in Neural Information Processing Systems, pp. 5027-5038, 2018
Marc-Alexandre Côté. First textworld problems: A reinforcement and language learning challenge. In NewrlPS Workshop, 2018. httpsstts:///ittt 1y/2553Ar
Mlee-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. arXiv preprint arXiv: 1533,, 2018.

---

Under review as a conference paper at ICLR 2020
Mark A DePristo and Robert Zubek. being-in-the-wer In Proceedings of the 2001 AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, pp. 31-34, 2001.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. What can you do with rock?
affordance extraction via word embeddings. arXiv preprint arXiv:/703.03429, 2017.
Samuel Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans
and animals: integrative framework. Annual review of psychology, 68:101-128, 2017.
Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
1rXiv:3308.0850 2013.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with natural language action space. arXiv preprint arXiv:1511.046:3 2015.
Leslie Pack Kaclbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Joel Lehman and Kenneth Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336, 2008
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 199-23, 2011.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint urXiv:I906. 03926, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(75400)529, 2015.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv: / 1504.08946, 2015.
Pierre-Yves Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009,
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16 17, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1543, 2014. wtR.bttp:/ wwww. ac web.. org/ htt///atooooyyy11-1111
Tim Salimans and Richard Chen. Learning montezuma's revenge from single demonstration, arXiv preprint arXiv: 1 8/2203338, 2018.
Jürgen Schmidhuber. Curious model-building control systems. Proc. international joint confer- ence on neural neiworks, pp. 1458-1463 1991a.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991b.
Jürgen Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection Science, 173-17773187,,,0 2006.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
10

---

Under review as a conference paper at ICLR 2020
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. A hierarchical recurrent encoder-decodee for generative context-aware query sug- gestion. In Proceedings of the 24th ACM International Conference on Information and Knowl- edge Management, pp. 553-562. ACM, 2015.
Rural Spaceman. List of text-based computer games, 2019. URL https::/en.wikippp gxg/wixi/List_offrextt
ettpeeee__pppee_eeeeeeeeeeeeeee games
[Online; accessed 21 21-Sepp2011
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Ruo Yu Tao, Marc-Alexandre Côté, Xingdi Yuan, and Layla El Asri Towards solving text-based games by producing adaptive action spaces. arXiv preprint 181.00855 2018.
Chen Tessler, Tom Zahavy, Deborah Cohen, Daniel J Mankowitz, and Shie Mannor. Action as- sembly: Sparse imitation learning for text based games with combinatorial action spaces. arXiv preprint 1.XXv:1905.097000 2019.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Xingdi Yuan, Alexande Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize in text- based games. arXiv preprint arXiv: 1525, 2018.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. In Advances Neural Information Processing Systems, pp. 3562-3573, 2018.
Mikuláš Zelinka. Using reinforcement learning learn how play text-based games. arXiv preprint arXiv: /80100999 2018

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 TEXT-GAMES
CoinCollector In the hard setting (mode 2), each room on the path to the coin has two distractor rooms, and the level (e.g. 30) indicates the shortest path from the starting point to the coin room.
Cooking World The game's complexity is determined by the number of skills and the types of skills that agent needs to master. The skills are:
recipe (1,2,3): number of ingredients in the recipe
take (1,2,3): number of ingredients to find (not already in the inventory)
open: whether containers/doors need to opened
cook: whether ingredients need to be cooked
cut: whether ingredients need to be cut
drop: whether the inventory has limited capacity
go {1,6,91123: number of locations
Thus the hardest game would be a recipe with 3 ingredients, which must picked up somewhere across 12 locations and then need to be cut and cooked, and get access some locations, several doors or objects need to be opened. The handicap of a limited capacity in the inventory makes the game difficult by requiring the agent drop object and later on take again if needed. The grammar used for the text-based games is the following:
go, look, examine, inventory, eat, open/close, take/drop, put/insert
cook X with Y grilled X (when Y is the BBQ)
cook X with Y roasted X (when Y is the oven)
cook X with Y fried X (when Y is the stove)
slice X with Y - sliced X
chop X with Y chopped X
dice with Y diced X
prepare meal
Where Y is something sharp (e.g. knife).
A.2 MODELS
Phase 1: explore until solved
Phase 2: generalize
Select state archive
Go
Explore
Update archive
Run Imitation learning on trajectory
from
Figure 3: High level intuition of the Go-Exlore algorithm. Figure taken from Ecoffet et al. (2019 with permission.
12

---

Under review as a conference paper at ICLR 2020
take
red
pepper


table
Qior, Verb)
Q(o; Adj1)
((o,,Nounl]
((,, Adj2)
Qior, Noun2)
LSTM MeanPool
Ot
Figure LSTM--DQ high level schema.
0t
LSTM
X
Q(or take red pepper from table)
take
red
pepper
from
table
Figure 5: DRRN high level schema.
A.3 PLOTS
13

---

Under review as a conference paper at ICLR 2020
Ot
LSTM Encoder
LSTM Decoder
Take red pepper from table
Figure 6: Seq2Seq for imitation learning.
0.0
rentbed Pgno
resped Fapo2
reripe respet-guz
recsed yoor
rer Alakk roul curl Hopper guodd
resied caseed coull cost d-rd-- gud
recipus eecieve cus rapert siror
respest crsade copertt cotoppin
moraa copeett draph taak cast
reciect
recpeet
renieed
recdeed cakkel
ner
(a) Single
Figure 7: Breakdown of results for the Cooking World games in the Single setting. The results are normalized and sorted by increasing difficulty level from left to right, averaged among the 20 games of each level.
LST
reseed
recped
neipe modeed Fightt
agop
medpet gosp
recipe cook
moddee Fakes makk cade opent sto3p
recdeakk akuue iddii
recieet Aocaak cute openi 2i
resiaae seake secondk orgae dinn gpus
recipeck cocot crrpp 2ol
openn p-12
rec takurz Aucaes roseeve diopi 22
recpett
reciped
recided
recspeek
recsieed eakud hogerm indtoan
wed cowet spper
restlec casll sitpp
ressinc task corkt carte vaaer vinprr
caste
recipadd eeripee
(a) Zero-Shot
Figure 8: Breakdown of results for the Cooking World games in the Zero-Shot setting. The results are normalized and sorted by increasing difficulty level from left to right, averaged among the 20 games of each level.
14

---

Under review as a conference paper at ICLR 2020
Figure Single all the games
stace reciead Ecotircat
reciped
reopect
red
LSTM-DON
LSMM-DON--AM
DARN
ExpExxlone
15

---

Under review as a conference paper at ICLR 2020
Figure 10: Joint all the games
reclos
red por
neipal
recpes
rediectial
redby
recpec
recbes
redsed
recped
recpo2
LSTM-DON
LSTM ADM
DRAM
Exprore
0.0
0.5
1.0
16

---

Under review as a conference paper at ICLR 2020
Figure 11: Zero-Shot all the games
pal saue watt
reopelee
rocper
rodred
recloe fodirs
repe
reopeletae
ped Fot
rociacs
redpesst
recpc
LSTN-DON
ADM
DRAN
Go-Explore
1.0
17


Reviews:
[{"review_id": "Byxbair19H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper considers the task of training an agent to play text-based computer games. One of the key challenges is the high-dimensional action space in these games, which poses a problem for many current methods. The authors propose to learn an LSTM-based decoder to output the action $a_t$ by greedily prediction one word at a time. They achieve this by training a sequence to sequence model on trajectories collected by running the game using a previously proposed exploration method (Go-Explore). While the results are promising, there might be limited novelty beyond training a sequence to sequence model on pre-collected trajectories. Further, the experiments are missing key elements in terms of proper comparison to baselines. \n\nPros:\n1. Nice idea for tackling the unbounded action space problem in text-based games. \n\nCons:\n1. The method depends on the assumption that we can get a set of trajectories with high rewards. This seems a pretty strong assumption. In fact, the authors use a smaller set of admissible actions in order to collect these trajectories in the first place - this seems to not be in line with the goal of solving the large action space problem. If we assume access to this admissible action function, why not just use it directly?\n2. Some of the empirical results may not be fair comparisons (unless I'm missing something). For example, all the baselines for the CookingWorld games use $\\epsilon$-greedy exploration. Since the Go-Explore method assumes access to extra trajectories at the start, this doesn't seem fair to the other baselines which may not observe the same high-reward trajectories.\n\nOther comments:\n1. Do you use the game rewards to train/finetune the seq2seq model or is it only trained in a supervised fashion on the trajectories? (like an imitation learning setup)\n2. How critical is the first step of producing high reward trajectories to the overall performance? Some more analysis or discusssion on this would be helpful to disentangle the contribution of GoExplore from the seq2seq action decoder.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByljVLmfcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an exploration approach of Go-Explore together with imitation learning for playing text games. It is shown to outperform existing solutions in solving text-based games with better sample efficiency and stronger generalization ability to unseen games.\n\nPros:\nSeq2seq imitation learning + Go-Explore is applied to more challenging text games and achieves better performance, higher sample complexity and better generalization ability.\n\nCons: \n•\tFrom modeling perspective, the policy network uses the standard sequence-to-sequence network with attention. And it is trained on the high-reward trajectories obtained with Go-Explore method using imitation learning. From this perspective, there is not much novelty in this paper.\n\nDetailed comments:\n•\tMore details about the mapping function f(x) in Phase 1 should be given.\n•\tIt is not clear why Phase 2 should be called “Robustification”. It seems to be just standard imitation learning of seq2seq model on the high-reward trajectories collected in Phase 1.\n•\tIn the paragraph after eqn. (1), H is defined to be the hidden states of the decoder. Shouldn’t it be the hidden states of the encoder?\n•\tIt seems to be unfair to compare the proposed method with advanced exploration strategy to other model-free baselines that only have very simple exploration strategies (e.g., epsilon-greedy). It is not surprising at all that Go-Explore should outperform them on sparse reward problems. More baselines with better exploration strategies should be compared.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJldEiamKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies the Go-Explore algorithm to the domain of text-based games and shows significant performance gains on Textworld's Coin Collector and Cooking sets of games. Additionally, the authors evaluate 3 different paradigms for training agents on (1) single games, (2) jointly on multiple games, and (3) training on a train set of games and testing on a held-out set of games. Results show that Go-Explore's policies outperform prior methods including DRRN and LSTM-DQN. In addition to better asymptotic performance Go-Explore is also more efficient in terms of the number of environment interactions needed to reach a good policy.\n\nBroadly I like how this paper shows the potency of Go-Explore applied to deterministic environments such as the CoinCollector/CookingWorld  games. It is an algorithm that should not be ignored by the text-based game playing community. I also like the fact that this paper clearly explains and demonstrates how efficient and effective Go-Explore can be, particularly when generalizing to unseen games.\n\nThe major drawback of the paper is a lack of novelty - the Go-Explore algorithm is already well known, and this paper seems to be a direct application of Go-Explore to text-based games. While the results are both impressive and relevant for the text-game-playing community - it's my feeling that this work may not be of general interest to the broader ICLR community due to the lack of new insights in deep learning / representation discovery. However, I am open to being convinced otherwise.\n\nMinor Comments:\n\nThe textworld cooking competition produced at least one highly performing agent (designed by Pedro Lima). While I'm not sure if the code or agent scores are available, it would be a relevant comparison to see how well Go-Explore compared to this agent. (See https://www.microsoft.com/en-us/research/blog/first-textworld-problems-the-competition-using-text-based-games-to-advance-capabilities-of-ai-agents/)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper applies the Go-Explore algorithm to text-based games and shows that it is able to solve text-based game with better sample efficiency and generalization than some alternatives.  The Go-Explore algorithm is used to extract high reward trajectories that can be used to train a policy using a seq2seq model that maps observations to actions.

Paper received 1 weak accept and 2 weak rejects.  Initially the paper received three weak rejects, with the author response and revision convincing one reviewer to increase their score to a weak accept.

Overall, the authors liked the paper and thought that it was well-written with good experiments.
However, there is concern that the paper lacks technical novelty and would not be of interest to the broader ICLR community (beyond those that are interested in text-based games).  Another concern reviewers expressed was that the proposed method was only compared against baselines with simple exploration strategies and that baselines with more advanced exploration strategies should be included.

The AC agrees with above concerns and encourage the authors to improve their paper based on the reviewer feedback, and to consider resubmitting to a venue that is more focused on text-based games (perhaps an NLP conference).
Decision: Reject