Exploration Based Language Learning for Text-Based Games

1 INTRODUCTION . Text-based games became popular in the mid 80s with the game series Zork ( Anderson & Galley , 1985 ) resulting in many different text-based games being produced and published ( Spaceman , 2019 ) . These games use a plain text description of the environment and the player has to interact with them by writing natural-language commands . Recently , there has been a growing interest in developing agents that can automatically solve text-based games ( Côté et al. , 2018 ) by interacting with them . These settings challenge the ability of an artificial agent to understand natural language , common sense knowledge , and to develop the ability to interact with environments using language ( Luketina et al. , 2019 ; Branavan et al. , 2012 ) . Since the actions in these games are commands that are in natural language form , the major obstacle is the extremely large action space of the agent , which leads to a combinatorially large exploration problem . In fact , with a vocabulary ofN words ( e.g . 20K ) and the possibility of producing sentences with at most m words ( e.g . 7 words ) , the total number of actions is O ( Nm ) ( e.g . 20K7 ≈ 1.28e30 ) . To avoid this large action space , several existing solutions focus on simpler text-based games with very small vocabularies where the action space is constrained to verb-object pairs ( DePristo & Zubek , 2001 ; Narasimhan et al. , 2015 ; Yuan et al. , 2018 ; Zelinka , 2018 ) . Moreover , many existing works rely on using predetermined sets of admissible actions ( He et al. , 2015 ; Tessler et al. , 2019 ; Zahavy et al. , 2018 ) . However , a more ideal , and still under explored , alternative would be an agent that can operate in the full , unconstrained action space of natural language that can systematically generalize to new text-based games with no or few interactions with the environment . To address this challenge , we propose to use the idea behind the recently proposed GoExplore ( Ecoffet et al. , 2019 ) algorithm . Specifically , we propose to first extract high reward trajectories of states and actions in the game using the exploration methodology proposed in Go-Explore and then train a policy using a Seq2Seq ( Sutskever et al. , 2014 ) model that maps observations to actions , in an imitation learning fashion . To show the effectiveness of our proposed methodology , we first benchmark the exploration ability of Go-Explore on the family of text-based games called CoinCollector ( Yuan et al. , 2018 ) . Then we use the 4,440 games of “ First TextWorld Problems ” ( Côté , 2018 ) , which are generated using the machinery introduced by Côté et al . ( 2018 ) , to show the generalization ability of our proposed methodology . In the former experiment we show that Go-Explore finds winning trajectories faster than existing solutions , and in the latter , we show that training a Seq2Seq model on the trajectories found by Go-Explore results in stronger generalization , as suggested by the stronger performance on unseen games , compared to existing competitive baselines ( He et al. , 2015 ; Narasimhan et al. , 2015 ) . Reinforcement Learning Based Approaches for Text-Based Games Among reinforcement learning based efforts to solve text-based games two approaches are prominent . The first approach assumes an action as a sentence of a fixed number of words , and associates a separate Qfunction ( Watkins , 1989 ; Mnih et al. , 2015 ) with each word position in this sentence . This method was demonstrated with two-word sentences consisting of a verb-object pair ( e.g . take apple ) ( DePristo & Zubek , 2001 ; Narasimhan et al. , 2015 ; Yuan et al. , 2018 ; Zelinka , 2018 ; Fulda et al. , 2017 ) . In the second approach , one Q-function that scores all possible actions ( i.e . sentences ) is learned and used to play the game ( He et al. , 2015 ; Tessler et al. , 2019 ; Zahavy et al. , 2018 ) . The first approach is quite limiting since a fixed number of words must be selected in advance and no temporal dependency is enforced between words ( e.g . lack of language modelling ) . In the second approach , on the other hand , the number of possible actions can become exponentially large if the admissible actions ( a predetermined low cardinality set of actions that the agent can take ) are not provided to the agent . A possible solution to this issue has been proposed by Tao et al . ( 2018 ) , where a hierarchical pointer-generator is used to first produce the set of admissible actions given the observation , and subsequently one element of this set is chosen as the action for that observation . However , in our experiments we show that even in settings where the true set of admissible actions is provided by the environment , a Q-scorer ( He et al. , 2015 ) does not generalize well in our setting ( Section 5.2 Zero-Shot ) and we would expect performance to degrade even further if the admissible actions were generated by a separate model . Less common are models that either learn to reduce a large set of actions into a smaller set of admissible actions by eliminating actions ( Zahavy et al. , 2018 ) or by compressing them in a latent space ( Tessler et al. , 2019 ) . Exploration in Reinforcement Learning In most text-based games rewards are sparse , since the size of the action space makes the probability of observing a reward extremely low when taking only random actions . Sparse reward environments are particularly challenging for reinforcement learning as they require longer term planning . Many exploration based solutions have been proposed to address the challenges associated with reward sparsity . Among these exploration approaches are novelty search ( Lehman & Stanley , 2008 ; 2011 ; Conti et al. , 2018 ; Achiam & Sastry , 2017 ; Burda et al. , 2018 ) , intrinsic motivation ( Schmidhuber , 1991b ; Oudeyer & Kaplan , 2009 ; Barto , 2013 ) , and curiosity based rewards ( Schmidhuber , 2006 ; 1991a ; Pathak et al. , 2017 ) . For text based games exploration methods have been studied by Yuan et al . ( 2018 ) , where the authors showed the effectiveness of the episodic discovery bonus ( Gershman & Daw , 2017 ) in environments with sparse rewards . This exploration method can only be applied in games with very small action and state spaces , since their counting methods rely on the state in its explicit raw form . 2 METHODOLOGY . Go-Explore ( Ecoffet et al. , 2019 ) differs from the exploration-based algorithms discussed above in that it explicitly keeps track of under-explored areas of the state space and in that it utilizes the determinism of the simulator in order to return to those states , allowing it to explore sparse-reward environments in a sample efficient way ( see Ecoffet et al . ( 2019 ) as well as section 4.1 ) . For the experiments in this paper we mainly focus on the final performance of our policy , not how that policy is trained , thus making Go-Explore a suitable algorithm for our experiments . Go-Explore is composed of two phases . In phase 1 ( also referred to as the “ exploration ” phase ) the algorithm explores the state space through keeping track of previously visited states by maintaining an archive . During this phase , instead of resuming the exploration from scratch , the algorithm starts exploring from promising states in the archive to find high performing trajectories . In phase 2 ( also referred to as the “ robustification ” phase , while in our variant we will call it “ generalization ” ) the algorithm trains a policy using the trajectories found in phase 1 . Following this framework , which is also shown in Figure 3 ( Appendix A.2 ) , we define the Go-Explore phases for text-based games . Let us first define text-based games using the same notation as Yuan et al . ( 2018 ) . A text-based game can be framed as a discrete-time Partially Observable Markov Decision Process ( POMDP ) ( Kaelbling et al. , 1998 ) defined by ( S , T , A , Ω , O , R ) , where : S is the set of the environment states , T is the state transition function that defines the next state probability , i.e . T ( st+1|at ; st ) ∀st ∈ S , A is the set of actions , which in our case is all the possible sequences of tokens , Ω is the set of observations , i.e . text observed by the agent every time has it to take an action in the game ( i.e . dialogue turn ) which is controlled by the conditional observation probability O , i.e . O ( ot|st , at−1 ) , and , finally , R is the reward function i.e . r = R ( s , a ) . Let us also define the observation ot ∈ Ω and the action at ∈ A. Text-based games provide some information in plain text at each turn and , without loss of generality , we define an observation ot as the sequence of tokens { o0t , · · · , ont } that form that text . Similarly , we define the tokens of an action at as the sequence { a0t , · · · , amt } . Furthermore , we define the set of admissible actions At ∈ A as At = { a0 , · · · , az } , where each ai , which is a sequence of tokens , is grammatically correct and admissible with reference to the observation ot . 2.1 PHASE 1 : EXPLORATION . In phase 1 , Go-Explore builds an archive of cells , where a cell is defined as a set of observations that are mapped to the same , discrete representation by some mapping function f ( x ) . Each cell is associated with meta-data including the trajectory towards that cell , the length of that trajectory , and the cumulative reward of that trajectory . New cells are added to the archive when they are encountered in the environment , and existing cells are updated with new meta-data when the trajectory towards that cells is higher scoring or equal scoring but shorter . At each iteration the algorithm selects a cell from this archive based on meta-data of the cell ( e.g . the accumulated reward , etc . ) and starts to randomly explore from the end of the trajectory associated with the selected cell . Phase 1 requires three components : the way that observations are embedded into cell representations , the cell selection , and the way actions are randomly selected when exploring from a selected cell . In our variant of the algorithm , f ( x ) is defined as follows : given an observation , we compute the word embedding for each token in this observation , sum these embeddings , and then concatenate this sum with the current cumulative reward to construct the cell representation . The resulting vectors are subsequently compressed and discretized by binning them in order to map similar observations to the same cell . This way , the cell representation , which is the key of the archive , incorporates information about the current observation of the game . Adding the current cumulative reward to the cell representation is new to our Go-Explore variant as in the original algorithm only down-scaled image pixels were used . It turned out to be a very very effective to increase the speed at which high reward trajectories are discovered . In phase 1 , we restrict the action space to the set of admissible actions At that are provided by the game at every step of the game 1 . This too is particularly important for the random search to find a high reward trajectory faster . Finally , we denote the trajectory found in phase 1 for game g as Tg = [ ( o0 , a0 , r0 ) , · · · , ( ot , at , rt ) ] . 1Note that the final goal is to generalize to test environments where admissible actions are not available . The assumption that admissible actions are available at training time holds in cases where we build the training environment for an RL agent ( e.g . a hand-crafted dialogue system ) , and a system trained in such an environment can be practically applied as long as the system does not rely on such information at test time . Thus , we assumed that these admissible commands are not available at test time .

Summary: This paper applies the Go-Explore algorithm to the domain of text-based games and shows significant performance gains on Textworld's Coin Collector and Cooking sets of games. Additionally, the authors evaluate 3 different paradigms for training agents on (1) single games, (2) jointly on multiple games, and (3) training on a train set of games and testing on a held-out set of games. Results show that Go-Explore's policies outperform prior methods including DRRN and LSTM-DQN. In addition to better asymptotic performance Go-Explore is also more efficient in terms of the number of environment interactions needed to reach a good policy.