Multi-Stage Episodic Control for Strategic Exploration in Text Games

1 INTRODUCTION . Text adventure games provide a unique test-bed for algorithms that integrate reinforcement learning ( RL ) with natural language understanding . Aside from the linguistic ingredient , a key challenge in these games is the combination of very large action spaces with sparse rewards , which calls for a delicate balance between exploration and exploitation . For instance , the game of ZORK1 can contain up to fifty valid action commands per state2 to choose from . Importantly , unlike other RL environments ( Bellemare et al. , 2013 ; Todorov et al. , 2012 ) , the set of valid action choices does not remain constant across states , with unseen actions frequently appearing in later states of the game . For example , Figure 1 shows several states from ZORK1 where a player has to issue unique action commands like ‘ kill troll with sword ’ , ‘ echo ’ or ‘ odysseus ’ to progress further in the game . This requires a game-playing agent to perform extensive exploration to determine the appropriateness of actions , which is hard to bootstrap from previous experience . On the other hand , since rewards are sparse , the agent only gets a few high-scoring trajectories to learn from , requiring vigorous exploitation in order to get back to the furthest point of the game and make progress thereon . Prior approaches to solving these games ( He et al. , 2016a ; Hausknecht et al. , 2020 ; Ammanabrolu & Hausknecht , 2020 ; Guo et al. , 2020 ) usually employ a single policy and action selection strategy , making it difficult to strike the right balance between exploration and exploitation . In this paper , we propose eXploit-Then-eXplore ( XTX ) , an algorithm for multi-stage control to explicitly decompose the exploitation and exploration phases within each episode . In the first phase , the agent selects actions according to an exploitation policy which is trained using self-imitation learning on a mixture of promising trajectories from its past experience sampled using a combination of factors such as episodic scores and path length . This policy allows the agent to return to a 1Our code is available in supplementary material . 2Valid actions are a feature in the Jericho simulator ( Hausknecht et al. , 2020 ) to improve computational tractability for RL . Without this handicap , the number of possible action commands is almost 200 billion . state at the frontier of the state space it has explored so far . Importantly , we ensure that this policy is trained on a mixture of trajectories with different scores , in order to prevent the agent from falling into a local minimum in the state space ( e.g . red space in Figure 1 ) . In the second phase , an exploration policy takes over and the agent chooses actions using a value function that is trained using a combination of a temporal difference ( TD ) loss and an auxiliary inverse dynamics loss ( Pathak et al. , 2017 ) . This allows the agent to perform strategic exploration around the frontier by reusing values of previously seen actions while exploring novel ones in order to find rewards and make progress in the game . To allow for more fine-grained control , we use a mixture of policies for both exploration and exploitation , and only change a single interpolation parameter to switch between phases . The two-stage approach to gameplay in XTX allows an agent to combine global decisions about which parts of the game space to advance , followed by local exploration of sub-strategies in that space . This is similar to how humans tackle these games : if a player were to lose to a troll in the dungeon , they would immediately head back to the dungeon after the game restarts and explore strategies thereon to try and defeat the troll . XTX ’ s multi-stage episodic control differs from prior approaches that add exploration biases to a single policy through curiosity bonuses ( Pathak et al. , 2017 ; Tang et al. , 2017 ) or use different reward functions to train a separate exploration policy ( Colas et al. , 2018 ; Schäfer et al. , 2021 ; Whitney et al. , 2021 ) . Moreover , in contrast to methods like GoExplore ( Ecoffet et al. , 2021 ; Madotto et al. , 2020 ) , XTX does not have global phases of random exploration followed by learning —instead , both our policies are continuously updated with new experience , allowing XTX to adapt and scale as the agent goes deeper into the game . XTX also does not make any assumptions about the environment being deterministic , and does not require access to underlying game simulator or additional memory archives to keep track of game trees . We evaluate XTX on a set of games from the Jericho benchmark ( Hausknecht et al. , 2020 ) , considering both deterministic and stochastic variants of the games . XTX outperforms competitive baselines on 8 out of 12 games , and achieves an average improvement of 2.7 % in terms of normalized scores across all games . For instance , on Zork1 , our method obtains a score of 103 in the deterministic setting and 67 in the stochastic setting — substantial improvements over baseline scores of 40 and 41 , respectively . We also perform ablation studies to demonstrate the importance of the multi-stage approach , as well as several key design choices in our exploitation and exploration policies . 2 RELATED WORK . Reinforcement learning for text-based games Prior work on building autonomous agents for text adventure games has explored several variants of reinforcement learning ( RL ) agents equipped with a language understanding module ( see Osborne et al . ( 2021 ) for a detailed survey ) . Innovations on the language representation side include using deep neural networks for handling text sequences trained using RL ( Narasimhan et al. , 2015 ; He et al. , 2016a ) , knowledge graphs to track states across trajectories ( Ammanabrolu & Hausknecht , 2020 ; Adhikari et al. , 2020 ; Xu et al. , 2020 ) , and incorporating question answering or reading comprehension modules ( Ammanabrolu et al. , 2020 ; Guo et al. , 2020 ) . While these approaches focus mainly on the issues of partial observability and language semantics , they all suffer from challenges due to the large action space and sparse rewards found in games from benchmarks like Jericho ( Hausknecht et al. , 2020 ) . Some approaches aim to navigate the large action space by filtering inadmissible actions ( Zahavy et al. , 2018 ; Jain et al. , 2020 ) , leveraging pre-trained language models for action selection ( Yao et al. , 2020 ; Jang et al. , 2020 ) or word embeddings for affordance detection ( Fulda et al. , 2017 ) . Recent work has also explored tackling sparse rewards by employing hierarchical policies ( Xu et al. , 2021 ) . Navigating the exploration-exploitation trade-off in RL The trade-off between exploration and exploitation is a well-known issue in reinforcement learning ( Sutton & Barto , 2018 ; François-Lavet et al. , 2018 ; Kearns & Singh , 2002 ; Brafman & Tennenholtz , 2002 ) . In this respect , we can broadly categorize prior techniques into two types . The first type includes methods with mixed objectives that balance exploration with exploitation . Oh et al . ( 2018 ) introduced the idea of self-imitation learning on high-scoring episodes to exploit good trajectories , as an auxiliary objective to standard actor-critic methods . Prior work has also explored the addition of curiosity bonuses to encourage exploration ( Pathak et al. , 2017 ; Tang et al. , 2017 ; Li et al. , 2020 ; Bellemare et al. , 2016 ; Machado et al. , 2020 ; Taiga et al. , 2021 ) . While we leverage self-imitation learning for exploitation and inverse dynamics bonuses for exploration , our multi-stage mixed policy allows for more strategic exploration in text games . Other works learn a mixture of policies for decoupling exploration and exploitation , either by using a conditional architecture with shared weights ( Badia et al. , 2020 ) , pre-defining an exploration mechanism for restricted policy optimization ( Shani et al. , 2019 ) , or learning separate task and exploration policies to maximize different reward functions ( Colas et al. , 2018 ; Schäfer et al. , 2021 ; Whitney et al. , 2021 ) . While we also train multiple policies , our multistage algorithm performs distinct exploitation and exploration phases within each episode , and does not require pre-defined exploration policies or phases . Further , our work considers environments with significantly larger action spaces that evolves dynamically with deeper progress in the game . The second class of algorithms explicitly separate exploitation and exploration in each episode . Methods like E3 ( Kearns & Singh , 2002 ; Henaff , 2019 ) maintain a set of dynamics models to encourage exploration . Policy-based Go-Explore ( Ecoffet et al. , 2021 ) uses self-imitation learning to ‘ exploit ’ high-reward trajectories , but requires choosing intermediate sub-goals for the agent to condition its policy on . PC-PG ( Agarwal et al. , 2020 ) uses a policy cover to globally choose state spaces to return to , followed by random exploration . Compared to these approaches , we perform more strategic local exploration due to the use of a Q-function with inverse dynamics bonus and do not require any assumptions about determinism or linearity of the MDP . We provide a more technical discussion on the novelty of our approach at the end of Section 3 . Directed exploration in text-based games As previously mentioned , the large dynamic action space in text games warrant specific strategies for directed exploration . Ammanabrolu et al . ( 2020 ) used a knowledge-graph based intrinsic motivation reward to encourage exploration . Jang et al . ( 2020 ) incorporated language semantics into action selection for planning using MCTS . Both above methods utilize the determinism of the game or require access to underlying simulator to restart the game from specific states . Madotto et al . ( 2020 ) modified the Go-Explore algorithm to test generalization in the CoinCollector ( Yuan et al. , 2018 ) and Cooking world domains ( Côté et al. , 2018 ) . Their method has two phases — the agent first randomly explores and collects trajectories and then a policy is learned through imitation of the best trajectories in the experience replay buffer . In contrast , our algorithm provides for better exploration of new , unseen actions in later stages of the game through the use of an inverse dynamics module and performs multiple rounds of imitation learning for continuous scaling to deeper trajectories in the game . Recently , Yao et al . ( 2021 ) used inverse dynamics to improve exploration and Yao et al . ( 2020 ) used a language model to generate action candidates that guide exploration . However , both approaches did not employ a two-stage rollout like our work , and the latter considers a different setup without any valid action handicap . 3 METHOD . Background Text-adventure games can be formalized as a Partially Observable Markov Decision Process ( POMDP ) ⟨S , T , A , O , R , γ⟩ . The underlying state space S contains all configurations of the game state within the simulator , which is unobserved by the agent . The agent receives observations from O from which it has to infer the underlying state s ∈ S. The action set A consists of short phrases from the game vocabulary , T ( s′|s , a ) is the transition function which determines the probability of moving to the next state s′ given the agent has taken action a in state s , R ( s , a ) determines the instantaneous reward , and γ is the reward discount factor . Existing RL approaches that tackle these games usually learn a value function using game rewards . One example is the Deep Reinforcement Relevance Network ( DRRN ) ( He et al. , 2016b ) which trains a deep neural network with parameters ϕ to approximate Qϕ ( o , a ) . This model encodes each observation o and action candidate a using two recurrent neural networks fo and fa and aggregates the representations to derive the Q-value through an MLP q : Qϕ ( o , a ) = q ( fo ( o ) , fa ( a ) ) . The parameters ϕ of the model are trained by minimizing the temporal difference ( TD ) loss on tuples ( o , a , r , o′ ) of observation , action , reward and the next observation sampled from an experience replay buffer : LTD ( ϕ ) = ( r + γmax a′∈A Qϕ ( o ′ , a′ ) −Qϕ ( o , a ) ) 2 ( 1 ) The agent samples actions using a softmax exploration policy π ( a|o ; ϕ ) ∝ exp ( Qϕ ( o , a ) ) . Challenges There are two unique aspects of text adventure games that make them challenging . First , the action space is combinatorially large – usually games can accept action commands of 1-4 words with vocabularies of up to 2257 words . This means an agent potentially has to choose between O ( 22574 ) = 2.6 ∗ 1013 actions at each state of the game . To make this problem more tractable , benchmarks like Jericho ( Hausknecht et al. , 2020 ) provide a valid action detector that filters out the set of inadmissible commands ( i.e . commands that are either unrecognized by the game engine or do not change the underlying state of the game ) . However , this still results in the issue of dynamic action spaces for the agent that change with the state . For example , the action ‘ echo ’ is unique to the Loud Room ( see Figure 1 ) . Second , these games have very sparse rewards and several bottleneck states ( Ammanabrolu et al. , 2020 ) , making learning difficult for RL agents that use the same policy for exploration and exploitation ( He et al. , 2016a ; Hausknecht et al. , 2020 ; Ammanabrolu & Riedl , 2019 ) . This results in issues of derailment ( Ecoffet et al. , 2021 ) , with agents unable to return to promising parts of the state space , resulting in a substantial gap between average episode performance of the agent and the maximum score it sees in the game ( Yao et al. , 2020 ) . 3.1 OUR ALGORITHM : EXPLOIT-THEN-EXPLORE ( XTX ) We tackle the two challenges outlined above using a multi-stage control policy that allows an agent to globally pick promising states of the game to visit while allowing for strategic local exploration thereafter . To this end , we develop eXploit-Then-eXplore , where an agent performs the following two distinct phases of action selection in each episode . PHASE 1 : Exploitation In the exploitation phase , the agent makes a global decision about revisiting promising states of the game it has seen in its past episodes . We sample k trajectories from the experience replay D using a combination of factors such as game score and trajectory length . These trajectories are then used to learn a policy cover πexploit using self-imitation learning ( Oh et al. , 2018 ) ( see Section 3.1.2 for details ) . The agent then samples actions from πexploit until it has reached either ( 1 ) the maximum score seen during training or ( 2 ) a number of steps in the episode equal to the longest of the k sampled trajectories . The second condition ensures the agent can always return to the longest of the k sampled trajectories3 . Once either of the above conditions is satisfied , the agent transitions to the exploration phase , adjusting its policy . PHASE 2 : Exploration In the exploration phase , the agent uses a different policy πexplore trained using a combination of the temporal difference loss and an auxiliary inverse dynamics bonus ( Pathak et al. , 2017 ; Yao et al. , 2021 ) ( see Section 3.1.3 for details ) . The intuition here is that the exploitation policy πexploit in phase 1 has brought the agent to the game frontier , which is under-explored and may contain a combination of common ( e.g . “ open door ” ) and novel ( e.g . “ kill troll with sword ” ) actions . Therefore , using a combination of Q-values and inverse dynamics bonus enables the agent to perform strategic , local exploration to further expand the frontier and discover new rewarding states . The agent continues sampling from πexplore until a terminal state or episode limit is reached . 3It is possible that the agent needs less steps to return to a particular part of the game space and hence wastes some steps , but we empirically didn ’ t find this to be a problem .

Summary: In this paper, the authors propose eXploit-Then-eXplore (XTX), a training strategy for agent solving human-generated text-based games.   XTX consists of two training phases:  1. In the exploitation phase, the agent samples high quality experiences (in terms of score and trajectory length) from its replay buffer. Using the sampled trajectories, an action generation module is trained. At a certain game step $t$, the action generation module takes the observation $o_t$, as well as the two most recent past actions $a_{t-1}$ and $a_{t-2}$ as input, and generates the new action $a_t$ in a word-by-word auto-regressive manner. This process is referred as self-imitation by the authors.   2. In the exploration phase, in addition to the Q-learning loss as used in DRRN, the authors use two auxiliary losses to encourage the model to capture useful representations. First, the inverse dynamics loss $L_{inv}$ optimizes a module that predicts an action $a_t$ given two consecutive observations $o_t$ and $o_{t+1}$, where $o_{t+1}$ is resulted by $a_t$ given $o_t$. The second loss $L_{dec}$ is a regularizer that optimizes a module reconstructs an action $a_t$ from its encoding $f_a(a_t)$.  During training, the two phases take control in an (almost) alternate manner, however, there is a coefficient $\lambda$ controls the interpolation between the phases. The authors show that it is beneficial to not having the exploitation take control solely.   On a subset of games from the Jericho suite, the authors show their agent outperform prior works.