Monte-Carlo Planning and Learning with Language Action Value Estimates

1 INTRODUCTION . Building an intelligent goal-oriented agent that can perceive and react via natural language is one of the grand challenges of artificial intelligence . In pursuit of this goal , we consider Interactive Fiction ( IF ) games ( Nelson , 2001 ; Montfort , 2005 ) , which are text-based simulation environments where the agent interacts with the environment only through natural language . They serve as a useful testbed for developing language-based goal-oriented agents , posing important challenges such as natural language understanding , commonsense reasoning , and non-myopic planning in the combinatorial search space of language actions . IF games naturally have a large branching factor , with at least hundreds of natural language actions that can affect the simulation of game states . This renders naive exhaustive search infeasible and raises the strong need for language-grounded planning ability , i.e . effective search space is too large to choose an optimal action , without inferring the future impact of language actions by understanding the environment state described in natural language . Still , standard planning methods such as Monte-Carlo tree search ( MCTS ) are language-agnostic and rely only on uncertainty-driven exploration , encouraging more search on less-visited states and actions . This simple uncertainty-based strategy is not sufficient to find an optimal language action under limited search time , especially when each language action is treated as a discrete token . On the other hand , recent reinforcement learning agents for IF games have started to leverage pre-trained word embeddings for language understanding ( He et al. , 2016 ; Fulda et al. , 2017 ; Hausknecht et al. , 2020 ) or knowledge graphs for commonsense reasoning ( Ammanabrolu & Hausknecht , 2020 ) , but their exploration strategies are still limited to the -greedy or the softmax policies , lacking more structured and non-myopic planning ability . As a consequence , current state-of-the-art agents for IF games still have not yet been up to the human-level play . In this paper , we introduce Monte-Carlo planning with Language Action Value Estimates ( MCLAVE ) , a planning algorithm for the environments with text-based interactions . MC-LAVE combines Monte-Carlo tree search with language-driven exploration , addressing the search inefficiency attributed to the lack of language understanding . It starts with credit assignment to language actions via Q-learning of the experiences collected from the past searches . Then , MC-LAVE assigns nonuniform search priorities to each language action based on the optimistically aggregated Q-estimates of the past actions that share similar meanings with the candidate action , so as to focus more on the semantically promising actions . This is in contrast to the previous methods that involve language understanding in the form of a knowledge graph , where the insignificant language actions are uniformly filtered out by the graph mask ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu et al. , 2020 ) . We show that the non-uniform search empowered by language understanding in MC-LAVE yields better search efficiency while not hurting the asymptotic guarantee of MCTS . We then present our reinforcement learning approach that uses MC-LAVE as a strong policy improvement operator . Since MCTS explores the combinatorial space of action sequences , its search results can be far better than the simple greedy improvement , as demonstrated in the game of Go ( Silver et al. , 2017 ) . This final algorithm , MC-LAVE-RL , alternates between planning via MC-LAVE and supervised learning of self-generated language actions . Experimental results demonstrate that MC-LAVE-RL achieves new high scores in various IF games provided in the Jericho framework ( Hausknecht et al. , 2020 ) , showing the effectiveness of language-grounded MC-LAVE planning . 2 BACKGROUND . 2.1 INTERACTIVE FICTION GAME . Interactive Fiction ( IF ) games are fully text-based environments where the observation and the action spaces are defined as natural language . The game-playing agent observes textual descriptions of the world , selects a language-based action , and receives the associated reward . IF games can be modeled as a special case of partially observable Markov decision processes ( POMDPs ) defined by tuple 〈S , A , Ω , T , O , R , γ〉 , where S is the set of environment states s , A is the set of language actions a , Ω is the set of text observations o , T ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) is the transition function , R ( s , a ) is the reward function for taking action a in state s , O ( s ) = o is the deterministic observation function in state s , and γ ∈ ( 0 , 1 ) is the discount factor . The history at time step t , ht = { o0 , a0 , . . . , ot−1 , at−1 , ot } , is a sequence of observations and actions . The goal is to find an optimal policy π∗ that maximizes the expected cumulative rewards , i.e . π∗ = arg maxπ Eπ [ ∑∞ t=0 γ tR ( st , at ) ] . We use the same definition of observation and action space as Hausknecht et al . ( 2020 ) ; Ammanabrolu & Hausknecht ( 2020 ) ; Côté et al . ( 2018 ) , i.e . An observation is defined by ot = ( otdesc , otgame , otinv , at−1 ) where otdesc is the textual description of the current location of the agent , otgame is the simulator response to the previous action taken by the agent , otinv is the information of agent ’ s inventory , and at−1 is the previous action taken by the agent . An action is denoted by a sequence of words at = ( a1t , a 2 t , . . . , a |at| t ) . Finally , we denote Avalid ( ot ) ⊆ A as the set of valid actions for the observation ot , which is provided by the Jericho environment interface . Figure 1 shows an example of observation and action in ZORK1 , one of the representative IF games . 2.2 CHALLENGES IN INTERACTIVE FICTION GAME . IF games pose important challenges for reinforcement learning agents , requiring natural language understanding , commonsense reasoning , and non-myopic language-grounded planning ability in combinatorial search space of language actions ( Hausknecht et al. , 2020 ) . More concretely , consider the particular game state of ZORK1 described in Figure 1 . In this example situation , a human player would be naturally capable of performing strategic planning via language understanding and commonsense reasoning : ( 1 ) the ‘ closed trap door ’ will have to be opened and be explored to proceed with the game , ( 2 ) however , acquiring the ‘ lantern ’ should precede entering the trap door since the cellar , which is expected to exist below the trap door , could be likely pitch-dark . Without such language-grounded reasoning and planning , the agent may need to try out every actions uniformly , most of them making it vulnerable to being eaten by a monster in the cellar who always appears when there is no light source . As a result , any agent that lacks the ability of long-term planning with language reasoning is prone to be stuck at a suboptimal policy , which enters the cellar to obtain an immediate reward and does nothing further to avoid encountering the monster that kills the agent immediately . To overcome the bottleneck , the agent should be able to infer that ‘ take lantern ’ is worthy enough to be chosen in preference to other actions even though it gives no immediate reward . In this work , we will precisely address such failures of myopic planning that arise from the lack of language-grounded exploration strategy . To this end , we first propose MCTS-based long-term planning for nonmyopic-planning . Then we further improve the search efficiency of MCTS-based planning by incorporating our novel language-grounded exploration strategy based on Language Action Value Estimates , which will be detailed in the subsequent section . 2.3 MONTE-CARLO TREE SEARCH . Monte-Carlo Tree Search ( MCTS ) ( Kocsis & Szepesvári , 2006 ; Coulom , 2006 ; Browne et al. , 2012 ) is a generic online planning algorithm that combines random sampling and tree search , which has become the de-facto standard method for large sequential decision-making problems . Starting from an empty tree , MCTS repeatedly performs the phases of selection , expansion , rollout , and backpropagation to evaluate the nodes of the search tree with increased accuracy . UCT ( Kocsis & Szepesvári , 2006 ) is the standard MCTS method , which adopts UCB ( Auer et al. , 2002 ) as an action selection rule at each internal node of the search tree : arg max a [ Q ( h , a ) + c √ logN ( h ) N ( h , a ) ] ( 1 ) where h is the history of past observations and actions , Q ( h , a ) is the average value of sampled returns with taking action a in h , N ( h ) is the number of simulations performed in h , N ( h , a ) is the number of times action a is selected in h , and c is a constant that balances exploration and exploitation . However , UCT suffers from severe search inefficiency in the problems with a large action space such as IF games , since it requires to take every action at least once and relies only on uncertainty-driven ( or visit-count-based ) exploration . PUCT ( Silver et al. , 2017 ; 2018 ) partially addresses the challenges of IF games by adopting PUCB ( Rosin , 2011 ) , which involves a prior action distribution π ( ·|h ) and eliminates the need for choosing every action at least once . The action selection rule is given by : arg max a [ Q ( h , a ) + cPUCTπ ( a|h ) √ N ( h ) N ( h , a ) + 1 ] ( 2 ) The prior distribution π ( ·|h ) in Eq . ( 2 ) can be trained by behavioral cloning of ( hroot , a∗root ) samples obtained by the result of tree search in previous time steps ( Anthony et al. , 2017 ; Silver et al. , 2018 ) . However , this procedure not only discards other search information such as abundant ( ht , at , rt , ht+1 ) samples obtained during the search but also hardly encourages information sharing across the semantically similar actions . Therefore , PUCT is still limited in search efficiency for the tasks with language action space , raising the need for a language-driven exploration strategy . 3 MONTE-CARLO PLANNING WITH LANGUAGE ACTION VALUE ESTIMATES . Language-grounded planning is essential to address the challenges of IF games . To this end , we need a mechanism to incorporate language understanding and commonsense reasoning into the ex- ploration strategy of search . In this section , we introduce Monte-Carlo planning with Language Action Value Estimates ( MC-LAVE ) , a novel planning algorithm that combines MCTS with languagedriven exploration . MC-LAVE does not rely on any handcrafted heuristic evaluation but starts tabula rasa , based only upon pre-trained word embeddings that can well define the semantic similarities between two language actions . In addition , it only requires a black-box simulator ( r , s′ , o′ ) ∼ G ( s , a ) , which yields ( o , a , r , o′ ) samples at each simulator query .

Summary: This paper presents a novel MCTS-based policy improvement operator called MC-LAVE designed specifically for environments with text-based action spaces. MC-LAVE adds an additional term to PUCT that shares information across semantically similar actions. This additional term for a given action $a$ is set to the soft maximum over the Q function evaluated at all $(o, \bar{a})$ pairs of transitions in the replay buffers whose transition action $\bar{a}$ is within some cosine distance $\delta$ of the action $a$.  As shown in Appendix C, addition of this term preserves the expected regret bounds of PUCB. The paper reports empirical improvements over existing methods on various interactive fiction (IF) games in the Jericho suite.