Monte-Carlo Planning and Learning with Language Action Value Estimates

1 INTRODUCTION . Building an intelligent goal-oriented agent that can perceive and react via natural language is one of the grand challenges of artificial intelligence . In pursuit of this goal , we consider Interactive Fiction ( IF ) games ( Nelson , 2001 ; Montfort , 2005 ) , which are text-based simulation environments where the agent interacts with the environment only through natural language . They serve as a useful testbed for developing language-based goal-oriented agents , posing important challenges such as natural language understanding , commonsense reasoning , and non-myopic planning in the combinatorial search space of language actions . IF games naturally have a large branching factor , with at least hundreds of natural language actions that can affect the simulation of game states . This renders naive exhaustive search infeasible and raises the strong need for language-grounded planning ability , i.e . effective search space is too large to choose an optimal action , without inferring the future impact of language actions by understanding the environment state described in natural language . Still , standard planning methods such as Monte-Carlo tree search ( MCTS ) are language-agnostic and rely only on uncertainty-driven exploration , encouraging more search on less-visited states and actions . This simple uncertainty-based strategy is not sufficient to find an optimal language action under limited search time , especially when each language action is treated as a discrete token . On the other hand , recent reinforcement learning agents for IF games have started to leverage pre-trained word embeddings for language understanding ( He et al. , 2016 ; Fulda et al. , 2017 ; Hausknecht et al. , 2020 ) or knowledge graphs for commonsense reasoning ( Ammanabrolu & Hausknecht , 2020 ) , but their exploration strategies are still limited to the -greedy or the softmax policies , lacking more structured and non-myopic planning ability . As a consequence , current state-of-the-art agents for IF games still have not yet been up to the human-level play . In this paper , we introduce Monte-Carlo planning with Language Action Value Estimates ( MCLAVE ) , a planning algorithm for the environments with text-based interactions . MC-LAVE combines Monte-Carlo tree search with language-driven exploration , addressing the search inefficiency attributed to the lack of language understanding . It starts with credit assignment to language actions via Q-learning of the experiences collected from the past searches . Then , MC-LAVE assigns nonuniform search priorities to each language action based on the optimistically aggregated Q-estimates of the past actions that share similar meanings with the candidate action , so as to focus more on the semantically promising actions . This is in contrast to the previous methods that involve language understanding in the form of a knowledge graph , where the insignificant language actions are uniformly filtered out by the graph mask ( Ammanabrolu & Hausknecht , 2020 ; Ammanabrolu et al. , 2020 ) . We show that the non-uniform search empowered by language understanding in MC-LAVE yields better search efficiency while not hurting the asymptotic guarantee of MCTS . We then present our reinforcement learning approach that uses MC-LAVE as a strong policy improvement operator . Since MCTS explores the combinatorial space of action sequences , its search results can be far better than the simple greedy improvement , as demonstrated in the game of Go ( Silver et al. , 2017 ) . This final algorithm , MC-LAVE-RL , alternates between planning via MC-LAVE and supervised learning of self-generated language actions . Experimental results demonstrate that MC-LAVE-RL achieves new high scores in various IF games provided in the Jericho framework ( Hausknecht et al. , 2020 ) , showing the effectiveness of language-grounded MC-LAVE planning . 2 BACKGROUND . 2.1 INTERACTIVE FICTION GAME . Interactive Fiction ( IF ) games are fully text-based environments where the observation and the action spaces are defined as natural language . The game-playing agent observes textual descriptions of the world , selects a language-based action , and receives the associated reward . IF games can be modeled as a special case of partially observable Markov decision processes ( POMDPs ) defined by tuple 〈S , A , Ω , T , O , R , γ〉 , where S is the set of environment states s , A is the set of language actions a , Ω is the set of text observations o , T ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) is the transition function , R ( s , a ) is the reward function for taking action a in state s , O ( s ) = o is the deterministic observation function in state s , and γ ∈ ( 0 , 1 ) is the discount factor . The history at time step t , ht = { o0 , a0 , . . . , ot−1 , at−1 , ot } , is a sequence of observations and actions . The goal is to find an optimal policy π∗ that maximizes the expected cumulative rewards , i.e . π∗ = arg maxπ Eπ [ ∑∞ t=0 γ tR ( st , at ) ] . We use the same definition of observation and action space as Hausknecht et al . ( 2020 ) ; Ammanabrolu & Hausknecht ( 2020 ) ; Côté et al . ( 2018 ) , i.e . An observation is defined by ot = ( otdesc , otgame , otinv , at−1 ) where otdesc is the textual description of the current location of the agent , otgame is the simulator response to the previous action taken by the agent , otinv is the information of agent ’ s inventory , and at−1 is the previous action taken by the agent . An action is denoted by a sequence of words at = ( a1t , a 2 t , . . . , a |at| t ) . Finally , we denote Avalid ( ot ) ⊆ A as the set of valid actions for the observation ot , which is provided by the Jericho environment interface . Figure 1 shows an example of observation and action in ZORK1 , one of the representative IF games . 2.2 CHALLENGES IN INTERACTIVE FICTION GAME . IF games pose important challenges for reinforcement learning agents , requiring natural language understanding , commonsense reasoning , and non-myopic language-grounded planning ability in combinatorial search space of language actions ( Hausknecht et al. , 2020 ) . More concretely , consider the particular game state of ZORK1 described in Figure 1 . In this example situation , a human player would be naturally capable of performing strategic planning via language understanding and commonsense reasoning : ( 1 ) the ‘ closed trap door ’ will have to be opened and be explored to proceed with the game , ( 2 ) however , acquiring the ‘ lantern ’ should precede entering the trap door since the cellar , which is expected to exist below the trap door , could be likely pitch-dark . Without such language-grounded reasoning and planning , the agent may need to try out every actions uniformly , most of them making it vulnerable to being eaten by a monster in the cellar who always appears when there is no light source . As a result , any agent that lacks the ability of long-term planning with language reasoning is prone to be stuck at a suboptimal policy , which enters the cellar to obtain an immediate reward and does nothing further to avoid encountering the monster that kills the agent immediately . To overcome the bottleneck , the agent should be able to infer that ‘ take lantern ’ is worthy enough to be chosen in preference to other actions even though it gives no immediate reward . In this work , we will precisely address such failures of myopic planning that arise from the lack of language-grounded exploration strategy . To this end , we first propose MCTS-based long-term planning for nonmyopic-planning . Then we further improve the search efficiency of MCTS-based planning by incorporating our novel language-grounded exploration strategy based on Language Action Value Estimates , which will be detailed in the subsequent section . 2.3 MONTE-CARLO TREE SEARCH . Monte-Carlo Tree Search ( MCTS ) ( Kocsis & Szepesvári , 2006 ; Coulom , 2006 ; Browne et al. , 2012 ) is a generic online planning algorithm that combines random sampling and tree search , which has become the de-facto standard method for large sequential decision-making problems . Starting from an empty tree , MCTS repeatedly performs the phases of selection , expansion , rollout , and backpropagation to evaluate the nodes of the search tree with increased accuracy . UCT ( Kocsis & Szepesvári , 2006 ) is the standard MCTS method , which adopts UCB ( Auer et al. , 2002 ) as an action selection rule at each internal node of the search tree : arg max a [ Q ( h , a ) + c √ logN ( h ) N ( h , a ) ] ( 1 ) where h is the history of past observations and actions , Q ( h , a ) is the average value of sampled returns with taking action a in h , N ( h ) is the number of simulations performed in h , N ( h , a ) is the number of times action a is selected in h , and c is a constant that balances exploration and exploitation . However , UCT suffers from severe search inefficiency in the problems with a large action space such as IF games , since it requires to take every action at least once and relies only on uncertainty-driven ( or visit-count-based ) exploration . PUCT ( Silver et al. , 2017 ; 2018 ) partially addresses the challenges of IF games by adopting PUCB ( Rosin , 2011 ) , which involves a prior action distribution π ( ·|h ) and eliminates the need for choosing every action at least once . The action selection rule is given by : arg max a [ Q ( h , a ) + cPUCTπ ( a|h ) √ N ( h ) N ( h , a ) + 1 ] ( 2 ) The prior distribution π ( ·|h ) in Eq . ( 2 ) can be trained by behavioral cloning of ( hroot , a∗root ) samples obtained by the result of tree search in previous time steps ( Anthony et al. , 2017 ; Silver et al. , 2018 ) . However , this procedure not only discards other search information such as abundant ( ht , at , rt , ht+1 ) samples obtained during the search but also hardly encourages information sharing across the semantically similar actions . Therefore , PUCT is still limited in search efficiency for the tasks with language action space , raising the need for a language-driven exploration strategy . 3 MONTE-CARLO PLANNING WITH LANGUAGE ACTION VALUE ESTIMATES . Language-grounded planning is essential to address the challenges of IF games . To this end , we need a mechanism to incorporate language understanding and commonsense reasoning into the ex- ploration strategy of search . In this section , we introduce Monte-Carlo planning with Language Action Value Estimates ( MC-LAVE ) , a novel planning algorithm that combines MCTS with languagedriven exploration . MC-LAVE does not rely on any handcrafted heuristic evaluation but starts tabula rasa , based only upon pre-trained word embeddings that can well define the semantic similarities between two language actions . In addition , it only requires a black-box simulator ( r , s′ , o′ ) ∼ G ( s , a ) , which yields ( o , a , r , o′ ) samples at each simulator query .

Summary: This paper introduces Monte-Carlo planning with language action value estimates to guide exploration. The method builds on top of MCTS w/ PUCT, where a policy distribution over actions is introduced to estimate Q for actions not seen during sampling. The modification proposed here is an additional term to the Q estimate, which is a weighted average of Q values of similar actions, where similarity is computed using word embeddings of words that make up an action. The authors show gains over MCTS  w/ PUCT on 8/9 real text games.