A⋆MCTS: SEARCH WITH THEORETICAL GUARANTEE USING POLICY AND VALUE FUNCTIONS

1 INTRODUCTION . Monte Carlo Tree Search ( MCTS ) is an online heuristic search algorithm commonly used to find optimal policies for problems in reinforcement learning . It is a vital component of recent breakthroughs for AI in game play , including AlphaGo and AlphaZero ( Silver et al. , 2016 ; 2017 ) . An interesting difference between the MCTS used in AlphaGo/AlphaZero and traditional MCTS is the adoption of neural networks to predict the value of the current state ( value network ) , and to prioritize the next action to search ( policy network ) . The value network replaces random rollouts used to evaluate a non-terminal state , which saves both online computation time and substantially reduces the variance of the estimate . The policy network ( coupled with PUCT ( Rosin , 2011 ) ) prioritizes promising actions for more fruitful exploration . Both networks are trained on pre-existing datasets ( either from human replays or self-play ) to incorporate prior knowledge . This strategy has been applied in other domains including Neural Architecture Search ( Wang et al. , 2018 ; Wistuba , 2017 ; Negrinho & Gordon , 2017 ; Wang et al. , 2019b ) . Despite the empirical success and recent popularity , theoretical foundations are lacking . ( Kocsis & Szepesvári , 2006 ; Coquelin & Munos , 2007 ) analyze MCTS with rollouts using a multi-armed bandit framework and gives asymptotic results without finite sample complexity bounds . Powering MCTS with value/policy neural networks instead of rollouts is even more poorly understood . To better understand how to efficiently learn optimal policies in decision trees using value and policy networks , we propose a novel algorithm , A ? MCTS , which is a combination of A ? ( Delling et al. , 2009 ; Kanoulas et al. , 2006 ) and MCTS . Like A ? , it uses a priority queue to store all leaves currently being explored and picks the most optimistic one to expand , according to an upper confidence bound heuristic function . Like MCTS , it uses policy and value networks to prioritize the next state to be explored . To facilitate the analysis , the policy and value networks are treated as black-box functions and a statistical model is built the accuracy of the predictions . In particular , we assume the standard deviation of the noisy estimates of intermediate state values decays ( either polynomially or exponentially ) as a function of the depth of the tree , so that the values of near-terminal states are estimated to a greater degree of accuracy . With this statistical model , we provide theoretical guarantees for the sample complexity ( expected number of state expansions / rollouts ) to determine optimal actions . We apply our theoretical framework to simple models for rewards and show that our algorithm enjoys a sample complexity that is polynomial in depth D. Our experiments validate the theoretical analysis and demonstrate the effectiveness of A ? MCTS over benchmark MCTS algorithms with value and policy networks . To our knowledge , our work is the first that studies tree search for optimal actions in the presence of pre-trained value and policy networks , and we hope that it inspires further progress on this important problem . 2 BACKGROUND AND RELATED WORK . Monte Carlo Tree Search ( Browne et al. , 2012 ) is an algorithm that tries to find optimal decisions in Markov Decision Processes by taking random samples in the decision space and building a tree based on the results . In each iteration , the agent traverses the tree by acting according to the tree policy ( which usually involves some exploration ) for the set of known states and then determines the value of an unexpanded state by performing random rollouts . MCTS is commonly used to search for a one-step optimal actions from the root . The agent then takes this action , observes the result , transitions to a new state , and repeats MCTS starting from this new state , with the benefit of the additional observation . The main theoretical question of interest in MCTS is how quickly and efficiently one can find this optimal one-step action , in terms of the number of random rollouts , or the number of nodes in the tree that should be expanded . On the theoretical side , there is a small body of literature that addresses the theoretical foundations of MCTS with random rollouts ( Kocsis & Szepesvári ( 2006 ) , Veness et al . ( 2011 ) ) . The algorithm , called UCT ( UCB applied to trees ) models the search as a complex multi-armed bandit problem where the rewards are allowed to drift over time . The analysis shows that after a period of time T , only the optimal branch will be followed . Veness et al . ( 2011 ) gives various ways of improving UCT with variance reduction techniques . Later followup work in Coquelin & Munos ( 2007 ) highlights some hard instances of UCT and offers some alternatives , including a modified UCT ( with a different exploration bonus ) , Flat UCB , and BAST ( smooth tree models ) . On the experimental side , MCTS was popularized by the story of AlphaGo ( Silver et al . ( 2016 ) ) , which used MCTS using a weighted combination of the value network ’ s output and random rollouts to defeat the leading champion in Go . Follow-up work in AlphaZero ( Silver et al . ( 2017 ) ) used MCTS with only policy and value neural networks , without random rollouts and reports faster training with better performance . However , there is no theoretical foundation for the search algorithm that AlphaZero uses . A ? ( Nosrati et al. , 2012 ) uses a priority queue to keep track of lowest cost actions . A ? combined with hand-designed heuristics , has been used in many path finding and routing problems ( AlShawi et al. , 2012 ; Rana & Zaveri , 2011 ; Ghaffari , 2014 ) . Recent work ( Wang et al. , 2019a ; Chen & Wei , 2011 ; Sigurdson & Bulitko , 2017 ) trains neural-based heuristic functions for A ? for specific problems . However , to the best of our knowledge , we are not aware of any prior work that performs theoretical analysis of such heuristic based A ? algorithms . 3 PRELIMINARIES AND SUMMARY OF CONTRIBUTIONS . In this work we consider a deterministic , finite D-horizon Markov decision process ( MDP ) with no intermediate rewards . With the set of states denoted by S , there is one deterministic root initial state s0 and the process terminates at horizon D with a termination reward that depends on the end state . We assume that there are exactly K possible actions at every intermediate state s and for simplicity that the process does not backtrack , i.e. , each action leads to a new and unique state . This is a Markov decision tree . The set of leaf states of the Markov decision tree are denoted as L. We shall denote an arbitrary state at depth d as sd and also a leaf state as l ∈ L. There is a value function V : S → R that takes as input a state s and returns the value of s , defined in the following way . If s ∈ L , then Vs is the termination reward of that particular leaf . For an intermediate node s , Vs = maxl∈L ( s ) Vl , where L ( s ) is the set of leaf nodes that share s as an ancestor , i.e. , the set of leaf nodes in the subtree rooted at s. We assume that the highest value V ∗ ≡ maxl∈L Vl = maxs∈S Vs is achieved at a unique leaf node l∗ = arg maxl∈L Vl . In this work , we focus on the problem of efficiently computing V ∗ exactly or approximately with a controlled accuracy without direct access to the value function V : S → R , as a stepping stone to finding the optimal one-step action from s0 . Instead , there is a pre-trained black-box resource , value network U : S → R that takes as input a state and returns a noisy value estimate , i.e. , for a state sd at depth d , Usd ≡ Vsd +Xsd , where Xsd are i.i.d . copies of a random variable Xd that depends only on the depth d. We assume that at depth D , which is the level with all the leaf nodes , Ul = Vl ( i.e. , the value network returns the ground truth values without any noise ) . The efficiency is defined in terms of sample complexity , which is the number of queries made to the value network for the value of a state . In the worst case , all leaves need to be queried and the sample complexity is KD . Another type of available pre-trained black box resource is a policy network . Let C ( s ) = { r1 , . . . rK } denote the set of K children of state s , each corresponding to the transition state of one of the K actions { a1 , . . . aK } . The policy network outputs a probability estimate of the form pri = e Uπri∑K k=1 e Uπrk , where each Uπri is a perturbed estimate of Vri . Here the superscript π indicates that this is the approximation from the policy network . We assume that if the child ri is at depth d then the estimate Uπri = Vri + X π ri , where X π ri are i.i.d . copies of a random variable X π d that depends only on the depth d. We assume that the distribution of Xd and Xπd are known a priori for each depth d. Further , the variances of Xd and Xπd decrease sufficiently rapidly as the depth d increases ( i.e. , as one traverses deeper in the tree , the noise level is smaller ) . This assumption is motivated by basic approximate dynamic programming principles , which is that near-terminal states require a smaller sample complexity to estimate well , since the backwards induction will propagate this noise to earlier states , making it harder to estimate the values of initial states well . 3.1 MAIN CONTRIBUTIONS . We develop and analyze ( giving specific sample complexity bounds ) algorithms to estimate V ∗ exactly and approximately using value and policy networks . Since our value and policy networks are trained functions that give fixed , deterministic noisy outputs for each state ( no matter how many times one calls the function on a particular state ) , where the noise variance is smaller at deeper levels of the search tree , we use our value and policy network estimates as signals for which are the promising states to expand . We rank the states based on our optimistic estimates Usd + cd in a priority queue , and we expand ( query the value network for the value estimate ) the children of the top nodes of the priority queue to get increasingly more accurate value estimates and determine the value of the optimal leaf V ∗ . The main contributions are organized in the following sections : • Section 4 introduces and analyzes A ? MCTS-V ∗ and A ? MCTS-π∗V ∗ that are our core techniques for estimating V ∗ exactly . A ? MCTS-V ∗ uses only a value network to determine V ∗ exactly and outputs the optimal one-step from s0 that leads to l∗ . We give an expected sample complexity in Theorem 1 . A ? MCTS-π∗V ∗ builds onA ? MCTS-V ∗ by using a policy network to further reduce the sample complexity , which we bound in Theorem 2 . • Section 5 extendsA ? MCTS-V ∗ andA ? MCTS-π∗V ∗ to the case when we tolerate δ-additive approximations of V ∗ . • Section 6 introduces some sample distribution models for the value functions and the noise and applies our results to those models . We show that for some reasonable models , our algorithms can achieve sample complexity that is polynomial in depth D. We also provide some experimental comparisons against benchmark MCTS implementations .

Summary: This paper proposes A*MCTS, which combines A* and MCTS with policy and value networks to prioritize the next state to be explored. It further establishes the sample complexity to determine optimal actions. Experimental results validate the theoretical analysis and demonstrate the effectiveness of A*MCTS over benchmark MCTS algorithms with value and policy networks.