A⋆MCTS: SEARCH WITH THEORETICAL GUARANTEE USING POLICY AND VALUE FUNCTIONS

1 INTRODUCTION . Monte Carlo Tree Search ( MCTS ) is an online heuristic search algorithm commonly used to find optimal policies for problems in reinforcement learning . It is a vital component of recent breakthroughs for AI in game play , including AlphaGo and AlphaZero ( Silver et al. , 2016 ; 2017 ) . An interesting difference between the MCTS used in AlphaGo/AlphaZero and traditional MCTS is the adoption of neural networks to predict the value of the current state ( value network ) , and to prioritize the next action to search ( policy network ) . The value network replaces random rollouts used to evaluate a non-terminal state , which saves both online computation time and substantially reduces the variance of the estimate . The policy network ( coupled with PUCT ( Rosin , 2011 ) ) prioritizes promising actions for more fruitful exploration . Both networks are trained on pre-existing datasets ( either from human replays or self-play ) to incorporate prior knowledge . This strategy has been applied in other domains including Neural Architecture Search ( Wang et al. , 2018 ; Wistuba , 2017 ; Negrinho & Gordon , 2017 ; Wang et al. , 2019b ) . Despite the empirical success and recent popularity , theoretical foundations are lacking . ( Kocsis & Szepesvári , 2006 ; Coquelin & Munos , 2007 ) analyze MCTS with rollouts using a multi-armed bandit framework and gives asymptotic results without finite sample complexity bounds . Powering MCTS with value/policy neural networks instead of rollouts is even more poorly understood . To better understand how to efficiently learn optimal policies in decision trees using value and policy networks , we propose a novel algorithm , A ? MCTS , which is a combination of A ? ( Delling et al. , 2009 ; Kanoulas et al. , 2006 ) and MCTS . Like A ? , it uses a priority queue to store all leaves currently being explored and picks the most optimistic one to expand , according to an upper confidence bound heuristic function . Like MCTS , it uses policy and value networks to prioritize the next state to be explored . To facilitate the analysis , the policy and value networks are treated as black-box functions and a statistical model is built the accuracy of the predictions . In particular , we assume the standard deviation of the noisy estimates of intermediate state values decays ( either polynomially or exponentially ) as a function of the depth of the tree , so that the values of near-terminal states are estimated to a greater degree of accuracy . With this statistical model , we provide theoretical guarantees for the sample complexity ( expected number of state expansions / rollouts ) to determine optimal actions . We apply our theoretical framework to simple models for rewards and show that our algorithm enjoys a sample complexity that is polynomial in depth D. Our experiments validate the theoretical analysis and demonstrate the effectiveness of A ? MCTS over benchmark MCTS algorithms with value and policy networks . To our knowledge , our work is the first that studies tree search for optimal actions in the presence of pre-trained value and policy networks , and we hope that it inspires further progress on this important problem . 2 BACKGROUND AND RELATED WORK . Monte Carlo Tree Search ( Browne et al. , 2012 ) is an algorithm that tries to find optimal decisions in Markov Decision Processes by taking random samples in the decision space and building a tree based on the results . In each iteration , the agent traverses the tree by acting according to the tree policy ( which usually involves some exploration ) for the set of known states and then determines the value of an unexpanded state by performing random rollouts . MCTS is commonly used to search for a one-step optimal actions from the root . The agent then takes this action , observes the result , transitions to a new state , and repeats MCTS starting from this new state , with the benefit of the additional observation . The main theoretical question of interest in MCTS is how quickly and efficiently one can find this optimal one-step action , in terms of the number of random rollouts , or the number of nodes in the tree that should be expanded . On the theoretical side , there is a small body of literature that addresses the theoretical foundations of MCTS with random rollouts ( Kocsis & Szepesvári ( 2006 ) , Veness et al . ( 2011 ) ) . The algorithm , called UCT ( UCB applied to trees ) models the search as a complex multi-armed bandit problem where the rewards are allowed to drift over time . The analysis shows that after a period of time T , only the optimal branch will be followed . Veness et al . ( 2011 ) gives various ways of improving UCT with variance reduction techniques . Later followup work in Coquelin & Munos ( 2007 ) highlights some hard instances of UCT and offers some alternatives , including a modified UCT ( with a different exploration bonus ) , Flat UCB , and BAST ( smooth tree models ) . On the experimental side , MCTS was popularized by the story of AlphaGo ( Silver et al . ( 2016 ) ) , which used MCTS using a weighted combination of the value network ’ s output and random rollouts to defeat the leading champion in Go . Follow-up work in AlphaZero ( Silver et al . ( 2017 ) ) used MCTS with only policy and value neural networks , without random rollouts and reports faster training with better performance . However , there is no theoretical foundation for the search algorithm that AlphaZero uses . A ? ( Nosrati et al. , 2012 ) uses a priority queue to keep track of lowest cost actions . A ? combined with hand-designed heuristics , has been used in many path finding and routing problems ( AlShawi et al. , 2012 ; Rana & Zaveri , 2011 ; Ghaffari , 2014 ) . Recent work ( Wang et al. , 2019a ; Chen & Wei , 2011 ; Sigurdson & Bulitko , 2017 ) trains neural-based heuristic functions for A ? for specific problems . However , to the best of our knowledge , we are not aware of any prior work that performs theoretical analysis of such heuristic based A ? algorithms . 3 PRELIMINARIES AND SUMMARY OF CONTRIBUTIONS . In this work we consider a deterministic , finite D-horizon Markov decision process ( MDP ) with no intermediate rewards . With the set of states denoted by S , there is one deterministic root initial state s0 and the process terminates at horizon D with a termination reward that depends on the end state . We assume that there are exactly K possible actions at every intermediate state s and for simplicity that the process does not backtrack , i.e. , each action leads to a new and unique state . This is a Markov decision tree . The set of leaf states of the Markov decision tree are denoted as L. We shall denote an arbitrary state at depth d as sd and also a leaf state as l ∈ L. There is a value function V : S → R that takes as input a state s and returns the value of s , defined in the following way . If s ∈ L , then Vs is the termination reward of that particular leaf . For an intermediate node s , Vs = maxl∈L ( s ) Vl , where L ( s ) is the set of leaf nodes that share s as an ancestor , i.e. , the set of leaf nodes in the subtree rooted at s. We assume that the highest value V ∗ ≡ maxl∈L Vl = maxs∈S Vs is achieved at a unique leaf node l∗ = arg maxl∈L Vl . In this work , we focus on the problem of efficiently computing V ∗ exactly or approximately with a controlled accuracy without direct access to the value function V : S → R , as a stepping stone to finding the optimal one-step action from s0 . Instead , there is a pre-trained black-box resource , value network U : S → R that takes as input a state and returns a noisy value estimate , i.e. , for a state sd at depth d , Usd ≡ Vsd +Xsd , where Xsd are i.i.d . copies of a random variable Xd that depends only on the depth d. We assume that at depth D , which is the level with all the leaf nodes , Ul = Vl ( i.e. , the value network returns the ground truth values without any noise ) . The efficiency is defined in terms of sample complexity , which is the number of queries made to the value network for the value of a state . In the worst case , all leaves need to be queried and the sample complexity is KD . Another type of available pre-trained black box resource is a policy network . Let C ( s ) = { r1 , . . . rK } denote the set of K children of state s , each corresponding to the transition state of one of the K actions { a1 , . . . aK } . The policy network outputs a probability estimate of the form pri = e Uπri∑K k=1 e Uπrk , where each Uπri is a perturbed estimate of Vri . Here the superscript π indicates that this is the approximation from the policy network . We assume that if the child ri is at depth d then the estimate Uπri = Vri + X π ri , where X π ri are i.i.d . copies of a random variable X π d that depends only on the depth d. We assume that the distribution of Xd and Xπd are known a priori for each depth d. Further , the variances of Xd and Xπd decrease sufficiently rapidly as the depth d increases ( i.e. , as one traverses deeper in the tree , the noise level is smaller ) . This assumption is motivated by basic approximate dynamic programming principles , which is that near-terminal states require a smaller sample complexity to estimate well , since the backwards induction will propagate this noise to earlier states , making it harder to estimate the values of initial states well . 3.1 MAIN CONTRIBUTIONS . We develop and analyze ( giving specific sample complexity bounds ) algorithms to estimate V ∗ exactly and approximately using value and policy networks . Since our value and policy networks are trained functions that give fixed , deterministic noisy outputs for each state ( no matter how many times one calls the function on a particular state ) , where the noise variance is smaller at deeper levels of the search tree , we use our value and policy network estimates as signals for which are the promising states to expand . We rank the states based on our optimistic estimates Usd + cd in a priority queue , and we expand ( query the value network for the value estimate ) the children of the top nodes of the priority queue to get increasingly more accurate value estimates and determine the value of the optimal leaf V ∗ . The main contributions are organized in the following sections : • Section 4 introduces and analyzes A ? MCTS-V ∗ and A ? MCTS-π∗V ∗ that are our core techniques for estimating V ∗ exactly . A ? MCTS-V ∗ uses only a value network to determine V ∗ exactly and outputs the optimal one-step from s0 that leads to l∗ . We give an expected sample complexity in Theorem 1 . A ? MCTS-π∗V ∗ builds onA ? MCTS-V ∗ by using a policy network to further reduce the sample complexity , which we bound in Theorem 2 . • Section 5 extendsA ? MCTS-V ∗ andA ? MCTS-π∗V ∗ to the case when we tolerate δ-additive approximations of V ∗ . • Section 6 introduces some sample distribution models for the value functions and the noise and applies our results to those models . We show that for some reasonable models , our algorithms can achieve sample complexity that is polynomial in depth D. We also provide some experimental comparisons against benchmark MCTS implementations .

Summary: This paper presents the search algorithm A*MCTS to find the optimal policies for problems in Reinforcement Learning. In particular, A*MCTS combines the A* and MCTS algorithms to use the pre-trained value networks for facilitating the exploration and making optimal decisions. A*MCTS refers the value network as a black box and builds a statistical model for the prediction accuracies, which provides theoretical guarantees for the sample complexity. The experiments verify the effectiveness of the proposed A*MCTS.