Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search

1 INTRODUCTION . Recently , Monte Carlo Tree Search ( MCTS ) algorithms such as UCT ( Kocsis et al. , 2006 ) have achieved great success in solving many challenging artificial intelligence ( AI ) benchmarks , including video games ( Guo et al. , 2016 ) and Go ( Silver et al. , 2016 ) . However , they rely on a large number ( e.g . millions ) of interactions with the environment emulator to construct search trees for decision-making , which leads to high time complexity ( Browne et al. , 2012 ) . For this reason , there has been an increasing demand for parallelizing MCTS over multiple workers . However , parallelizing MCTS without degrading its performance is difficult ( Segal , 2010 ; Mirsoleimani et al. , 2018a ; Chaslot et al. , 2008 ) , mainly due to the fact that each MCTS iteration requires information from all previous iterations to provide effective exploration-exploitation tradeoff . Specifically , parallelizing MCTS would inevitably obscure these crucial information , and we will show in Section 2.2 that this loss of information potentially results in a significant performance drop . The key question is therefore how to acquire and utilize more available information to mitigate the information loss caused by parallelization and help the algorithm to achieve better exploration-exploitation tradeoff . To this end , we propose WU-UCT ( Watch the Unovserved in UCT ) , a novel parallel MCTS algorithm that attains linear speedup with only limited performance loss . This is achieved by a conceptual innovation ( Section 3.1 ) as well as an efficient real system implementation ( Section 3.2 ) . Specifically , the key idea in WU-UCT to overcome the aforementioned challenge is a set of statistics that tracks the number of on-going yet incomplete simulation queries ( named as unobserved samples ) . We combine these newly devised statistics with the original statistics of observed samples to modify UCT ’ s policy in the selection steps in a principled manner , which , as we shall show in Section 4 , effectively retains exploration-exploitation tradeoff during parallelization . Our proposed 1Code is available at https : //github.com/liuanji/WU-UCT . approach has been successfully deployed in a production system for efficiently and accurately estimating the rate at which users pass levels ( termed user pass-rate ) in a mobile game “ Joy City ” , with the purpose of reducing their design cycles . On this benchmark , we show that WU-UCT achieves near-optimal linear speedup and superior performance in predicting user pass-rate ( Section 5.1 ) . We further evaluate WU-UCT on the Atari Game benchmark and compare it to state-of-the-art parallel MCTS algorithms ( Section 5.2 ) , which also demonstrate our superior speedup and performance . 2 ON THE DIFFICULTIES OF PARALLELIZING MCTS . We first introduce the MCTS and the UCT algorithms , along with their difficulties in parallelization . 2.1 MONTE CARLO TREE SEARCH AND UPPER CONFIDENCE BOUND FOR TREES ( UCT ) . We consider the Markov Decision Process ( MDP ) 〈S , A , R , P , γ〉 , where an agent interacts with the environment in order to maximize a long-term cumulative reward . Specifically , an agent at state st ∈ S takes an action at ∈ A according to a policy π , so that the MDP transits to the next state st+1 ∼ P ( st+1|st , at ) and emits a reward R ( st , at ) .2 The objective of the agent is to learn an optimal policy π∗ such that the long-term cumulative reward is maximized : max π Eat∼π , st+1∼P [ ∞∑ t=0 γtR ( st , at ) |s0=s ] , ( 1 ) where s ∈ S denotes the initial state and γ ∈ ( 0 , 1 ] is the discount factor.3 Many reinforcement learning ( RL ) algorithms have been developed to solve the above problem ( Sutton & Barto , 2018 ) , including model-free algorithms ( Mnih et al. , 2013 ; 2016 ; Williams , 1992 ; Konda & Tsitsiklis , 2000 ; Schulman et al. , 2015 ; 2017 ) and model-based algorithms ( Nagabandi et al. , 2018 ; Weber et al. , 2017 ; Bertsekas , 2005 ; Deisenroth & Rasmussen , 2011 ) . Monte Carlo Tree Search ( MCTS ) is a model-based RL algorithm that plans the best action at each time step ( Browne et al. , 2012 ) . Specifically , it uses the MDP model ( or its sampler ) to identify the best action at each time step by constructing a search tree ( Figure 1 ( a ) ) , where each node s represents a visited state , each edge from s denotes an action as that can be taken at that state , and the landing node s′ denotes the state it transits to after taking as . As shown in Figure 1 ( a ) , MCTS repeatedly performs four sequential steps : selection , expansion , simulation and backpropagation . The selection step traverses over the existing search tree until the leaf node ( or other termination conditions are satisfied ) by choosing actions ( edges ) as at each node s according to a tree policy . One widely used node-selection policy is the one used in the Upper Confidence bound for Trees ( UCT ) ( Kocsis et al. , 2006 ) : as = argmax s′∈C ( s ) { Vs′ + β √ 2 logNs Ns′ } , ( 2 ) where C ( s ) denotes the set of all child nodes for s ; the first term Vs′ is an estimate for the long-term cumulative reward that can be received when starting from the state represented by node s′ , and the second term represents the uncertainty ( size of the confidence interval ) of that estimate . The confidence interval is calculated based on the Upper Confidence Bound ( UCB ) ( Auer et al. , 2002 ; Auer , 2002 ) usingNs andNs′ , which denote the number of times that the nodes s and s′ have been visited , respectively . Therefore , the key idea of the UCT policy ( 2 ) is to select the best action according to an optimistic estimation ( i.e. , the upper confidence bound ) of the expected return , which strikes a balance between the exploitation ( first term ) and the exploration ( second term ) with β controlling their tradeoff . Once the selection process reaches a leaf node of the search tree ( or other termination conditions are met ) , we will expand the node according to a prior policy by adding a new child node . Then , in the simulation step , we estimate its value function ( cumulative reward ) V̂s by running the environment simulator with a default ( simulation ) policy . Finally , during backpropagation , we update the statistics Vs and Ns from the leaf node sT to the root node s0 of the selected path by recursively performing the following update ( i.e. , from t = T − 1 to t = 0 ) : Nst ← Nst + 1 , V̂st ← R ( st , at ) + γV̂st+1 , Vst ← ( ( Nst − 1 ) Vst + V̂st ) /Nst , ( 3 ) 2In the context of MCTS , the action space A is assumed to be finite and the transition P is assumed to be deterministic , i.e. , the next state st+1 is determined by the current state st and action at . 3We assume certain regularity conditions hold so that the cumulative reward ∑∞ t=0 γ tR ( st , at ) is always bounded ( Sutton & Barto , 2018 ) . where V̂sT is the simulation return of sT ; at denotes the action selected following ( 2 ) at state st . 2.2 THE INTRINSIC DIFFICULTIES OF PARALLELIZING MCTS . The above discussion implies that the MCTS algorithm is intrinsically sequential : each selection step in a new rollout requires the previous rollouts to complete in order to deliver the updated statistics , Vs and Ns , for the UCT tree policy ( 2 ) . Although this requirement of up-to-date statistics is not mandatory for implementation , it is in practice intensively required to achieve effective explorationexploitation tradeoff ( Auer et al. , 2002 ) . Specifically , up-to-date statistics best help the UCT tree policy to identify and prune non-rewarding branches as well as extensively visiting rewarding paths for additional planning depth . Likewise , to achieve the best possible performance , when multiple workers are used , it is also important to ensure that each worker uses the most recent statistics ( the colored Vs andNs in Figure 1 ( b ) ) in its own selection step . However , this is impossible in parallelizing MCTS based on the following observations . First , the expansion step and the simulation step are generally more time-consuming compared to the other two steps , because they involve a large number of interactions with the environment ( or its simulator ) . Therefore , as exemplified by Figure 1 ( c ) , when a worker C initiates a new selection step , the other workers A and B are most likely still in their simulation or expansion steps . This prevents them from updating the ( global ) statistics for other workers like C , which happens at their respective backpropagation steps . Using outdated statistics ( the gray-colored Vs and Ns ) at different workers could lead to a significant performance loss given a fixed target speedup , due to behaviors like collapes of exploration or exploitation failure , which we shall discuss thoroughly in Section 4 . To give an example , Figure 1 ( c ) illustrates the collapse of exploration , where worker C traverses over the same path as the worker A in its selection step due to the determinism of ( 2 ) . Specifically , if the statistics are unchanged between the moments that worker A and C begin their own selection steps , they will choose the same node according to ( 2 ) , which greatly reduces the diversity of exploration . Therefore , the key question that we want to address in parallelizing MCTS is how to track the correct statistics and modify the UCT policy in a principled manner , with the hope of retaining effective exploration-exploitation tradeoff at different workers . 3 WU-UCT . In this section , we first develop the conceptual idea of our WU-UCT algorithm ( Section 3.1 ) , and then we present a real system implementation using a master-worker architecture ( Section 3.2 ) . 3.1 WATCH THE UNOBSERVED SAMPLES IN UCT TREE POLICY . As we pointed out earlier , the key question we want to address in parallelizing MCTS is how to deliver the most up-to-date statistics { Vs , Ns } to each worker so that they can achieve effective exploration-exploitation tradeoff in its selection step . This is assumed to be the case in the ideal parallelization in Figure 1 ( b ) . Algorithmically , it is equivalent to the sequential MCTS except that the rollouts are performed in parallel by different workers . Unfortunately , in practice , the statistics { Vs , Ns } available to each worker are generally outdated because of the slow and incomplete simulation and expansion steps at the other workers . Specifically , since the estimated value V̂s is unobservable before simulations complete and workers should not wait for the updated statistics to proceed , the ( partial ) loss of statistics { Vs , Ns } is unavoidable . Now the question becomes : is there an alternative way to addressing the issue ? The answer is in the affirmative and is explained below . Aiming at bridging the gap between naive parallelization and the ideal case , we closely examine their difference in terms of the availability of statistics . As illustrated by the colors of the statistics , their only difference in { Vs , Ns } is caused by the on-going simulation process . As suggested by ( 3 ) , although Vs can only be updated after a simulation step is completed , the newest Ns information can actually be available as early as a worker initiates a new rollout . This is the key insight that we leverage to enable effective parallelization in our WU-UCT algorithm . Motivated by this , we introduce another quantity , Os , to count the number of rollouts that have been initiated but not yet completed , which we name as unobserved samples . That is , our new statistics , Os , watch the number of unobserved samples , and are then used to correct the UCT tree policy ( 2 ) into the following form : as = argmax s′∈C ( s ) { Vs′ + β √ 2 log ( Ns +Os ) Ns′ +Os′ } . ( 4 ) The intuition of the above modified node-selection policy is that when there are Os workers simulating ( querying ) node s , the confidence interval at node s will eventually be shrunk after they complete . Therefore , adding Os and Os′ to the exploration term considers such a fact beforehand and let other workers be aware of it . Despite its simple form , ( 4 ) provides a principled way to retain effective exploration-exploitation tradeoff under parallel settings ; it corrects the confidence bound towards better exploration-exploitation tradeoff . As the confidence level is instantly updated ( i.e. , at the beginning of simulation ) , more recent workers are guaranteed to observe additional statistics , which prevent them from extensively querying the same node as well as find better nodes for them to query . For example , when multiple children are in demand for exploration , ( 4 ) allows them to be explored evenly . In contrast , when a node has been sufficiently visited ( i.e. , large Ns and Ns′ ) , adding Os and Os′ from the unobserved samples have little effect on ( 4 ) because the confidence interval is sufficiently shrunk around Vs′ , allowing extensively exploitation of the best-valued child .

Summary: This paper introduces a new algorithm for parallelizing Monte-Carlo Tree Search (MCTS). Specifically, when expanding a new node in the search tree, the algorithm updates the parent nodes’ statistics of the visit counts but not their values; it is only when the expansion and simulation steps are complete that the values are updated as well. This has the effect of shrinking the UCT exploration term, and making other workers less likely to explore that part of the tree even before the simulation is complete. This algorithm is evaluated in two domains, a mobile game called “Joy City” as well as on Atari. The proposed algorithm results in large speedups compared to serial MCTS with seemingly little impact in performance, and also results in higher scores on Atari than existing parallelization methods.