On Effective Parallelization of Monte Carlo Tree Search

Abstract: Despite its groundbreaking success in Go and computer games, Monte Carlo Tree Search (MCTS) is computationally expensive as it requires a substantial number of rollouts to construct the search tree, which calls for effective parallelization. However, how to design effective parallel MCTS algorithms has not been systematically studied and remains poorly understood. In this paper, we seek to lay its first theoretical foundation, by examining the potential performance loss caused by parallelization when achieving a desired speedup. In particular, we discover two necessary conditions for achieving a desirable parallelization performance and highlight two of their practical benefits. First, by examining whether existing parallel MCTS algorithms satisfy these conditions, we identify key design principles that should be inherited by future algorithms. Specifically, we find that, although no existing algorithm satisfies both conditions in general (i.e., they are still far from optimal), pre-adjusting the nodes visit counts (used in VL-UCT (Segal, 2010) and WU-UCT (Liu et al., 2020)) is essential. In particular, WU-UCT pre-adjusts visit counts in a way that always satisfies one necessary condition while its other designs meet the other condition when the maximum tree depth is 2. We theoretically establish that they together facilitate O(ln⁡n+M/ln⁡n) cumulative regret under the depth-2 setting, where n is the number of rollouts and M is the number of workers. A regret of this form is highly desirable, as compared to O(ln⁡n) regret incurred by a sequential counterpart, its excess part approaches zero as n increases. Second, and more importantly, we demonstrate how the proposed necessary conditions can be adopted to design more effective parallel MCTS algorithms. To illustrate this, we propose a new parallel MCTS algorithm, called BU-UCT, by following our theoretical guidelines. The newly proposed algorithm, albeit preliminary, out-performs four competitive baselines on 11 out of 15 Atari games. We hope our theoretical results could inspire future work of more effective parallel MCTS.

Reviews and author responses:

Thank you for your constructive feedback.

- Virtual loss: 
  - We would like to clarify that we have considered the AlphaGo [1] version of virtual loss that adjusts both the value and the visit count. Specifically, in Section 4.3 we wrote “... or use virtual loss (Segal, 2010) to penalize action value and visit counts of nodes with high incomplete visit count (e.g., VL-UCT) ...”; also, in Appendix A we categorized virtual loss algorithms (i.e., VL-UCT) into two types: VL-UCT with hard penalty and VL-UCT with soft penalty. The soft penalty version adjusts both action values (i.e., Q) and visit counts (i.e., N) in the exact same way that AlphaGo did (see the third paragraph of Appendix A). In the revised manuscript, we further clarify that VL-UCT adjusts both the action value and the visit count. Related revisions are summarized below:

     - Abstract: We clarified that both VL-UCT and WU-UCT adjusts the visit count: “... pre-adjusting the nodes visit count (used in VL-UCT (Segal, 2010) and WU-UCT (Liu et al., 2020)) is essential …..”
     - Introduction, paragraph 4: “... for example pre-adjusting the node visit counts, which is used in VL-UCT and WU-UCT …..”
     - Figure 1: We modified the VL-UCT figure to better reflect the fact that VL-UCT adjusts both the action value and the visit count.

  - Additionally, we want to clarify that virtual loss is not adjusting action values and visit counts in the same manner as WU-UCT or BU-UCT. Specifically, in the AlphaGo paper, they used $n_{vl}$ to pre-adjust both the value and the visit count:

     $N_{r} (s_{t}, a_{t})  \leftarrow N_{r} (s_{t}, a_{t}) - n_{vl} + 1$;

     $W_{r} (s_{t}, a_{t}) \leftarrow W_{r} (s_{t}, a_{t}) + n_{vl} + z_{t}$ (here $z_{t}$ is the simulation return)

     (quoted from [1]: page 7, right column, line 6 of the third paragraph)

     Action value is then computed by $Q = W_{r} / N_{r}$. When, n_{vl} is set to 1 to match the adjusted visit counts in WU-UCT, its action values are changed compared to not adding virtual loss (i.e., n_{vl} = 0). In contrast, WU-UCT uses the incomplete visit count statistics (i.e., $O$) to only adjust the visit count but not the action value, which is essentially equivalent to including a dynamic virtual action value that is always equal to the current action value estimate. Due to this difference, the positive result of WU-UCT given in Theorem 2 does not apply to VL-UCT, no matter how its hyperparameters are set. We have added a discussion about the differences between VL-UCT and WU-UCT in Appendix A.1. 

- Limitations of Theorem 2: 
  - The big picture of our theoretical results is best viewed when combining Theorems 1 and 2 together. Specifically, as shown by Theorem 1, none of the existing parallel MCTS algorithms satisfy both necessary conditions in the general MCTS setting (i.e., maximum tree depth > 2). Hence none of them have vanishing excess regret when considering the general MCTS setup. To further investigate the tightness of the necessary conditions (i.e., whether an algorithm can enjoy superior theoretical performance if it satisfies the necessary conditions), we examine the performance of WU-UCT in the depth-2 tree search case (i.e., Theorem 2) since this is the only setup where both necessary conditions are satisfied. We have revised Sections 3 and 4.3 to better position Theorem 2 in the big picture: we emphasized that WU-UCT will not have vanishing excess regret when the maximum tree depth > 2 since the first necessary condition is not satisfied in this setting, and the main goal of Theorem 2 is to examine “whether satisfying both necessary conditions (in the depth-2 setup) offer noticeable gain in WU-UCT’s performance from a theoretical perspective”.

  - Additionally, the main purpose of analyzing the depth-2 setting for WU-UCT is to identify its performance boundary. Specifically, since WU-UCT does not satisfy both necessary conditions when the maximum depth of the search tree is greater than 2, it will not have vanishing excess regret anymore. For this reason, it is not necessary to further derive the regret bound for WU-UCT in the general MCTS setting. Rather, a more meaningful effort would be to develop algorithms that better satisfy the suggested necessary conditions.

[1] Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." nature 529.7587 (2016): 484-489.

Thank you for your constructive feedback.

- MDP definition (“the discount factor is not part of the MDP”): We have fixed the definition in our revised paper based on your suggestion. 

- Simple regret vs. cumulative regret: Thank you for suggesting the related work on simple regret vs. cumulative regret in the context of MCTS. We found the papers inspiring and helpful in justifying our choice of using cumulative regret (both types of regret are important for MCTS algorithms, though in different ways). In the revised paper, We have also added citations to these papers in the related work section. Besides, we have also added the following discussion in the last paragraph of Section 4.1:

 > “... In particular, Tolpin & Shimony (2012) highlighted this contradiction in MCTS and proposed a simple yet effective solution — minimizing simple regret when selecting children of the root node while minimizing cumulative regret at non-root nodes.  ……  As argued in these works (e.g., Hay et al. (2014); Tolpin & Shimony (2012)), both types of regret are both useful metrics that have a strong correlation with MCTS algorithms’ performance, hence both are worth studying. …”

- Limitations of the theoretical analysis: The analysis of the depth-2 setting for WU-UCT in Theorem 2 should be viewed together with Theorem 1, and it is used to identify the performance boundary of WU-UCT. Specifically, we can conclude from Theorem 1 that WU-UCT suffers from non-vanishing excess regret when the maximum tree depth is greater than 2 (since WU-UCT does not satisfy the first necessary condition in such cases). On the other hand, Theorem 2 demonstrates that WU-UCT enjoys vanishing excess regret in the depth-2 setting. In contrast, none of the other algorithms has vanishing excess regret since they violate both necessary conditions even in this simplified setting. Putting them together, we conclude that (i) WU-UCT enjoys better theoretical properties compared to other existing algorithms, and (ii) WU-UCT is still far from being an optimal parallel MCTS algorithm for depth>2. We have revised Sections 3 and 4.3 to better position Theorem 2 in the big picture: we emphasized that WU-UCT will not have vanishing excess regret when the maximum tree depth > 2 since the first necessary condition is not satisfied in this setting, and the main goal of Theorem 2 is to examine “whether satisfying both necessary conditions (in the depth-2 setup) offer noticeable gain in WU-UCT’s performance from a theoretical perspective”.

- Empirical evaluation: As pointed out by Reviewer #3, we did not choose our setting in any way deliberately biased. Instead, we follow the exact same setting as the one used in the WU-UCT paper [1]. Specifically, we evaluated all algorithms on exactly the same 15 Atari games used in the WU-UCT paper and did not cherry-pick any specific domains out of them (this is clarified in the experiment setup paragraph of the revised paper). In this regard, these tasks do not favor our proposed BU-UCT algorithm in any way. 

[1] Liu, Anji, et al. "Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search." ICLR 2020.


- Theorem 1 seems trivial: At a high-level, we agree that any parallel MCTS algorithm should make its behavior as close to a virtual sequential MCTS as possible in order to close the performance gap. However, it is highly non-trivial in designing a parallel MCTS algorithm that can approach this insight. Our theoretical analysis aims to give a preliminary answer to this question by identifying workable conditions in practice. Leveraging our theoretical results, we developed an explicitly accessible quantity --- action value gap, that can be used to design parallel MCTS algorithms towards this intuition. Specifically, based on our theory, we can design better parallel MCTS algorithms by minimizing a certain surrogate gap that approximates the action value gap.

Thank you for your constructive feedback. 

- Introduction to previous work: We decided to use the dedicated related work section as a recap of all the relevant works, which put various lines of related research in the same picture. We hope this can help readers to better understand the relation between this paper and previous works.

- References to proofs: Thanks for the suggestion. Under each theorem in the revised manuscript,  we have added the reference to the Appendix that includes the corresponding proof. 

- The proposed new algorithm: We thank the reviewer for the appreciation of our theoretical work in inspiring new algorithms. We do not provide theoretical analysis for BU-UCT because we use BU-UCT as an example to show: (i) how to use our theory to design better algorithms, and (ii) that minimizing the action value gap (or the surrogate gap) can indeed lead to better algorithms. We believe BU-UCT is still far from optimal and hope it can encourage more work based on our theory.

- “aggregation” in BU-UCT: Aggregation is proposed to encourage exploration in the “earlier stage” of a node $s$, where “earlier stage” is defined to be the period when node $s$ has at least one child node without returned simulation. When a node $s$ is in its “earlier stages”, we aggregate the simulation returns originated from each of its child nodes into one single simulation return (i.e., we only count it as one simulation and increment N only by 1). When all the child nodes of $s$ have at least 1 simulation return, we set back to the normal backpropagation strategy. To clarify the details for “aggregation”, we have revised Section 5, paragraph “Algorithm Details” of the paper to include more explanations, and we also add a detailed Algorithm Table in Appendix G for further reference.

- Empirical result: As noted by the reviewer,  our experiments are not biased towards the BU-UCT. In particular, we followed the exact same experiment setup as the WU-UCT paper and evaluated the algorithms on the same 15 Atari games therein. Therefore, we believe the results are not in favor of BU-UCT. 

Thank you for your constructive feedback. 

- General algorithm framework: As referred to in the first paragraph of Section 4.2, Appendix B.3 includes detailed demonstrations on how LeafP, RootP, TreeP, VL-UCT, and WU-UCT can be viewed as special cases of the general parallel MCTS framework.

The paper discusses the theory behind parallelization of Monte Carlo Tree Search, and establishes some necessary conditions for excess regret to vanish asymptotically. The paper also provides a limited theoretical analysis of regret of WU-UCT, an earlier introduced parallel MTCS variant and the current state-of-the art, for a settings of tree depth bounded to 2 (that is, essentially flat MBA). A different algorithm, BU-UCT, is introduced in the paper which satisfies, like WU-UCT, the necessary condition. The algorithm builds on intuition of WU-UCT but also attempts to decrease the action value gap.  The algorithm is evaluated on 15 Atari games, showing performance comparable or exceeding that of WU-UCT and other parallel MCTS variants.

I find the paper an interesting and easy to follow. Parallel MTCS (UCT) variants are indeed not analysed theoretically well enough (and in fact sequential UCT analysis is not flawless either) and any contribution in this direction is very welcome. This paper spots the importance of proper estimation of visit count and value gap (theorem 1). My main concerns with the paper are:

1) somewhat imprecise definitions
2)  superficial discussion of cumulative vs simple regret, with citations missing
3) the theoretical analysis of WU-UCT, limited to depth 2 and nontrivial to extend to unlimited depth, is a minor contribution since behavior of bandit algorithms in trees can be very different than in this special case
4)  limited empirical evaluation, on one domain and with settings to chosen for that particular domain.

In detail,

* the paper defines MDP as a 5-tuple, including the discount factor. The discount factor is not a part of MDP, it is a trick used under certain conditions to analyse and search for policies in MDPs. In game settings in particular, the discounted MDP is often *not used* because the reward of individual actions is zero, and the value of a rollout is determined by the final action (win/loss and by what score) only.

* the paper defines the cumulative regret (7) in a way that can be understood by someone who knows what cumulative regret is, but the same definition can be used for simple regret (!) with the same formula but different discussion. Bibliographic references cite papers on flat bandits rather than trees, and it would be hard to connect cumulative regret for MBA to the formula provided. Simple vs cumulative regret  was analysed  theoretically quite well (https://arxiv.org/abs/1207.5536 AAAI, https://arxiv.org/abs/1408.2048 UAI) and I believe these works should be cited, and the choice for cumulative regret justified in their light.

* the theoretical analysis for depth tree 2 is not obvious to extend to a more general settings, and thus its importance is questionable. 

* BU-UCT introduced to highlight importance of design based on the theoretical analysis in the paper, analysed on a single domain with conditions tuned for that domain compared to algorithms chosen by paper authors. Performance numbers in this setting indicate that the domain for empirical evaluation is chosen well in favor of the proposed algorithm, but not necessarily that the algorithm is better than others. There are may be settings in which BU-UCT does not perform as well. I would prefer to either see insights (detailed analysis) why other algorithms failed or underpeformed when BU-UCT succeeded on selected problem, or an empirical evaluation which gives the impression of being unbiased. 

This paper builds on WU-UCT and further explores effective parallelization of MCTS. It introduces two necessary conditions (Q, N) that are further used to motivate BU-UCT (and extension of WU-UCT). These conditions are then used for theoretical analysis in simplified settings (tree of depth 2). Furthermore, the paper includes interesting analysis, motivation and results on Atari games.

Strong points:

- Previous work is well cited (though I think in some cases misinterpreted, see the weak points).

- The figure 2b is interesting, and nicely fits the presented motivation

- BU-UCT is well motivated

- I enjoyed the empirical analysis on Atari games.


Weak points:

- The paper (as well as the previous work of WU-UCT) claims that the virtual loss used in ALphaGo [Mastering
the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.] modifies value, while the innovation here is increasing the visit counts. This is not true, virtual loss (as used an described in AlphaGo and followup papers) modifies visit counts in the very same way WU-UCT and BU-UCT does!

- The theoretical framework / claims is very limited, and follows easily from the bandit settings. It has limited connections to the MCTS settings - I appreciate though that the authors are very open about those limitations.

-  The necessary conditions really just say "one needs to have the same/similar visit counts (N) and value (Q) as if one ran sequential MCTS".

Recommendation:
I think this is an ok paper - "weak accept". While the theory is not amazing, the motivations are solid and this paper brings some interesting insights. 

Additional feedback:
Please see the virtual loss comment and fix the corresponding text and description, as it not correctly represents the previous work.

The paper looks at the issue of parallelising Monte Carlo Tree Search (MCTS). While existing techniques for doing this are well known and have been studied in practice, this paper makes theoretical contributions to understanding what properties a parallelisation scheme should have in order to minimise the performance losses inherent in parallelisation. Based on this, they propose a new parallelisation method and provide results showing it outperforming existing techniques.

The paper provides a sensible background to previous approaches. While it is rather brief due to the paper size, the important previous work is described at an understandable level. The related work section at the end of the paper is rather odd. Much of the research mentioned in here has already been mentioned much earlier in the paper - often in greater depth. It would be better to merge anything that only appears in this section into the introduction or preliminary material.

The presentation of the theoretical results is handled well. The results are first stated and explained in clear language with intuitive explanation of why they should help the search. This is later formalised in terms of regret. The main theoretical result in the paper have proofs in the supplemental material but this is not stated in the main paper - e.g. a reference to appendix C.1 is needed just above or below Theorem 1. The necessary conditions on Q appears hard to guarantee in practice (though future work might provide bounds to it or other approximations).

The new algorithm proposed is inspired by the theoretical results developed. That is to say, one of the necessary conditions appears essentially impossible for an algorithm to meet in practice so the technique derived aims to reduce the amount by which it is violated. As a result, the novel algorithm does not have any theoretically guaranteed behaviour, but the presented results indicate that it works well in practice.

The description of the "aggregation" in the algorithm is rather weak. What exactly is aggregated and when is it backpropagated? i.e. are statistics gathered and then the average only backpropagated when N have been seen (and if so what is N) or does aggregation only happen if several workers are currently exploring the same node? There are several ways that multiple scores could be aggregated and further description of how this works is definitely needed to make the algorithm understandable.

As the paper acknowledges the results are tentative rather than comprehensive. The setting doesn't appear in any way deliberately biased, but there is only a single setting of the algorithm considered. Given that the paper also presents both theoretical results and a novel technique, a slightly weak results section can probably be accepted.

The authors consider a general family of parallel MCTS algorithms, and  derive some properties which algorithms in this family must satisfy in order to converge to zero excess regret on any MDP. They then analyze various parallel MCTS algorithms against these properties and propose a new algorithm (BU-UCT) which satisfies them. Empirical results are given on Atari, showing BU-UCT outperforming baselines.

I was unable to verify the key claim that all of the prior parallel MCTS algorithms are indeed members of the family of algorithms considered. 

Meta-review: As the reviewer confidence of the reviews of 2.8 or lower, I made a full and detailed pass of submission as it was requested by the PCs. 

The paper studies how to parallelization of MCTS affects its performance and provides the analysis of the excess regret - how much  "error" we incur by  parallelizing as opposed to single-threaded execution.  The submission however does not give convincing arguments on why existing parallel methods perform empirically under their sequential counterparts. This would be particularly useful for the settings described in the submission, which are only resembling the actual parallel solutions. 

Furthermore, the paper analysis of a "cumulative regret" in which it counts the errors made for the exploration - but the setting here is a MCTS (search, planning), when we access an oracle (a model of the environment) and we should be rather interested in the sample complexity, studied for example in https://www.cis.upenn.edu/~mkearns/papers/sparsesampling-journal.pdf that gave early results and many follow ups until OR  we should study how good the final policy it --- and in your parallel setting (which is indeed very timely), how small the excess error of this final policy is. [Or you only focus on a single action recommended by MCTS and you study some pure-exploration measure: best-arm identification, simple regret, \eps-BAI, ... ]

From the theoretical side modeling practice,  I don't see the theoretical framework provided to be suited with practical considerations that an MCTS is facing.  Specifically the need of condition 1 for the consequence of Theorem 1 to hold is very worrying.  Briefly it states that if  the value and the counts are not the same as reference method, then the results obtained from the that parallel executions would not match the results reference method.  While this true, it simply tells us that it would be good if the parallel threads came up with the same value as the reference method - the main issue that this does not tell the practitioner how to assure the condition states in Eq 8. It would be much more interesting to characterize that, i.e., under which conditions (in particular, that the practitioner can influence OR  at least verify prior to the execution) is the property stated in the Eq. 8 satisfied. 

Theorem 2 bring additional concerns. A book of Munos on MCTS (https://hal.archives-ouvertes.fr/hal-00747575v4/document) has Section 2.3 devoted why vanilla UCT has poor performance. There is work by  Kocsis and Szepesvári studying under which conditions UCT can have favorable regret -  there is at least a discussion missing why Theorem 2 would be able to bypass known hardness for UCT for higher depths, and if not I question its utility. Finally, from finite time result excess I would expect more finite time lessons learned beyond "regret term that converges to zero as n increases". 


Decision: withdrawn-rejected-submissions