Title: Monte Carlo Tree Search with Boltzmann Exploration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Monte Carlo Tree Search

Planning

Entropy

Reinforcement Learning
Abstract: Monte-Carlo Tree Search (MCTS) methods, such as Upper Confidence Bound applied to Trees (UCT), are instrumental to automated planning techniques. However, UCT can be slow to explore an optimal action when it initially appears inferior to other actions. Maximum ENtropy Tree-Search (MENTS) incorporates the maximum entropy principle into an MCTS approach, utilising Boltzmann policies to sample actions, naturally encouraging more exploration. In this paper, we highlight a major limitation of MENTS: optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective. We introduce two algorithms, Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS), that address these limitations and preserve the benefits of Boltzmann policies, such as allowing actions to be sampled faster by using the Alias method. Our empirical analysis shows that our algorithms show consistent high performance across several benchmark domains, including the game of Go.

Monte Carlo Tree Search with Boltzmann Exploration

Abstract

Monte-Carlo Tree Search (MCTS) methods, such as Upper Confidence Bound applied to Trees (UCT), are instrumental to automated planning techniques. However, UCT can be slow to explore an optimal action when it initially appears inferior to other actions. Maximum ENtropy Tree-Search (MENTS) incorporates the maximum entropy principle into an MCTS approach, utilising Boltzmann policies to sample actions, naturally encouraging more exploration. In this paper, we highlight a major limitation of MENTS: optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective. We introduce two algorithms, Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS), that address these limitations and preserve the benefits of Boltzmann policies, such as allowing actions to be sampled faster by using the Alias method. Our empirical analysis shows that our algorithms show consistent high performance across several benchmark domains, including the game of Go.

1 Introduction

Planning under uncertainty is a core problem in Artificial Intelligence, commonly modelled as a Markov Decision Process (MDP) or variant thereof. MDPs can be solved using dynamic programming techniques to obtain an optimal policy [3]. However, computing a full optimal policy does not scale to large state-spaces, necessitating the use of heuristic solvers [18, 4] and online, sampling-based, planners based on Monte-Carlo Tree-Search (MCTS), such as the Upper Confidence Bound applied to Trees (UCT) algorithm [22].

The UCT search policy is designed to minimise cumulative regret , so manages a trade-off between exploration and exploitation. To exploit, UCT often selects the same action on successive trials, which can result in it getting stuck in local optima. Conversely, Maximum ENtropy Tree Search (MENTS) places a greater emphasis on exploration by combining MCTS with techniques from maximum entropy policy optimisation [38, 16, 17]. MENTS jointly maximises cumulative rewards and policy entropy, where a temperature parameter controls the weight of the entropy objective. However, MENTS is sensitive to this temperature parameter, and may not converge to the reward maximising policy or require a prohibitively low temperature to do so.

In this work, we consider scenarios where MCTS methods are used with a simulator to plan how an agent should act. We introduce two algorithms for this scenario that address the above limitations. First, we present Boltzmann Tree Search (BTS) which uses a Boltzmann search policy like MENTS, but optimises for reward maximisation only. Secondly, we introduce Decaying ENtropy Tree Search (DENTS), which adds entropy backups to BTS, but is still consistent (i.e. it converges to the reward maximising policy in the limit).

The main contributions of this paper are: (1) Demonstrating that the maximum entropy objective used in MENTS can be misaligned with reward maximisation, thus preventing it from converging to the optimal policy; (2) Introducing two new algorithms, BTS and DENTS, which preserve the benefits

of using Boltzmann search policies while being as simple to implement as UCT and MENTS, but converge to the reward maximising policy; (3) Analysing MENTS, BTS and DENTS through the lens of simple regret to provide theoretical convergence results; (4) Highlighting and demonstrating that the Alias method [35, 34] can be used with stochastic action selection to improve the asymptotic complexity of running a fixed number of trials over existing MCTS algorithms; and (5) Demonstrating the performance improvements of Boltzmann search policies used in BTS and DENTS in benchmark gridworld environments and the game of Go.

2 Background

2.1 Markov Decision Processes

We define a (finite-horizon) MDP as a tuple M = ( S , A , p, R, H ) , where S is the set of states; A is the set of actions; p : S × A × S → [0 , 1] is the transition function where p ( s ′ | s, a, ) is the probability of moving to state s ′ given that action a was taken in state s ; R : S × A → R is the reward function; and H ∈ N is the finite horizon. Let Succ ( s, a ) denote the set of successor states of ( s, a ) , i.e. Succ ( s, a ) = { s ′ ∈ S | p ( s ′ | s, a ) > 0 } .

A policy π maps a state and timestep to a distribution over actions, and we denote the probability of executing a at state s and timestep t as π ( a | s, t ) . Let s t denote the state after t time-steps and a t the action selected at s t , according to π . The expected value V π and expected state-action value Q π of π are defined as:

The goal is to find the optimal policy π ∗ with the maximum expected reward: π ∗ = arg max π V π . The optimal value functions are then defined as V ∗ = V π ∗ , Q ∗ = Q π ∗ . For an MDP, there always exists an optimal policy π ∗ which is deterministic [24].

2.2 Maximum entropy policy optimization

In planning and reinforcement learning, the agent usually aims to maximise the expected sum of rewards. In maximum entropy policy optimisation, the objective is augmented with the expected entropy of the policy [16, 38]. Formally, this is expressed as:

where α ≥ 0 is a temperature parameter, and H is the Shannon entropy function. The temperature determines the relative importance of the entropy against the reward and thus controls the stochasticity of the optimal policy. The conventional reward maximisation objective can be recovered by setting α = 0 .

An optimal value function for maximum entropy optimization is obtained using the soft Bellman optimality equations [17]:

which corresponds to a standard Bellman backup, with the max replaced by a softmax , shown in Equation (5). The optimal soft policy π ∗ sft = arg max π V π sft can be computed directly [26] as follows:

Note that the soft policy is always stochastic for any α > 0 . Henceforth, we will use soft value to refer to value functions indexed with 'sft', (optimal) soft policy to refer to policies of the form given in Equation (6), and standard value and (optimal) standard policy for values and policies of the form given in Section 2.1, unless it is clear from the context.

For the remainder of this paper we will drop the timestep t from policies and value functions to simplify notation.

2.3 Monte-Carlo tree search

MCTS methods build a search tree T using Monte-Carlo trials. Each trial is split into two phases: starting from the root node, actions are chosen according to a search policy and states sampled from the transition distribution until the first state not in T is reached. A new node is added to T and its value is initialised using some function V init , often using a rollout policy to select actions until the time horizon H is reached. In the second phase, the return for the trial is back-propagated up (or 'backed up') the tree to update the values of nodes in T . For a reader unfamiliar with MCTS, we refer to [6] for a review of the MCTS literature, as many variants of MCTS exist and may vary from our description.

Two critical choices in designing an MCTS algorithm are the search policy (which needs to balance exploration and exploitation) and the backups (how values are updated). MCTS algorithms are often designed to achieve consistency (i.e. convergence to the optimal action in the limit), which implies that running more trials will increase the probability that the optimal action is recommended.

To simplify notation we assume that each node in the search tree corresponds to a unique state, so we may represent nodes using states. Our algorithms and results do not make use of this assumption, and generalise to when this assumption does not hold.

UCT UCT[22] applies the upper confidence bound (UCB) in its search policy to balance exploration and exploitation. The n th trial of UCT operates as follows: let T be the current search tree and let τ = ( s 0 , a 0 , ..., a h -1 , s h ) denote the trajectory of the n th trial, where s h ̸∈ T or h = H . At each node s t the UCT search policy π UCT will select a random action that has not previously been selected, otherwise, it will select the action with maximum UCB value:

where, ¯ Q ( s, a ) is the current empirical Q-value estimate, N ( s ) (and N ( s, a ) ) is how many times s has been visited (and action a selected) and c is an exploration parameter. Then, s h is added to the tree: T ← { s h } ∪ T . The backup consists of updating empirical estimates for t = h -1 , ..., 0 :

where ¯ R ( t ) = V init ( s h ) + ∑ h -1 i = t R ( s i , a i ) , and V init ( s h ) = ∑ H i = h R ( s i , a i ) if using a rollout policy.

MENTS MENTS [37] combines maximum entropy policy optimization [16, 38] with MCTS. Algorithmically, it is similar to UCT. The two differences are: (1) the search policy follows a stochastic Boltzmann policy, and (2) it uses soft values that are updated with dynamic programming backups. The MENTS search policy π MENTS is given by:

where λ s = min(1 , ϵ/ log( e + N ( s ))) , ϵ ∈ (0 , ∞ ) is an exploration parameter and ˆ V sft ( s ) (and ˆ Q sft ( s, a ) ) are the current soft (Q-)value estimates. The soft value of the new node is initialised ˆ V sft ( s h ) ← V init ( s h ) and the soft values are updated with backups for t = h -1 , ..., 0 :

Each ˆ Q sft ( s, a ) is initialised using another function Q init sft ( s, a ) (but is typically zero).

1

aR

2

aR

3

D

aR

Rf

D-1

D-2

D-3

0

D

D

D

2.4 Simple regret

UCB [1] is frequently used in MCTS methods to minimise cumulative regret during the tree search. Cumulative regret is most appropriate in scenarios where the actions taken during tree search have an associated real-world cost. However, MCTS methods often use a simulator during the tree search, where the only significant real-world cost is associated with taking the recommended action after the tree search. In such scenarios, simple regret [7, 8] is more appropriate for analysing the performance of algorithms, as it only considers the cost of the actions that are actually executed. Under simple regret, algorithms are not penalised for under-exploiting during the search, thus can explore more, which leads to better recommendations by allowing algorithms to confirm that bad actions are indeed of lower value.

We consider the problem of MDP planning as a sequential decision problem, where for each round n :

the forecaster algorithm produces a search policy π n and samples a trajectory τ ∼ π n ,

the environment returns the rewards R ( s t , a t ) for each s t , a t pair in τ ,

the forecaster algorithm produces a recommendation policy ψ n ,

if environment sends stop signal, then end, else return to step 1.

The simple regret of the forecaster on it's n th round is then:

In MENTS the recommendation policy suggested in [37] can be written:

We can now formally define consistency : an algorithm is consistent if and only if its recommendation policy ψ n converges to an expected simple regret of zero: E [ reg ( s, ψ n )] → 0 as n → ∞ . Note that because we are considering randomised algorithms, there is a distribution over the possible recommendation policies that could have been produced. If a policy has a simple regret of zero then it implies it is an optimal policy.

3 Limitations of prior MCTS methods

In this section we use the D-chain problem introduced in [9] (Figure 1) to highlight the limitations of UCT and MENTS. In the D-chain problem, when an agent chooses action a L , from some state d , it moves to an absorbing state and receives a reward of ( D -d ) /D . In state D , action a R corresponds to an absorbing state with reward R f = 1 . The optimal standard policy always selects action a R .

In the 10-chain problem ( D = 10 ), UCT will recommend action a L from state 1 (Figure 2a). UCT requires Ω(exp( ... exp(1) ... )) many trials ( D composed exponential functions) to recommend the optimal policy that reaches the reward of R f = 1 [9]. This highlights the first limitation mentioned in Section 1: UCT quickly disregards action a R at the initial state, to exploit the reward of 0 . 9 .

When MENTS is run on the 10-chain problem, with the help of the entropy term it quickly finds the final reward of R f = 1 (Figure 2a). However, consider the modified 10-chain with R f = 1 / 2 instead. Repeated applications of Equations (4) and (5) for α = 1 gives the optimal soft values of Q ∗ sft (1 , a R ) = log(exp(1 / 2) + ∑ 8 i =0 exp( i/ 10)) ≈ 2 . 74 and Q ∗ sft (1 , a L ) = 0 . 9 . So in the modified

(a) 10-chain.

(b) Modified 10-chain.

1.0

0.9

Monte-Carlo Value Estimate

0.9

Monte-CarloValue Estimate

0.8

0.8

0.7

0.7

0.6

0.6

Algorithm

0.5

Algorithm

UCT

UCT

0.5

DENTS

0.4

DENTS

MENTS(1.0)

MENTS(1.0)

0.4

MENTS(0.01)

0.3

MENTS(0.01)

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

NumberofTrials

10-chain problem, we have Q ∗ sft (1 , a R ) > Q ∗ sft (1 , a L ) with α = 1 , whereas Q ∗ (1 , a R ) < Q ∗ (1 , a L ) . Thus, when MENTS converges, it will recommend the wrong action with respect to the standard objective (Figure 2b), i.e. it is not consistent. The modified 10-chain is an example of Proposition 3.1, which states that MENTS will not always converge to the standard optimal policy.

Proposition 3.1. There exists an MDP M and temperature α such that E [ reg ( s 0 , ψ n MENTS )] ̸→ 0 as n →∞ . That is, MENTS is not consistent.

We can reduce the value of α to decrease the importance of entropy in the soft objective E [ R ( s t , a t ) + α H ( π ( ·| s t ))] . If α is small enough, then MENTS recommendations can converge to the optimal standard policy (Theorem 3.2). Hence, MENTS with a low temperature can solve the modified 10-chain problem (Figure 2b). However, in practice, a low temperature will often cause MENTS to not sufficiently explore, as demonstrated in the original D-chain (Figure 2a).

̸

Theorem 3.2. For any MDP M , after running n trials of the MENTS algorithm with α ≤ ∆ M / 3 H log |A| , there exists constants C, k > 0 such that: E [ reg ( s 0 , ψ MENTS )] ≤ C exp( -kn ) , where ∆ M = min { Q ∗ ( s, a ) -Q ∗ ( s, a ′ ) | Q ∗ ( s, a ) = Q ∗ ( s, a ′ ) , s ∈ S , a, a ′ ∈ A , t ∈ N } .

Proof outline. We can show ˆ Q sft ( s, a ) p → Q ∗ sft ( s, a ) (Corollary E.12.1) similarly to Thoerem 4.2. The bound on α is required to ensure that π ∗ ( s 0 ) = π ∗ sft ( s 0 ) .

In conclusion, similar MDPs can require vastly different temperatures for MENTS to be effective. We discuss MENTS sensitivity to the temperature parameter further in Appendix D.2, and demonstrate this parameter sensitivity in the Frozen Lake environment (Section 5.1) in Figure 27 in the appendix.

4 Boltzmann search

We now introduce two algorithms that utilise Boltzmann search policies similar to MENTS and admit bounded simple regrets that converge to zero without restrictive constraints on parameters. Thus, they do not suffer from sensitivity to parameter selection that MENTS does. Both algorithms use action selection and value backups that are easy to implement and use. We designed these algorithms with consistency in mind, which in practice, means that if we run more trials then we (with high probability) will recommend a better solution (note that Proposition 3.1 implies that this is not always the case for MENTS).

4.1 Boltzmann Tree Search

Our first approach, put simply, replaces the use of soft values in MENTS with Bellman values. We call this algorithm Boltzmann Tree Search (BTS). BTS promotes exploration through the stochastic

Boltzmann search policy, like MENTS, while using backups that optimise for the standard objective, like UCT. The search policy π BTS and backups for the n th trial are given by:

for t = h -1 , ..., 0 , where ˆ V and ˆ Q are the current Bellman (Q-)value estimates, λ s = min(1 , ϵ/ log( e + N ( s ))) , ϵ ∈ (0 , ∞ ) is an exploration parameter and α is a search temperature (unrelated to entropy). Each ˆ V ( s ) and ˆ Q ( s, a ) are initialised using V init and Q init functions similarly to MENTS. The Bellman values are used for recommendations:

By using Bellman backups, we can guarantee that the BTS recommendation policy converges to the optimal standard policy for any temperature α , given enough time. In other words, BTS is consistent.

Theorem 4.1. For any MDP M , after running n trials of the BTS algorithm with a root node of s 0 , there exists constants C, k > 0 such that for all ε > 0 we have E [ reg ( s 0 , ψ BTS )] ≤ C exp( -kn ) , and also ˆ V ( s 0 ) p → V ∗ ( s 0 ) as n →∞ .

Proof outline. This result is a special case of Theorem 4.2 by setting β ( m ) = 0 .

4.2 Decaying Entropy Tree Search

Secondly, we present Decaying ENtropy Tree Search (DENTS), which can effectively interpolate between the MENTS and BTS algorithms. DENTS also uses the dynamic programming backups from equations (17) and (18), but adds an entropy backup . The entropy values are weighted by a bounded non-negative function β ( N ( s )) in the DENTS search policy π DENTS:

for t = h -1 , ..., 0 , where H V ( s ) and H Q ( s, a ) are the entropy values of the search policy rooted at s and ( s, a ) respectively, and α, λ s are the same as for BTS, as described in Section 4.1. Initial values are the same as Section 4.1, and the entropy values are initialised to zero. In DENTS we can view ˆ Q ( s, a ) + β ( N ( s )) H Q ( s, a ) as a soft value for ( s, a ) . Hence, by setting β ( m ) = α , the DENTS search will mimic the MENTS search (demonstrated in Appendix D.4), and if β ( m ) = 0 then the algorithm reduces to the BTS algorithm. By using a decaying function for β we amplify values using entropy as an exploration bonus early in the search while allowing for more exploitation later. Recommendations still use Bellman values:

Because the recommendation policy ψ DENTS uses the Bellman values, we can guarantee that it will converge to the optimal standard policy, and is consistent for any β .

Theorem 4.2. For any MDP M , after running n trials of the DENTS algorithm with a root node of s 0 , if β is a bounded function, then there exists constants C, k > 0 such that for all ε > 0 we have E [ reg ( s 0 , ψ DENTS )] ≤ C exp( -kn ) , and also ˆ V ( s 0 ) p → V ∗ ( s 0 ) as n →∞ .

̸

Proof outline. Let 0 < η < π DENTS ( a | s ) for all s, a , which exists because exp( · ) > 0 and 1 / |A| > 0 in Equation 20 (Lemma E.1), which can be used with Hoeffding bounds to show for appropriate constants and any event E that { Pr( E ) ≤ C 0 exp( -k 0 ε 2 N ( s t )) } iff { Pr( E ) ≤ C 1 exp( -k 1 ε 2 N ( s t , a t )) } iff { Pr( E ) ≤ C 2 exp( -k 2 ε 2 N ( s t +1 )) } . We can then show Pr( | ˆ Q ( s 0 , a ) -Q ∗ ( s 0 , a ) | > ε ) < C 3 exp ( -k 3 ε 2 n ) by induction, where the base case ˆ V ( s H +1 ) = V ∗ ( s H +1 ) = 0 holds vacuously (Lemmas E.10, E.16 and Theorem E.17). Let ∆ M be a small constant (Equation (122)) such that ∀ s, a. | ˆ Q ( s 0 , a ) -Q ∗ ( s 0 , a ) | ≤ ∆ M / 2 = ⇒ arg max a ˆ Q ( s, a ) = arg max a Q ∗ ( s, a ) . Setting ε = ∆ M / 2 gives a bound on Pr( ψ DENTS = π ∗ ) , which can then be used in the definition of simple regret to give the result.

4.3 Using the Alias method

The Alias method [35, 34] can be used to sample from a categorical distribution with m categories in O (1) time, with a preprocessing step of O ( m ) time. Given any stochastic search policy, we can sample actions in amortised O (1) time, by computing an alias table every |A| visits to a node, and then sampling from that table. Note that when using the Alias method we are making a trade off between using the most up to date policy and the speed of sampling actions.

In Appendix C.1 we discuss this idea in more detail, and give an informal analysis of the complexity to run n trials. BTS, DENTS and MENTS can run n trials in O ( n ( H log( |A| ) + |A| )) time when using the Alias method, as opposed to the typical complexity of O ( nH |A| ) .

4.4 Limitations and benefits

The main limitations of BTS and DENTS are as follows: (1) the DENTS decay function β can be nontrivial to set and tune; (2) the focus on simple regret and exploration means they are not appropriate to use when actions taken during the tree search/planning phase have a real-world cost; (3) the backups implemented directly as presented above are computationally more costly than computing the average returns that UCT uses; (4) when it is desirable for an agent to follow the maximum entropy policy, then MENTS would be preferable, for example if the agent needs to explore to learn and discover an unknown environment.

The main benefits of using a stochastic policy for action selection are: (1) they allow the Alias method (Section 4.3) to be used to speed up trials; (2) they naturally encourage exploration as actions are sampled randomly, which is useful for discovering sparse or delayed rewards and for confirming that actions with low values do in fact have a low value; and (3) the entropy of a stochastic policy can be computed and used as an exploration bonus. In Appendix C.3 we summarise and compare the differences between the algorithms considered in this work in more detail.

5 Results

This section compares the proposed BTS and DENTS against MENTS and UCT on a set of goal-based MDPs and in the game of Go. For additional baselines, we also compare with the RENTS and TENTS algorithms [10], which use relative and Tsalis entropy in place of Shannon entropy respectively, and the H-MCTS algorithm [20] which combines UCT and Sequential Halving.

5.1 Gridworlds

To evaluate an algorithm with search tree T , we complete the partial recommendation policy ψ alg as follows:

̸

(b) 6x6 Sailing Problem.

Monte-Carlo Value Estimate

0.8

Monte-Carlo Value Estimate

-20

Algorithm

Algorithm

UCT

UCT

MENTS

MENTS

BTS

-40

BTS

DENTS

-60

DENTS

0.6

RENTS

-80

RENTS

0.4

TENTS

-100

TENTS

0.2

0.0

HMCTS

-120

HMCTS

0

100

200

300

400

500

0

50

100

150

200

250

300

NumberofTrials(x1000)

Number ofTrials(x1000)

We sample a number of trajectories from ψ , and take the average return to estimate V ψ . Although we are evaluating the algorithms in an offline planning setting, it still indicates how the algorithms perform in an online setting where we interleave planning in simulation with letting the agent act.

5.1.1 Domains

To validate our approach, we use the Frozen Lake environment [5], and the Sailing Problem [28], commonly used to evaluate tree search algorithms [28, 22, 33, 13]. We chose these environments to compare our algorithms in a domain with a sparse and dense reward respectively.

The (Deterministic) Frozen Lake is a grid world environment with one goal state. The agent can move in any cardinal direction at each time step, and walking into a wall leaves the agent in the same location. Trap states exist where the agent falls into a hole and the trial ends. If the agent arrives at the goal state after t timesteps, then a reward of 0 . 99 t is received.

The Sailing Problem is a grid world environment with one goal state, at the opposite corner to the starting location of the agent. The agent has 8 different actions to travel each of the 8 adjacent states. In each state, the wind is blowing in a given direction and will stochastically change after every transition. The agent cannot sail directly into the wind. The cost of each action depends on the tack , the angle between the direction of the agent's travel and the wind.

5.1.2 Results

We used an 8x12 Frozen Lake environment and a 6x6 Sailing Problem for evaluation, more environment details are given in Appendix D.1. Parameters were selected using a hyper-parameter search (Appendix D.3). Each algorithm is run 25 times on each environment and evaluated every 250 trials using 250 trajectories. A horizon of 100 was used for Frozen Lake and 50 for the Sailing Problem.

In Frozen Lake (Figure 3a), entropy proved to be a useful exploration bonus for the sparse reward . Values in UCT and BTS remain at zero until a trial successfully reaches the goal. However, entropy guides agents to avoid trap states, where the entropy is zero. DENTS was able to perform similarly to MENTS, and BTS was able to improve its policy over time more than UCT.

In the Sailing Problem (Figure 3b) UCT performs well due to the dense reward. BTS and DENTS also manage to keep up with UCT. MENTS and TENTS appear to be slightly hindered by entropy in this environment. The relative entropy encourages RENTS to pick the same actions over time, so it tends to pick a direction and stick with it regardless of cost.

Finally, BTS and DENTS were able to perform well in both domains with a sparse and dense reward structure, whereas the existing methods performed better on one than the other, hence making BTS and DENTS good candidates for a general purpose MCTS algorithm.

5.2 Go

For a more challenging domain we ran a round-robin tournament using the game of Go, which has widely motivated the development of MCTS methods [14, 30, 32]. In each match, each algorithm played 50 games as black and 50 as white. Area scoring is used to score the games, with a komi of 7 . 5 . We used an openly available value network ˜ V and policy network ˜ π from KataGo [36]. Our baseline was the PUCT algorithm [2], as described in Alpha Go Zero [32] using prioritised UCB [29] to utilise the policy neural network. Each algorithm was limited to 5 seconds of compute time per move, allowed to use 32 search threads per move, and had access to 80 Intel Xeon E5-2698V4 CPUs clocked at 2.2GHz, and a single Nvidia V100 GPU on a shared compute cluster.

To use Boltzmann search in Go, we adapted the algorithms to account for an opponent that wishes to minimise the value of a two-player game. This is achieved by appropriately negating values used in the search policy and backups, which is described precisely in Appendix C.2.

Additionally, we found that adapting the algorithms to use average returns (recall Equation (8)) outperformed using Bellman backups for Go (Appendix D.5.1). The Bellman backups were sensitive to and propogated noise from the neural network evaluations. We use the prefix 'AR' to denote the algorithms using average returns, such as AR-DENTS. Full details for these algorithms are given in Appendix B.

5.2.1 Using neural networks with Boltzmann search

This section describes how to use value and policy networks in BTS. Adapting MENTS and DENTS are similar (Appendix C.2). Values can be initialised with the neural networks as ˆ Q ( s, a ) ← log ˜ π ( a | s ) + B and ˆ V ( s ) ← ˜ V ( s ) , where B is a constant (adapted from Xiao et al. [37]). With such an initialisation, the initial BTS policy is ρ BTS ( a | s ) ∝ ˜ π ( a | s ) 1 /α . For these experiments we set a value of B = -1 |A| ∑ a ∈A log ˜ π ( a | s ) . Additionally, the stochastic search policy naturally lends itself to mixing in a prior policy, so we can replace BTS search policy π BTS (Equation (15)) with π BTS,mix:

where λ ˜ π = min(1 , ϵ ˜ π / log( e + N ( s ))) , and ϵ ˜ π ∈ (0 , ∞ ) controls the weighting for the prior policy.

5.2.2 Results

Results of the round-robin are summarised in Table 1, and we discuss how parameters were selected in Appendix D.5.1. BTS was able to run the most trials per move and beat all of the other algorithms other than DENTS which it drew. We used the optimisations outlined in Appendix C.1 which allowed the Boltzmann search algorithms to run significantly more trials per move than PUCT. BTS and DENTS were able to beat PUCT with results of 57-43 and 58-42 respectively. Using entropy did not seem to have much benefit in these experiments, as can be witnessed by MENTS only beating TENTS, and DENTS drawing 50-50 with BTS. This is likely because the additional exploration provided by entropy is vastly outweighed by utilising the information contained in the neural networks ˜ V and ˜ π . Interestingly RENTS had the best performance out of the prior works, losing 43-57 to PUCT, and the use of relative entropy appears to take advantage of a heuristic for Go that the RA VE [15] algorithm used: the value of a move is typically unaffected by other moves on the board.

To validate the strength of our PUCT agent, we also compared it directly with KataGo [36], limiting each algorithm to 1600 trials per move. Our PUCT agent won 61-39 in 9x9 Go, and lost 35-65 in 19x19 Go, suggesting that our PUCT agent is strong enough to provide a meaningful comparison for our other general purpose algorithms. Finally, note that we did not fine-tune the neural networks, so the Boltzmann search algorithms directly used the networks that were trained for use in PUCT.

6 Related work

UCT [22, 23] is a widely used variant of MCTS. Polynomial UCT [2], replaces the logarithmic term in UCB with a polynomial one (such as a square root), which has been further popularised by its use

in Alpha Go and Alpha Zero [30, 32, 31]. Coquelin and Munos introduce the Flat-UCB and BAST algorithms to adapt UCT for the D-chain problem [9]. However, we consider an alternative approach for search in MCTS rather than adapting UCB.

Maximum entropy policy optimization methods are well-known in the reinforcement learning literature [16, 17, 38]. MENTS [37] is the first method to combine the principle of maximum entropy and MCTS. Kozakowski et al. [25] extend MENTS to arrive at Adaptive Entropy Tree Search (ANTS), adapting parameters throughout the search to match a prescribed entropy value. Dam et al. [10] also extend MENTS using Relative and Tsallis entropy to arrive at the RENTS and TENTS algorithms. Our work is closely related to MENTS, however, we focus on reward maximisation and consider how entropy can be used in MCTS without altering our planning objective.

Bubeck et al. [7, 8] introduce simple regret in the context of multi-armed bandit problems (MABs). They alternate between pulling arms for exploration and outputting a recommendation . They show for MABs that a uniform exploration produces an exponential bound on the simple regret of recommendations. We use simple regret, but in the context of sequential decision-making, to analyse the convergence of MCTS algorithms.

Tolpin and Shimony [33] extend simple regret to MDP settings, showing an O (exp( - √ n )) bound on simple regret after n trials by adapting UCT. The subsequent work of Hay et al. [19] extends [33] to consider a metalevel decision problem , incorporating computation costs into the objective. Pepels et al. [27] introduce a Hybrid MCTS (H-MCTS) motivated by the notion of simple regret. H-MCTS uses a mixture of Sequential Halving [20], and UCT. Feldman et al. [13, 11, 12] introduce the Best Recommendation with Uniform Exploration (BRUE) algorithm. BRUE splits trials up to explicitly focus on exploration and value estimation one at a time. BRUE achieves an exponential bound O (exp( -n )) on simple regret after n trials [13]. Prior work that considers simple regret in MCTS has focused on adaptations to UCT, whereas this work focuses on algorithms that sample actions from Boltzmann distributions, rather than using UCB for action seclection.

7 Conclusion

We considered the recently introduced MENTS algorithm, compared and contrasted it to UCT, and discussed the limitations of both. We introduced two new algorithms, BTS and DENTS, that are consistent, converge to the optimal standard policy, while preserving the benefits that come with using a stochastic Boltzmann search policy. Finally, we compared our algorithms in gridworld environments and Go, demonstrating the performance benefits of utilising the Alias method, that entropy can be a useful exploration bonus with sparse rewards, and more generally, demonstrating the advantage of prioritising exploration in planning, by using Boltzmann search policies.

An interesting area of future work may include investigating good heuristics for setting parameters in BTS and DENTS. We noticed that the best value for the search temperature tended to be the same order of magnitude as the optimal value at the root node V ∗ ( s 0 ) , which suggests a heuristic similar to [21] may be reasonable.

References

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

David Auger, Adrien Couetoux, and Olivier Teytaud. Continuous upper confidence trees with polynomial exploration-consistency. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 194-209. Springer, 2013.

Richard Bellman. A markovian decision process. Journal of mathematics and mechanics , 6(5):679-684, 1957.

Blai Bonet and Hector Geffner. Labeled rtdp: Improving the convergence of real-time dynamic programming. In ICAPS , volume 3, pages 12-21, 2003.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games , 4(1):1-43, 2012.

Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In International conference on Algorithmic learning theory , pages 23-37. Springer, 2009.

Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in finitely-armed and continuous-armed bandits. Theoretical Computer Science , 412(19):1832-1852, 2011.

Pierre-Arnaud Coquelin and Rémi Munos. Bandit algorithms for tree search. In Uncertainty in Artificial Intelligence , 2007.

Tuan Q Dam, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Convex regularization in monte-carlo tree search. In International Conference on Machine Learning , pages 2365-2375. PMLR, 2021.

Zohar Feldman and Carmel Domshlak. Monte-carlo planning: Theoretically fast convergence meets practical efficiency. arXiv preprint arXiv:1309.6828 , 2013.

Zohar Feldman and Carmel Domshlak. On mabs and separation of concerns in monte-carlo planning for mdps. In Twenty-Fourth International Conference on Automated Planning and Scheduling , 2014.

Zohar Feldman and Carmel Domshlak. Simple regret optimization in online planning for markov decision processes. Journal of Artificial Intelligence Research , 51:165-205, 2014.

Sylvain Gelly and David Silver. Combining online and offline knowledge in uct. In Proceedings of the 24th international conference on Machine learning , pages 273-280, 2007.

Sylvain Gelly and David Silver. Monte-carlo tree search and rapid action value estimation in computer go. Artificial Intelligence , 175(11):1856-1875, 2011.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning , pages 1352-1361. PMLR, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Eric A Hansen and Shlomo Zilberstein. Lao*: A heuristic search algorithm that finds solutions with loops. Artificial Intelligence , 129(1-2):35-62, 2001.

Nicholas Hay, Stuart Russell, David Tolpin, and Solomon Eyal Shimony. Selecting computations: Theory and applications. arXiv preprint arXiv:1408.2048 , 2014.

Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning , pages 1238-1246. PMLR, 2013.

Thomas Keller and Patrick Eyerich. Prost: Probabilistic planning based on uct. In Twenty-Second International Conference on Automated Planning and Scheduling , 2012.

Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning , pages 282-293. Springer, 2006.

Levente Kocsis, Csaba Szepesvári, and Jan Willemson. Improved monte-carlo search. Univ. Tartu, Estonia, Tech. Rep , 1, 2006.

Andrey Kolobov. Planning with Markov Decision Processes: An AI Perspective , volume 6. Morgan & Claypool Publishers, 2012.

Piotr Kozakowski, Mikołaj Pacek, and Piotr Miło´ s. Planning and learning using adaptive entropy tree search. arXiv preprint arXiv:2102.06808 , 2021.

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems , 30, 2017.

Tom Pepels, Tristan Cazenave, Mark HM Winands, and Marc Lanctot. Minimizing simple and cumulative regret in monte-carlo tree search. In Workshop on Computer Games , pages 1-15. Springer, 2014.

Laurent Péret and Frédérick Garcia. On-line search for solving markov decision processes via heuristic sampling. learning , 16:2, 2004.

Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence , 61(3):203-230, 2011.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

David Tolpin and Solomon Shimony. Mcts based on simple regret. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 26, pages 570-576, 2012.

Michael D Vose. A linear algorithm for generating random numbers with a given distribution. IEEE Transactions on software engineering , 17(9):972-975, 1991.

Alastair J Walker. New fast method for generating discrete random numbers with arbitrary frequency distributions. Electronics Letters , 8(10):127-128, 1974.

David J Wu. Accelerating self-play learning in go. arXiv preprint arXiv:1902.10565 , 2019.

Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems , 32, 2019.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. 2008.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces two variants of MCTS designed to perform more exploration during search than the commonly used UCT search policy, while resolving an issue with a previously introduced MCTS variant, Maximum ENtropy Tree Search (MENTS), which was designed to explore more than UCT, but which does not converge to the optimal policy / value function with an increasing number of MCTS trials.\n\nTo address this issue, the authors modify MCTS in two different ways. In the first variant, Boltzmann Tree Search (BTS), the authors use a Boltzmann search policy during the selection/search phase of MCTS, instead of using UCT, but perform a regular Bellman backup to compute state and action values. In the second variant, Decaying ENtropy Tree Search (DENTS), they modify MENTs to separately keep track of the policy entropy and the (unmodified) value functions, allowing them to slowly decay the entropy contribution over time, while using the entropy-augmented value functions to drive exploration during search. Both variants of MCTS are consistent -- i.e. they value function they compute for the root node converges to the optimal value as the number of MCTS trials increases, and the simple regret decays exponentially with respect to the number of MCTS trials.\n\nThe authors evaluate BTS and DENTS against baseline methods on both toy MDP problems, and in a 9x9 Go round-robin tournament. They find that BTS and DENTS perform similarly or slightly better than existing baselines on the toy problems, and consistently outcompete other methods (modified to make use of neural-networks) in the Go tournament.\n\nSTRENGTHS:\n- The paper clearly explains an existing problem with MENTS not converging to the optimal root-node policy with increasing trials  (e.g. using the D-chain problem to illustrate the point).\n- The \"fix\" that this paper proposes to MENTS is intuitive and clearly explained.\n- The theoretical results appear to be sound, with extensive proofs (though I did not check these carefully).\n- The empirical results demonstrate that BTS & DENTS can improve upon baseline search policies like UCT and MENTS in non-trivial problems (i.e. Go), while generally performing no worse than the baselines on simpler problems.\n\nWEAKNESSES:\n- The motivation for increasing exploration during planning/search could be better explained.\n  - In particular, I think the authors could better highlight that UCT was justified in the context of selecting actions in the real-world, and that per Tolpin & Shimony et al [1, 2], using UCT in *simulation* doesn't make as much sense, since computational actions are less costly than actual actions.\n  - This better motivates the use of a Boltzmann search policy, since exploring more when \"thinking\" could ultimately lead to better exploitation when finally acting / executing.\n  - Similarly, the authors could better explain their choice of using simple regret (instead of cumulative regret) as the metric for an MCTS algorithm, following the discussion / reasoning in [1, 2]. Cumulative regret doesn't matter if the cost of computation is effectively zero (and only matters slightly if computation cost is much smaller the actual cost of acting).\n\n- I don't think this should count very much against the paper, given the extensive effort at formally proving the stated results (which I think is valuable, and should be recognized), but it's arguably the case that BTS are DENTS are fairly incremental and straightforward extensions of MENTS.\n\n[1] David Tolpin and Solomon Shimony. Mcts based on simple regret. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 570–576, 2012.\n\n[2] Nicholas Hay, Stuart Russell, David Tolpin, and Solomon Eyal Shimony. Selecting computations: Theory and applications. arXiv preprint arXiv:1408.2048, 2014.\n\nQUESTIONS:\n- Why does the search temperature $\\alpha$ in DENTS differ from the entropy weighting term $\\beta$? By default (e.g. in MENTS) this should be the same -- is there a reason why it makes sense have them vary separately? Why not decay $\\alpha$ with increasing node visits as well?\n\n- In the AR version of DENTS used for the Go experiments, both $\\alpha$ and $\\beta$ are decayed. AR-MENTS is then described as a variant of AR-DENTS, where $\\alpha$ is set equal to $\\beta$. But in AR-MENTS, are $\\alpha$ and $\\beta$ set to a fixed value, or decayed with increasing node visits? I'm assuming it's the former, in which case that should be clarified in the Appendix, but if it's the latter, how is AR-MENTS different from AR-DENTS?\n\n- What is the value of DENTS over BTS, or vice versa? Are their high-level rules for when one should be preferred over the other?\n\nMinor comments:\n- There seems to be some inconsistency in notation in equations 11 and 19 vs. the text surrounding them -- $N(s)$ and $N(s, a)$ are used in the equations, but the text uses $m_s$ and $m_a$.\n\nPost-Rebuttal Response:\n\nThe author's have addressed my questions, and provided additional experiments showing how their proposed methods compare favorably against existing baselines. I continue to recommend acceptance.\n\nLIMITATIONS:\nOne benefit of computing the max entropy policy, as MENTS does, is that it might help with exploration when *acting* (not just planning) --- this is perhaps helpful when the agent doesn't have an accurate model of the world, and needs to learn it over time. Adding some discussion on this point would be good, since it seems like a potential limitation of \n\nApart from that, the authors have addressed the limitation of BTS and DENTS that the require $O(|\\mathcal{A}|)$ backups relative to UCT, which may hinder performance when actions spaces are large.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two new algorithms, Boltzmann Tree Search (BTS) and Decaying Entropy Tree Search (DENTS), based on the Maximum Entropy Tree Search (MENTS) method. The authors prove that in certain situations, MENTS fails to converge to the optimal policy. They analyze BTS and DENTS using simple regret and show that these algorithms will recommend the optimal actions. Finally, the authors evaluate various MCTS algorithms, where BTS and DENTS significantly outperform other MCTS algorithms.\n\nSTRENGTHS:\n1. The paper is well-structured, with clear explanations of the concepts and methods used.\n2. The paper identifies the limitations of MENTS and proposes two new algorithms, BTS and DENTS, to address the issues of MENTS. The authors conduct theoretical analysis and perform experiments to validate their proposed solutions.\n3. The experiments show that the proposed algorithms outperform other MCTS algorithms.\n\nWEAKNESSES:\nIt is not quite convincing for the baseline program used in the Go experiments. The Komi in 9x9 Go is usually set to 7 (a balanced setting for both players). Black will have an advantage when Komi is set to 6.5. In addition, in Table 1, does PUCT have the same performance (strength) as KataGo? Why not compare the AR-* methods to KataGo directly? Besides, it is quite confusing why the author uses 128 CPU threads. To the best of my knowledge, this is not the best machine configuration for running KataGo. Using 128 CPU threads may even hurt the performance of KataGo. Moreover, in line 266, the authors said they added Dirichlet noise at the root node. In AlphaGo Zero, there is no need to add Dirichlet noise during evaluation.\n\nQUESTIONS:\n1. The authors mention that MENTS may struggle to converge under certain specific circumstances by using the D-Chain as an example. However, it is unclear whether this phenomenon generally happens in other environments.\n\n2. The author mentions the difficulty of selecting β. The experiments also show that BTS (DENTS with β=0) performs better than DENTS. Do the authors conduct any analysis or have any suggestions for β? In what scenarios where DENTS will perform better than BTS?\n\n3. The author claims that DENTS will be similar to MENTS when setting β(m) = α, is there any experimental/theoretical explanation for this part? The relationship is a bit unclear from the mathematical formula.\n\n4. How does MCTS operate in a stochastic environment for Sailing problems?\n\n5. Why were RENTS and TENTS only tested on the 6x6 Sailing Problem and not on the 8x12 Frozen Lake?\n\nOther minor suggestions:\n1. The lines in Figure 2 are overlapping, making it difficult to see clearly.\n2. In equation (4), should it be $V_{\\text{sft}}^{\\pi}(s', t)$ or $V^{\\pi}_{\\text{sft}}(s', t + 1)$?\n\nLIMITATIONS:\nThe authors addressed their limitations in section 5.3.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n<< I have read the authors' rebuttal and have raised my score based on the discussion >>\n\nThe paper addresses challenges in online tree-based search algorithms, such as UCT, which have shortcomings such as being prone to local optima and a slower convergence rate to the optimal action. An alternative approach, Maximum ENtropy Tree Search (MENTS), despite theoretically optimal performance, has practical limitations like sensitivity to the temperature parameter, a tendency to over explore, and increased computational expense. The authors present two new algorithms to address these issues: Boltzmann Tree Search (BTS) and Decaying ENtropy Tree Search (DENTS). BTS employs a Boltzmann search policy, focusing solely on reward maximisation, whereas DENTS introduces entropy backups that can be used with varying weighting in the search. Both algorithms preserve the exploratory benefits of MENTS, and importantly, they consistently converge to the optimal reward maximising policy. Key contributions of the paper include highlighting the misalignment of the entropy objective in MENTS with reward maximisation, introducing BTS and DENTS, providing an analysis of MENTS, BTS, and DENTS through the lens of simple regret, and demonstrating the improved performance of BTS and DENTS over other MCTS algorithms in specific environments and games.\n\nSTRENGTHS:\nNovelty: The presented Boltzmann Tree Search (BTS) algorithm is a variant of the Maximum ENtropy Tree Search (MENTS), substituting the soft update value in equation 13 with the max operator in equation 19. The Decaying ENtropy Tree Search (DENTS) proposes a novel method for decaying exploration to ultimately converge to the optimal solution, an aspect MENTS struggles with due to its focus on maintaining entropy in the search policy. The novelty of this paper's contributions, while discernible, seems to be limited.\n\nClarity: While the paper is generally well-written, comprehension is somewhat impeded by the multitude of mathematical symbols and technical jargon used throughout. An effort to streamline and elucidate the language could significantly enhance readability.\n\nSignificance: The paper's experimental results highlight DENTS' robustness to the adjustment of the temperature hyperparameter, a critical element in MENTS. Moreover, the simplicity of the proposed changes augments their potential for practical adoption in the field.\n\nWEAKNESSES:\nW1: The novelty of the work seems somewhat limited. It remains unclear whether the improvements result from the incorporation of entropy in the case of DENTS or from the use of hard updates instead of soft updates in BTS.\n\nW2: The paper emphasizes the sensitivity of MENTS to the temperature hyperparameter. It is true that many online search algorithms rely on hyperparameter tuning for optimal performance. However, it is noteworthy that MENTS outperforms both BTS and DENTS on the 10-chain and gridworld tasks when the temperature hyperparameter is appropriately tuned. Therefore, it could be argued that the only advantage of using BTS or DENTS is their lesser dependency on temperature. Yet, with the correct tuning of this singular parameter, MENTS could potentially offer a superior solution.\n\nQUESTIONS:\nQ1: I found it somewhat surprising that MENTS performed relatively poorly on the game of Go, especially considering its strong performance in other domains. Can you confirm whether the temperature hyperparameter for MENTS was appropriately tuned in this case?\n\nQ2: What methodologies were employed by the authors to tune the hyperparameters for different baseline algorithms, including BTS and DENTS?\n\nQ3: To better understand the influence of hyperparameters on BTS and DENTS and gauge their robustness, would it be possible for the authors to compare their performance for varying values of the alpha parameter?\n\nLIMITATIONS:\nThe authors discussed the limitations of their work. No discussion needed regarding potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents two MCTS based algorithms called  Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS). These algorithms overcome the limitation of two state-of-the-art algorithms pUCT and MENTS of under and over exploration respectively. Further, the paper provides simple “simple regret” analysis and shows state-of-the-art performance on benchmark tasks.\n\nSTRENGTHS:\n1. The paper is very well structured and easy to follow for the most part.\n2. The empirical results are impressive, especially on Go.\n3. The algorithms are novel to my knowledge and are inspiring.\n\nWEAKNESSES:\n1. The motivation for analyzing simple regret instead of cumulative regret is not clear. This limits the contribution of the proposed approach to some extent. (I will update the score post rebuttal, given the authors' response).\n2. The paper puts all the theoretical proof in the appendix. Please provide at least a proof sketch in the main paper. \n3. Section 6.1 is not clearly written. What do you mean by “ To utilize ... a terminal/goal state.” (lines 232-234)\n\nMinor\n1. Graphs in Figure 2 are of poor quality. Please consider using transparent lines or a variety of dashed lines (different patterns of dashes) to represent different algorithms.\n2. Please increase the font size of the text in Figure 3.\n\nNote: I haven't carefully taken a look at the proofs in the appendix. I might update my score based on correctness.\n\nQUESTIONS:\n1. Line number 18 - What do you mean by “exact methods”?\n2. Why H-MCTS [27] (or other algorithms that focus on simple regret) is not used as a baseline?\n3. Are there no approaches that study how to decrease alpha in MENTS so that Theorem 2-like statement holds?\n4. How to set the value of alpha in BTS?\n5. Why are RENTS and TENTS not compared on Frozen Lake?\n\nLIMITATIONS:\nLimitations are clearly mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}