SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michael Painter, Mohamed Baioumy, Nick Hawes, Bruno Lacerda
Oxford Robotics Institute
{mpainter, mohamed, nickh, bruno}@robots.ox.ac.uk
Monte-Carlo Tree Search (MCTS) methods, such as Upper Confidence Bound applied to Trees (UCT), are instrumental to automated planning techniques. However, UCT can be slow to explore an optimal action when it initially appears inferior to other actions. Maximum ENtropy Tree-Search (MENTS) incorporates the maximum entropy principle into an MCTS approach, utilising Boltzmann policies to sample actions, naturally encouraging more exploration. In this paper, we highlight a major limitation of MENTS: optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective. We introduce two algorithms, Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS), that address these limitations and preserve the benefits of Boltzmann policies, such as allowing actions to be sampled faster by using the Alias method. Our empirical analysis shows that our algorithms show consistent high performance across several benchmark domains, including the game of Go.
Planning under uncertainty is a core problem in Artificial Intelligence, commonly modelled as a Markov Decision Process (MDP) or variant thereof. MDPs can be solved using dynamic programming techniques to obtain an optimal policy [3]. However, computing a full optimal policy does not scale to large state-spaces, necessitating the use of heuristic solvers [18, 4] and online, sampling-based, planners based on Monte-Carlo Tree-Search (MCTS), such as the Upper Confidence Bound applied to Trees (UCT) algorithm [22].
The UCT search policy is designed to minimise cumulative regret, so manages a trade-off between exploration and exploitation. To exploit, UCT often selects the same action on successive trials, which can result in it getting stuck in local optima. Conversely, Maximum ENtropy Tree Search (MENTS) places a greater emphasis on exploration by combining MCTS with techniques from maximum entropy policy optimisation [38, 16, 17]. MENTS jointly maximises cumulative rewards and policy entropy, where a temperature parameter controls the weight of the entropy objective. However, MENTS is sensitive to this temperature parameter, and may not converge to the reward maximising policy or require a prohibitively low temperature to do so.
In this work, we consider scenarios where MCTS methods are used with a simulator to plan how an agent should act. We introduce two algorithms for this scenario that address the above limitations. First, we present Boltzmann Tree Search (BTS) which uses a Boltzmann search policy like MENTS, but optimises for reward maximisation only. Secondly, we introduce Decaying ENtropy Tree Search (DENTS), which adds entropy backups to BTS, but is still consistent (i.e. it converges to the reward maximising policy in the limit).
The main contributions of this paper are: (1) Demonstrating that the maximum entropy objective used in MENTS can be misaligned with reward maximisation, thus preventing it from converging to the optimal policy; (2) Introducing two new algorithms, BTS and DENTS, which preserve the benefits
of using Boltzmann search policies while being as simple to implement as UCT and MENTS, but converge to the reward maximising policy; (3) Analysing MENTS, BTS and DENTS through the lens of simple regret to provide theoretical convergence results; (4) Highlighting and demonstrating that the Alias method [35, 34] can be used with stochastic action selection to improve the asymptotic complexity of running a fixed number of trials over existing MCTS algorithms; and (5) Demonstrating the performance improvements of Boltzmann search policies used in BTS and DENTS in benchmark gridworld environments and the game of Go.
We define a (finite-horizon) MDP as a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},p,R,H)$ , where  $\mathcal{S}$  is the set of states;  $\mathcal{A}$  is the set of actions;  $p:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$  is the transition function where  $p(s^{\prime}|s,a,\cdot)$  is the probability of moving to state  $s^\prime$  given that action  $a$  was taken in state  $s$ ;  $R:\mathcal{S}\times \mathcal{A}\rightarrow \mathbb{R}$  is the reward function; and  $H\in \mathbb{N}$  is the finite horizon. Let  $\operatorname {Succ}(s,a)$  denote the set of successor states of  $(s,a)$ , i.e.  $\operatorname {Succ}(s,a) = \{s'\in S\mid p(s'|s,a) > 0\}$ .
A policy  $\pi$  maps a state and timestep to a distribution over actions, and we denote the probability of executing  $a$  at state  $s$  and timestep  $t$  as  $\pi(a|s,t)$ . Let  $s_t$  denote the state after  $t$  time-steps and  $a_t$  the action selected at  $s_t$ , according to  $\pi$ . The expected value  $V^{\pi}$  and expected state-action value  $Q^{\pi}$  of  $\pi$  are defined as:
$$
V ^ {\pi} (s, t) = \mathbb {E} _ {\pi} \left[ \sum_ {i = t} ^ {H} R \left(s _ {i}, a _ {i}\right) \mid s _ {t} = s \right], \tag {1}
$$
$$
Q ^ {\pi} (s, a, t) = R (s, a) + \mathbb {E} _ {s ^ {\prime} \sim p (\cdot | s, a)} [ V ^ {\pi} (s ^ {\prime}, t + 1) ]. \tag {2}
$$
The goal is to find the optimal policy  $\pi^{*}$  with the maximum expected reward:  $\pi^{*} = \arg \max_{\pi} V^{\pi}$ . The optimal value functions are then defined as  $V^{*} = V^{\pi^{*}}$ ,  $Q^{*} = Q^{\pi^{*}}$ . For an MDP, there always exists an optimal policy  $\pi^{*}$  which is deterministic [24].
2.2 Maximum entropy policy optimization
In planning and reinforcement learning, the agent usually aims to maximise the expected sum of rewards. In maximum entropy policy optimisation, the objective is augmented with the expected entropy of the policy [16, 38]. Formally, this is expressed as:
$$
V _ {\mathrm {s f t}} ^ {\pi} (s, t) = \mathbb {E} _ {\pi} \left[ \sum_ {i = t} ^ {H} R \left(s _ {i}, a _ {i}\right) + \alpha \mathcal {H} (\pi (\cdot | s _ {i}, i)) \mid s _ {t} = s \right], \tag {3}
$$
where  $\alpha \geq 0$  is a temperature parameter, and  $\mathcal{H}$  is the Shannon entropy function. The temperature determines the relative importance of the entropy against the reward and thus controls the stochasticity of the optimal policy. The conventional reward maximisation objective can be recovered by setting  $\alpha = 0$ .
An optimal value function for maximum entropy optimization is obtained using the soft Bellman optimality equations [17]:
$$
Q _ {\mathrm {s f t}} ^ {*} (s, a, t) = R (s, a) + \mathbb {E} _ {s ^ {\prime} \sim p (\cdot | s, a)} \left[ V _ {\mathrm {s f t}} ^ {*} \left(s ^ {\prime}, t\right) \right], \tag {4}
$$
$$
V _ {\mathrm {s f t}} ^ {*} (s, t) = \alpha \log \sum_ {a \in \mathcal {A}} \exp \left(Q _ {\mathrm {s f t}} ^ {*} (s, a, t) / \alpha\right), \tag {5}
$$
which corresponds to a standard Bellman backup, with the max replaced by a softmax, shown in Equation (5). The optimal soft policy  $\pi_{\mathrm{sft}}^{*} = \arg \max_{\pi} V_{\mathrm{sft}}^{\pi}$  can be computed directly [26] as follows:
$$
\pi_ {\mathrm {s f t}} ^ {*} (a | s, t) = \exp \left(\left(Q _ {\mathrm {s f t}} ^ {*} (s, a, t) - V _ {\mathrm {s f t}} ^ {*} (s, t)\right) / \alpha\right). \tag {6}
$$
Note that the soft policy is always stochastic for any  $\alpha > 0$ . Henceforth, we will use soft value to refer to value functions indexed with 'sft', (optimal) soft policy to refer to policies of the form given in Equation (6), and standard value and (optimal) standard policy for values and policies of the form given in Section 2.1, unless it is clear from the context.
For the remainder of this paper we will drop the timestep  $t$  from policies and value functions to simplify notation.
2.3 Monte-Carlo tree search
MCTS methods build a search tree  $\mathcal{T}$  using Monte-Carlo trials. Each trial is split into two phases: starting from the root node, actions are chosen according to a search policy and states sampled from the transition distribution until the first state not in  $\mathcal{T}$  is reached. A new node is added to  $\mathcal{T}$  and its value is initialised using some function  $V^{\mathrm{init}}$ , often using a rollout policy to select actions until the time horizon  $H$  is reached. In the second phase, the return for the trial is back-propagated up (or 'backed up') the tree to update the values of nodes in  $\mathcal{T}$ . For a reader unfamiliar with MCTS, we refer to [6] for a review of the MCTS literature, as many variants of MCTS exist and may vary from our description.
Two critical choices in designing an MCTS algorithm are the search policy (which needs to balance exploration and exploitation) and the backups (how values are updated). MCTS algorithms are often designed to achieve consistency (i.e. convergence to the optimal action in the limit), which implies that running more trials will increase the probability that the optimal action is recommended.
To simplify notation we assume that each node in the search tree corresponds to a unique state, so we may represent nodes using states. Our algorithms and results do not make use of this assumption, and generalise to when this assumption does not hold.
UCT UCT [22] applies the upper confidence bound (UCB) in its search policy to balance exploration and exploitation. The  $n$ th trial of UCT operates as follows: let  $\mathcal{T}$  be the current search tree and let  $\tau = (s_0, a_0, \dots, a_{h-1}, s_h)$  denote the trajectory of the  $n$ th trial, where  $s_h \notin \mathcal{T}$  or  $h = H$ . At each node  $s_t$  the UCT search policy  $\pi_{\mathrm{UCT}}$  will select a random action that has not previously been selected, otherwise, it will select the action with maximum UCB value:
$$
\pi_ {\mathrm {U C T}} (s) = \max  _ {a \in \mathcal {A}} \bar {Q} (s, a) + c \sqrt {\frac {\log N (s)}{N (s , a)}}, \tag {7}
$$
where,  $\bar{Q}(s,a)$  is the current empirical Q-value estimate,  $N(s)$  (and  $N(s,a)$ ) is how many times  $s$  has been visited (and action  $a$  selected) and  $c$  is an exploration parameter. Then,  $s_h$  is added to the tree:  $\mathcal{T} \gets \{s_h\} \cup \mathcal{T}$ . The backup consists of updating empirical estimates for  $t = h - 1,\dots,0$ :
$$
\bar {Q} \left(s _ {t}, a _ {t}\right) \leftarrow \bar {Q} \left(s _ {t}, a _ {t}\right) + \frac {\bar {R} (t) - \bar {Q} \left(s _ {t} , a _ {t}\right)}{N \left(s _ {t} , a _ {t}\right) + 1}, \tag {8}
$$
where  $\bar{R}(t) = V^{\mathrm{init}}(s_h) + \sum_{i=t}^{h-1} R(s_i, a_i)$ , and  $V^{\mathrm{init}}(s_h) = \sum_{i=h}^{H} R(s_i, a_i)$  if using a rollout policy.
MENTS MENTS [37] combines maximum entropy policy optimization [16, 38] with MCTS. Algorithmically, it is similar to UCT. The two differences are: (1) the search policy follows a stochastic Boltzmann policy, and (2) it uses soft values that are updated with dynamic programming backups. The MENTS search policy  $\pi_{\mathrm{MENTS}}$  is given by:
$$
\pi_ {\text {M E N T S}} (a | s) = \left(1 - \lambda_ {s}\right) \rho_ {\text {M E N T S}} (a | s) + \frac {\lambda_ {s}}{| \mathcal {A} |}, \tag {9}
$$
$$
\rho_ {\text {M E N T S}} (a | s) = \exp \left(\frac {1}{\alpha} \left(\hat {Q} _ {\text {s f t}} (s, a) - \hat {V} _ {\text {s f t}} (s)\right)\right) \tag {10}
$$
where  $\lambda_s = \min(1, \epsilon / \log(e + N(s))), \epsilon \in (0, \infty)$  is an exploration parameter and  $\hat{V}_{\mathrm{sft}}(s)$  (and  $\hat{Q}_{\mathrm{sft}}(s, a)$ ) are the current soft (Q-)value estimates. The soft value of the new node is initialized  $\hat{V}_{\mathrm{sft}}(s_h) \gets V^{\mathrm{init}}(s_h)$  and the soft values are updated with backups for  $t = h - 1, \dots, 0$ :
$$
\hat {Q} _ {\mathrm {s f t}} \left(s _ {t}, a _ {t}\right) \leftarrow R \left(s _ {t}, a _ {t}\right) + \sum_ {s ^ {\prime} \in \operatorname {S u c c} (s, a)} \left(\frac {N \left(s ^ {\prime}\right)}{N \left(s _ {t} , a _ {t}\right)} \hat {V} _ {\mathrm {s f t}} \left(s ^ {\prime}\right)\right), \tag {11}
$$
$$
\hat {V} _ {\mathrm {s f t}} \left(s _ {t}\right) \leftarrow \alpha \log \sum_ {a \in \mathcal {A}} \exp \left(\frac {1}{\alpha} \hat {Q} _ {\mathrm {s f t}} \left(s _ {t}, a\right)\right). \tag {12}
$$
Each  $\hat{Q}_{\mathrm{sft}}(s,a)$  is initialised using another function  $Q_{\mathrm{sft}}^{\mathrm{init}}(s,a)$  (but is typically zero).
Figure 1: An illustration of the (modified)  $D$ -chain problem, where 1 is the starting state, transitions are deterministic and values next to states represent rewards for arriving in that state.
2.4 Simple regret
UCB [1] is frequently used in MCTS methods to minimise cumulative regret during the tree search. Cumulative regret is most appropriate in scenarios where the actions taken during tree search have an associated real-world cost. However, MCTS methods often use a simulator during the tree search, where the only significant real-world cost is associated with taking the recommended action after the tree search. In such scenarios, simple regret [7, 8] is more appropriate for analysing the performance of algorithms, as it only considers the cost of the actions that are actually executed. Under simple regret, algorithms are not penalised for under-exploiting during the search, thus can explore more, which leads to better recommendations by allowing algorithms to confirm that bad actions are indeed of lower value.
We consider the problem of MDP planning as a sequential decision problem, where for each round  $n$ :
1. the forecaster algorithm produces a search policy  $\pi^n$  and samples a trajectory  $\tau \sim \pi^n$ ,
2. the environment returns the rewards  $R(s_{t},a_{t})$  for each  $s_t, a_t$  pair in  $\tau$ ,
3. the forecaster algorithm produces a recommendation policy  $\psi^n$ ,
4. if environment sends stop signal, then end, else return to step 1.
The simple regret of the forecaster on it's  $n$ th round is then:
$$
\operatorname {r e g} (s, \psi^ {n}) = V ^ {*} (s) - V ^ {\psi^ {n}} (s). \tag {13}
$$
In MENTS the recommendation policy suggested in [37] can be written:
$$
\psi_ {\text {M E N T S}} (s) = \underset {a \in \mathcal {A}} {\arg \max } \hat {Q} _ {\text {s f t}} (s, a). \tag {14}
$$
We can now formally define consistency: an algorithm is consistent if and only if its recommendation policy  $\psi^n$  converges to an expected simple regret of zero:  $\mathbb{E}[\mathrm{reg}(s,\psi^n)]\to 0$  as  $n\to \infty$ . Note that because we are considering randomised algorithms, there is a distribution over the possible recommendation policies that could have been produced. If a policy has a simple regret of zero then it implies it is an optimal policy.
3 Limitations of prior MCTS methods
In this section we use the  $D$ -chain problem introduced in [9] (Figure 1) to highlight the limitations of UCT and MENTS. In the D-chain problem, when an agent chooses action  $a_{L}$ , from some state  $d$ , it moves to an absorbing state and receives a reward of  $(D - d) / D$ . In state  $D$ , action  $a_{R}$  corresponds to an absorbing state with reward  $R_{f} = 1$ . The optimal standard policy always selects action  $a_{R}$ .
In the 10-chain problem  $(D = 10)$ , UCT will recommend action  $a_{L}$  from state 1 (Figure 2a). UCT requires  $\Omega (\exp (\dots \exp (1)\dots))$  many trials  $(D$  composed exponential functions) to recommend the optimal policy that reaches the reward of  $R_{f} = 1$  [9]. This highlights the first limitation mentioned in Section 1: UCT quickly disregards action  $a_{R}$  at the initial state, to exploit the reward of 0.9.
When MENTS is run on the 10-chain problem, with the help of the entropy term it quickly finds the final reward of  $R_{f} = 1$  (Figure 2a). However, consider the modified 10-chain with  $R_{f} = 1 / 2$  instead. Repeated applications of Equations (4) and (5) for  $\alpha = 1$  gives the optimal soft values of  $Q_{\mathrm{sft}}^{*}(1,a_{R}) = \log (\exp (1 / 2) + \sum_{i = 0}^{8}\exp (i / 10))\approx 2.74$  and  $Q_{\mathrm{sft}}^{*}(1,a_{L}) = 0.9$ . So in the modified
(a) 10-chain.
(b) Modified 10-chain.
Figure 2: A comparison of MENTS, DENTS and UCT when run on the (modified) 10-chain.
10-chain problem, we have  $Q_{\mathrm{sft}}^*(1, a_R) > Q_{\mathrm{sft}}^*(1, a_L)$  with  $\alpha = 1$ , whereas  $Q^*(1, a_R) < Q^*(1, a_L)$ . Thus, when MENTS converges, it will recommend the wrong action with respect to the standard objective (Figure 2b), i.e. it is not consistent. The modified 10-chain is an example of Proposition 3.1, which states that MENTS will not always converge to the standard optimal policy.
Proposition 3.1. There exists an MDP  $\mathcal{M}$  and temperature  $\alpha$  such that  $\mathbb{E}[\mathrm{reg}(s_0, \psi_{\mathrm{MENTS}}^n)] \not\to 0$  as  $n \to \infty$ . That is, MENTS is not consistent.
Proof. Proof is by example with  $\alpha = 1$  in the modified 10-chain (Figure 1).
We can reduce the value of  $\alpha$  to decrease the importance of entropy in the soft objective  $\mathbb{E}\left[R\left(s_{t},a_{t}\right) + \alpha \mathcal{H}\left(\pi \left(\cdot \mid s_{t}\right)\right)\right]$ . If  $\alpha$  is small enough, then MENTS recommendations can converge to the optimal standard policy (Theorem 3.2). Hence, MENTS with a low temperature can solve the modified 10-chain problem (Figure 2b). However, in practice, a low temperature will often cause MENTS to not sufficiently explore, as demonstrated in the original D-chain (Figure 2a).
Theorem 3.2. For any MDP  $\mathcal{M}$ , after running  $n$  trials of the MENTS algorithm with  $\alpha \leq \Delta_{\mathcal{M}} / 3H\log |\mathcal{A}|$ , there exists constants  $C,k > 0$  such that:  $\mathbb{E}[\mathrm{reg}(s_0,\psi_{\mathrm{MENTS}})]\leq C\exp (-kn)$  where  $\Delta_{\mathcal{M}} = \min \{Q^{*}(s,a) - Q^{*}(s,a^{\prime})|Q^{*}(s,a)\neq Q^{*}(s,a^{\prime}),s\in \mathcal{S},a,a^{\prime}\in \mathcal{A},t\in \mathbb{N}\}$ .
Proof outline. We can show  $\hat{Q}_{\mathrm{sft}}(s,a)\xrightarrow{p}Q_{\mathrm{sft}}^{*}(s,a)$  (Corollary E.12.1) similarly to Theorem 4.2. The bound on  $\alpha$  is required to ensure that  $\pi^{\ast}(s_0) = \pi_{\mathrm{sft}}^{\ast}(s_0)$ .
In conclusion, similar MDPs can require vastly different temperatures for MENTS to be effective. We discuss MENTS sensitivity to the temperature parameter further in Appendix D.2, and demonstrate this parameter sensitivity in the Frozen Lake environment (Section 5.1) in Figure 27 in the appendix.
4 Boltzmann search
We now introduce two algorithms that utilise Boltzmann search policies similar to MENTS and admit bounded simple regrets that converge to zero without restrictive constraints on parameters. Thus, they do not suffer from sensitivity to parameter selection that MENTS does. Both algorithms use action selection and value backups that are easy to implement and use. We designed these algorithms with consistency in mind, which in practice, means that if we run more trials then we (with high probability) will recommend a better solution (note that Proposition 3.1 implies that this is not always the case for MENTS).
4.1 Boltzmann Tree Search
Our first approach, put simply, replaces the use of soft values in MENTS with Bellman values. We call this algorithm Boltzmann Tree Search (BTS). BTS promotes exploration through the stochastic
Boltzmann search policy, like MENTS, while using backups that optimise for the standard objective, like UCT. The search policy  $\pi_{\mathrm{BTS}}$  and backups for the  $n$ th trial are given by:
$$
\pi_ {\mathrm {B T S}} (a | s) = \left(1 - \lambda_ {s}\right) \rho_ {\mathrm {B T S}} (a | s) + \frac {\lambda_ {s}}{| \mathcal {A} |}, \tag {15}
$$
$$
\rho_ {\mathrm {B T S}} (a | s) \propto \exp \left(\frac {1}{\alpha} (\hat {Q} (s, a))\right), \tag {16}
$$
$$
\hat {Q} \left(s _ {t}, a _ {t}\right) \leftarrow R \left(s _ {t}, a _ {t}\right) + \sum_ {s ^ {\prime} \in \operatorname {S u c c} \left(s _ {t}, a _ {t}\right)} \left(\frac {N \left(s ^ {\prime}\right)}{N \left(s _ {t} , a _ {t}\right)} \hat {V} \left(s ^ {\prime}\right)\right), \tag {17}
$$
$$
\hat {V} (s _ {t}) \leftarrow \max  _ {a \in \mathcal {A}} \hat {Q} (s _ {t}, a), \tag {18}
$$
for  $t = h - 1, \dots, 0$ , where  $\hat{V}$  and  $\hat{Q}$  are the current Bellman (Q-)value estimates,  $\lambda_s = \min(1, \epsilon / \log(e + N(s)))$ ,  $\epsilon \in (0, \infty)$  is an exploration parameter and  $\alpha$  is a search temperature (unrelated to entropy). Each  $\hat{V}(s)$  and  $\hat{Q}(s, a)$  are initialised using  $V^{\mathrm{init}}$  and  $Q^{\mathrm{init}}$  functions similarly to MENTS. The Bellman values are used for recommendations:
$$
\psi_ {\mathrm {B T S}} (s) = \underset {a \in \mathcal {A}} {\arg \max } \hat {Q} (s, a). \tag {19}
$$
By using Bellman backups, we can guarantee that the BTS recommendation policy converges to the optimal standard policy for any temperature  $\alpha$ , given enough time. In other words, BTS is consistent.
Theorem 4.1. For any MDP  $\mathcal{M}$ , after running  $n$  trials of the BTS algorithm with a root node of  $s_0$ , there exists constants  $C, k > 0$  such that for all  $\varepsilon > 0$  we have  $\mathbb{E}[\mathrm{reg}(s_0, \psi_{\mathrm{BTS}})] \leq C \exp(-kn)$  and also  $\hat{V}(s_0) \xrightarrow{p} V^*(s_0)$  as  $n \to \infty$ .
Proof outline. This result is a special case of Theorem 4.2 by setting  $\beta(m) = 0$ .
4.2 Decaying Entropy Tree Search
Secondly, we present Decaying ENtropy Tree Search (DENTS), which can effectively interpolate between the MENTS and BTS algorithms. DENTS also uses the dynamic programming backups from equations (17) and (18), but adds an entropy backup. The entropy values are weighted by a bounded non-negative function  $\beta(N(s))$  in the DENTS search policy  $\pi_{\mathrm{DENTS}}$ :
$$
\pi_ {\text {D E N T S}} (a | s) = \left(1 - \lambda_ {s}\right) \rho_ {\text {D E N T S}} (a | s) + \frac {\lambda_ {s}}{| \mathcal {A} |}, \tag {20}
$$
$$
\rho_ {\text {D E N T S}} (a | s) \propto \exp \left(\frac {1}{\alpha} \left(\hat {Q} (s, a) + \beta (N (s)) \mathcal {H} _ {Q} (s, a)\right)\right), \tag {21}
$$
$$
\mathcal {H} _ {V} \left(s _ {t}\right) \leftarrow \mathcal {H} \left(\pi_ {\text {D E N T S}} (\cdot | s _ {t})\right) + \sum_ {a \in \mathcal {A}} \pi_ {\text {D E N T S}} (a | s _ {t}) \mathcal {H} _ {Q} \left(s _ {t}, a\right), \tag {22}
$$
$$
\mathcal {H} _ {Q} \left(s _ {t}, a _ {t}\right) \leftarrow \sum_ {s ^ {\prime} \in \operatorname {S u c c} \left(s _ {t}, a _ {t}\right)} \frac {N \left(s ^ {\prime}\right)}{N \left(s _ {t} , a _ {t}\right)} \mathcal {H} _ {V} \left(s ^ {\prime}\right), \tag {23}
$$
for  $t = h - 1, \dots, 0$ , where  $\mathcal{H}_V(s)$  and  $\mathcal{H}_Q(s, a)$  are the entropy values of the search policy rooted at  $s$  and  $(s, a)$  respectively, and  $\alpha, \lambda_s$  are the same as for BTS, as described in Section 4.1. Initial values are the same as Section 4.1, and the entropy values are initialised to zero. In DENTS we can view  $\hat{Q}(s, a) + \beta(N(s))\mathcal{H}_Q(s, a)$  as a soft value for  $(s, a)$ . Hence, by setting  $\beta(m) = \alpha$ , the DENTS search will mimic the MENTS search (demonstrated in Appendix D.4), and if  $\beta(m) = 0$  then the algorithm reduces to the BTS algorithm. By using a decaying function for  $\beta$  we amplify values using entropy as an exploration bonus early in the search while allowing for more exploitation later. Recommendations still use Bellman values:
$$
\psi_ {\text {D E N T S}} (s) = \underset {a \in \mathcal {A}} {\arg \max } \hat {Q} (s, a). \tag {24}
$$
Because the recommendation policy  $\psi_{\mathrm{DENTS}}$  uses the Bellman values, we can guarantee that it will converge to the optimal standard policy, and is consistent for any  $\beta$ .
Theorem 4.2. For any MDP  $\mathcal{M}$ , after running  $n$  trials of the DENTS algorithm with a root node of  $s_0$ , if  $\beta$  is a bounded function, then there exists constants  $C, k > 0$  such that for all  $\varepsilon > 0$  we have  $\mathbb{E}[\mathrm{reg}(s_0, \psi_{\mathrm{DENTS}})] \leq C \exp(-kn)$ , and also  $\hat{V}(s_0) \xrightarrow{p} V^*(s_0)$  as  $n \to \infty$ .