SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof outline. Let  $0 < \eta < \pi_{\mathrm{DENTS}}(a|s)$  for all  $s, a$ , which exists because  $\exp(\cdot) > 0$  and  $1 / |\mathcal{A}| > 0$  in Equation 20 (Lemma E.1), which can be used with Hoeffding bounds to show for appropriate constants and any event  $E$  that  $\{\operatorname*{Pr}(E) \leq C_0 \exp(-k_0 \varepsilon^2 N(s_t))\}$  iff  $\{\operatorname*{Pr}(E) \leq C_1 \exp(-k_1 \varepsilon^2 N(s_t, a_t))\}$  iff  $\{\operatorname*{Pr}(E) \leq C_2 \exp(-k_2 \varepsilon^2 N(s_{t+1}))\}$ . We can then show  $\operatorname*{Pr}(|\hat{Q}(s_0, a) - Q^*(s_0, a)| > \varepsilon) < C_3 \exp(-k_3 \varepsilon^2 n)$  by induction, where the base case  $\hat{V}(s_{H+1}) = V^*(s_{H+1}) = 0$  holds vacuously (Lemmas E.10, E.16 and Theorem E.17). Let  $\Delta_{\mathcal{M}}$  be a small constant (Equation (122)) such that  $\forall s, a. |\hat{Q}(s_0, a) - Q^*(s_0, a)| \leq \Delta_{\mathcal{M}} / 2 \Rightarrow \arg \max_a \hat{Q}(s, a) = \arg \max_a Q^*(s, a)$ . Setting  $\varepsilon = \Delta_{\mathcal{M}} / 2$  gives a bound on  $\operatorname*{Pr}(\psi_{\mathrm{DENTS}} \neq \pi^*)$ , which can then be used in the definition of simple regret to give the result.
4.3 Using the Alias method
The Alias method [35, 34] can be used to sample from a categorical distribution with  $m$  categories in  $O(1)$  time, with a preprocessing step of  $O(m)$  time. Given any stochastic search policy, we can sample actions in amortised  $O(1)$  time, by computing an alias table every  $|\mathcal{A}|$  visits to a node, and then sampling from that table. Note that when using the Alias method we are making a trade off between using the most up to date policy and the speed of sampling actions.
In Appendix C.1 we discuss this idea in more detail, and give an informal analysis of the complexity to run  $n$  trials. BTS, DENTS and MENTS can run  $n$  trials in  $O(n(H\log (|\mathcal{A}|) + |\mathcal{A}|))$  time when using the Alias method, as opposed to the typical complexity of  $O(nH|\mathcal{A}|)$ .
4.4 Limitations and benefits
The main limitations of BTS and DENTS are as follows: (1) the DENTS decay function  $\beta$  can be nontrivial to set and tune; (2) the focus on simple regret and exploration means they are not appropriate to use when actions taken during the tree search/planning phase have a real-world cost; (3) the backups implemented directly as presented above are computationally more costly than computing the average returns that UCT uses; (4) when it is desirable for an agent to follow the maximum entropy policy, then MENTS would be preferable, for example if the agent needs to explore to learn and discover an unknown environment.
The main benefits of using a stochastic policy for action selection are: (1) they allow the Alias method (Section 4.3) to be used to speed up trials; (2) they naturally encourage exploration as actions are sampled randomly, which is useful for discovering sparse or delayed rewards and for confirming that actions with low values do in fact have a low value; and (3) the entropy of a stochastic policy can be computed and used as an exploration bonus. In Appendix C.3 we summarise and compare the differences between the algorithms considered in this work in more detail.
This section compares the proposed BTS and DENTS against MENTS and UCT on a set of goal-based MDPs and in the game of Go. For additional baselines, we also compare with the RENTS and TENTS algorithms [10], which use relative and Tsalis entropy in place of Shannon entropy respectively, and the H-MCTS algorithm [20] which combines UCT and Sequential Halving.
5.1 Gridworlds
To evaluate an algorithm with search tree  $\mathcal{T}$ , we complete the partial recommendation policy  $\psi_{\mathrm{alg}}$  as follows:
$$
\psi (a | s) = \left\{ \begin{array}{l l} 1 & \text {i f} s \in \mathcal {T} \text {a n d} a = \psi_ {\mathrm {a l g}} (s), \\ 0 & \text {i f} s \in \mathcal {T} \text {a n d} a \neq \psi_ {\mathrm {a l g}} (s), \\ \frac {1}{| \mathcal {A} |} & \text {o t h e r w i s e .} \end{array} \right. \tag {25}
$$
(a) 8x12 Frozen Lake.
(b) 6x6 Sailing Problem.
Figure 3: Results for gridworld environments. Further results are given in Appendix D.3.
We sample a number of trajectories from  $\psi$ , and take the average return to estimate  $V^{\psi}$ . Although we are evaluating the algorithms in an offline planning setting, it still indicates how the algorithms perform in an online setting where we interleave planning in simulation with letting the agent act.
5.1.1 Domains
To validate our approach, we use the Frozen Lake environment [5], and the Sailing Problem [28], commonly used to evaluate tree search algorithms [28, 22, 33, 13]. We chose these environments to compare our algorithms in a domain with a sparse and dense reward respectively.
The (Deterministic) Frozen Lake is a grid world environment with one goal state. The agent can move in any cardinal direction at each time step, and walking into a wall leaves the agent in the same location. Trap states exist where the agent falls into a hole and the trial ends. If the agent arrives at the goal state after  $t$  timesteps, then a reward of  $0.99^t$  is received.
The Sailing Problem is a grid world environment with one goal state, at the opposite corner to the starting location of the agent. The agent has 8 different actions to travel each of the 8 adjacent states. In each state, the wind is blowing in a given direction and will stochastically change after every transition. The agent cannot sail directly into the wind. The cost of each action depends on the tack, the angle between the direction of the agent's travel and the wind.
5.1.2 Results
We used an 8x12 Frozen Lake environment and a 6x6 Sailing Problem for evaluation, more environment details are given in Appendix D.1. Parameters were selected using a hyper-parameter search (Appendix D.3). Each algorithm is run 25 times on each environment and evaluated every 250 trials using 250 trajectories. A horizon of 100 was used for Frozen Lake and 50 for the Sailing Problem.
In Frozen Lake (Figure 3a), entropy proved to be a useful exploration bonus for the sparse reward. Values in UCT and BTS remain at zero until a trial successfully reaches the goal. However, entropy guides agents to avoid trap states, where the entropy is zero. DENTS was able to perform similarly to MENTS, and BTS was able to improve its policy over time more than UCT.
In the Sailing Problem (Figure 3b) UCT performs well due to the dense reward. BTS and DENTS also manage to keep up with UCT. MENTS and TENTS appear to be slightly hindered by entropy in this environment. The relative entropy encourages RENTS to pick the same actions over time, so it tends to pick a direction and stick with it regardless of cost.
Finally, BTS and DENTS were able to perform well in both domains with a sparse and dense reward structure, whereas the existing methods performed better on one than the other, hence making BTS and DENTS good candidates for a general purpose MCTS algorithm.
5.2 Go
For a more challenging domain we ran a round-robin tournament using the game of Go, which has widely motivated the development of MCTS methods [14, 30, 32]. In each match, each algorithm played 50 games as black and 50 as white. Area scoring is used to score the games, with a komi of 7.5. We used an openly available value network  $\tilde{V}$  and policy network  $\tilde{\pi}$  from KataGo [36]. Our baseline was the PUCT algorithm [2], as described in Alpha Go Zero [32] using prioritised UCB [29] to utilise the policy neural network. Each algorithm was limited to 5 seconds of compute time per move, allowed to use 32 search threads per move, and had access to 80 Intel Xeon E5-2698V4 CPUs clocked at  $2.2\mathrm{GHz}$ , and a single Nvidia V100 GPU on a shared compute cluster.
To use Boltzmann search in Go, we adapted the algorithms to account for an opponent that wishes to minimise the value of a two-player game. This is achieved by appropriately negating values used in the search policy and backups, which is described precisely in Appendix C.2.
Additionally, we found that adapting the algorithms to use average returns (recall Equation (8)) outperformed using Bellman backups for Go (Appendix D.5.1). The Bellman backups were sensitive to and propagated noise from the neural network evaluations. We use the prefix 'AR' to denote the algorithms using average returns, such as AR-DENTS. Full details for these algorithms are given in Appendix B.
5.2.1 Using neural networks with Boltzmann search
This section describes how to use value and policy networks in BTS. Adapting MENTS and DENTS are similar (Appendix C.2). Values can be initialised with the neural networks as  $\hat{Q}(s, a) \gets \log \tilde{\pi}(a|s) + B$  and  $\hat{V}(s) \gets \tilde{V}(s)$ , where  $B$  is a constant (adapted from Xiao et al. [37]). With such an initialisation, the initial BTS policy is  $\rho_{\mathrm{BTS}}(a|s) \propto \tilde{\pi}(a|s)^{1/\alpha}$ . For these experiments we set a value of  $B = \frac{-1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} \log \tilde{\pi}(a|s)$ . Additionally, the stochastic search policy naturally lends itself to mixing in a prior policy, so we can replace BTS search policy  $\pi_{\mathrm{BTS}}$  (Equation (15)) with  $\pi_{\mathrm{BTS,mix}}$ :
$$
\begin{array}{l} \pi_ {\mathrm {B T S}, \operatorname {m i x}} (a | s) = \lambda_ {\tilde {\pi}} \tilde {\pi} (a | s) + (1 - \lambda_ {\tilde {\pi}}) \pi_ {\mathrm {B T S}} (a | s) (26) \\ = \lambda_ {\tilde {\pi}} \tilde {\pi} (a | s) + (1 - \lambda_ {\tilde {\pi}}) (1 - \lambda_ {s}) \rho_ {\mathrm {B T S}} (a | s) + \frac {(1 - \lambda_ {\tilde {\pi}}) \lambda_ {s}}{| \mathcal {A} |}, (27) \\ \end{array}
$$
where  $\lambda_{\bar{\pi}} = \min (1,\epsilon_{\bar{\pi}} / \log (e + N(s)))$  , and  $\epsilon_{\bar{\pi}}\in (0,\infty)$  controls the weighting for the prior policy.
Results of the round-robin are summarised in Table 1, and we discuss how parameters were selected in Appendix D.5.1. BTS was able to run the most trials per move and beat all of the other algorithms other than DENTS which it drew. We used the optimisations outlined in Appendix C.1 which allowed the Boltzmann search algorithms to run significantly more trials per move than PUCT. BTS and DENTS were able to beat PUCT with results of 57-43 and 58-42 respectively. Using entropy did not seem to have much benefit in these experiments, as can be witnessed by MENTS only beating TENTS, and DENTS drawing 50-50 with BTS. This is likely because the additional exploration provided by entropy is vastly outweighed by utilising the information contained in the neural networks  $\tilde{V}$  and  $\tilde{\pi}$ . Interestingly RENTS had the best performance out of the prior works, losing 43-57 to PUCT, and the use of relative entropy appears to take advantage of a heuristic for Go that the RAVE [15] algorithm used: the value of a move is typically unaffected by other moves on the board.
To validate the strength of our PUCT agent, we also compared it directly with KataGo [36], limiting each algorithm to 1600 trials per move. Our PUCT agent won 61-39 in 9x9 Go, and lost 35-65 in 19x19 Go, suggesting that our PUCT agent is strong enough to provide a meaningful comparison for our other general purpose algorithms. Finally, note that we did not fine-tune the neural networks, so the Boltzmann search algorithms directly used the networks that were trained for use in PUCT.
UCT [22, 23] is a widely used variant of MCTS. Polynomial UCT [2], replaces the logarithmic term in UCB with a polynomial one (such as a square root), which has been further popularised by its use
Black \White	PUCT	AR-M	AR-R	AR-T	AR-B	AR-D	Trials/move
PUCT	-	33-17	27-23	42-8	17-33	15-35	1054
AR-MENTS	12-48	-	13-37	38-12	10-40	12-38	4851
AR-RENTS	20-30	24-26	-	39-11	18-32	14-36	3672
AR-TENTS	8-42	11-39	9-41	-	6-44	10-40	5206
AR-BTS	25-25	35-15	31-19	34-16	-	15-35	5375
AR-DENTS	23-27	36-14	29-21	36-14	15-35	-	4677
Table 1: Results for the Go round-robin tournament. The first column gives the agent playing as black. The final column gives the average trials run per move across the entire round-robin. In the top row, we abbreviate the algorithm names for space.
in Alpha Go and Alpha Zero [30, 32, 31]. Coquelin and Munos introduce the Flat-UCB and BAST algorithms to adapt UCT for the D-chain problem [9]. However, we consider an alternative approach for search in MCTS rather than adapting UCB.
Maximum entropy policy optimization methods are well-known in the reinforcement learning literature [16, 17, 38]. MENTS [37] is the first method to combine the principle of maximum entropy and MCTS. Kozakowski et al. [25] extend MENTS to arrive at Adaptive Entropy Tree Search (ANTS), adapting parameters throughout the search to match a prescribed entropy value. Dam et al. [10] also extend MENTS using Relative and Tsallis entropy to arrive at the RENTS and TENTS algorithms. Our work is closely related to MENTS, however, we focus on reward maximisation and consider how entropy can be used in MCTS without altering our planning objective.
Bubeck et al. [7, 8] introduce simple regret in the context of multi-armed bandit problems (MABs). They alternate between pulling arms for exploration and outputting a recommendation. They show for MABs that a uniform exploration produces an exponential bound on the simple regret of recommendations. We use simple regret, but in the context of sequential decision-making, to analyse the convergence of MCTS algorithms.
Tolpin and Shimony [33] extend simple regret to MDP settings, showing an  $O(\exp(-\sqrt{n}))$  bound on simple regret after  $n$  trials by adapting UCT. The subsequent work of Hay et al. [19] extends [33] to consider a metalevel decision problem, incorporating computation costs into the objective. Pepels et al. [27] introduce a Hybrid MCTS (H-MCTS) motivated by the notion of simple regret. H-MCTS uses a mixture of Sequential Halving [20], and UCT. Feldman et al. [13, 11, 12] introduce the Best Recommendation with Uniform Exploration (BRUE) algorithm. BRUE splits trials up to explicitly focus on exploration and value estimation one at a time. BRUE achieves an exponential bound  $O(\exp(-n))$  on simple regret after  $n$  trials [13]. Prior work that considers simple regret in MCTS has focused on adaptations to UCT, whereas this work focuses on algorithms that sample actions from Boltzmann distributions, rather than using UCB for action seclusion.
We considered the recently introduced MENTS algorithm, compared and contrasted it to UCT, and discussed the limitations of both. We introduced two new algorithms, BTS and DENTS, that are consistent, converge to the optimal standard policy, while preserving the benefits that come with using a stochastic Boltzmann search policy. Finally, we compared our algorithms in gridworld environments and Go, demonstrating the performance benefits of utilising the Alias method, that entropy can be a useful exploration bonus with sparse rewards, and more generally, demonstrating the advantage of prioritising exploration in planning, by using Boltzmann search policies.
An interesting area of future work may include investigating good heuristics for setting parameters in BTS and DENTS. We noticed that the best value for the search temperature tended to be the same order of magnitude as the optimal value at the root node  $V^{*}(s_{0})$ , which suggests a heuristic similar to [21] may be reasonable.
[24] Andrey Kolobov. Planning with Markov Decision Processes: An AI Perspective, volume 6. Morgan & Claypool Publishers, 2012.
[25] Piotr Kozakowski, Mikołaj Pacek, and Piotr Miłos. Planning and learning using adaptive entropy tree search. arXiv preprint arXiv:2102.06808, 2021.
[26] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
[27] Tom Pepels, Tristan Cazenave, Mark HM Winands, and Marc Lanctot. Minimizing simple and cumulative regret in monte-carlo tree search. In Workshop on Computer Games, pages 1-15. Springer, 2014.
[28] Laurent Péret and Frédérique Garcia. On-line search for solving markov decision processes via heuristic sampling. learning, 16:2, 2004.
[29] Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61(3):203–230, 2011.
[30] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
[31] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-1144, 2018.
[32] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
[33] David Tolpin and Solomon Shimony. Mcts based on simple regret. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 570-576, 2012.
[34] Michael D Vose. A linear algorithm for generating random numbers with a given distribution. IEEE Transactions on software engineering, 17(9):972-975, 1991.
[35] Alastair J Walker. New fast method for generating discrete random numbers with arbitrary frequency distributions. *Electronics Letters*, 8(10):127-128, 1974.
[36] David J Wu. Accelerating self-play learning in go. arXiv preprint arXiv:1902.10565, 2019.
[37] Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems, 32, 2019.
[38] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. 2008.