SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overall, we experimentally demonstrated that our novel proposed method can significantly improve the performance of our agent and its planning capabilities without penalising the reasoning time. We also demonstrated that our proposal can improve the decision-making process by using only the information generated during the tree search process and, in contrast to  $\rho$ -POMCP, without relying on the explicit representation of latent functions (e.g., the observation function).
Ablation study - To evaluate the impact of our proposed modifications and enhancements, we performed an ablation study over our method. We consider 2 different variations of IB-POMCP that partially implement its key points in this experiment: (a) the Information-guided Particle Reinvigoration POMCP (IPR-POMCP), which implements the modifications to the particle filter reinvigoration process (explained in Section 4, steps (i) and (ii)); and (b) the I-UCB POMCP, which implements the proposed modifications to the search and simulation process (explained in Section 4, steps (iii) and (iv)). We consider 4 different scenarios, one for each environment presented in Table 1, to run our experiments. The results are depicted in Figure 2. Both additional methods proposed for the study are available in our GitHub repository together with the complete code of this work.
Figure 2: Ablation study of IB-POMCP in four different scenarios.
By analysing the graphs, it is clear that the I-UCB and implementation of an information-guided particle reinvigoration process for the Tiger, Maze, and Foraging problems directly enhance the reasoning capabilities of our planning method, which is translated here in terms of reward collection. For the Tiger and Foraging problems, diversifying the set of particles (I-UCB POMCP) alone presents improvements for POMCP; however, it has less impact than including entropy in the search process (IPR-POMCP). On the other hand, in the Maze environment, both approaches present similar improvements for the POMCP algorithm. However, when combined, they significantly improved upon the baseline results. Our intuition behind these results is that the IPR-POMCP proposal is responsible for delivering a better set of particles and a better initial estimation of the current state to the agent before simulating actions, whereas I-UCB continually guides and affects the planning process from the beginning of the simulations until the decision of the best action.
In this study, we propose IB-POMCP, a novel algorithm for planning under uncertainty that is capable of aggregating information entropy into a decision-making algorithm using our modified version of the UCB function, I-UCB. We present the theoretical properties for convergence under certain assumptions, which are supported by empirical results collected in five different domains. Overall, we increase the reward collection by up to 10 times in comparison with TB  $\rho$ -POMCP (U-shaped, F1), in addition to reducing the reasoning time by up to  $93\%$  compared to  $\rho$ -POMCP (MazeHoles).
We gratefully acknowledge the financial support provided by a PhD studentship from the Faculty of Science and Technology (FST) at Lancaster University, United Kingdom. We also thank FST for granting us access to their High-End Computing Cluster (HEC 3.0), which was pivotal to this research. We extend our sincere gratitude to Dr Mike Pacey for his expertise and assistance provided to correctly setting up HEC 3.0 and running our experiments. Our special thanks go to Dr Ronghui Mu for providing insightful comments and feedback on an earlier draft of this paper.
[1] D. Agudelo-Espana, S. Gomez-Gonzalez, S. Bauer, B. Scholkopf, and J. Peters. Bayesian online prediction of change points. In Conference on Uncertainty in Artificial Intelligence, pages 320-329. PMLR, 2020.
[2] S. V. Albrecht and P. Stone. Reasoning about hypothetical agent behaviours and their parameters. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS '17, page 547-555, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
[3] S. Amini, M. Palhang, and N. Mozayani. POMCP-based decentralized spatial task allocation algorithms for partially observable environments. Applied Intelligence, 53(10):12613-12631, 2023.
[4] M. Araya, O. Buffet, V. Thomas, and F. Charpillet. A POMDP extension with belief-dependent rewards. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23, 2010.
[5] A. Bai, F. Wu, Z. Zhang, and X. Chen. Thompson sampling based monte-carlo planning in POMDPs. In International Conference on Automated Planning and Scheduling (ICAPS), page 29-37. AAAI Press, 2014. ISBN 9781577356608.
[6] A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial observability. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 44-52, 2022.
[7] S. Barrett, A. Rosenfeld, S. Kraus, and P. Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-171, 2017.
[8] S. R. Chowdhury, R. Oliveira, and F. Ramos. Active learning of conditional mean embeddings via bayesian optimisation. In Conference on Uncertainty in Artificial Intelligence, pages 1119-1128. PMLR, 2020.
[9] M. A. do Carmo Alves, A. Varma, Y. Elkhatib, and L. Soriano Marcolino. AdLeap-MAS: An open-source multi-agent simulator for ad-hoc reasoning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1893-1895, 2022.
[10] D. Fox. Adapting the sample size in particle filters through kld-sampling. The international Journal of robotics research, 22(12):985-1003, 2003.
[11] M. Ghavamzadeh, S. Mannor, J. Pineau, A. Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359–483, 2015.
[12] M. Gini. Multi-robot allocation of tasks with temporal and ordering constraints. In AAAI Conference on Artificial Intelligence, 2017.
[13] A. Guez, D. Silver, and P. Dayan. Efficient Bayes-adaptive reinforcement learning using sample-based search. International Conference on Neural Information Processing Systems, 25, 2012.
[14] A. Hayashi, D. Ruiken, T. Hasegawa, and C. Goerick. Reasoning about uncertain parameters and agent behaviors through encoded experiences and belief planning. Artif. Intell., 280(C), Mar. 2020. ISSN 0004-3702.
[33] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550 (7676):354-359, 2017.
[34] A. Somani, N. Ye, D. Hsu, and W. S. Lee. DESPOT: Online POMDP planning with regularization. Advances in neural information processing systems, 26, 2013.
[35] Z. N. Sunberg and M. J. Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. In International Conference on Automated Planning and Scheduling, 2018.
[36] V. Thomas, G. Hutin, and O. Buffet. Monte Carlo information-oriented planning. In European Conference On Artificial Antigellence, 2020.
[37] C. Xiao, R. Huang, J. Mei, D. Schuurmans, and M. Müller. Maximum entropy Monte-Carlo planning. Advances in Neural Information Processing Systems, 32, 2019.
[38] A. Yadav, R. Noothigattu, E. Rice, L. Onasch-Vera, L. Soriano Marcolino, and M. Tambe. Please be an influencer? contingency-aware influence maximization. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), page 1423-1431, 2018.
[39] E. S. Yourdshahi, T. Pinder, G. Dhawan, L. S. Marcolino, and P. Angelov. Towards large scale ad-hoc teamwork. In International Conference on Agents (ICA), pages 44-49. IEEE, 2018.
[40] W. J. Zhou, B. Subagdja, A.-H. Tan, and D. W.-S. Ong. Hierarchical control of multi-agent reinforcement learning team in real-time strategy (RTS) games. Expert Systems with Applications, 186:115707, 2021.
[41] M. Zuccotto, A. Castellini, and A. Farinelli. Learning state-variable relationships for improving POMCP performance. In ACM/SIGAPP Symposium on Applied Computing (SAC), pages 739-747, 2022.
In this section, we provide the pseudo-code of the particle filter update step (Algorithm 1), and for IB-POMCP's main routine (Algorithm 2). To clarify our approach better and support the text presented in the main paper, both algorithms highlight the differences between our approach and POMCP [32]. See Section 3 in the main paper for further details about our rationale and methodology.
Algorithm 1 Particle Filter Update. The starred lines (with a light grey background) highlight the differences between POMCP and our proposal, IB-POMCP. Union considers repetition here.
1: procedure PARTICLEFILTERUPDATE( $F_{\mathfrak{r}}$ ,  $\tilde{P}(z|ha)$ ,  $k$ )  
2:  $\hat{\mathbf{B}} \gets \mathcal{F}_{\mathfrak{r}}, \mathcal{F}_{\mathfrak{r}} \gets \{\}$  ▷  $\hat{\mathbf{B}}$  is a copy of  $F_{\mathfrak{r}}$   
*3: while  $|\mathcal{F}_{\mathfrak{r}}| < \tilde{P}(z|ha)k$  do  
*4:  $b \sim \hat{\mathbf{B}}, \mathcal{F}_{\mathfrak{r}} \gets \mathcal{F}_{\mathfrak{r}} \cup b$  ▷ Adding  $\tilde{P}(z|ha)k$  particles to the new  $F_{\mathfrak{r}}$   
*5: while  $|\mathcal{F}_{\mathfrak{r}}| < k$  do  
*6:  $b \sim \mathcal{U}_z, \mathcal{F}_{\mathfrak{r}} \gets \mathcal{F}_{\mathfrak{r}} \cup b$  ▷ Adding  $(1 - \tilde{P}(z|ha))k$  particles to the new  $F_{\mathfrak{r}}$   
*7: return  $F_{\mathfrak{r}}$
Algorithm 1 Particle Filter Update - Details and Further Discussion: We save some space here with the intention of assuring the reader's understanding of the novelty that lies on our particle filter update proposal. Our method employs a dynamic threshold derived from the dynamic value of  $\tilde{P}(z|ha)$  (or specifically,  $1 - \tilde{P}(z|ha)$ ) that determines the ratio of uniform samples in the reinvigorated particle filter. Unlike some traditional approaches which use a fixed value for that threshold, or some other state-of-the-art contributions, which mostly rely on additional knowledge and expensive calculations [17, 14, 10], IB-POMCP does not need an explicit true distribution of observations, transition function or training data in this process by using our proposed  $\tilde{P}(z|h_{\mathrm{r}}a)$  probability.
It is important to note that the action selection process within the planning phase has a direct impact on the  $\tilde{P}(z|h_{\mathfrak{r}}a)$  value. Therefore, the strategy used to choose actions for simulation during planning will affect the particle filter, since it affects the dynamic threshold  $\tilde{P}(z|h_{\mathfrak{r}}a)$ . Hence, note that the I-UCB equation used in the search tree also has an impact on belief tracking.
Our suggested algorithm to investigate the impact of this proposal (Ablation study in Section 6), IPR-POMCP, employs our dynamic threshold and shows significant improvement to the conventional POMCP, which follows a standard particle reinvigoration paradigm (with a fixed parameter for reinvigoration). Hence, just using the dynamic threshold  $\tilde{P}(z|h_{\mathrm{r}}a)$  improves results. Furthermore, the ablation study indicates that combining the dynamic threshold with the I-UCB equation yields even more favourable results and that using solely the I-UCB equation without the dynamic threshold (IUCB-POMCP baseline) results in worse outcomes than the full IB-POMCP approach. Hence, both the dynamic threshold and the I-UCB equation are important to improve results, and their simultaneous application produces even greater enhancements than when employed individually.
What is perhaps still left open is whether the I-UCB equation leads to better  $\tilde{P}(z|h_{\mathfrak{r}}a)$  values than the UCB equation. Since our approach takes the estimated entropy into consideration for action selection we believe that I-UCB may lead to better threshold values.
For example, in cases where a node possesses high entropy and no reward in the reasoning horizon, the I-UCB equation could prioritise visiting it more often than the UCB equation would, increasing the  $\tilde{P}(z|h_{\mathfrak{r}}a)$  value for that particular node (note that it will not imply in worse probability estimation because the entropy is based on the estimation of observations and the particle filter gives the probabilities of states). The high entropy indicates that the node accommodates a more diverse set of particles in the particle filter and hence sampling particles from a uniform distribution during particle reinvigoration (to increase diversity) becomes less useful. Hence, the implications of the I-UCB equation appear favourable as it leads to fewer particle samples from the uniform distribution in this case (as we use the ratio  $1 - \tilde{P}(z|h_{\mathfrak{r}}a)$ ). On the other hand, if a node has low entropy, the particle filter may easily get biased to a small set of states. In this case, I-UCB would lead to a higher number of particles sampled from the uniform distribution than the UCB equation in the particle reinvigoration,
since it would have a lower  $\tilde{P}(z|h_{\mathfrak{r}}a)$  value in I-UCB. Therefore, we believe that I-UCB should be better than UCB for defining the dynamic threshold  $\tilde{P}(z|h_{\mathfrak{r}}a)$ , and hence better in belief tracking.
Algorithm 2 IB-POMCP's Planning. The starred lines in a light grey background highlight the difference between POMCP and our proposal. We suggest the reader to Silver and Veness (2010) [32] for further details.  $\gamma$  is the historical discount factor,  $depth_{max}$  is the maximum depth for the tree, and  $z_{h}$  represents the last observation found (associated with node  $h$ ).
1: procedure SEARCH(hr) 1: procedure SIMULATE(s,h,d,α)  
2: while Timeout() is False do 2: if d < depthmax then  
3: if  $\mathcal{F}_{\mathrm{r}} = \emptyset$  then s ~ Uz  
4: else s ~ F  
*5: α := e ln(N(hr)) ∑i=1N(hr) H(i(hr) / N(hr) max H(i(hr) i=1  
*6: Simulate(s,hr,0,α)  
*7: return arg max (1 - α)V(ha) + αH(ha) a∈A  
8: (s',z,r) ~ G(s,a)  
*9: r', Zm t+1 ← Simulate(s', haz,d+1,α)  
10: Fh ← Fh ∪ {s}  
11: N(h) ← N(h) + 1  
12: N(ha) ← N(ha) + 1  
13: V(ha) ← V(ha) + R-V(ha)/N(ha)  
1: procedure EXPAND_NODE(h)  
2: for a ∈ A do  
3: T(ha) ← (ha, Vinit(ha), Ninit(ha), Φ)  
1: procedure ROLLOUT(s,h,d,γ)  
2: if d < depthmax then return 0  
3: a ~ πrollout(h,·)  
4: (s',z,r) ~ G(s,a)  
5: return r + γ Rollout(s', haz,d+1)  
*17: return R, Zm t+1 ∪ z
B Detailed Theoretical Analysis
In this section, we offer a comprehensive theoretical analysis of our proposed method. In addition to the assumption, lemma, theorems, and corollary presented in the main paper, our complete analysis encompasses all proofs, along with our definition (Definition 1) and lemma (Lemma 2).
Definition 1 An estimated  $\hat{\mathcal{V}}(s)$  is said to be  $\epsilon$ -optimal if  $|\hat{\mathcal{V}}(s) - \mathcal{V}^*(s)| \leq \epsilon$ . From the literature, actions taken from the result of  $\epsilon$ -optimal value function estimations are known as  $\epsilon$ -optimal actions.
Assumption 2 Given a generic  $POMDP$  represented by the tuple  $(\mathbf{S},\mathbf{A},\mathcal{R},\mathcal{T},\mathbf{Z},\mathcal{Z},\mathbf{H})$ , the  $\mathcal{R}$  is bounded and there exists a positive maximum reward value  $r_{max} \in \mathcal{R}$ .
Lemma 1  $\alpha(h) \in [0,1]$  converges to  $0$  as the number of visits  $\mathcal{N}(h)$  approaches the infinity. Mathematically,  $\lim_{\mathcal{N}(h) \to \infty} \alpha(h) = 0$ .
Proof: Considers our equation for a root node with history  $h$ :
$$
\alpha (h) = \frac{e\ln(\mathcal{N}(h))}{\mathcal{N}(h)}\frac{\sum_{i = 1}^{\mathcal{N}(h)}\mathcal{H}_{i}(h)}{\mathcal{N}(h)\max_{i = 1}^{\mathcal{N}(h)}\mathcal{H}_{i}(h)}
$$
First, we can re-organise the equation as follows:
$$
\alpha (h) = \frac {e \ln (\mathcal {N} (h))}{\mathcal {N} (h)} \frac {\sum_ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}{\mathcal {N} (h) \max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)} = \frac {e \ln (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} \frac {\sum_ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}{\max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}
$$
Analysing the limit over the new expression, we can find that:
$$
\lim _ {\mathcal {N} (h) \to \infty} \frac {e \ln (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} \frac {\sum_ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}{\max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)} = 0
$$
by following the rationale:
$$
0 \leq \sum_ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h) \leq \mathcal {N} (h) \max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h) \Rightarrow 0 \leq \alpha (h) \leq \frac {e \ln (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} \frac {\mathcal {N} (h) \max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}{\max _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}
$$
Therefore, since we know that:
$$
\lim  _ {\mathcal {N} (h)) \rightarrow \infty} \frac {e l n (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} \frac {\mathcal {N} (h) \max  _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)}{\max  _ {i = 1} ^ {\mathcal {N} (h)} \mathcal {H} _ {i} (h)} =
$$
$$
\frac {e l n (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} \mathcal {N} (h) = e \frac {\mathcal {N} (h) l n (\mathcal {N} (h))}{\mathcal {N} (h) ^ {2}} = 0, \text {b y L ’ H o p i t a l ’ s R u l e}
$$
using the sandwich theorem, we find that  $\alpha (h)\to 0$  as  $\mathcal{N}(h)\to \infty$
Now, we introduce Lemma 2, which will be used as a key result that supports Theorem 1. In other words, we will use Lemma 2's proof to reach a contradiction later in order to prove Theorem 1.
Lemma 2 Assume an action  $a_{i}$  that is taken a finite number of times and  $a_{j}$  that is taken infinitely during IB-POMCP's search for any node  $h$ . There  $\exists t'' > t'$  such that  $I-UCB_{t''}(ha_{i}) \geq I-UCB_{t''}(ha_{j})$  where  $t'$  is a finite iteration number after which  $a_{i}$  is never taken.
Proof: Let us assume that there exists an iteration time  $t$  that is greater than  $t'$  where  $a_i$  is never taken again for simulation. Consider the expression  $\mathrm{I - UCB}_t(ha_i) - \mathrm{I - UCB}_t(ha_j)$ , which is equal to:
$$
\begin{array}{l} \mathrm {I - U C B} _ {t} (h a _ {i}) - \mathrm {I - U C B} _ {t} (h a _ {j}) = \left[ \mathcal {V} _ {t} (h a _ {i}) - \mathcal {V} _ {t} (h a _ {j}) \right] + \\ (1 - \alpha_ {t} (h)) \sqrt {\ln (\mathcal {N} _ {t} (h))} \left[ \frac {1}{\sqrt {\mathcal {N} _ {t} (h a _ {i})}} - \frac {1}{\sqrt {\mathcal {N} _ {t} (h a _ {j})}} \right] + \alpha_ {t} (h) \left[ \hat {\mathcal {H}} _ {t} (h a _ {j}) - \hat {\mathcal {H}} _ {t} (h a _ {i}) \right] \\ \end{array}
$$
Trivially,  $\left(\frac{1}{\sqrt{\mathcal{N}_t(ha_i)}} - \frac{1}{\sqrt{\mathcal{N}_t(ha_j)}}\right) \to \frac{1}{\sqrt{\mathcal{N}_{t'}(ha_i)}}$  as  $\mathcal{N}_t(ha_j) \to \infty$ , since  $\frac{1}{\sqrt{\mathcal{N}_t(ha_j)}} \to 0$  and  $\mathcal{N}_t(ha_i)$  will be constant under our assumption. Note also that  $\frac{1}{\sqrt{\mathcal{N}_t(ha_i)}} - \frac{1}{\sqrt{\mathcal{N}_t(ha_j)}}$  is increasing and will become positive at some time  $t^*$  when  $\mathcal{N}_{t^*}(ha_j) > \mathcal{N}_{t^*}(ha_i)$ . After then (i.e.  $t > t^*$ ), the term  $\sqrt{\ln(\mathcal{N}_t(h))} \left( \frac{1}{\sqrt{\mathcal{N}_t(ha_i)}} - \frac{1}{\sqrt{\mathcal{N}_t(ha_j)}} \right)$  will diverge to  $\infty$  as  $t \to \infty$ ,  $\ln (\mathcal{N}_t(h)) \to \infty$ . Further, note that, in a single time step, the maximum difference in rewards is bounded by  $r_{max}$ , therefore  $\mathcal{V}_t(ha_i) - \mathcal{V}_t(ha_j) \geq -\sum_{k=0}^{\infty} \gamma^k r_{max} = \frac{-r_{max}}{1 - \gamma}$  (This is because  $|\mathcal{V}_t(ha_i) - \mathcal{V}_t(ha_j)| \leq \frac{r_{max}}{1 - \gamma}$ , as shown in Theorem 3 and this implies that  $\frac{-r_{max}}{1 - \gamma} \leq \mathcal{V}_t(ha_i) - \mathcal{V}_t(ha_j) \leq \frac{r_{max}}{1 - \gamma}$ ) and  $\alpha_t(h)(\hat{\mathcal{H}}_t(ha_j) - \hat{\mathcal{H}}_t(ha_i)) \geq -1$  as  $0 \leq \alpha_t(h) \leq 1$  and  $0 \leq \hat{\mathcal{H}}_t(h) \leq 1$ . Therefore,  $\exists t''$  where I-UCB $t''(ha_i) - I-UCB_{t''}(ha_j) \geq 0$ .
Theorem 1 For any non-empty history  $h \neq \emptyset \in \mathbf{H}$  and  $\forall a \in \mathbf{A}$ , as  $\mathcal{N}(h) \to \infty$ ,  $\mathcal{N}(ha) \to \infty$ , we have that all states  $b$  in  $\mathcal{B}(h, b)$  which  $P(s = b \mid h) > 0$  will be visited infinitely many times.
Proof: Let us assume that not all actions are taken infinitely many times. There must exist at least one action  $a_i \in \mathbf{A}$  that is taken a finite number of times and, hence,  $\exists t'$  after which the action  $a_i$  will never be taken. Now, consider any action  $a_j$  that is taken infinitely many times as  $\mathcal{N}(h) \to \infty$ . In Lemma 2, we show that  $\exists t'' > t'$  such that:
$$
\mathrm {I} - \mathrm {U C B} _ {t ^ {\prime \prime}} \left(h a _ {i}\right) > \mathrm {I} - \mathrm {U C B} _ {t ^ {\prime \prime}} \left(h a _ {j}\right)
$$
We know that the second term in I-UCB grows faster for  $a_i$  than  $a_j$ , as seen from the I-UCB function, since  $\frac{1}{\mathcal{N}_t(ha_i)}$  is constant for  $t > t'$ . Consequently, we reach a conclusion that  $a_i$  is chosen over  $a_j$  again after  $t'$ . This further means that for all actions  $a_k$ 's, we can find a such a finite time instant after which  $a_i$  is preferred over  $a_k$ . Therefore, at some point, the action  $a_i$  has to be chosen over all others. This is a contradiction to our original claim. Therefore, all action  $a \in \mathbf{A}$  are taken infinitely many times and, hence, all states will be visited infinitely many times.
Proof: We adapt the proof from the UCT convergence in Shah et al. 2020 and POMCP convergence in Silver and Veness 2010. We induct over the depth of the tree, starting from the leaf up to the root. From Theorem 1, the leaf nodes will have an unbiased estimation of the expected reward  $r_h^a = \mathbb{E}_{b\in \mathcal{B}(h)}(\mathcal{R}(b,a))$ ,  $\forall a\in \mathbf{A}$ . Note that we are using  $r_h^a$  as the expected reward given a  $h$ , which is different from an immediate reward  $r_b^a = R(b,a)$  given a state  $b$ . Similarly, any state at level  $D - 1$  has a simple UCB-like problem since each state is visited infinitely many times and the one-step rewards have an unbiased estimation. In detail, consider the leaves at level  $|h| = D$ , as  $\mathcal{N}(h_{\mathfrak{r}})\to \infty \Rightarrow \mathcal{N}(h)\to \infty$ ,  $\alpha \rightarrow 0$ , and in the absence of children's entropy  $\mathcal{H}(ha)$  (as it is the leaf node and has no children),  $r_h^a$  converges unbiasedly  $\forall h\in \mathbf{H}, a\in \mathbf{A}$ . Consequently,  $\mathcal{V}(h)\rightarrow \mathcal{V}^{*}(h)$  since it is a single decision process, akin to multi-armed bandits. Now, we assume convergence for all levels from the leaves up to  $|h'| = |h| + 1$ , and as we know  $r_h^a = \sum_{s\in S}P(s_t = b|h)r_b^a$ . As we know, the value function satisfies:
$$
\mathcal {V} ^ {*} (h) = \max _ {a \in \mathbf {A}} \left[ r _ {h} ^ {a} + \gamma \sum P (h _ {t + 1} = h ^ {\prime} | h _ {t} a _ {t} = h a) \mathcal {V} ^ {*} (h ^ {\prime}) \right]
$$
From our hypothesis, the  $\mathcal{V}^*(h')$  must converge and moreover, through the sampling process, the value function must converge for level  $|h|$  [32]. Consequently, by backward induction, the whole tree's values must converge.
Theorem 3 Given any  $\epsilon >0$ , there is a finite horizon  $D$  for the problem, such that  $|\mathcal{V}^{*}(h_{\mathfrak{r}}) - \mathcal{V}_{D}^{*}(h_{\mathfrak{r}})|\leq \epsilon$ , where  $h_\tau$  represents the root of the tree.
Proof: Consider  $\pi^{*}$  and  $\pi_{D}^{*}$  to be the optimal policies for the infinite and finite horizon problem, respectively. Also, assume that  $\epsilon$  is an arbitrary, positive real number. Now, let the value functions for these problems be  $\mathcal{V}^{*}(h_{\mathfrak{r}})$  and  $\mathcal{V}_{D}^{*}(h_{\mathfrak{r}})$ . For the root  $\mathfrak{r}$ , we can find that: