SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our insight is to use  $\alpha$  to augment the current uncertainty level in the tree's search policy. On the left-hand side of the multiplication, we design a function that represents the chance of finding new information by exploring a node, which decreases as the number of visits to the node increases.  $e$  is the Euler's constant, which will be used as our amortisation factor for  $\ln (\mathcal{N}(h_{\mathfrak{r}})) / \mathcal{N}(h_{\mathfrak{r}})$ . Applying both together, we can describe a function that slowly decreases and maintains, in the infinity and under some assumptions, theoretical guarantees for belief approximation (see Section 5). The right-hand side of the multiplication expresses the "general surprise trend", which is calculated through the division between the actual cumulative entropy  $\sum_{i=1}^{\mathcal{N}(h_{\mathfrak{r}})} \mathcal{H}_i(h_{\mathfrak{r}})$  and the estimated maximum cumulative entropy  $\mathcal{N}(h_{\mathfrak{r}}) \max_{i=1}^{\mathcal{N}(h_{\mathfrak{r}})} \mathcal{H}_i(h_{\mathfrak{r}})$ , which is the multiplication of the total number of visits to the node and the maximum entropy. By multiplying both sides of the equation, we can estimate the current uncertainty of our system, in this case, of our tree. In addition to improving reasoning, we discard the need for prior knowledge about the problem of adjusting and fixing the tree constant  $c$ . Note that we adjust  $\alpha(h_{\mathfrak{r}})$  for each traversal on the tree, i.e., from the root  $\mathfrak{r}$  to some leaf.
(b) Entropy Calculation - We adapt Shannon's Entropy equation to measure the level of information in the IB-POMCP's search process based on the collection of observations of our agent while performing Simulations (see step (iv)), which is designed as  $\mathcal{H}(h) = -\sum_{z\in \tilde{\mathbf{Z}}}P_h(z)\ln (P_h(z))$ , where  $P_{h}(z)$  is the probability of finding the observation  $z\in \tilde{\mathbf{Z}}$  by simulating actions from the node with history  $h$ . We use  $\tilde{\mathbf{Z}}$ , which is an estimated set of observations, in the calculation of the entropy because we consider that a compact representation of the full observations space  $\mathbf{Z}$  may not be available.
In order to build  $\tilde{\mathbf{Z}}$  as a reliable estimation of  $\mathbf{Z}$ , we collect all the observations found during all traversals within the tree and save them first as the multiset  $\tilde{\mathbf{Z}}^m$  - avoiding losing information, e.g., the frequency of the observations - and then we translate it as the set  $\tilde{\mathbf{Z}}$  when necessary.
Each node has its own estimated multiset of observation  $\tilde{\mathbf{Z}}_h^m$ . Every time we visit a node  $h$ , we update the  $\tilde{\mathbf{Z}}_h^m$  using the collected observation information, which is saved and back-propagated as  $\tilde{\mathbf{Z}}_t^m$ . On the other hand,  $\tilde{\mathbf{Z}}_t^m$  is the multiset that saves the observations from a single traversal starting from the node at the tree level  $t$  to the maximum length of the path  $D$ . Therefore, each node  $h$  has its own multiset that saves all possible observations to be found by performing a simulation from it. Formally, we can define  $\tilde{\mathbf{Z}}_h^m = \bigcup_{i=1}^{\mathcal{N}(h)} \tilde{\mathbf{Z}}_{t,i}^m$ , where  $\tilde{\mathbf{Z}}_{t,i}^m$  is the back-propagated  $\tilde{\mathbf{Z}}_t^m$  at  $i$ -th visit to the node, and  $\tilde{\mathbf{Z}}_{t,i}^m = \tilde{\mathbf{Z}}_{t+1,i}^m \cup z_t, \forall t = 0, 1, \ldots, D$ . Note that  $\tilde{\mathbf{Z}}_{t,i}^m = \emptyset, \forall t > D$ . Let's put it as an example:
Consider  $h = h_0$  as the root node,  $D = 3$  and a single traversal in the tree. Under the backpropagation perspective, we start our update from the last node visited, related to  $\tilde{\mathbf{Z}}_{t+3}^m$ . Following our definition,  $\tilde{\mathbf{Z}}_{t+3}^m = \{z_{t+3}\}$  and, since it is a leaf node, only  $z_{t+3}$  will be included in  $\tilde{\mathbf{Z}}_{h_3}^m$  as a new possible observation to be found from node  $h_3$ . Now, in  $\tilde{\mathbf{Z}}_{t+2}^m$ , we will add the  $z_{t+2}$  to our backpropagation multiset  $\tilde{\mathbf{Z}}_{t+3}^m$  and, consequently, we will add  $\{z_{t+3}, z_{t+2}\}$  to  $\tilde{\mathbf{Z}}_{h_2}^m$ . We repeat this process until we reach  $h_0$ , where we include all found observations  $\tilde{\mathbf{Z}}_t^m = \tilde{\mathbf{Z}}_{t + 1}^m \cup z_0 = \{z_3,z_2,z_1,z_0\}$  to the root multiset  $\tilde{\mathbf{Z}}_h^m$ . Figure 1 illustrates this example and our proposed observation back-propagation procedure. Figures 1a and 1d present a high-level perspective of the process, showing only the observation nodes of the tree to facilitate visualisation. Figures 1b and 1c present a closer perspective of the process as a one-step execution. The red particles are states and the blue ones are observations.
(a) Simulation procedure from a high-level perspective
(b) One-step simulation
(c) One-step backpropagation (BP)
(d) BP through the observation nodes of the tree
Figure 1: Illustration of IB-POMCP's search process.
With  $\tilde{\mathbf{Z}}_h^m$  in hands, we now can approximate  $P_{h}(z)$  by calculating the frequency of the observation  $z$  in  $\tilde{\mathbf{Z}}_h^m$ , following  $P_{h}(z) \approx \tilde{P}_{h}(z) = \frac{1}{|\tilde{\mathbf{Z}}_{h}^{m}|}\sum_{z^{\prime} \in \tilde{\mathbf{Z}}_{h}^{m}}1_{\{z^{\prime} = z\}}$ . Therefore, our final entropy function will be  $\mathcal{H}(h) = -\sum_{z \in \tilde{\mathbf{Z}}_h}\tilde{P}_h(z)\ln (\tilde{P}_h(z))$ . We use this defined entropy to calculate the current entropy of the system. However, since IB-POMCP plans in an online manner and the  $\tilde{\mathbf{Z}}_h^m$  is updated at each visit to the node, we propose the estimation of our tree's entropy using the update-equation:
$$
\mathcal {H} _ {\mathcal {N} (h)} (h) := \mathcal {H} _ {\mathcal {N} (h) - 1} (h) + \frac {(\mathcal {H} (h) - \mathcal {H} _ {\mathcal {N} (h) - 1} (h))}{\mathcal {N} (h)} \tag {2}
$$
where  $\mathcal{N}(h)$  is the number of visits to the node  $h$ ,  $\mathcal{H}(h)$  is the above defined entropy calculation, and  $\mathcal{H}_i(h), i = 1,2,\dots,\mathcal{N}(h)$  is the estimated entropy at the visit  $i$ . Note that, by following this final definition for the entropy, we can also calculate  $\alpha$  (Equation 1) and update it in an online fashion.
(c) Entropy Normalisation - Another issue that arises is that, since we may not have access to the true observation distribution in advance, our current approach will result in entropy calculations that are not standardised under the same image function. This problem occurs because the size of the set  $\tilde{\mathbf{Z}}$  grows every time an observation never found before is delivered by the environment. As a consequence, the maximum value for the entropy also grows, and nodes that receive a higher number of different observations will calculate higher values for  $\mathcal{H}(h)$ , which can create bias by including these overestimated values in the decision-making process. Therefore, we propose its normalisation  $a$  posteriori, following:  $\hat{\mathcal{H}}(h) \coloneqq \frac{\mathcal{H}_{\mathcal{N}(h)}(h)}{\max_{j=1}^{N(h)} \mathcal{H}_j(h)}$ , where  $\mathcal{H}_j(h)$  represents the entropy calculated at the  $j$ -th visit to the node  $h$ . This approach guarantees  $\hat{\mathcal{H}}(h) \in [0,1], \forall h$ , i.e., for all nodes. Moreover,
$\mathcal{H}(h) = 1$  when  $\mathcal{N}(h) = 0$  by definition, since we consider that when there is no information, the uncertainty is maximised, hence, the entropy is also maximised.
(iv) Simulation - After adjusting our  $\alpha$  function, we start the simulation process. While expanding the tree, we propose a modified version of UCB1 to guide the tree search process based on the information level of each node, the  $I$ -UCB function, which follows:
$$
\operatorname {I} - \operatorname {U C B} (h, a, \alpha) := \mathcal {V} (h a) + (1 - \alpha) \sqrt {\ln (\mathcal {N} (h)) / \mathcal {N} (h a)} + \alpha \hat {\mathcal {H}} (h a) \tag {3}
$$
Our proposal intends to guide the expansion and planning process based on entropy updates that occur within the tree, defining what we denote as the information-guided planning process. In this type of planning, a trade-off between exploration and exploitation still exists, but now we also bring information value to the discussion. Our main novelty here moves past the traditional idea, where the more we explore, the better we can exploit the rewards, thereby improving the agent's performance. Instead, we expand the discussion and enhance the algorithm capabilities by introducing an information-gain perspective to the tree-planning process using entropy calculations.
Intuitively, our action selection function considers that (a) if the system uncertainty is high (hence,  $\alpha$  value is high), we weight the action selection based on the advantage of information gain (which may expand the tree in depth), trying to decrease the entropy value and accumulate knowledge, or; (b) if the system uncertainty is low (hence,  $\alpha$  value is low), we weight the action selection in the advantage of the exploration gain (which may expand the tree in breadth), trying to increase the entropy value and increase the number of possibilities to reason over. Therefore, I-UCB performs adaptive planning that also depends on the system entropy, besides the upper confidence estimations.
Another direct advantage of the application of I-UCB instead of UCB1 is that our method rarely fails to deliver an action justified by metrics, that is, it rarely delivers a randomly chosen action as the best one. Usually, this issue arises in problems where rewards are sparsely distributed over a long horizon of actions, which can lead to several ties in the Q-values (e.g., at zero). Proposing a solution that works under these conditions requires (a) to increase the reasoning horizon of the solution, which usually leads to the expenditure of significantly more computational resources, or (b) the capability to handle it within the available reasoning horizon, which is a non-trivial task. IB-POMCP solves this problem by following a non-trivial solution considering (a) the inclusion of a novel metric for evaluation – the system entropy, which frequently is non-zero since the observation distribution is diverse and; (b) the adaptive value of alpha, which often promotes an action branch to find (at least) a non-zero value (reward or entropy) or to be frequently visited during the reasoning process.
(v) Best action selection - we decide the best action by picking the action with the highest score using  $a_{best} = \arg\max_{a \in \mathbf{A}} (1 - \alpha)\mathcal{V}(ha) + \alpha\hat{\mathcal{H}}(ha)$ . If there is a tie, we break it using the number of visits for each possible node. If it persists, we run a random policy to select the "best" action. Because of the proposed entropy calculation (and in contrast to POMCP), IB-POMCP has the ability to choose actions even without experiencing non-zero rewards while planning.
Lastly, considering the above discussion on adapting and inserting  $\alpha$  inside I-UCB, we consider scaling  $\alpha$  value to be within an interval  $[q, 1 - q] \subset [0, 1]$ ,  $0 < q < 0.5$ , as a practical enhancement. This strategy allows our method to avoid ignoring part of the I-UCB result by multiplying one term by zero. This step will not be considered in the theoretical analysis.
Algorithm/Pseudo-code: To highlight the difference between POMCP and IB-POMCP, we present the complete pseudocode of our algorithm in Appendix A<sup>1</sup>.
In this section, we analyse IB-POMCP's capability to plan optimally in partially observable finite horizon problems and  $\epsilon$ -optimally for infinite horizon partially observable problems. For the complete proofs see Appendix B. We start with the following assumption:
Assumption 1 Given a generic POMDP represented by the tuple  $(\mathbf{S},\mathbf{A},\mathcal{R},\mathcal{T},\mathbf{Z},\mathcal{Z},\mathbf{H})$ , the  $\mathcal{R}$  is bounded and there exists a positive maximum reward value  $r_{max} \in \mathcal{R}$ .
<sup>1</sup>IB-POMCP's implementation publicly available on GitHub: https://github.com/lsmcolab/ib-pomcp/
Under this simple assumption and to establish the IB-POMCP's planning capabilities, we must examine our action selection procedure, specifically through the application of I-UCB. Consequently, we first analyse how the  $\alpha$  coefficient behaves and impacts the IB-POMCP's search process:
Lemma 1  $\alpha(h) \in [0,1]$  converges to  $0$  as the number of visits  $\mathcal{N}(h)$  approaches the infinity.
Theorem 1 For any non-empty history  $h \neq \emptyset \in \mathbf{H}$  and  $\forall a \in \mathbf{A}$ , as  $\mathcal{N}(h) \to \infty$ ,  $\mathcal{N}(ha) \to \infty$ , we have that all states  $b$  in  $\mathcal{B}(h, b)$  which  $P(s = b \mid h) > 0$  will be simulated infinitely many times.
Using the previous results, we can show the convergence for a finite horizon:
Theorem 2 IB-POMCP's nodes converge to the optimal value for a fixed horizon  $D$  as the number of search iterations goes to infinite (i.e.,  $\mathcal{N}(h_{\mathfrak{r}}) \to \infty$ ).
We now move on to the infinite horizon case. Consider  $\mathcal{V}^*(h_{\mathfrak{r}})$  as the optimal value function for the root  $\mathfrak{r}$  and  $\mathcal{V}_D^*(h_{\mathfrak{r}})$  as the optimal value function for the root  $\mathfrak{r}$  for a finite horizon D. We find that:
Theorem 3 Given any  $\epsilon$ , there is a finite horizon  $D$  for the problem, such that  $|\mathcal{V}^{*}(h_{\mathfrak{r}}) - \mathcal{V}_{D}^{*}(h_{\mathfrak{r}})| \leq \epsilon$ , where  $h_{\mathfrak{r}}$  represents the root of the tree.
Given the previous results, we can prove the following for the convergence in the infinite horizon:
Corollary 1 IB-POMCP converges to  $\epsilon$ -optimal actions at the root  $(h_{\mathfrak{r}})$  in a  $\gamma$ -discounted infinite horizon scenario.
Evaluation Settings - We define five well-known domains as our benchmarks. See Appendix C for more information. (i) The Tiger (T0) environment presents an agent must choose between two doors: one hides a treasure; the other, a tiger. The agent can decide which door to open or listen to before making the final decision. (ii) The Maze (M0-M3) environment [36], which is an Active Localization Problem in which an agent navigates through a toroidal grid and tries to localize its own position by collecting local information. The reward is based on the entropy of the current belief. (iii) The RockSample (R0-R3) problem [36] considers that a robot has to navigate through a grid-world and maximise the number of good rocks it picks up and analyses instead of bad rocks. If it picks up (samples) a bad rock, it gets a negative reward. Otherwise, it gets a positive reward and the good rock becomes bad. (iv) The Tag/LaserTag (LT0-LT1) environment [34], where an agent is trying to find and tag a target opponent agent that intentionally moves away. The agent only knows its own position but can observe the target's position if they are in the same position (Tag) or using an 8-directional laser (in LaserTag). If the agent successfully tags the target it is rewarded, otherwise it is penalised. (v) The Foraging (F0-F4) problem is a famous scenario to evaluate online planning algorithms [2, 30] that presents an agent that collects items displaced in a rectangular grid world. All experiments were implemented using AdLeap-MAS [9] because it implements all five environments and enables the quick execution of our experiments. Each run was performed in a single node of a high-performance cluster containing 16 cores of Intel Ivy Bridge processors and 64 GB RAM.
Baselines - In this work, we compare IB-POMCP against: (a) POMCP [32], since it is a relevant state-of-art proposal and represents the basis of this work; (b)  $\rho$ -POMCP [36], representing our main competitor in terms of using information theory to perform the online planning procedure, and (c) TB  $\rho$ -POMCP [36], as a faster alternative to  $\rho$ -POMCP that employs an explicit POMDP model.
Metrics - Two different evaluation metrics were used: (i) the average reward (R) across the experiment's mean reward; and (ii) the average planning time (t) spent by the agent to plan the next actions. Mean results were calculated across 50 executions. Every experiment ran independently; thus, no knowledge was carried from one execution to another. The calculated errors  $(\pm Err)$  represent the  $95\%$  confidence interval of a two-sample t-test. Note that our results and baselines are separated into two categories (i) Quick Planning: grouping methods that perform the decision-making process within a reasonable time window; and (ii) Long Planning: presenting the  $\rho$ -POMCP's results (without time constraints). We separated them to make their understanding easier.
Finally, we refer the reader to our Appendix D to information about our hyperparameters set.
Benchmarks study - All results are presented in Table 1. In addition, we highlight the pros and cons of our method, analysing our limitations while discussing the outcome for each environment.
Table 1: Summarized results. The bold underlined values in Quick Planning highlight the best result with statistical significance across all Quick Planning baselines for the respective metric.
Problem	Quick Planning	Long Planning ρ-POMCP
POMCP	TB ρ-POMCP	IB-POMCP	R ± Err	t ± Err (sec)
R ± Err	t ± Err (sec)	R ± Err	t ± Err (sec)	R ± Err	t ± Err (sec)
Tiger (T0)	-4.25 × 102±0.80 × 102	0.09 ± 0.01	-0.16 × 102±0.03 × 102	0.20 ± 0.00	-0.52 × 102±0.15 × 102	0.11 ± 0.01	-0.21 × 102±0.03 × 102	1.45 ± 0.04
MazeCross (M0)	1.11 × 10-2±0.08 × 10-2	2.47 ± 0.02	1.12 × 10-2±0.07 × 10-2	3.00 ± 0.00	1.23 × 10-2±0.04 × 10-2	2.43 ± 0.01	1.19 × 10-2±0.07 × 10-2	28.16 ± 0.68
MazeHoles (M1)	1.56 × 10-2±0.50 × 10-2	3.72 ± 0.03	1.32 × 10-2±0.46 × 10-2	4.00 ± 0.00	3.80 × 10-2±0.86 × 10-2	3.60 ± 0.04	2.34 × 10-2±0.44 × 10-2	48.41 ± 2.82
MazeDots (M2)	1.20 × 10-2±0.28 × 10-2	2.59 ± 0.02	0.79 × 10-2±0.15 × 10-2	3.00 ± 0.00	2.99 × 10-2±0.85 × 10-2	2.53 ± 0.02	1.25 × 10-2±0.20 × 10-2	36.71 ± 1.39
MazeGridX (M3)	2.80 × 10-2±0.60 × 10-2	1.86 ± 0.02	0.99 × 10-2±0.15 × 10-2	2.00 ± 0.00	0.96 × 10-2±0.02 × 10-2	1.82 ± 0.01	1.37 × 10-2±0.30 × 10-2	19.79 ± 0.32
RockSample22 (R0)	0.06 × 10-1±0.01 × 10-1	2.08 ± 0.03	0.03 × 10-1±0.01 × 10-1	5.00 ± 0.00	0.07 × 10-1±0.01 × 10-1	2.33 ± 0.04	0.03 × 10-1±0.01 × 10-1	2.70 ± 0.43
RockSample40 (R1)	0.12 × 10-1±0.01 × 10-1	2.08 ± 0.03	0.11 × 10-1±0.02 × 10-1	5.00 ± 0.00	0.16 × 10-1±0.01 × 10-1	2.40 ± 0.04	0.10 × 10-1±0.01 × 10-1	2.77 ± 0.32
RockSample44 (R2)	0.07 × 10-1±0.01 × 10-1	4.67 ± 0.09	0.06 × 10-1±0.01 × 10-1	5.00 ± 0.00	0.09 × 10-1±0.02 × 10-1	5.10 ± 0.06	0.04 × 10-1±0.02 × 10-1	23.18 ± 3.10
RockSample17 (R3)	-0.04 × 10-1±0.02 × 10-1	4.72 ± 0.09	-0.06 × 10-1±0.02 × 10-1	5.00 ± 0.00	0.00 × 10-1±0.01 × 10-1	5.11 ± 0.06	-0.09 × 10-1±0.02 × 10-1	25.49 ± 3.28
Tag (LT0)	-0.47 × 10-1±0.09 × 10-1	1.22 ± 0.20	-0.65 × 10-1±0.09 × 10-1	3.00 ± 0.00	-0.36 × 10-1±0.07 × 10-1	0.86 ± 0.14	-0.65 × 10-1±0.10 × 10-1	11.23 ± 1.52
LaserTag (LT1)	-0.90 × 10-1±0.06 × 10-1	4.72 ± 0.09	-0.86 × 10-1±0.09 × 10-1	3.00 ± 0.00	-0.92 × 10-1±0.06 × 10-1	5.11 ± 0.06	-0.96 × 10-1±0.51 × 10-1	19.41 ± 1.02
TheCorridor (F0)	4.29 × 10-2±0.51 × 10-2	0.96 ± 0.06	4.36 × 10-2±0.38 × 10-2	1.00 ± 0.00	6.89 × 10-2±0.28 × 10-2	0.87 ± 0.10	5.15 × 10-2±0.18 × 10-2	8.18 ± 0.71
U-shaped (F1)	0.70 × 10-2±0.34 × 10-2	3.67 ± 0.02	0.53 × 10-2±0.03 × 10-2	3.00 ± 0.00	5.10 × 10-2±0.14 × 10-2	2.99 ± 0.19	5.30 × 10-2±0.31 × 10-2	37.36 ± 1.29
U-obstacles (F2)	1.17 × 10-2±0.34 × 10-2	1.90 ± 0.03	1.67 × 10-2±0.23 × 10-2	2.00 ± 0.00	5.50 × 10-2±0.34 × 10-2	1.88 ± 0.08	3.75 × 10-2±0.45 × 10-2	15.44 ± 0.37
Warehouse (F3)	5.20 × 10-2±0.45 × 10-2	3.02 ± 0.11	4.42 × 10-2±1.03 × 10-2	3.00 ± 0.00	10.73 × 10-2±0.41 × 10-2	2.91 ± 0.25	8.99 × 10-2±0.96 × 10-2	20.40 ± 0.67
TheOffice (F4)	0.43 × 10-2±0.21 × 10-2	1.90 ± 0.03	0.72 × 10-2±0.14 × 10-2	2.00 ± 0.00	2.34 × 10-2±0.27 × 10-2	1.88 ± 0.08	1.61 × 10-2±0.17 × 10-2	15.44 ± 0.37
In the Tiger domain, TB  $\rho$ -POMCP presents the best average reward among the Quick Planning methods ( $\rho < 0.01$ ). IB-POMCP still outperforms POMCP, showing significant improvement in terms of reward ( $\rho < 0.01$ ). For this specific problem, since the action "listen" generates variation in the entropy, IB-POMCP keeps performing it repeatedly until other action value outcomes the "listen" value in the best action selection procedure (Section 4, step (v)), a circumstance which leads our method to reduce its average reward collection. Hence, when facing problems where seeking spots of high uncertainty produces small penalties, IB-POMCP may collect penalties until it significantly decreases the uncertainty and chooses another path.
In the Maze domain, we observe that IB-POMCP presents a significantly better average reward in 3 out of 4 proposed scenarios ( $\rho \leq 0.015$ ), except for M3, for which POMCP presents a better result. Note that reducing uncertainty leads to increasing reward; i.e. the faster an agent can access areas with high uncertainty, the higher its received reward. IB-POMCP's results match our expectations (given the developed rationale throughout our methodology) since we build it to, besides tracking the rewards available in the scenario, often seek paths that lead to spots with high entropy in order to decrease uncertainty, what increases reward in this scenario. In terms of time, we present significantly faster reasoning in all scenarios ( $\rho < 0.01$ ). Investigating why IB-POMCP runs faster than POMCP, we found that our information-guided planning leads the algorithm to perform more transitions during the rollout phase than while performing the simulation inside our actual search tree, with a rate  $\frac{\#\text{ rollout}}{\# \text{ simulation}} = 1.61$ , whereas POMCP presents a rate of 1.28. Because rollout transitions run faster than simulation transitions inside the tree, we can save time by performing them frequently.
In the Rock Sample problem, we can see that IB-POMCP shows a significant improvement in terms of reward collection  $(\rho \leq 0.02)$  in 3 out of 4 scenarios - except for the simplest scenario RockSample22 (R0), which presents a p-value of  $\rho \leq 0.11$ . In terms of reasoning time, POMCP is slightly faster than all Quick Planning methods in 3 out of 4 scenarios  $(\rho \leq 0.01)$ , except for RockSample17 (R3).
In Tag, IB-POMCP presents significant improvement in terms of reward collection and reasoning time against all baselines ( $\rho \leq 0.04$ ). As in the Maze, we believe that IB-POMCP is faster because it simulates transitions in rollouts more often (in a ratio of  $\frac{\# \text{rollout}}{\# \text{simulation}} = 1.7$  against 1.61 for POMCP).
In LaserTag, we have a tie between all methods, with no statistically significant difference spotted across metrics ( $\rho \geq 0.32$  for reward collection and  $\rho \geq 0.06$  for reasoning time).
Finally, in the Foraging domain, our proposed method significantly outperformed all the baselines  $(\rho < 0.01)$ . The Foraging problem represents our most complex scenario, and empirically shows how IB-POMCP can overcome the necessity of adjusting the reasoning horizon to perform planning in settings that deliver rewards sparsely. The only reward available in these scenarios comes from the collection of tasks. Consequently, while attempting to solve a task, the number of actions that the agent needs to plan and execute in sequence may approach or exceed the reasoning horizon size. In this case, the probability of experiencing this reward is low, and algorithms that only follow the reward in planning will rarely obtain it in their simulations; hence, they fail to plan effectively.