SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \mathcal {V} ^ {*} \left(h _ {\mathfrak {r}}\right) - \mathcal {V} _ {D} ^ {*} \left(h _ {\mathfrak {r}}\right) \right| \leq \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {m a x} = \frac {\gamma^ {D}}{1 - \gamma} r _ {m a x} \leq \epsilon
$$
since we can calculate  $D = \log_{\gamma} \frac{\epsilon(1 - \gamma)}{r_{max}}$ , we can also ensure that the relation holds.
Proof: From Theorem 6, we can always find a depth  $D$  such that the value function of the finite-horizon problem is arbitrarily close to the infinite-horizon problem. From Theorem 5, we know that we can converge to the optimal value in a finite horizon POMDP. Therefore, since  $\alpha \to 0$  as  $\mathcal{N}(h_{\mathfrak{r}}) \to \infty$ , IB-POMCP will converge to  $\epsilon$ -optimal actions by choosing  $D$  appropriately.
C Benchmark Domains and Scenarios
All environments used were obtained from the literature. The Tiger problem is a well-known standard problem [36]. For the Maze domain, we based our design on Thomas et al. 2020 [36]. For the RockSample problem, we designed our own scenarios but based the implementation on Thomas et al. 2020 [36]. For Tag and LaserTag, we used the scenarios proposed by Somani et al. (2013) [34]. For Foraging [2], we proposed our own configurations for each scenario.
Tiger Environment: The Tiger problem is a classical benchmark where an agent must choose between two doors: one hides a treasure; the other, a tiger. The objective is to find out which door hides the treasure. The agent can decide which door to open or wait and listen to the tiger (with a certain probability of mishearing it behind one of the doors) before making the final decision.
The agent has a  $15\%$  probability of mishearing the tiger. A maximum of 20 actions are allowed per experiment. Choosing the right door gives the agent a reward of  $+0.1$ , the wrong door  $-1$ , and the listening action penalises the agent with a  $-0.01$  reward. Thus, each agent could try to listen to the tiger 19 times before making the final decision to open either the left or right door.
Maze Environment: This is an Active Localisation Problem where an agent navigates through a toroidal grid and tries to localise its own position by collecting information about its current position. The reward is based on the entropy of the current belief. There is a change of  $15\%$  to miss observe the colour of the cell. The belief follows a Bayesian Update process. Figure 1 illustrates the scenarios.
(a) Maze Cross - This is a 5x5 scenario in which black cells are positioned to ease the agent's localisation problem. Figure 1a illustrates the scenario's configuration.
(b) Maze Holes – This is an 8x8 scenario, which requires the agent to reason one step in advance to search for the missing black cell in this regular configuration. Figure 1b illustrates the scenario.
(c) Maze Dots - This is a 6x6 scenario, identical to Maze Hole except that black cells are separated by several white cells. Figure 1c illustrates the scenario.
(d) Maze Grid-X - This is a 3x3 scenario, with an upper white triangle and a bottom black triangle for the localisation. Figure 1c illustrates the scenario.
(a) Maze Cross.
(b) Maze Holes.
(c) Maze Dots.
Figure 1: Maze environment scenario's configuration.
(d) Maze Grid-X.
RockSample Environment: The context of this problem illustrates a rover exploring an unknown planet. The rover's objective is to earn rewards by both sampling rocks in the environment and leaving the planet with the samples. While the positions of the rover and the rocks are known, not all rocks hold scientific value, which are referred to as "good" rocks. Given the cost associated with rock sampling, the rover is equipped with a noisy long-range sensor, enabling it to assess a rock's potential scientific value before deciding whether to approach and sample it. The environment is a grid map of size  $N \times N$  with  $k$  rocks. The POMDP model for RockSample( $N, k$ ) follows: The state space is a combination of  $k + 1$  features, where Positions represent the rover's location  $\{(1,1), (1,2), \dots, (N, N)\}$ , and  $k$  binary features, RockType $_i$ , indicating whether each rock is "Good" or "Bad". There is a terminal state (portal) at the top-right corner of the map. The rover has a choice of  $k + 5$  actions: {North, South, East, West, Sample, Check $_1$ , ..., Check $_k$ }. The first four are deterministic single-step motion actions. The Sample action involves sampling the rock at the rover's current location. If the rock is determined to be "Good", the rover receives a reward of  $+1$ , and the rock transitions to "Bad", signifying no further benefit from sampling. If the rock is "Bad", a penalty of  $-1$  is delivered to the agent. Moving into the exit portal yields a reward of  $+0.0001$ . All other actions have no associated cost or reward. Each Check $_i$  action employs the rover's long-range sensor to observe Rock $_i$ , yielding a noisy observation of either "Good" or "Bad". The noise in the long-range sensor reading is influenced by the efficiency  $\eta$ , which decreases exponentially as a function of Euclidean distance from the target, following  $\eta = \exp(-0.2 \text{Euclidean Distance}(\text{rover}, \text{Rock}_i))$ . Initially, every rock is assumed to have an equal probability of being "Good" or "Bad". Figure 2 shows the four scenario configurations used.
(a) RockSample22 - This scenario is set on a 5x5 grid, featuring 2 good rocks and 2 bad rocks. The rover initiates its mission from the middle of these rocks and must locate each specific good rock among the bad rocks. Figure 2a illustrates the scenario's configuration.
(b) RockSample40 – In this 5x5 scenario, there are 4 good rocks initially placed in the environment, with no bad rocks present. In contrast to RockSample22, the rover's goal here is to collect more rewards as there are more good rocks available. The rover starts its journey from the centre of these rocks. Figure 2b illustrates the scenario's configuration.
(c) RockSample44 – This 10x10 scenario features 4 good rocks and 4 bad rocks. The rover's starting point is the bottom-left corner of the map. Similar to RockSample22 but on a larger scale, the rover's challenge is to locate each specific good rock among the bad rocks to optimize its reward. Figure 2c illustrates the scenario's configuration.
(d) RockSample17 – In this 10x10 scenario, there's only 1 good rock and 7 bad rocks, and the rover begins its journey in the bottom-left corner of the map. Unlike RockSample44, the expectation here is for the rover to collect fewer rewards. However, if the rover finds the only good rock available, it can significantly improve its performance in terms of reward collection. Figure 2c illustrates the scenario's configuration.
(a) RockSample22.
(b) RockSample40.
(c) RockSample44.
Figure 2: RockSample environment scenario's configuration.
(d) RockSample17.
Tag and LaserTag Environment: In the Tag scenario (Figure 3a), the agent's primary objective is to locate and tag a target that actively moves away. Both the agent and the target navigate within a grid featuring 29 possible positions. While the agent is aware of its own location, it can only observe the target's position when they occupy the same spot. The agent has the option to either stay in its current position or move to any of the four adjacent positions, incurring a cost of  $-0.1$  for each move. Additionally, it can opt to execute the tag action, wherein it receives a reward of  $+1$  for a successful tag but receives a penalty of  $-1$  if it fails to tag the target.
LaserTag (Figure 3b) is an augmented version of Tag, where an agent is acting in a 7x11 grid world containing randomly placed obstacles. The agent's behaviour and that of its opponent align with the rules established in Tag. However, in LaserTag, there's a key distinction: the agent possesses prior knowledge of its initial location. Additionally, the agent is equipped with a laser system capable of providing distance estimations (in cells' units) in 8 directions. The laser readings are generated from a normal distribution centred around the agent's true distance from the nearest obstacle in each direction, with a standard deviation of 2.5 units. These readings are then discretised into whole units, resulting in an observation comprising a set of 8 integers. Figure 3 illustrates the scenarios.
(a) Tag.
Figure 3: Tag and LaserTag scenarios' configuration.
(b) LaserTag.
Foraging Environment: This is a common problem used for evaluating online planning algorithms [2, 7]. This domain presents an agent that must collect boxes displaced in a rectangular gridworld. The problem is defined over partial observability and the agent does not know how the tasks are distributed. The problem ends when the agent collects all boxes. A reward of  $+1$  is delivered to the agent every time a box is collected. The radius and the angle of vision (integer numbers) for the agents are, respectively,  $20\%$  of the diagonal dimension in the environment in cells' unit and  $90^o$  - for example, if we have a  $10\times 10$  environment, so our vision radius will be radius  $= 0.2\sqrt{x_{dim}^2 + y_{dim}^2} = 0.2\sqrt{100 + 100} = 0.2\sqrt{200} = 2$ . Additionally, agents have memory, i.e., after seeing (in the real world) a state or position in the environment, they will sample the next possible world configuration considering this information. Moreover, obstacles block the agent's vision, hence, it can not see tasks across the walls. Figure 4 illustrates the scenarios.
(a) The Corridor – This is a simple (20, 2) scenario, where the agent starts in the left part of a corridor with a box at its side and another at the end of the corridor. The idea here is to test the agent's capability to find isolated reward spots. Figure 4a illustrates the scenario.
(b) U-Shaped - A complex version of The Corridor, the U-shaped is a  $15 \times 15$  scenario, where the agent starts at the tip of a U-shaped corridor with a box at the beginning, one in the middle and another at the end. The idea here is to test the agent's capability to perform simple planning, but facing sparse rewards collection. Figure 4b illustrates the scenario.
(c) U-Obstacles - A complex  $20 \times 10$  scenario, where the agent needs to collect the boxes distributed in a room, but there are obstacles (U-shaped walls) that block its vision. This scenario is commonly used to study local minima issues that emerge in robotic navigation problems [22]. The idea here is to test the agent's capability to perform planning with sparse reward collections, and the vision is blocked by obstacles. Figure 4c illustrates the scenario.
(d) The Warehouse - This scenario is our largest in terms of dimensions,  $20 \times 20$ . The agent's objective is to locate specific reward spots (groups of boxes). These reward clusters are situated at considerable distances from one another. Consequently, the agent cannot observe multiple reward clusters from a single position, making planning more challenging. Figure 4d illustrates the scenario.
(e) The Office – Our most complex  $15 \times 10$  scenario, where the agent needs to collect the boxes distributed in an office. The idea here is to test the agent's capability to perform complex planning when tasks are distributed in different rooms and its vision is blocked by walls. The agent needs to enter each room to finish the problem. Figure 4e illustrates the scenario.
D Hyperparameters
We used a single hyperparameter set for all Monte-Carlo Tree Search-based methods:
- Historical weight/Discount factor:  $\gamma = 0.95$
Maximum depth for the tree: 20
Maximum number of simulations: 250 (per search)
For  $\rho$ -POMCP and TB  $\rho$ -POMCP specific hyperparameters, we consider the following:
- Small bag size:  $|\mathfrak{B}| = 10$
- TB  $\rho$ -POMCP runs within the same time window of IB-POMCP's average planning time.
Finally, considering IB-POMCP practical enhancement, we applied  $q = 0.2$ , hence  $\alpha \in [0.2, 0.8]$ .
(a) The Corridor.
(b) U-shaped.
(d) The Warehouse.
(c) U-Obstacles.
(e) The Office.
Figure 4: Foraging environment scenario's configuration.