SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matheus Aparecido do Carmo Alves
Lancaster, United Kingdom
m.a.docarmoalves@lancaster.ac.uk
Yehia Elkhatib
yehia.elkhatib@glasgow.ac.uk
Amokh Varma
Indian Institute of Technology
amokhvarma@gmail.com
Leandro Soriano Marcolino
l.marcolino@lancaster.ac.uk
This paper presents IB-POMCP, a novel algorithm for online planning under partial observability. Our approach enhances the decision-making process by using estimations of the world belief's entropy to guide a tree search process and surpass the limitations of planning in scenarios with sparse reward configurations. By performing what we denominate as an information-guided planning process, the algorithm, which incorporates a novel I-UCB function, shows significant improvements in reward and reasoning time compared to state-of-the-art baselines in several benchmark scenarios, along with theoretical convergence guarantees.
Decision-making agents are increasingly being used under uncertainty in nontrivial systems [17]. Based on the available information, such agents must evaluate potential decisions in the current environment to build a robust plan that either accomplishes the task or leads to a better situation. Consider, for instance, a scenario in which drones are deployed to rescue people in a hazardous environment. Drones need to quickly identify critical locations where they can offer support and save lives. If no clear target is found, they must rapidly formulate a plan based on their current knowledge and observable information to find an area in need of assistance from their current position.
The above context describes a partial observability problem that is recurrent in the literature [2, 7, 12]. The Partially Observable Monte-Carlo Planning (POMCP) [32] algorithm is commonly suggested as a method to address these problems because it enables agents to perform planning in an online manner while handling uncertainty [6, 36, 38]. However, several state-of-the-art solutions that rely on POMCP often struggle when rewards are delivered sparsely in time or outside their reasoning horizon. For example, in foraging, there may be no clear target to spot from the current observation and/or estimated map so far. Here, we find a fundamental academic challenge: How can we improve agent performance when rewards are delivered sparsely in time or out of their reasoning horizon?
Every time an agent makes an observation of the world, it gains information that can be used to improve its internal planning process. Based on this, some works improve the traditional planning process by handling the lack of prior information using the approximation of the world's dynamic models [16, 27], embedding supportive planning methods within the tree [2, 7], extracting non-explicit information from the observations [6, 16], enhancing the method's inner calculations [1, 8, 26] or employing neural networks-based techniques to improve the search quality [18, 37].
Overall, these strategies can improve search quality within a defined simulation horizon by using extensions of traditional partially observable models and enabling the integration of additional knowledge into planning. However, such knowledge may not be available to agents beforehand (e.g., conditional observation probabilities), or may demand significantly more computational resources to perform the task (e.g., time and memory). While it is important to note that incorporating more knowledge benefits efficiency in terms of objective accomplishment, it does not resolve challenges in sparse-reward scenarios, as most of the solutions remain reward-driven.
Hence, we present Information-based POMCP (IB-POMCP), an online planning algorithm that uses a novel information framework to boost the decision-making process under partial observability even when no rewards are available in the reasoning horizon. Our framework refines the traditional UCB1 action selection strategy by implementing our proposed Information-guided UCB (I-UCB) function, which is capable of leveraging entropy and estimating information gain, using both real-world and internally generated observations to identify promising states in the search process that leads the agent to quickly accomplish its objective. Its application together with our new particle filter reinvigoration strategy, which considers the current system's entropy to calibrate the reinvigoration, indicates that this approach may lead the agent to act hybridly when solving a partially observable problem by better weighing the exploration, exploitation, and information levels within the tree. We ran experiments across five benchmarks and compared them with state-of-the-art baselines, obtaining significantly higher average rewards (up to 10 times) while performing faster reasoning (up to  $93\%$ ).
In the current state-of-the-art, POMCP [32] remains a relevant solution due to its adaptability and problem-solving capability across the most diverse domains [3, 23, 24, 41]. However, many proposals are tailored to specific problems, leaving room for improvement, especially when handling sparse reward scenarios. Our approach addresses this gap by shifting POMCP's reward-based planning to our "information-guided planning strategy", capable of leveraging observation space entropy (information) and guiding the agent to promising spots from where it solves such scenarios.
Other methods suggest approximating the latent world functions to handle the lack of knowledge [11, 13, 19, 35]. However, these approaches are directly bounded by the belief in space dimensionality, which grows exponentially and hence requires exponentially more resources as the complexity of the problem increases [17]. We propose the extraction of information using only the observations received from the simulations, improving the reasoning without requiring significantly more resources.
Some studies recommend belief-dependent POMDP models to improve POMCP's reasoning capabilities [4, 21]. Under certain assumptions, these models have shown great efficiency when solving problems with large action and observation spaces. A recent advancement is  $\rho$ -POMCP [36], an efficient online planner that boosts the POMCP's search process by estimating belief-dependent rewards. However, this method relies on explicit access to the observation and transition function, struggling while reasoning under time constraints. TB  $\rho$ -POMCP [36] tries to handle scenarios with limited time to reason, penalising efficiency in order to do so. IB-POMCP can improve planning using less information and without significant penalties for reasoning time.
Time-constrained problems are prevalent in the evaluation of online planning algorithms. In real-time strategy games, a common strategy is to use macro and micro predefined coordination scripts to support the decision-making [25, 40]. Even though they handle several levels of coordination, these methods still require behavioural templates to plan. Similar problems arise in the multi-agent swarm research [28] where they avoid developing a complex planning process in order to save time (using, for example, Flat Monte Carlo approaches). We propose the execution of efficient planning under partial observability from scratch and without penalising the planning quality.
The Upper Confidence Bound (UCB1) [20] is a relevant algorithm in the literature for evaluating and selecting actions in online planning solutions and is recurrently used inside Monte-Carlo decision-making frameworks [15, 29, 35]. While we acknowledge the existence of other action selection methods available in the literature, such as the Thompson Sampling method [5] and the PUCT algorithm [33], in this paper, we focus on augmenting the POMCP's capabilities by modifying the UCB1. Differently from PUCT (AlphaGo) [33], for example, our proposed I-UCB well-fits partially observable scenarios while enhancing POMCP planning capabilities through an entropy-based perspective, without relying on pre-trained models or a large amount of data.
Recently, MENTS and ANTS improved the Monte-Carlo Tree Search (MCTS) performance using maximum entropy policy optimisation networks [18, 37]. In contrast to both works, we focus our study on partially observable models and, instead of directly optimising latent functions, we improve planning by embedding knowledge into the search process. Furthermore, both methods rely on prior problem-specific knowledge and data to train neural networks, including information on the maximum episode length, which helps them handle issues like sparse rewards. However, we assume that the agent must reason under a limited number of steps (often lower than necessary to accomplish the problem) and execute the problem from scratch, carrying no knowledge between executions.
Finally, although we focus on improving POMCP, our contributions can also benefit other algorithms, such as DESPOT-based solutions [34]. DESPOT's main focus is on filtering a large observation space to plan using a sparse approximation of the problem, hence, a smaller reasoning space. However, the algorithm still struggles in large action spaces, and/or sparse reward scenarios where the optimal policy could be long (requiring many steps). Our proposal addresses POMDPs featuring sparse rewards in an efficient manner while running the search process from scratch.
Markovian Models - The Markov Decision Process (MDP) is a mathematical framework to model stochastic processes in a discrete-time flow. The model specifies that each process is composed of a set of states  $\mathbf{S}$ , a set of actions  $\mathbf{A}$ , a reward function  $\mathcal{R}:\mathbf{S}\times \mathbf{A}\rightarrow \mathbb{R}$ , and a transition function  $\mathcal{T}$ . On the other hand, the Partially Observable MDP (POMDP) extends the MDP representation to problems in which the agent cannot directly access the current state. Instead, the agent receives an observation  $z\in \mathbf{Z}$ , determined by observation probabilities  $\mathcal{Z}_{s,z}^{a} = P(z_{t + 1} = z\mid s_{t + 1} = s;a_{t} = a)$ . The belief function  $\mathcal{B}(h)$  is a distribution over the state space  $\mathbf{S}$ , which describes the conditional probability of being in a particular state  $b\in \mathbf{S}$  given the current history  $h$ ,  $\mathcal{B}(h,b) = P(s_t = b\mid h_t)$ , where the history  $h\in \mathbf{H}$  is a sequence of action-observation pairs  $h_t = \{a_0,z_0,\dots ,a_t,z_t\}$ . The decision process follows a policy  $\pi (h,a) = P(a_{t + 1} = a|h_t)$ , while choosing actions.
Monte-Carlo Tree Search (MCTS) - MCTS algorithms aim to determine the optimal action  $a^*$  for a given state  $s$  by simulating the world steps within a tree structure. Each node in the Monte-Carlo tree  $\mathbf{T}$  is represented by  $(s, \mathcal{V}, \mathcal{N})$ , i.e., a tuple with a state  $s$ , a value  $\mathcal{V}(s, a)$ , and a visitation count  $\mathcal{N}(s, a)$  for each action  $a \in \mathbf{A}$ . The value of the node represents the expected cumulative reward for the simulated states. The number of visits to the state  $s$  is represented by  $\mathcal{N}(s) = \sum_{a \in \mathbf{A}} \mathcal{N}(s, a)$ . The simulations are divided into two main stages: search and rollout. Each state in the search tree is viewed as a multi-armed bandit taking actions usually chosen by the Upper Confidence Bound (UCB1) algorithm. UCB1 is a well-known algorithm that tries to increase the value of less-explored actions by attaching a bonus inversely proportional to the number of times each action is tried, following the update-equation  $UCB1(s, a) := \mathcal{V}(s, a) + c \sqrt{\frac{\log(\mathcal{N}(s))}{\mathcal{N}(s, a)}}$ . The scalar  $c$  is the exploration constant responsible for weighting the exploration value within the UCB1 function. With a correct definition of  $c$ , the value function converges in probability to the optimal value,  $\mathcal{V}(s, a) \xrightarrow{p} \mathcal{V}^*(s, a)$ .
Partially Observable MC Planning (POMCP) - POMCP is an extension of MCTS for partially observable problems. Analogously to it, each node is represented by a tuple  $(h,\mathcal{V}(h),\mathcal{N}(h),\mathcal{F}_h)$  and the algorithm is divided into the search and rollout stages. In contrast, POMCP uses an unweighted particle filter  $\mathcal{F}_h$  to approximate the belief state at each node  $h$ . This strategy requires a Monte-Carlo simulator  $\mathcal{G}$ , which can sample a state  $s'$ , reward  $r$  and observation  $z$  given a state-action pair. Recursively down the tree, all sampled information is added to a particle filter  $\mathcal{F}$ . At the end of the search stage, the best action is chosen and a real observation is received from the environment. Considering the current action-observation pair, we move to the new root node and perform the particle reinvigorating process [32], which is responsible for updating and boosting the new root's particle filter belief state approximation. After that, we continue the planning procedure (starting the search process) as described above. We refer the reader to Silver and Veness (2010) for further detail.
Shannon entropy in Information Theory - A traditional approach to measure uncertainty in stochastic processes is to use Shannon's Entropy function  $\mathcal{H}(X) = -\sum_{i=1}^{n} P(x_i) \log(P(x_i))$ , where  $X$  is a discrete random variable and  $x_i$  are the possible outcomes from  $X$ . The core idea is to find a reliable measure for "information", that is, to calculate the degree of surprise of an event given the available space of possibilities. If an event often occurs, the information inside the event is likely
not "novel". Hence, there is no "surprise" when it occurs. On the other hand, if an event rarely occurs, the "surprise" is greater. Comparing both situations, we can follow events with higher surprise levels to explore the space of possibilities, and safely follow events with lower surprise levels to exploit the environment, hence, we can adapt an agent's behaviour according to the entropy of the current state.
4 Information-based Partially Observable Monte Carlo Planning
Algorithm Outline - Our algorithm starts with the (i) initialisation of our tree structure, where we create our root and calculate the probability of stepping into this node given our current knowledge. Since the initial knowledge about the problem is assumed to be none, we can not calculate this probability, hence, we initialise it as 0. Subsequently, we (ii) initialise the particle filter of our root node, generating possible states from which we can simulate the agents' actions and search for a solution to the target problem. Initially, the beliefs are generated through a uniform distribution.
Our search process is iterative, as commonly found in the literature [32, 39, 2], where we sample a state from our current belief to perform multiple rounds of simulation. However, unlike the typical approach, we propose the implementation of an adaptive exploration coefficient and of our novel I-UCB function instead of the usual UCB1. Therefore, at every iteration of our search (before starting a simulation), we first perform our (iii) exploration coefficient adaptation, adjusting it using the entropy (level of information) estimated over the set of observations collected through the simulations. Then, with the updated coefficient in hands, we start our (iv) simulation with the I-UCB function. While choosing actions, our proposal searches for solutions and takes actions based on observations' entropy besides the collection of rewards. At the end of each simulation, we re-update the exploration coefficient based on the information gained during the path traversal. When we finish the search, we (v) select the best action based on the estimated rewards, entropy, and the number of visits of each action's node to determine the most promising path to follow in the real environment.
After taking an action and receiving a new real observation from the environment, we restart the IB-POMCP algorithm. However, the initial steps are slightly changed to maintain the knowledge and update the current information, i.e., perform online planning. Unlike the first iteration, we now  $(i)$  update the root node by traversing the tree based on the action taken and the most recent observation from the environment. Consequently, we recalculate the probability of stepping into the new root node using the information available in the tree. With the probability in our hands, we now  $(ii)$  update the particle filter of our root node by reinvigorating our belief, generating new states using the uniform distribution while maintaining promising states for simulation. Then, after performing these updates, we restart the search and repeat the whole process. Now, let us discuss these steps in detail.
(i) Root Initialisation and Update - This step is responsible for maintaining the tree structure  $\mathbf{T}$  coherent with the world and the information gained while performing online planning. Consider  $\mathfrak{r}$  as the tree  $\mathbf{T}$ 's root node,  $h_{\mathfrak{r}}$  as the history of the root  $\mathfrak{r}$  (our current history),  $a$  as the action taken by the agent in the real world and  $z$  as the most recent observation received from the real world (after the agent's action). As in POMCP, this step considers three possible procedures:
- Initialisation: if  $T$  does not exist, we create a root node  $\mathfrak{r}$  using the current history  $h_{\mathfrak{r}}$ , else;
- Update: if  $T$  exists and the nodes  $h_{\mathfrak{r}}a$  and  $h_{\mathfrak{r}}az$  also exist, we walk in the existing tree, traversing the nodes that correspond to the actions taken by the agent  $a$  and the most recent world observation  $z$ , and update the root, i.e., the node  $h_{\mathfrak{r}}az$  becomes the new  $\mathfrak{r}$ , hence,  $h_{\mathfrak{r}} = h_{\mathfrak{r}}az$ , else;
- Re-initialisation: if  $T$  exists but, node  $h_{\mathfrak{r}}a$  or  $h_{\mathfrak{r}}az$  does not exist, the tree is re-initialised. That is, we create a new node  $(h,\mathcal{V}(h),\mathcal{N}(h),\mathcal{F}_h)$  with  $h$  as the current history,  $\mathcal{V}(h) = 0, \mathcal{N}(h) = 0$  and  $\mathcal{F}_h$  is empty. Afterwards, we assign this new node as our new root  $\mathfrak{r}$  of  $T$ , which means that all other simulations already made in the agent's head are discarded and we restart the algorithm.
In contrast to POMCP, we estimate the current probability  $P(z|h_{\mathfrak{r}}a)$ , in this step, when finding the observation  $z$  after taking the action  $a$  considering our history  $h_{\mathfrak{r}}$ . Overall, we aim to use this probability to enhance the Particle Filter Update (see step (ii)) procedure by weighting its diversification and reinvigoration levels according to the agent's uncertainty when stepping into the new root - which would be the same as calculating the agent's "surprise" on finding the received observation in the real world after taking the chosen action. However, we consider that a compact representation of transition  $\mathcal{T}$  or observation probabilities  $\mathcal{Z}$  may not be available for complex problems. Hence, we propose the approximation of this probability using the knowledge within our particle filter  $\mathcal{F}_{\mathfrak{r}}$  (without requiring a POMDP explicit model) by  $P(z|h_{\mathfrak{r}}a)\approx \tilde{P} (z|h_{\mathfrak{r}}a) = \frac{\mathcal{N}(h_{\mathfrak{r}}az)}{\mathcal{N}(h_{\mathfrak{r}}a)}$ ,
where  $\mathcal{N}(h_{\mathfrak{r}}az)$  and  $\mathcal{N}(h_{\mathfrak{r}}a)$  are the number of visits of node  $h_{\mathfrak{r}}az$  and  $h_{\mathfrak{r}}a$ , respectively (the new root node and its parent). If  $h_{\mathfrak{r}}$  is empty or no new root node is found,  $\tilde{P}(z|h_{\mathfrak{r}}a) = 0$ . Note also that  $\mathcal{N}(haz) \leq \mathcal{N}(ha), \forall h \in \mathbf{T}$  and, consequently,  $\mathcal{N}(haz) / \mathcal{N}(ha) \in [0,1]$ .
(ii) Particle Filter Initialisation and Update - This process is responsible for initialising the particle filter of the root node  $\mathcal{F}_{\mathrm{r}}$  (if it is empty or does not exist) or performing the particle reinvigoration process based on the probability  $\tilde{P}(z|h_{\mathrm{r}}a)$  calculated in the Root Update process (step (i)). Directly, the initialisation is made through the sampling of  $k$  particles (which generate the observation  $z$ ) from the uniform distribution  $\mathcal{U}_z$ . On the other hand, the update considers the  $\tilde{P}(z|h_{\mathrm{r}}a)$  as the weight that balances the particle reinvigorating process over  $\mathcal{F}_{\mathrm{r}}$ . The idea is to diversify (or boost) the new root's particle filter as a reliable approximation of the belief state function. If the probability of stepping into this new root is high, i.e.,  $\tilde{P}(z|h_{\mathrm{r}}a)$  is high, we assume that the particles in the new root's particle filter will well approximate the real world, since through the simulations we recurrently found the observation  $z$  after taking the action  $a$  from the last root node  $h$ . In contrast, when this probability is low, we diversify the new root's particle filter  $\mathcal{F}_{\mathrm{r}}$  by uniformly generating particles that may better represent the real world using  $\mathcal{U}_z$  instead of  $\mathcal{F}_{\mathrm{r}}$ . Consequently, coherent with the above rationale, we reinvigorate the new root's particle filter  $\mathcal{F}_{\mathrm{r}}$  by maintaining  $\lfloor k\tilde{P}(z|h_{\mathrm{r}}a)\rfloor$  particles sampled from itself, i.e.,  $\mathcal{F}_{haz}$ , and sampling new  $\lceil k(1-\tilde{P}(z|h_{\mathrm{r}}a))\rceil$  particles from the uniform distribution  $\mathcal{U}_z$ . This update on the new root's particle filter may offer a better start for the algorithm when performing the search process since the sampling of belief states may consider an enhanced set of particles through our uncertainty-guided reinvigoration process. We kindly refer the reader to Appendix A for further discussion and pseudo-code to implement this procedure.
(iii) Updating the Tree Exploration Coefficient - To explain how we update our tree exploration coefficient, we first introduce the (a) adaptation made to the traditional exploration coefficient, then we present our (b) modified entropy function, which is used in the update, we explain how to calculate it and, in the end, we show our (c) strategy to normalise the entropy in the online planning context.
(a) Exploration Coefficient Adaptation - Before the actual tree simulation process, IB-POMCP first adjusts the value of our tree's exploration parameter based on the estimated entropy for the current system (the tree) in the root level  $\mathfrak{r}$  with history  $h_\mathfrak{r}$ . Directly, our approach considers the replacement of the traditional  $c$  constant by the function  $(1 - \alpha(h_\mathfrak{r})) \in [0,1]$ , which we define as:
$$
\alpha \left(h _ {\mathfrak {r}}\right) := \frac {e \ln \left(\mathcal {N} \left(h _ {\mathfrak {r}}\right)\right)}{\mathcal {N} \left(h _ {\mathfrak {r}}\right)} \frac {\sum_ {i = 1} ^ {\mathcal {N} \left(h _ {\mathfrak {r}}\right)} \mathcal {H} _ {i} \left(h _ {\mathfrak {r}}\right)}{\mathcal {N} \left(h _ {\mathfrak {r}}\right) \max  _ {i = 1} ^ {\mathcal {N} \left(h _ {\mathfrak {r}}\right)} \mathcal {H} _ {i} \left(h _ {\mathfrak {r}}\right)} \tag {1}
$$