Title: Information-guided Planning: An Online Approach for Partially Observable Problems
Venue: NeurIPS 2023
Accepted: yes
Keywords: Information-guided planning

Planning under uncertainty

Sequential decision making
Abstract: This paper presents IB-POMCP, a novel algorithm for online planning under partial observability. Our approach enhances the decision-making process by using estimations of the world belief's entropy to guide a tree search process and surpass the limitations of planning in scenarios with sparse reward configurations. By performing what we denominate as an *information-guided planning process*, the algorithm, which incorporates a novel I-UCB function, shows significant improvements in reward and reasoning time compared to state-of-the-art baselines in several benchmark scenarios, along with theoretical convergence guarantees.

Information-guided Planning: An Online Approach for Partially Observable Problems

Abstract

This paper presents IB-POMCP, a novel algorithm for online planning under partial observability. Our approach enhances the decision-making process by using estimations of the world belief's entropy to guide a tree search process and surpass the limitations of planning in scenarios with sparse reward configurations. By performing what we denominate as an information-guided planning process , the algorithm, which incorporates a novel I-UCB function, shows significant improvements in reward and reasoning time compared to state-of-the-art baselines in several benchmark scenarios, along with theoretical convergence guarantees.

1 Introduction

Decision-making agents are increasingly being used under uncertainty in nontrivial systems [17]. Based on the available information, such agents must evaluate potential decisions in the current environment to build a robust plan that either accomplishes the task or leads to a better situation. Consider, for instance, a scenario in which drones are deployed to rescue people in a hazardous environment. Drones need to quickly identify critical locations where they can offer support and save lives. If no clear target is found, they must rapidly formulate a plan based on their current knowledge and observable information to find an area in need of assistance from their current position.

The above context describes a partial observability problem that is recurrent in the literature [2, 7, 12]. The Partially Observable Monte-Carlo Planning (POMCP) [32] algorithm is commonly suggested as a method to address these problems because it enables agents to perform planning in an online manner while handling uncertainty [6, 36, 38]. However, several state-of-the-art solutions that rely on POMCP often struggle when rewards are delivered sparsely in time or outside their reasoning horizon. For example, in foraging, there may be no clear target to spot from the current observation and/or estimated map so far. Here, we find a fundamental academic challenge: How can we improve agent performance when rewards are delivered sparsely in time or out of their reasoning horizon?

Every time an agent makes an observation of the world, it gains information that can be used to improve its internal planning process. Based on this, some works improve the traditional planning process by handling the lack of prior information using the approximation of the world's dynamic models [16, 27], embedding supportive planning methods within the tree [2, 7], extracting non-explicit information from the observations [6, 16], enhancing the method's inner calculations [1, 8, 26] or employing neural networks-based techniques to improve the search quality [18, 37].

Amokh Varma

Leandro Soriano Marcolino

Overall, these strategies can improve search quality within a defined simulation horizon by using extensions of traditional partially observable models and enabling the integration of additional knowledge into planning. However, such knowledge may not be available to agents beforehand (e.g., conditional observation probabilities), or may demand significantly more computational resources to perform the task (e.g., time and memory). While it is important to note that incorporating more knowledge benefits efficiency in terms of objective accomplishment, it does not resolve challenges in sparse-reward scenarios, as most of the solutions remain reward-driven.

Hence, we present Information-based POMCP (IB-POMCP) , an online planning algorithm that uses a novel information framework to boost the decision-making process under partial observability even when no rewards are available in the reasoning horizon. Our framework refines the traditional UCB1 action selection strategy by implementing our proposed Information-guided UCB (I-UCB) function, which is capable of leveraging entropy and estimating information gain, using both real-world and internally generated observations to identify promising states in the search process that leads the agent to quickly accomplish its objective. Its application together with our new particle filter reinvigoration strategy, which considers the current system's entropy to calibrate the reinvigoration, indicates that this approach may lead the agent to act hybridly when solving a partially observable problem by better weighing the exploration, exploitation, and information levels within the tree. We ran experiments across five benchmarks and compared them with state-of-the-art baselines, obtaining significantly higher average rewards (up to 10 times) while performing faster reasoning (up to 93%).

2 Related Work

In the current state-of-the-art, POMCP [32] remains a relevant solution due to its adaptability and problem-solving capability across the most diverse domains [3, 23, 24, 41]. However, many proposals are tailored to specific problems, leaving room for improvement, especially when handling sparse reward scenarios. Our approach addresses this gap by shifting POMCP's reward-based planning to our ' information-guided planning strategy ', capable of leveraging observation space entropy (information) and guiding the agent to promising spots from where it solves such scenarios.

Other methods suggest approximating the latent world functions to handle the lack of knowledge [11, 13, 19, 35]. However, these approaches are directly bounded by the belief in space dimensionality, which grows exponentially and hence requires exponentially more resources as the complexity of the problem increases [17]. We propose the extraction of information using only the observations received from the simulations, improving the reasoning without requiring significantly more resources.

Some studies recommend belief-dependent POMDP models to improve POMCP's reasoning capabilities [4, 21]. Under certain assumptions, these models have shown great efficiency when solving problems with large action and observation spaces. A recent advancement is ρ -POMCP [36], an efficient online planner that boosts the POMCP's search process by estimating belief-dependent rewards. However, this method relies on explicit access to the observation and transition function, struggling while reasoning under time constraints. TB ρ -POMCP [36] tries to handle scenarios with limited time to reason, penalising efficiency in order to do so. IB-POMCP can improve planning using less information and without significant penalties for reasoning time.

Time-constrained problems are prevalent in the evaluation of online planning algorithms. In real-time strategy games, a common strategy is to use macro and micro predefined coordination scripts to support the decision-making [25, 40]. Even though they handle several levels of coordination, these methods still require behavioural templates to plan. Similar problems arise in the multi-agent swarm research [28] where they avoid developing a complex planning process in order to save time (using, for example, Flat Monte Carlo approaches). We propose the execution of efficient planning under partial observability from scratch and without penalising the planning quality.

The Upper Confidence Bound (UCB1) [20] is a relevant algorithm in the literature for evaluating and selecting actions in online planning solutions and is recurrently used inside Monte-Carlo decisionmaking frameworks [15, 29, 35]. While we acknowledge the existence of other action selection methods available in the literature, such as the Thompson Sampling method [5] and the PUCT algorithm [33], in this paper, we focus on augmenting the POMCP's capabilities by modifying the UCB1. Differently from PUCT ( AlphaGo ) [33], for example, our proposed I-UCB well-fits partially observable scenarios while enhancing POMCP planning capabilities through an entropy-based perspective, without relying on pre-trained models or a large amount of data.

Recently, MENTS and ANTS improved the Monte-Carlo Tree Search (MCTS) performance using maximum entropy policy optimisation networks [18, 37]. In contrast to both works, we focus our study on partially observable models and, instead of directly optimising latent functions, we improve planning by embedding knowledge into the search process. Furthermore, both methods rely on prior problem-specific knowledge and data to train neural networks, including information on the maximum episode length, which helps them handle issues like sparse rewards. However, we assume that the agent must reason under a limited number of steps (often lower than necessary to accomplish the problem) and execute the problem from scratch, carrying no knowledge between executions.

Finally, although we focus on improving POMCP, our contributions can also benefit other algorithms, such as DESPOT-based solutions [34]. DESPOT's main focus is on filtering a large observation space to plan using a sparse approximation of the problem, hence, a smaller reasoning space. However, the algorithm still struggles in large action spaces, and/or sparse reward scenarios where the optimal policy could be long (requiring many steps). Our proposal addresses POMDPs featuring sparse rewards in an efficient manner while running the search process from scratch.

3 Background

Markovian Models The Markov Decision Process ( MDP ) is a mathematical framework to model stochastic processes in a discrete-time flow. The model specifies that each process is composed of a set of states S , a set of actions A , a reward function R : S × A → R , and a transition function T . On the other hand, the Partially Observable MDP ( POMDP ) extends the MDP representation to problems in which the agent cannot directly access the current state. Instead, the agent receives an observation z ∈ Z , determined by observation probabilities Z a s,z = P ( z t +1 = z | s t +1 = s ; a t = a ) . The belief function B ( h ) is a distribution over the state space S , which describes the conditional probability of being in a particular state b ∈ S given the current history h , B ( h, b ) = P ( s t = b | h t ) , where the history h ∈ H is a sequence of action-observation pairs h t = { a 0 , z 0 , . . . , a t , z t } . The decision process follows a policy π ( h, a ) = P ( a t +1 = a | h t ) , while choosing actions.

Monte-Carlo Tree Search (MCTS) MCTS algorithms aim to determine the optimal action a ∗ for a given state s by simulating the world steps within a tree structure. Each node in the Monte-Carlo tree T is represented by ( s , V , N ), i.e., a tuple with a state s , a value V ( s, a ) , and a visitation count N ( s, a ) for each action a ∈ A . The value of the node represents the expected cumulative reward for the simulated states. The number of visits to the state s is represented by N ( s ) = ∑ a ∈ A N ( s, a ) . The simulations are divided into two main stages: search and rollout . Each state in the search tree is viewed as a multi-armed bandit taking actions usually chosen by the Upper Confidence Bound (UCB1) algorithm. UCB1 is a well-known algorithm that tries to increase the value of less-explored actions by attaching a bonus inversely proportional to the number of times each action is tried, following the update-equation UCB 1( s, a ) := V ( s, a ) + c √ log ( N ( s )) N ( s,a ) . The scalar c is the exploration constant responsible for weighting the exploration value within the UCB1 function. With a correct definition of c , the value function converges in probability to the optimal value, V ( s, a ) p - → V ∗ ( s, a ) .

Partially Observable MC Planning (POMCP) POMCP is an extension of MCTS for partially observable problems. Analogously to it, each node is represented by a tuple ( h , V ( h ) , N ( h ) , F h ) and the algorithm is divided into the search and rollout stages. In contrast, POMCP uses an unweighted particle filter F h to approximate the belief state at each node h . This strategy requires a Monte-Carlo simulator G , which can sample a state s ′ , reward r and observation z given a state-action pair. Recursively down the tree, all sampled information is added to a particle filter F . At the end of the search stage, the best action is chosen and a real observation is received from the environment. Considering the current action-observation pair, we move to the new root node and perform the particle reinvigorating process [32], which is responsible for updating and boosting the new root's particle filter belief state approximation. After that, we continue the planning procedure (starting the search process) as described above. We refer the reader to Silver and Veness (2010) for further detail.

Shannon entropy in Information Theory A traditional approach to measure uncertainty in stochastic processes is to use Shannon's Entropy function H ( X ) = -∑ n i =1 P ( x i ) log( P ( x i )) , where X is a discrete random variable and x i are the possible outcomes from X . The core idea is to find a reliable measure for 'information', that is, to calculate the degree of surprise of an event given the available space of possibilities. If an event often occurs, the information inside the event is likely

not 'novel'. Hence, there is no 'surprise' when it occurs. On the other hand, if an event rarely occurs, the 'surprise' is greater. Comparing both situations, we can follow events with higher surprise levels to explore the space of possibilities, and safely follow events with lower surprise levels to exploit the environment, hence, we can adapt an agent's behaviour according to the entropy of the current state.

4 Information-based Partially Observable Monte Carlo Planning

Algorithm Outline Our algorithm starts with the (i) initialisation of our tree structure , where we create our root and calculate the probability of stepping into this node given our current knowledge. Since the initial knowledge about the problem is assumed to be none, we can not calculate this probability, hence, we initialise it as 0. Subsequently, we (ii) initialise the particle filter of our root node , generating possible states from which we can simulate the agents' actions and search for a solution to the target problem. Initially, the beliefs are generated through a uniform distribution.

Our search process is iterative, as commonly found in the literature [32, 39, 2], where we sample a state from our current belief to perform multiple rounds of simulation. However, unlike the typical approach, we propose the implementation of an adaptive exploration coefficient and of our novel I-UCB function instead of the usual UCB1. Therefore, at every iteration of our search (before starting a simulation), we first perform our (iii) exploration coefficient adaptation , adjusting it using the entropy (level of information) estimated over the set of observations collected through the simulations. Then, with the updated coefficient in hands, we start our (iv) simulation with the I-UCB function . While choosing actions, our proposal searches for solutions and takes actions based on observations' entropy besides the collection of rewards. At the end of each simulation, we re-update the exploration coefficient based on the information gained during the path traversal. When we finish the search, we (v) select the best action based on the estimated rewards, entropy, and the number of visits of each action's node to determine the most promising path to follow in the real environment.

After taking an action and receiving a new real observation from the environment, we restart the IB-POMCP algorithm. However, the initial steps are slightly changed to maintain the knowledge and update the current information, i.e., perform online planning. Unlike the first iteration, we now (i) update the root node by traversing the tree based on the action taken and the most recent observation from the environment. Consequently, we recalculate the probability of stepping into the new root node using the information available in the tree. With the probability in our hands, we now (ii) update the particle filter of our root node by reinvigorating our belief, generating new states using the uniform distribution while maintaining promising states for simulation. Then, after performing these updates, we restart the search and repeat the whole process. Now, let us discuss these steps in detail.

(i) Root Initialisation and Update This step is responsible for maintaining the tree structure T coherent with the world and the information gained while performing online planning. Consider r as the tree T 's root node, h r as the history of the root r (our current history), a as the action taken by the agent in the real world and z as the most recent observation received from the real world (after the agent's action). As in POMCP , this step considers three possible procedures:

· Initialisation: if T does not exist, we create a root node r using the current history h r , else;

· Update: if T exists and the nodes h r a and h r az also exist, we walk in the existing tree, traversing the nodes that correspond to the actions taken by the agent a and the most recent world observation z , and update the root, i.e., the node h r az becomes the new r , hence, h r = h r az , else;

· Re-initialisation: if T exists but, node h r a or h r az does not exist, the tree is re-initialised. That is, we create a new node ( h, V ( h ) , N ( h ) , F h ) with h as the current history, V ( h ) = 0 , N ( h ) = 0 and F h is empty. Afterwards, we assign this new node as our new root r of T , which means that all other simulations already made in the agent's head are discarded and we restart the algorithm.

In contrast to POMCP , we estimate the current probability P ( z | h r a ) , in this step, when finding the observation z after taking the action a considering our history h r . Overall, we aim to use this probability to enhance the Particle Filter Update (see step (ii)) procedure by weighting its diversification and reinvigoration levels according to the agent's uncertainty when stepping into the new root - which would be the same as calculating the agent's 'surprise' on finding the received observation in the real world after taking the chosen action. However, we consider that a compact representation of transition T or observation probabilities Z may not be available for complex problems. Hence, we propose the approximation of this probability using the knowledge within our particle filter F r (without requiring a POMDP explicit model) by P ( z | h r a ) ≈ ˜ P ( z | h r a ) = N ( h r az ) N ( h r a ) ,

where N ( h r az ) and N ( h r a ) are the number of visits of node h r az and h r a , respectively (the new root node and its parent). If h r is empty or no new root node is found, ˜ P ( z | h r a ) = 0 . Note also that N ( haz ) ≤ N ( ha ) , ∀ h ∈ T and, consequently, N ( haz ) / N ( ha ) ∈ [0 , 1] .

(ii) Particle Filter Initialisation and Update This process is responsible for initialising the particle filter of the root node F r (if it is empty or does not exist) or performing the particle reinvigoration process based on the probability ˜ P ( z | h r a ) calculated in the Root Update process (step (i)). Directly, the initialisation is made through the sampling of k particles (which generate the observation z ) from the uniform distribution U z . On the other hand, the update considers the ˜ P ( z | h r a ) as the weight that balances the particle reinvigorating process over F r . The idea is to diversify (or boost) the new root's particle filter as a reliable approximation of the belief state function. If the probability of stepping into this new root is high, i.e., ˜ P ( z | h r a ) is high, we assume that the particles in the new root's particle filter will well approximate the real world, since through the simulations we recurrently found the observation z after taking the action a from the last root node h . In contrast, when this probability is low, we diversify the new root's particle filter F r by uniformly generating particles that may better represent the real world using U z instead of F r . Consequently, coherent with the above rationale, we reinvigorate the new root's particle filter F r by maintaining ⌊ k ˜ P ( z | h r a ) ⌋ particles sampled from itself, i.e., F haz , and sampling new ⌈ k (1 -˜ P ( z | h r a )) ⌉ particles from the uniform distribution U z . This update on the new root's particle filter may offer a better start for the algorithm when performing the search process since the sampling of belief states may consider an enhanced set of particles through our uncertainty-guided reinvigoration process. We kindly refer the reader to Appendix A for further discussion and pseudo-code to implement this procedure.

(iii) Updating the Tree Exploration Coefficient To explain how we update our tree exploration coefficient, we first introduce the (a) adaptation made to the traditional exploration coefficient , then we present our (b) modified entropy function , which is used in the update, we explain how to calculate it and, in the end, we show our (c) strategy to normalise the entropy in the online planning context.

(a) Exploration Coefficient Adaptation Before the actual tree simulation process, IB-POMCP first adjusts the value of our tree's exploration parameter based on the estimated entropy for the current system (the tree) in the root level r with history h r . Directly, our approach considers the replacement of the traditional c constant by the function (1 -α ( h r )) ∈ [0 , 1] , which we define as:

Our insight is to use α to augment the current uncertainty level in the tree's search policy. On the left-hand side of the multiplication, we design a function that represents the chance of finding new information by exploring a node, which decreases as the number of visits to the node increases. e is the Euler's constant, which will be used as our amortisation factor for ln( N ( h r )) / N ( h r ) . Applying both together, we can describe a function that slowly decreases and maintains, in the infinity and under some assumptions, theoretical guarantees for belief approximation (see Section 5). The righthand side of the multiplication expresses the ' general surprise trend ', which is calculated through the division between the actual cumulative entropy ∑ N ( h r ) i =1 H i ( h r ) and the estimated maximum cumulative entropy N ( h r ) max N ( h r ) i =1 H i ( h r ) , which is the multiplication of the total number of visits to the node and the maximum entropy. By multiplying both sides of the equation, we can estimate the current uncertainty of our system, in this case, of our tree. In addition to improving reasoning, we discard the need for prior knowledge about the problem of adjusting and fixing the tree constant c . Note that we adjust α ( h r ) for each traversal on the tree, i.e., from the root r to some leaf.

(b) Entropy Calculation Weadapt Shannon's Entropy equation to measure the level of information in the IB-POMCP's search process based on the collection of observations of our agent while performing Simulations (see step (iv)), which is designed as H ( h ) = -∑ z ∈ ˜ Z P h ( z ) ln( P h ( z )) , where P h ( z ) is the probability of finding the observation z ∈ ˜ Z by simulating actions from the node with history h . We use ˜ Z , which is an estimated set of observations, in the calculation of the entropy because we consider that a compact representation of the full observations space Z may not be available.

In order to build ˜ Z as a reliable estimation of Z , we collect all the observations found during all traversals within the tree and save them first as the multiset ˜ Z m - avoiding losing information, e.g., the frequency of the observations - and then we translate it as the set ˜ Z when necessary.

Each node has its own estimated multiset of observation ˜ Z m h . Every time we visit a node h , we update the ˜ Z m h using the collected observation information, which is saved and back-propagated as ˜ Z t m . On the other hand, ˜ Z t m is the multiset that saves the observations from a single traversal starting from the node at the tree level t to the maximum length of the path D . Therefore, each node h has its own multiset that saves all possible observations to be found by performing a simulation from it. Formally, we can define ˜ Z m h = ⋃ N ( h ) i =1 ˜ Z m t,i , where ˜ Z m t,i is the back-propagated ˜ Z t m at i -th visit to the node, and ˜ Z m t,i = ˜ Z t m +1 ,i ∪ z t , ∀ t = 0 , 1 , ..., D . Note that ˜ Z m t,i = ∅ , ∀ t > D . Let's put it as an example:

Consider h = h 0 as the root node, D = 3 and a single traversal in the tree. Under the backpropagation perspective, we start our update from the last node visited, related to ˜ Z t m +3 . Following our definition, ˜ Z t m +3 = { z t +3 } and, since it is a leaf node, only z t +3 will be included in ˜ Z m h 3 as a new possible observation to be found from node h 3 . Now, in ˜ Z t m +2 , we will add the z t +2 to our backpropagation multiset ˜ Z t m +3 and, consequently, we will add { z t +3 , z t +2 } to ˜ Z m h 2 . Werepeat this process until we reach h 0 , where we include all found observations ˜ Z t m = ˜ Z t m +1 ∪ z 0 = { z 3 , z 2 , z 1 , z 0 } to the root multiset ˜ Z m h . Figure 1 illustrates this example and our proposed observation back-propagation procedure. Figures 1a and 1d present a high-level perspective of the process, showing only the observation nodes of the tree to facilitate visualisation. Figures 1b and 1c present a closer perspective of the process as a one-step execution. The red particles are states and the blue ones are observations.

...

h

0

h

1

h

2

h

3

st

st

+1

at, z

t

+1

st

+2

at

+1

, z

t

+2

st

+3

at

+2

, z

t

+3

(a) Simulation procedure from

a high-level perspective

...

F

h

h

0

a

1

F

ha

1

z

1

h

1

a

2

F

ha

1

z

′

s

s

′

, z

1

, r

(b) One-step

simulation

˜

Z

m

h

2

F

h

2

a

1

˜

Z

m

h

3

F

h

3

a

2

F

h

2

a

1

z

′′

˜

Z

m

t

+3

=

{

z

3

}

˜

Z

m

t

+2

= ˜

Z

m

t

+3

∪

z

2

(c) One-step back-

propagation (BP)

˜

Z

m

t

+3

=

{

zt

+3

}

˜

Z

m

t

+2

= ˜

Z

m

t

+3

∪

zt

+2

˜

Z

m

t

+1

= ˜

Z

m

t

+2

∪

zt

+1

˜

Z

m

t

= ˜

Z

m

t

+1

∪

zt

h

0

h

1

h

2

h

3

(d) BP through the observation

nodes of the tree

With ˜ Z m h in hands, we now can approximate P h ( z ) by calculating the frequency of the observation z in ˜ Z m h , following P h ( z ) ≈ ˜ P h ( z ) = 1 | ˜ Z m h | ∑ z ′ ∈ ˜ Z m h 1 { z ′ = z } . Therefore, our final entropy function will be H ( h ) = -∑ z ∈ ˜ Z h ˜ P h ( z ) ln( ˜ P h ( z )) . We use this defined entropy to calculate the current entropy of the system. However, since IB-POMCP plans in an online manner and the ˜ Z m h is updated at each visit to the node, we propose the estimation of our tree's entropy using the update-equation:

where N ( h ) is the number of visits to the node h , H ( h ) is the above defined entropy calculation, and H i ( h ) , i = 1 , 2 ..., N ( h ) is the estimated entropy at the visit i . Note that, by following this final definition for the entropy, we can also calculate α (Equation 1) and update it in an online fashion.

(c) Entropy Normalisation Another issue that arises is that, since we may not have access to the true observation distribution in advance, our current approach will result in entropy calculations that are not standardised under the same image function. This problem occurs because the size of the set ˜ Z grows every time an observation never found before is delivered by the environment. As a consequence, the maximum value for the entropy also grows, and nodes that receive a higher number of different observations will calculate higher values for H ( h ) , which can create bias by including these overestimated values in the decision-making process. Therefore, we propose its normalisation a posteriori , following: ˆ H ( h ) := H N ( h ) ( h ) max N ( h ) j =1 H j ( h ) , where H j ( h ) represents the entropy calculated at the j -th visit to the node h . This approach guarantees ˆ H ( h ) ∈ [0 , 1] , ∀ h , i.e., for all nodes. Moreover,

H ( h ) = 1 when N ( h ) = 0 by definition, since we consider that when there is no information, the uncertainty is maximised, hence, the entropy is also maximised.

(iv) Simulation After adjusting our α function, we start the simulation process. While expanding the tree, we propose a modified version of UCB1 to guide the tree search process based on the information level of each node, the I-UCB function, which follows:

Our proposal intends to guide the expansion and planning process based on entropy updates that occur within the tree, defining what we denote as the information-guided planning process . In this type of planning, a trade-off between exploration and exploitation still exists, but now we also bring information value to the discussion. Our main novelty here moves past the traditional idea, where the more we explore, the better we can exploit the rewards, thereby improving the agent's performance. Instead, we expand the discussion and enhance the algorithm capabilities by introducing an information-gain perspective to the tree-planning process using entropy calculations.

Intuitively, our action selection function considers that (a) if the system uncertainty is high (hence, α value is high), we weight the action selection based on the advantage of information gain (which may expand the tree in depth), trying to decrease the entropy value and accumulate knowledge, or; (b) if the system uncertainty is low (hence, α value is low), we weight the action selection in the advantage of the exploration gain (which may expand the tree in breadth), trying to increase the entropy value and increase the number of possibilities to reason over. Therefore, I-UCB performs adaptive planning that also depends on the system entropy, besides the upper confidence estimations.

Another direct advantage of the application of I-UCB instead of UCB1 is that our method rarely fails to deliver an action justified by metrics, that is, it rarely delivers a randomly chosen action as the best one. Usually, this issue arises in problems where rewards are sparsely distributed over a long horizon of actions, which can lead to several ties in the Q-values (e.g., at zero). Proposing a solution that works under these conditions requires (a) to increase the reasoning horizon of the solution, which usually leads to the expenditure of significantly more computational resources, or (b) the capability to handle it within the available reasoning horizon, which is a non-trivial task. IB-POMCP solves this problem by following a non-trivial solution considering (a) the inclusion of a novel metric for evaluation - the system entropy, which frequently is non-zero since the observation distribution is diverse and; (b) the adaptive value of alpha, which often promotes an action branch to find (at least) a non-zero value (reward or entropy) or to be frequently visited during the reasoning process.

(v) Best action selection we decide the best action by picking the action with the highest score using a best = argmax a ∈ A (1 -α ) V ( ha ) + α ˆ H ( ha ) . If there is a tie, we break it using the number of visits for each possible node. If it persists, we run a random policy to select the 'best' action. Because of the proposed entropy calculation (and in contrast to POMCP), IB-POMCP has the ability to choose actions even without experiencing non-zero rewards while planning.

Lastly, considering the above discussion on adapting and inserting α inside I-UCB, we consider scaling α value to be within an interval [ q, 1 -q ] ⊂ [0 , 1] , 0 < q < 0 . 5 , as a practical enhancement. This strategy allows our method to avoid ignoring part of the I-UCB result by multiplying one term by zero. This step will not be considered in the theoretical analysis.

Algorithm/Pseudo-code: To highlight the difference between POMCP and IB-POMCP, we present the complete pseudocode of our algorithm in Appendix A 1 .

5 Theoretical Analysis

In this section, we analyse IB-POMCP's capability to plan optimally in partially observable finite horizon problems and ϵ -optimally for infinite horizon partially observable problems. For the complete proofs see Appendix B. We start with the following assumption:

Assumption 1 Given a generic POMDP represented by the tuple ( S , A , R , T , Z , Z , H ) , the R is bounded and there exists a positive maximum reward value r max ∈ R .

Under this simple assumption and to establish the IB-POMCP's planning capabilities, we must examine our action selection procedure, specifically through the application of I-UCB. Consequently, we first analyse how the α coefficient behaves and impacts the IB-POMCP's search process:

Lemma 1 α ( h ) ∈ [0 , 1] converges to 0 as the number of visits N ( h ) approach the infinity.

̸

Theorem 1 For any non-empty history h = ∅ ∈ H and ∀ a ∈ A , as N ( h ) →∞ , N ( ha ) →∞ , we have that all states b in B ( h, b ) which P ( s = b | h ) > 0 will be simulated infinitely many times.

Using the previous results, we can show the convergence for a finite horizon:

Theorem 2 IB-POMCP's nodes converge to the optimal value for a fixed horizon D as the number of search iterations goes to infinite (i.e., N ( h r ) →∞ ).

We now move on to the infinite horizon case. Consider V ∗ ( h r ) as the optimal value function for the root r and V ∗ D ( h r ) as the optimal value function for the root r for a finite horizon D. We find that:

Theorem 3 Given any ϵ , there is a finite horizon D for the problem, such that |V ∗ ( h r ) -V ∗ D ( h r ) | ≤ ϵ , where h r represents the root of the tree.

Given the previous results, we can prove the following for the convergence in the infinite horizon:

Corollary 1 IB-POMCP converges to ϵ -optimal actions at the root ( h r ) in a γ -discounted infinite horizon scenario.

6 Results

Evaluation Settings We define five well-known domains as our benchmarks. See Appendix C for more information. (i) The Tiger (T0) environment presents an agent must choose between two doors: one hides a treasure; the other, a tiger. The agent can decide which door to open or listen to before making the final decision. (ii) The Maze (M0-M3) environment [36], which is an Active Localisation Problem in which an agent navigates through a toroidal grid and tries to localise its own position by collecting local information. The reward is based on the entropy of the current belief. (iii) The RockSample (R0-R3) problem [36] considers that a robot has to navigate through a grid-world and maximise the number of good rocks it picks up and analyses instead of bad rocks . If it picks up (samples) a bad rock, it gets a negative reward. Otherwise, it gets a positive reward and the good rock becomes bad. (iv) The Tag/LaserTag (LT0-LT1) environment [34], where an agent is trying to find and tag a target opponent agent that intentionally moves away . The agent only knows its own position but can observe the target's position if they are in the same position (Tag) or using an 8-directional laser (in LaserTag). If the agent successfully tags the target it is rewarded, otherwise it is penalised. (v) The Foraging (F0-F4) problem is a famous scenario to evaluate online planning algorithms [2, 30] that presents an agent that collects items displaced in a rectangular grid world . All experiments were implemented using AdLeap-MAS [9] because it implements all five environments and enables the quick execution of our experiments. Each run was performed in a single node of a high-performance cluster containing 16 cores of Intel Ivy Bridge processors and 64 GB RAM.

Baselines In this work, we compare IB-POMCP against: (a) POMCP [32], since it is a relevant state-of-art proposal and represents the basis of this work; (b) ρ -POMCP [36], representing our main competitor in terms of using information theory to perform the online planning procedure, and (c) TB ρ -POMCP [36], as a faster alternative to ρ -POMCP that employs an explicit POMDP model.

Metrics Two different evaluation metrics were used: (i) the average reward ( R ) across the experiment's mean reward; and (ii) the average planning time ( t ) spent by the agent to plan the next actions. Mean results were calculated across 50 executions . Every experiment ran independently; thus, no knowledge was carried from one execution to another. The calculated errors ( ± Err ) represent the 95% confidence interval of a two-sample t-test. Note that our results and baselines are separated into two categories (i) Quick Planning : grouping methods that perform the decision-making process within a reasonable time window; and (ii) Long Planning : presenting the ρ -POMCP's results (without time constraints). We separated them to make their understanding easier.

Finally, we refer the reader to our Appendix D to information about our hyperparameters set.

Benchmarks study All results are presented in Table 1. In addition, we highlight the pros and cons of our method, analysing our limitations while discussing the outcome for each environment.

In the Tiger domain , TB ρ -POMCP presents the best average reward among the Quick Planning methods ( ρ < 0 . 01 ). IB-POMCP still outperforms POMCP, showing significant improvement in terms of reward ( ρ < 0 . 01 ). For this specific problem, since the action 'listen' generates variation in the entropy, IB-POMCP keeps performing it repeatedly until other action value outcomes the 'listen' value in the best action selection procedure (Section 4, step (v)), a circumstance which leads our method to reduce its average reward collection. Hence, when facing problems where seeking spots of high uncertainty produces small penalties, IB-POMCP may collect penalties until it significantly decreases the uncertainty and chooses another path.

In the Maze domain , we observe that IB-POMCP presents a significantly better average reward in 3 out of 4 proposed scenarios ( ρ ≤ 0 . 015 ), except for M3, for which POMCP presents a better result. Note that reducing uncertainty leads to increasing reward; i.e. the faster an agent can access areas with high uncertainty, the higher its received reward. IB-POMCP's results match our expectations (given the developed rationale throughout our methodology) since we build it to, besides tracking the rewards available in the scenario, often seek paths that lead to spots with high entropy in order to decrease uncertainty, what increases reward in this scenario. In terms of time, we present significantly faster reasoning in all scenarios ( ρ < 0 . 01 ). Investigating why IB-POMCP runs faster than POMCP, we found that our information-guided planning leads the algorithm to perform more transitions during the rollout phase than while performing the simulation inside our actual search tree, with a rate # rollout # simulation = 1 . 61 , whereas POMCP presents a rate of 1 . 28 . Because rollout transitions run faster than simulation transitions inside the tree, we can save time by performing them frequently.

In the Rock Sample problem , we can see that IB-POMCP shows a significant improvement in terms of reward collection ( ρ ≤ 0 . 02 ) in 3 out of 4 scenarios - except for the simplest scenario RockSample22 (R0), which presents a p-value of ρ ≤ 0 . 11 . In terms of reasoning time, POMCP is slightly faster than all Quick Planning methods in 3 out of 4 scenarios ( ρ ≤ 0 . 01 ), except for RockSample17 (R3).

In Tag , IB-POMCP presents significant improvement in terms of reward collection and reasoning time against all baselines ( ρ ≤ 0 . 04 ). As in the Maze, we believe that IB-POMCP is faster because it simulates transitions in rollouts more often (in a ratio of # rollout # simulation = 1 . 7 against 1 . 61 for POMCP).

In LaserTag , we have a tie between all methods, with no statistically significant difference spotted across metrics ( ρ ≥ 0 . 32 for reward collection and ρ ≥ 0 . 06 for reasoning time).

Finally, in the Foraging domain , our proposed method significantly outperformed all the baselines ( ρ < 0 . 01 ). The Foraging problem represents our most complex scenario, and empirically shows how IB-POMCP can overcome the necessity of adjusting the reasoning horizon to perform planning in settings that deliver rewards sparsely. The only reward available in these scenarios comes from the collection of tasks. Consequently, while attempting to solve a task, the number of actions that the agent needs to plan and execute in sequence may approach or exceed the reasoning horizon size. In this case, the probability of experiencing this reward is low, and algorithms that only follow the reward in planning will rarely obtain it in their simulations; hence, they fail to plan effectively.

Overall, we experimentally demonstrated that our novel proposed method can significantly improve the performance of our agent and its planning capabilities without penalising the reasoning time. We also demonstrated that our proposal can improve the decision-making process by using only the information generated during the tree search process and, in contrast to ρ -POMCP, without relying on the explicit representation of latent functions (e.g., the observation function).

Ablation study To evaluate the impact of our proposed modifications and enhancements, we performed an ablation study over our method. We consider 2 different variations of IB-POMCP that partially implement its key points in this experiment: (a) the Information-guided Particle Reinvigoration POMCP (IPR-POMCP) , which implements the modifications to the particle filter reinvigoration process (explained in Section 4, steps (i) and (ii)); and (b) the I-UCB POMCP, which implements the proposed modifications to the search and simulation process (explained in Section 4, steps (iii) and (iv)). We consider 4 different scenarios, one for each environment presented in Table 1, to run our experiments. The results are depicted in Figure 2. Both additional methods proposed for the study are available in our GitHub repository together with the complete code of this work.

(a) Tiger (T0)

0

25

50

75

100

125

150

175

200

Iteration

5

4

3

2

1

0

Cumulative Reward

(b) MazeDots (M2)

0

25

50

75

100

125

150

175

200

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

3.0

POMCP

IPR-POMCP

IUCB-POMCP

IB-POMCP

(c) U-obstacles (F2)

0

25

50

75

100

125

150

175

200

Iteration

0

1

2

3

4

5

(d) TheOffice (F4)

0

25

50

75

100

125

150

175

200

Iteration

0

1

2

3

4

By analysing the graphs, it is clear that the I-UCB and implementation of an information-guided particle reinvigoration process for the Tiger, Maze, and Foraging problems directly enhance the reasoning capabilities of our planning method, which is translated here in terms of reward collection. For the Tiger and Foraging problems, diversifying the set of particles (I-UCB POMCP) alone presents improvements for POMCP; however, it has less impact than including entropy in the search process (IPR-POMCP). On the other hand, in the Maze environment, both approaches present similar improvements for the POMCP algorithm. However, when combined, they significantly improved upon the baseline results. Our intuition behind these results is that the IPR-POMCP proposal is responsible for delivering a better set of particles and a better initial estimation of the current state to the agent before simulating actions, whereas I-UCB continually guides and affects the planning process from the beginning of the simulations until the decision of the best action.

7 Conclusion

In this study, we propose IB-POMCP , a novel algorithm for planning under uncertainty that is capable of aggregating information entropy into a decision-making algorithm using our modified version of the UCB function, I-UCB . We present the theoretical properties for convergence under certain assumptions, which are supported by empirical results collected in five different domains. Overall, we increase the reward collection by up to 10 times in comparison with TB ρ -POMCP (U-shaped, F1), in addition to reducing the reasoning time by up to 93% compared to ρ -POMCP (MazeHoles).

Acknowledgments and Disclosure of Funding

We gratefully acknowledge the financial support provided by a PhD studentship from the Faculty of Science and Technology (FST) at Lancaster University, United Kingdom. We also thank FST for granting us access to their High-End Computing Cluster (HEC 3.0), which was pivotal to this research. We extend our sincere gratitude to Dr Mike Pacey for his expertise and assistance provided to correctly setting up HEC 3.0 and running our experiments. Our special thanks go to Dr Ronghui Mu for providing insightful comments and feedback on an earlier draft of this paper.

References

D. Agudelo-España, S. Gomez-Gonzalez, S. Bauer, B. Schölkopf, and J. Peters. Bayesian online prediction of change points. In Conference on Uncertainty in Artificial Intelligence , pages 320-329. PMLR, 2020.

S. V. Albrecht and P. Stone. Reasoning about hypothetical agent behaviours and their parameters. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , AAMAS '17, page 547-555, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.

S. Amini, M. Palhang, and N. Mozayani. POMCP-based decentralized spatial task allocation algorithms for partially observable environments. Applied Intelligence , 53(10):12613-12631, 2023.

M. Araya, O. Buffet, V. Thomas, and F. Charpillet. A POMDP extension with belief-dependent rewards. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems , volume 23, 2010.

A. Bai, F. Wu, Z. Zhang, and X. Chen. Thompson sampling based monte-carlo planning in POMDPs. In International Conference on Automated Planning and Scheduling (ICAPS) , page 29-37. AAAI Press, 2014. ISBN 9781577356608.

A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial observability. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages 44-52, 2022.

S. Barrett, A. Rosenfeld, S. Kraus, and P. Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence , 242:132-171, 2017.

S. R. Chowdhury, R. Oliveira, and F. Ramos. Active learning of conditional mean embeddings via bayesian optimisation. In Conference on Uncertainty in Artificial Intelligence , pages 1119-1128. PMLR, 2020.

M. A. do Carmo Alves, A. Varma, Y. Elkhatib, and L. Soriano Marcolino. AdLeap-MAS: An open-source multi-agent simulator for ad-hoc reasoning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages 1893-1895, 2022.

D. Fox. Adapting the sample size in particle filters through kld-sampling. The international Journal of robotics research , 22(12):985-1003, 2003.

M. Ghavamzadeh, S. Mannor, J. Pineau, A. Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning , 8(5-6):359-483, 2015.

M. Gini. Multi-robot allocation of tasks with temporal and ordering constraints. In AAAI Conference on Artificial Intelligence , 2017.

A. Guez, D. Silver, and P. Dayan. Efficient Bayes-adaptive reinforcement learning using sample-based search. International Conference on Neural Information Processing Systems , 25, 2012.

A. Hayashi, D. Ruiken, T. Hasegawa, and C. Goerick. Reasoning about uncertain parameters and agent behaviors through encoded experiences and belief planning. Artif. Intell. , 280(C), Mar. 2020. ISSN 0004-3702.

M. Hoerger and H. Kurniawati. An on-line POMDP solver for continuous observation spaces. In International Conference on Robotics and Automation (ICRA) , pages 7643-7649. IEEE, 2021.

S. Katt, F. A. Oliehoek, and C. Amato. Learning in POMDPs with Monte Carlo Tree Search. In International Conference on Machine Learning (ICML) , pages 1819-1827, 2017.

S. Katt, F. A. Oliehoek, and C. Amato. Bayesian Reinforcement Learning in Factored POMDPs. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) , 2019.

P. Kozakowski, M. Pacek, and P. Milo´ s. Planning and learning using adaptive entropy tree search. In International Joint Conference on Neural Networks (IJCNN) . IEEE, 2022.

H. Kurniawati, D. Hsu, and W. S. Lee. SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and systems , 2008.

T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985. ISSN 0196-8858.

M. Lauri and R. Ritala. Planning for robotic exploration based on forward simulation. Robotics and Autonomous Systems , 83:15-31, 2016.

L. S. Marcolino and L. Chaimowicz. No robot left behind: Coordination to overcome local minima in swarm navigation. In International Conference on Robotics and Automation (ICRA) , pages 1904-1909. IEEE, 2008.

G. Mazzi, D. Meli, A. Castellini, and A. Farinelli. Learning logic specifications for soft policy guidance in POMCP. arXiv preprint arXiv:2303.09172 , 2023.

J. Mern, A. Yildiz, Z. Sunberg, T. Mukerji, and M. J. Kochenderfer. Bayesian optimized Monte Carlo planning. In AAAI Conference on Artificial Intelligence , volume 35. No. 13, pages 11880-11887, 2021.

R. O. Moraes, J. R. Marino, L. H. Lelis, and M. A. Nascimento. Action abstractions for combinatorial multi-armed bandit tree search. In Artificial Intelligence and Interactive Digital Entertainment Conference , 2018.

L. Niss and A. Tewari. What you see may not be what you get: UCB bandit algorithms robust to ε -contamination. In Conference on Uncertainty in Artificial Intelligence , pages 450-459. PMLR, 2020.

G. Papoudakis, F. Christianos, and S. Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems , 34, 2021.

L. Pelcner, S. Li, M. Aparecido do Carmo Alves, L. Soriano Marcolino, and A. Collins. Realtime learning and planning in environments with swarms: A hierarchical and a parameter-based simulation approach. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) , page 1019-1027, 2020.

F. Riccio, R. Capobianco, and D. Nardi. Loop: Iterative learning for optimistic planning on robots. Robotics and Autonomous Systems , 136:103693, 2021.

E. Shafipour Yourdshahi, M. A. do Carmo Alves, A. Varma, L. Soriano Marcolino, J. Ueyama, and P. Angelov. On-line estimators for ad-hoc task execution: learning types and parameters of teammates for effective teamwork. Autonomous Agents and Multi-Agent Systems , 36(2):45, 2022.

D. Shah, Q. Xie, and Z. Xu. Non-asymptotic analysis of Monte Carlo Tree Search. In Abstracts of the SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer Systems , pages 31-32, 2020.

D. Silver and J. Veness. Monte-carlo planning in large POMDPs. In Neural Information Processing Systems , pages 2164--2172, 2010.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of Go without human knowledge. Nature , 550 (7676):354-359, 2017.

A. Somani, N. Ye, D. Hsu, and W. S. Lee. DESPOT: Online POMDP planning with regularization. Advances in neural information processing systems , 26, 2013.

Z. N. Sunberg and M. J. Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. In International Conference on Automated Planning and Scheduling , 2018.

V. Thomas, G. Hutin, and O. Buffet. Monte Carlo information-oriented planning. In European Conference On Artificial Antelligence , 2020.

C. Xiao, R. Huang, J. Mei, D. Schuurmans, and M. Müller. Maximum entropy Monte-Carlo planning. Advances in Neural Information Processing Systems , 32, 2019.

A. Yadav, R. Noothigattu, E. Rice, L. Onasch-Vera, L. Soriano Marcolino, and M. Tambe. Please be an influencer? contingency-aware influence maximization. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) , page 1423-1431, 2018.

E. S. Yourdshahi, T. Pinder, G. Dhawan, L. S. Marcolino, and P. Angelov. Towards large scale ad-hoc teamwork. In International Conference on Agents (ICA) , pages 44-49. IEEE, 2018.

W. J. Zhou, B. Subagdja, A.-H. Tan, and D. W.-S. Ong. Hierarchical control of multi-agent reinforcement learning team in real-time strategy (RTS) games. Expert Systems with Applications , 186:115707, 2021.

M. Zuccotto, A. Castellini, and A. Farinelli. Learning state-variable relationships for improving POMCP performance. In ACM/SIGAPP Symposium on Applied Computing (SAC) , pages 739-747, 2022.

A Algorithms

In this section, we provide the pseudo-code of the particle filter update step (Algorithm 1), and for IB-POMCP's main routine (Algorithm 2). To clarify our approach better and support the text presented in the main paper, both algorithms highlight the differences between our approach and POMCP [32]. See Section 3 in the main paper for further details about our rationale and methodology.

Algorithm 1 Particle Filter Update. The starred lines (with a light grey background) highlight the differences between POMCP and our proposal, IB-POMCP. Union considers repetition here .

Algorithm 1 Particle Filter Update - Details and Further Discussion: We save some space here with the intention of assuring the reader's understanding of the novelty that lies on our particle filter update proposal. Our method employs a dynamic threshold derived from the dynamic value of ˜ P ( z | ha ) (or specifically, 1 -˜ P ( z | ha ) that determines the ratio of uniform samples in the reinvigorated particle filter). Unlike some traditional approaches which use a fixed value for that threshold, or some other state-of-the-art contributions, which mostly rely on additional knowledge and expensive calculations [17, 14, 10], IB-POMCP does not need an explicit true distribution of observations, transition function or training data in this process by using our proposed ˜ P ( z | h r a ) probability.

It is important to note that the action selection process within the planning phase has a direct impact on the ˜ P ( z | h r a ) value. Therefore, the strategy used to choose actions for simulation during planning will affect the particle filter, since it affects the dynamic threshold ˜ P ( z | h r a ) . Hence, note that the I-UCB equation used in the search tree also has an impact on belief tracking.

Our suggested algorithm to investigate the impact of this proposal (Ablation study in Section 6), IPR-POMCP, employs our dynamic threshold and shows significant improvement to the conventional POMCP, which follows a standard particle reinvigoration paradigm (with a fixed parameter for reinvigoration). Hence, just using the dynamic threshold ˜ P ( z | h r a ) improves results. Furthermore, the ablation study indicates that combining the dynamic threshold with the I-UCB equation yields even more favourable results and that using solely the I-UCB equation without the dynamic threshold (IUCB-POMCP baseline) results in worse outcomes than the full IB-POMCP approach. Hence, both the dynamic threshold and the I-UCB equation are important to improve results, and their simultaneous application produces even greater enhancements than when employed individually.

What is perhaps still left open is whether the I-UCB equation leads to better ˜ P ( z | h r a ) values than the UCB equation. Since our approach takes the estimated entropy into consideration for action selection we believe that I-UCB may lead to better threshold values.

For example, in cases where a node possesses high entropy and no reward in the reasoning horizon, the I-UCB equation could prioritise visiting it more often than the UCB equation would, increasing the ˜ P ( z | h r a ) value for that particular node (note that it will not imply in worse probability estimation because the entropy is based on the estimation of observations and the particle filter gives the probabilities of states). The high entropy indicates that the node accommodates a more diverse set of particles in the particle filter and hence sampling particles from a uniform distribution during particle reinvigoration (to increase diversity) becomes less useful. Hence, the implications of the I-UCB equation appear favourable as it leads to fewer particle samples from the uniform distribution in this case (as we use the ratio 1 -˜ P ( z | h r a ) ). On the other hand, if a node has low entropy, the particle filter may easily get biased to a small set of states. In this case, I-UCB would lead to a higher number of particles sampled from the uniform distribution than the UCB equation in the particle reinvigoration,

since it would have a lower ˜ P ( z | h r a ) value in I-UCB. Therefore, we believe that I-UCB should be better than UCB for defining the dynamic threshold ˜ P ( z | h r a ) , and hence better in belief tracking.

Algorithm 2 IB-POMCP's Planning. The starred lines in a light grey background highlight the difference between POMCP and our proposal. We suggest the reader to Silver and Veness (2010) [32] for further details. γ is the historical discount factor, depth max is the maximum depth for the tree, and z h represents the last observation found (associated with node h ).

B Detailed Theoretical Analysis

In this section, we offer a comprehensive theoretical analysis of our proposed method. In addition to the assumption, lemma, theorems, and corollary presented in the main paper, our complete analysis encompasses all proofs, along with our definition (Definition 1) and lemma (Lemma 2).

Definition 1 An estimated ˆ V ( s ) is said to be ϵ -optimal if | ˆ V ( s ) -V ∗ ( s ) | ≤ ϵ . From the literature, actions taken from the result of ϵ -optimal value function estimations are known as ϵ -optimal actions.

Assumption 2 Given a generic POMDP represented by the tuple ( S , A , R , T , Z , Z , H ) , the R is bounded and there exists a positive maximum reward value r max ∈ R .

Lemma 1 α ( h ) ∈ [0 , 1] converges to 0 as the number of visits N ( h ) approach the infinity. Mathematically, lim N ( h ) →∞ α ( h ) = 0 .

Proof: Considers our equation for a root node with history h :

First, we can re-organise the equation as follows:

Analysing the limit over the new expression, we can find that:

by following the rationale:

Therefore, since we know that:

using the sandwich theorem, we find that α ( h ) → 0 as N ( h ) →∞ ■

Now, we introduce Lemma 2, which will be used as a key result that supports Theorem 1. In other words, we will use Lemma 2's proof to reach a contradiction later in order to prove Theorem 1.

Lemma 2 Assume an action a i that is taken a finite number of times and a j that is taken infinitely during IB-POMCP's search for any node h . There ∃ t ′′ > t ′ such that I-UCB t ′′ ( ha i ) ≥ I-UCB t ′′ ( ha j ) , where t ′ is a finite iteration number after which a i is never taken.

Proof: Let us assume that there exists an iteration time t that is greater than t ′ where a i is never taken again for simulation. Consider the expression I-UCB t ( ha i ) -I-UCB t ( ha j ) , which is equal to:

Trivially, ( 1 √ N t ( ha i ) -1 √ N t ( ha j ) ) → 1 √ N t ′ ( ha i ) as N t ( ha j ) → ∞ , since 1 √ N t ( ha j ) → 0 and N t ( ha i ) will be constant under our assumption. Note also that 1 √ N t ( ha i ) -1 √ N t ( ha j ) is increasing and will become positive at some time t ∗ when N t ∗ ( ha j ) > N t ∗ ( ha i ) . After then (i.e. t > t ∗ ), the term √ ln( N t ( h ))( 1 √ N t ( ha i ) -1 √ N t ( ha j ) ) will diverge to ∞ as t → ∞ , ln( N t ( h )) → ∞ . Further, note that, in a single time step, the maximum difference in rewards is bounded by r max , therefore V t ( ha i ) -V t ( ha j ) ≥ -∑ ∞ k =0 γ k r max = -r max 1 -γ (This is because |V t ( ha i ) -V t ( ha j ) | ≤ r max 1 -γ , as shown in Theorem 3 and this implies that -r max 1 -γ ≤ V t ( ha i ) - V t ( ha j ) ≤ r max 1 -γ ) and α t ( h )( ˆ H t ( ha j ) -ˆ H t ( ha i )) ≥ -1 as 0 ≤ α t ( h ) ≤ 1 and 0 ≤ ˆ H t ( h ) ≤ 1 . Therefore, ∃ t ′′ where I-UCB t ′′ ( ha i ) -I-UCB t ′′ ( ha j ) ≥ 0 . ■

̸

Theorem 1 For any non-empty history h = ∅ ∈ H and ∀ a ∈ A , as N ( h ) →∞ , N ( ha ) →∞ , we have that all states b in B ( h, b ) which P ( s = b | h ) > 0 will be visited infinitely many times.

Proof: Let us assume that not all actions are taken infinitely many times. There must exist at least one action a i ∈ A that is taken a finite number of times and, hence, ∃ t ′ after which the action a i will never be taken. Now, consider any action a j that is taken infinitely many times as N ( h ) →∞ . In Lemma 2, we show that ∃ t ′′ > t ′ such that:

We know that the second term in I-UCB grows faster for a i than a j , as seen from the I-UCB function, since 1 N t ( ha i ) is constant for t > t ′ . Consequently, we reach a conclusion that a i is chosen over a j again after t ′ . This further means that for all actions a k 's, we can find a such a finite time instant after which a i is preferred over a k . Therefore, at some point, the action a i has to be chosen over all others. This is a contradiction to our original claim. Therefore, all action a ∈ A are taken infinitely many times and, hence, all states will be visited infinitely many times. ■

Theorem 2 IB-POMCP's nodes converge to the optimal value for a fixed horizon D as the number of search iterations goes to infinite (i.e., N ( h r ) →∞ ).

Proof: We adapt the proof from the UCT convergence in Shah et al. 2020 and POMCP convergence in Silver and Veness 2010. We induct over the depth of the tree, starting from the leaf up to the root. From Theorem 1, the leaf nodes will have an unbiased estimation of the expected reward r a h = E b ∈B ( h ) ( R ( b, a )) , ∀ a ∈ A . Note that we are using r a h as the expected reward given a h , which is different from an immediate reward r a b = R ( b, a ) given a state b . Similarly, any state at level D -1 has a simple UCB-like problem since each state is visited infinitely many times and the one-step rewards have an unbiased estimation. In detail, consider the leaves at level | h | = D , as N ( h r ) →∞⇒N ( h ) →∞ , α → 0 , and in the absence of children's entropy H ( ha ) (as it is the leaf node and has no children), r a h converges unbiasedly ∀ h ∈ H , a ∈ A . Consequently, V ( h ) →V ∗ ( h ) since it is a single decision process, akin to multi-armed bandits. Now, we assume convergence for all levels from the leaves up to | h ′ | = | h | +1 , and as we know r a h = ∑ s ∈ S P ( s t = b | h ) r a b . As we know, the value function satisfies:

From our hypothesis, the V ∗ ( h ′ ) must converge and moreover, through the sampling process, the value function must converge for level | h | [32] . Consequently, by backward induction, the whole tree's values must converge. ■

Theorem 3 Given any ϵ > 0 , there is a finite horizon D for the problem, such that |V ∗ ( h r ) -V ∗ D ( h r ) | ≤ ϵ , where h r represents the root of the tree.

Proof: Consider π ∗ and π ∗ D to be the optimal policies for the infinite and finite horizon problem, respectively. Also, assume that ϵ is an arbitrary, positive real number . Now, let the value functions for these problems be V ∗ ( h r ) and V ∗ D ( h r ) . For the root r , we can find that:

since we can calculate D = log γ ϵ (1 -γ ) r max , we can also ensure that the relation holds. ■

Corollary 1 IB-POMCP converges to ϵ -optimal actions at the root ( h r ) in a γ -discounted infinite horizon scenario.

Proof: From Theorem 6, we can always find a depth D such that the value function of the finitehorizon problem is arbitrarily close to the infinite-horizon problem. From Theorem 5, we know that we can converge to the optimal value in a finite horizon POMDP. Therefore, since α → 0 as N ( h r ) →∞ , IB-POMCP will converge to ϵ -optimal actions by choosing D appropriately. ■

C Benchmark Domains and Scenarios

All environments used were obtained from the literature. The Tiger problem is a well-known standard problem [36]. For the Maze domain, we based our design on Thomas et al. 2020 [36]. For the RockSample problem, we designed our own scenarios but based the implementation on Thomas et al. 2020 [36]. For Tag and LaserTag, we used the scenarios proposed by Somani et al. (2013) [34]. For Foraging [2], we proposed our own configurations for each scenario.

Tiger Environment: The Tiger problem is a classical benchmark where an agent must choose between two doors: one hides a treasure; the other, a tiger. The objective is to find out which door hides the treasure. The agent can decide which door to open or wait and listen to the tiger (with a certain probability of mishearing it behind one of the doors) before making the final decision.

The agent has a 15% probability of mishearing the tiger. A maximum of 20 actions are allowed per experiment. Choosing the right door gives the agent a reward of +0 . 1 , the wrong door -1 , and the listening action penalises the agent with a -0 . 01 reward. Thus, each agent could try to listen to the tiger 19 times before making the final decision to open either the left or right door.

Maze Environment: This is an Active Localisation Problem where an agent navigates through a toroidal grid and tries to localise its own position by collecting information about its current position. The reward is based on the entropy of the current belief. There is a change of 15% to miss observe the colour of the cell. The belief follows a Bayesian Update process. Figure 1 illustrates the scenarios.

(a) Maze Cross This is a 5x5 scenario in which black cells are positioned to ease the agent's localisation problem. Figure 1a illustrates the scenario's configuration.

(b) Maze Holes This is an 8x8 scenario, which requires the agent to reason one step in advance to search for the missing black cell in this regular configuration. Figure 1b illustrates the scenario.

(c) Maze Dots This is a 6x6 scenario, identical to Maze Hole except that black cells are separated by several white cells. Figure 1c illustrates the scenario.

(d) Maze Grid-X This is a 3x3 scenario, with an upper white triangle and a bottom black triangle for the localisation. Figure 1c illustrates the scenario.

(b) Maze Holes.

600

600

ooQ

RockSample Environment: The context of this problem illustrates a rover exploring an unknown planet. The rover's objective is to earn rewards by both sampling rocks in the environment and leaving the planet with the samples. While the positions of the rover and the rocks are known, not all rocks hold scientific value, which are referred to as 'good' rocks. Given the cost associated with rock sampling, the rover is equipped with a noisy long-range sensor, enabling it to assess a rock's potential scientific value before deciding whether to approach and sample it. The environment is a grid map of size N × N with k rocks. The POMDP model for RockSample ( N,k ) follows: The state space is a combination of k +1 features, where Positions represent the rover's location { (1 , 1) , (1 , 2) , ..., ( N,N ) } , and k binary features, RockType i , indicating whether each rock is 'Good' or 'Bad'. There is a terminal state (portal) at the top-right corner of the map. The rover has a choice of k +5 actions: { North,South,East,West,Sample,Check 1 , ..., Check k } . The first four are deterministic single-step motion actions. The Sample action involves sampling the rock at the rover's current location. If the rock is determined to be 'Good', the rover receives a reward of +1 , and the rock transitions to 'Bad', signifying no further benefit from sampling. If the rock is 'Bad', a penalty of -1 is delivered to the agent. Moving into the exit portal yields a reward of +0 . 0001 . All other actions have no associated cost or reward. Each Check i action employs the rover's long-range sensor to observe Rock i , yielding a noisy observation of either 'Good' or 'Bad'. The noise in the long-range sensor reading is influenced by the efficiency η , which decreases exponentially as a function of Euclidean distance from the target, following η = exp ( -0 . 2 EuclideanDistance ( rover, Rock i )) . Initially, every rock is assumed to have an equal probability of being 'Good' or 'Bad'. Figure 2 shows the four scenario configurations used.

(a) RockSample22 This scenario is set on a 5x5 grid, featuring 2 good rocks and 2 bad rocks. The rover initiates its mission from the middle of these rocks and must locate each specific good rock among the bad rocks. Figure 2a illustrates the scenario's configuration.

(b) RockSample40 In this 5x5 scenario, there are 4 good rocks initially placed in the environment, with no bad rocks present. In contrast to RockSample22, the rover's goal here is to collect more rewards as there are more good rocks available. The rover starts its journey from the centre of these rocks. Figure 2b illustrates the scenario's configuration.

(c) RockSample44 This 10x10 scenario features 4 good rocks and 4 bad rocks. The rover's starting point is the bottom-left corner of the map. Similar to RockSample22 but on a larger scale, the rover's challenge is to locate each specific good rock among the bad rocks to optimize its reward. Figure 2c illustrates the scenario's configuration.

(d) RockSample17 In this 10x10 scenario, there's only 1 good rock and 7 bad rocks, and the rover begins its journey in the bottom-left corner of the map. Unlike RockSample44, the expectation here is for the rover to collect fewer rewards. However, if the rover finds the only good rock available, it can significantly improve its performance in terms of reward collection. Figure 2c illustrates the scenario's configuration.

(a) RockSample22.

(b) RockSample40.

(c) RockSample44.

(d) RockSample17.

000

000

000

Tag and LaserTag Environment: In the Tag scenario (Figure 3a), the agent's primary objective is to locate and tag a target that actively moves away. Both the agent and the target navigate within a grid featuring 29 possible positions. While the agent is aware of its own location, it can only observe the target's position when they occupy the same spot. The agent has the option to either stay in its current position or move to any of the four adjacent positions, incurring a cost of -0 . 1 for each move. Additionally, it can opt to execute the tag action, wherein it receives a reward of +1 for a successful tag but receives a penalty of -1 if it fails to tag the target.

LaserTag (Figure 3b) is an augmented version of Tag, where an agent is acting in a 7x11 grid world containing randomly placed obstacles. The agent's behaviour and that of its opponent align with the rules established in Tag. However, in LaserTag, there's a key distinction: the agent possesses prior knowledge of its initial location. Additionally, the agent is equipped with a laser system capable of providing distance estimations (in cells' units) in 8 directions. The laser readings are generated from a normal distribution centred around the agent's true distance from the nearest obstacle in each direction, with a standard deviation of 2 . 5 units. These readings are then discretised into whole units, resulting in an observation comprising a set of 8 integers. Figure 3 illustrates the scenarios.

(a) Tag.

(b) LaserTag.

R

R

Foraging Environment: This is a common problem used for evaluating online planning algorithms [2, 7]. This domain presents an agent that must collect boxes displaced in a rectangular gridworld . The problem is defined over partial observability and the agent does not know how the tasks are distributed. The problem ends when the agent collects all boxes . A reward of +1 is delivered to the agent every time a box is collected. The radius and the angle of vision (integer numbers) for the agents are, respectively, 20% of the diagonal dimension in the environment in cells' unit and 90 o - for example, if we have a 10 × 10 environment, so our vision radius will be radius = 0 . 2 √ x 2 dim + y 2 dim = 0 . 2 √ 100 + 100 = 0 . 2 √ 200 = 2 . Additionally, agents have memory, i.e., after seeing (in the real world) a state or position in the environment, they will sample the next possible world configuration considering this information. Moreover, obstacles block the agent's vision, hence, it can not see tasks across the walls. Figure 4 illustrates the scenarios.

(a) The Corridor This is a simple (20 , 2) scenario, where the agent starts in the left part of a corridor with a box at its side and another at the end of the corridor. The idea here is to test the agent's capability to find isolated reward spots. Figure 4a illustrates the scenario.

(b) U-Shaped A complex version of The Corridor, the U-shaped is a 15 × 15 scenario, where the agent starts at the tip of a U-shaped corridor with a box at the beginning, one in the middle and another at the end. The idea here is to test the agent's capability to perform simple planning, but facing sparse rewards collection. Figure 4b illustrates the scenario.

(c) U-Obstacles A complex 20 × 10 scenario, where the agent needs to collect the boxes distributed in a room, but there are obstacles (U-shaped walls) that block its vision. This scenario is commonly used to study local minima issues that emerge in robotic navigation problems [22]. The idea here is to test the agent's capability to perform planning with sparse reward collections, and the vision is blocked by obstacles. Figure 4c illustrates the scenario.

(d) The Warehouse This scenario is our largest in terms of dimensions, 20 × 20 . The agent's objective is to locate specific reward spots (groups of boxes). These reward clusters are situated at considerable distances from one another. Consequently, the agent cannot observe multiple reward clusters from a single position, making planning more challenging. Figure 4d illustrates the scenario.

(e) The Office Our most complex 15 × 10 scenario, where the agent needs to collect the boxes distributed in an office. The idea here is to test the agent's capability to perform complex planning when tasks are distributed in different rooms and its vision is blocked by walls. The agent needs to enter each room to finish the problem. Figure 4e illustrates the scenario.

D Hyperparameters

We used a single hyperparameter set for all Monte-Carlo Tree Search-based methods:

Historical weight/Discount factor: γ = 0 . 95

Maximum depth for the tree: 20

Maximum number of simulations: 250 (per search)

For ρ -POMCP and TB ρ -POMCP specific hyperparameters, we consider the following:

Small bag size: | B | = 10

TB ρ -POMCP runs within the same time window of IB-POMCP's average planning time.

Finally, considering IB-POMCP practical enhancement, we applied q = 0 . 2 , hence α ∈ [0 . 2 , 0 . 8] .

(a) The Corridor.

(b) U-shaped.

(c) U-Obstacles.

(d) The Warehouse.

A

0

(e) The Office.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an extension to the POMCP algorithm for online planning under partial observability, called Information-Based POMCP (IB-POMCP). The distinguishing feature of IB-POMCP is that it uses information entropy to enhance the tree search procedure. One effect of this extension is that IB-POMCP is better able to handle sparse rewards, performing its search based on which branches of the tree are likely to yield higher information gain, unlike previous methods which consider only exploration of the tree. \n\nThe first modification to POMCP is that when an action is taken in IB-POMCP, new particles are sampled after an action is taken, depending on how unlikely the resulting observation is. If an observation is received that had a high probability, then the particles in the new b root node are expected to capture the belief well. On the other hand, if the probability is low, then more new particles are generated from a uniform distribution. The intention of this is to improve the algorithm’s starting point at each step. \n\nThe second modification is the introduction of a new exploration coefficient which is recalculated at each iteration of the search - replacing the alpha coefficient from UCB1 with a new version for I-UCB. The I-UCB alpha coefficient captures information about visit counts similar to UCB1, but includes an additional term which favours nodes with high entropy. The idea of this term is to seek information gain during the search procedure, where the entropy is estimated from the set of observations seen so far. This is a more nuanced search procedure than standard UCB1, which only considers the number of times each node has been visited, but not the distribution of observations received.\n\nSTRENGTHS:\nThe paper presents an approach to planning for partially observable problems that makes a clear original contribution to the literature on planning under partial observability.  \n\nThe authors justify the benefits of their information-guided approach to POMCP-style planning in a manner that makes intuitive sense. The paper also provides a strong experimental justification for the approach, evaluating against relevant baselines in standard domains, as well as providing an ablation study that justifies the individual components of their algorithm. The experiments cover a good range of domains, and the intuition behind the results is explained fairly well. \n\nThe paper is overall well structured and written, and presents its key ideas in a clear and logical manner.\n\nWEAKNESSES:\nAfter discussion, I believe that I was a bit too negative regarding the substantiality of the contribution, so I'm increasing the score. Please do not that the section with the theoretical results needs improvement. I think adding intuition and discussion around those results is more important than adding a new domain to the experiments.\n\n---------\nMy main concern with the paper is the incremental aspect of its contributions. The two improvements over POMCP are (i) adding some standard tricks to the particle filter so that it tracks the belief more accurately, and (ii) a new action selection equation which simply considers entropy of the belief as an extra term. This is a bit on the light side. I’d expect a strong theoretical analysis and justification of Equation (3) to bring the significance of the contribution up. Furthermore, the technical section is simply a listing of results without any context or intuition - not enough insight is given to the reader.\n\nOther issues:\n\n* Equation 2 could do with more justification/explanation of how the update works.\n* Line 128 the terminology “the probability of stepping into this node” is unclear and needs further explanation at this point in the text. It made sense to me only after I got up to Section 4(ii). \n* Figure 1 caption should explain in more detail what is being shown (e.g. “bright red dots represent X, faded red dots represent Y, blue dots represent Z”). This is mentioned in the text in Line 232 but more detail would be helpful to clarify the diagram. \n* Following from the above, I think the authors should make a clearer distinction between the (online) planning phase and action execution. For example, I don’t think it’s clear from the paper that one *only* updates the belief after executing an action, and the POMCP tree keeps histories as nodes but does not do any explicit belief update.\n* Lines 264-270 could do with a clearer explanation of the terms “information gain” vs “exploration gain”.\n\nQUESTIONS:\n* Please argue for the significance of the contribution presented in the paper. How is this a substantial step from the state-of-the-art?\n* Unclear to me why $\\alpha(h_\\tau)$ (Equation 1) is defined only at the root node rather than at every subsequent node - is this to make the algorithm simpler, or is there an actual justification for why this is preferable or doesn’t matter?\n* Line 351-354 is a bit of a long-winded sentence and unclear what is meant by the final bit - “often seek paths that lead to spots with high entropy in order to decrease uncertainty”, what uncertainty is being decreased here?  \n* Line 369 what is meant by “without penalising reasoning time”? Is this just saying that IB-POMCP performs well without taking longer to plan than other methods, or something more specific?\n\nLIMITATIONS:\nThe experiments section might be strengthened by the addition of another baseline which uses some sort of latent world model (e.g. SARSOP?). It would be interesting to see a comparison with a model that does better than IB-POMCP on performance but requires much more computation time to do so. This would provide a concrete justification for the claim in Lines 51-55 that such modelling is too computationally intensive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents IB-POMCP, a novel algorithm for online planning under partial observability, that uses entropy estimation to guide the tree search. The main concept is to replace the traditional explore-exploit balancer (UCB1) with (an IUCB score) which is a weighted mixture of value, exploration score, and entropy score. This idea is motivated by the failure modes of other algorithms tackling POMDP problems: poor performance in the case of sparse rewards and especially sparse rewards that are often delivered outside of the algorithm's planning horizon. IB-POMCP is compared against POMCP and TB rho-POMCP on 3 tasks: Tiger, Maze, and Foraging. The experiments show a clear advantage of IB-POMCP. There is also an ablation study which gives even more insight into the method.\n\nSTRENGTHS:\nThe experiments are carefully made (with confidence intervals) and clearly show the advantage of the method over baselines. Also, there is a time comparison, which shows another desired property of IB-POMCP: it is usually faster than baselines. There is an exhaustive discussion of the results. The paper contains an ablation study, which compares the effect of two different aspects of IB-POMCP:\na) information guided particle reinvigoration\nb) U-UCB score\nIt is demonstrated that both of the above contribute to the effectiveness of IB-POMCP and there is an interesting comment about these results.  \nThe paper is well-written, images are helpful. There are mathematical theorems guaranteeing theoretical convergence to optimal solution or near optimal (with a limited planning horizon). Mathematical formulas are explained via insightful comments. The main ideas of the paper is clear simple and easy to uderstand.\n\nWEAKNESSES:\nIt is impossible for me to describe the scientifically important weaknesses of this paper since it is far outside my area of expertise. I can only provide minor weaknesses:\n\n1. The paper assumes a reader has a lot of prior knowledge in the area of POMDP. It is a problem or not depending on the desired reader. As a person who is not familiar with the topic, I like to have more preliminary comments.\n2. Notation may be not entirely clear for a person not familiar with the topic, i.e.  line 169, what exactly means: \"haz\"?\n\nQUESTIONS:\n1. The tiger environment has a limit of 20 actions. What exactly happens when the agent chooses to listen 19 times in a row? In the 20th step, the action space is different and consists of only choosing the door to open or can the agent listen again?\n2. In practice: can the entropy of the whole tree behave non-monotonically over time? If yes, how often it happens?\n3. line 159: how exactly is the tree re-initialized?\n\nLIMITATIONS:\nThere is little about the limitations of the method. I would like to know for example: can IB-POMCP be used on any POMDP? How the resources needed to solve a task with IB-POMCP depend on the problem's complexity? What is the worst-case scenario performance? What is the memory-usage?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a novel approach for online planning under uncertainty. It leverages the information entropy to guide a tree search process.

STRENGTHS:
* This paper is welled structured.
* The paper conducts both empirical and theoretical analysis.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes several improvements to the POMCP algorithm. The improvements introduce adaptive sample size, entropy regularized tree exploration and action selection.\n\nSTRENGTHS:\nThe research considers an important problem of online planning in POMDP --- dealing with sparse rewards. A few improvements to POMCP are proposed, theoretically analysed and empirically evaluated.\n\nWEAKNESSES:\n1) The authors' view on MCTS planning is partial and may be misleading. The authors argue that MCTS is based on UCB1. It is not, UCB1 is just option, there are others. One example is Thompson sampling, https://dl.acm.org/doi/10.5555/3038794.3038799 . Also, AlphaZero does not use UCB1, just as an illustration.\n\n2) The idea of adapting particle set size based on the entropy / KL divergence is nice and useful, but not novel: https://journals.sagepub.com/doi/10.1177/0278364903022012001 . Particle set size should be adapted based on diversity, either intrinsically or extrinsically estimated. \n\n3) The algorithm is not compared with DESPOT, https://papers.nips.cc/paper_files/paper/2013/hash/c2aee86157b4a40b78132f1e71a9e6f1-Abstract.html, the much better performing POMCP variant. There are follow up research publications on DESPOT in the last 10 years, too. DESPOT exhibits much better performance.\n\n4) The domains on which the algorithm is evaluated are very small and are NOT the domains on which POMCP, DESPOT and variants are evaluated for performance in the literature (Tag, LaserTag, RockSample just to name a few examples). I  am not convinced that estimates and modifications in the proposed POMCP variant scale well beyond the toy domains included in the evaluation.\n\nQUESTIONS:\nBelief state particle set reinvigoration involves estimating of the observation given the action.  In all domains used for evaluation, the probability of an observation given the state is given, so this probability can be estimated from the particle state itself rather than from action-observation statistics. How accurate is your estimate compared to the particle set estimate? How does plugging in particle set based estimate affect performance of the algorithm?\n\nLIMITATIONS:\nThe authors do not discuss limitations of their proposed algorithm. I suspect that the advantage of the algorithm is limited to toy domains only.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}