Title: Online POMDP Planning with Anytime Deterministic Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: POMDPs

Planning under uncertainty

Robotics
Abstract: Autonomous agents operating in real-world scenarios frequently encounter uncertainty and make decisions based on incomplete information. Planning under uncertainty can be mathematically formalized using partially observable Markov decision processes (POMDPs). However, finding an optimal plan for POMDPs can be computationally expensive and is feasible only for small tasks. In recent years, approximate algorithms, such as tree search and sample-based methodologies, have emerged as state-of-the-art POMDP solvers for larger problems. Despite their effectiveness, these algorithms offer only probabilistic and often asymptotic guarantees toward the optimal solution due to their dependence on sampling. To address these limitations, we derive a deterministic relationship between a simplified solution that is
easier to obtain and the theoretically optimal one. First, we derive bounds for selecting a subset of the observations to branch from while computing a complete belief at each posterior node. Then, since a complete belief update may be computationally demanding, we extend the bounds to support reduction of both the state and the observation spaces. We demonstrate how our guarantees can be integrated with existing state-of-the-art solvers that sample a subset of states and observations. As a result, the returned solution holds deterministic bounds relative to the optimal policy. Lastly, we substantiate our findings with supporting experimental results.

Online POMDP Planning with Anytime Deterministic Guarantees

Abstract

Autonomous agents operating in real-world scenarios frequently encounter uncertainty and make decisions based on incomplete information. Planning under uncertainty can be mathematically formalized using partially observable Markov decision processes (POMDPs). However, finding an optimal plan for POMDPs can be computationally expensive and is feasible only for small tasks. In recent years, approximate algorithms, such as tree search and sample-based methodologies, have emerged as state-of-the-art POMDP solvers for larger problems. Despite their effectiveness, these algorithms offer only probabilistic and often asymptotic guarantees toward the optimal solution due to their dependence on sampling. To address these limitations, we derive a deterministic relationship between a simplified solution that is easier to obtain and the theoretically optimal one. First, we derive bounds for selecting a subset of the observations to branch from while computing a complete belief at each posterior node. Then, since a complete belief update may be computationally demanding, we extend the bounds to support reduction of both the state and the observation spaces. We demonstrate how our guarantees can be integrated with existing state-of-the-art solvers that sample a subset of states and observations. As a result, the returned solution holds deterministic bounds relative to the optimal policy. Lastly, we substantiate our findings with supporting experimental results.

1 Introduction

Partially Observable Markov Decision Processes (POMDPs) serve as a comprehensive mathematical framework for addressing uncertain sequential decision-making problems. Despite their applicability, most problems framed as POMDPs struggle to achieve optimal solutions, largely due to factors such as large state spaces and an extensive range of potential future scenarios. The latter tends to grow exponentially with the horizon, rendering the solution process computationally prohibitive. The advent of approximate online, tree-based solvers has expanded the capacity of POMDPs, enabling them to tackle larger problems by providing a more scalable approach to problem-solving.

A prominent search algorithm addressing the challenges posed by large state and observation spaces in POMDPs is POMCP [Silver and Veness, 2010]. POMCP is a forward search algorithm which handles the large state and observation spaces by aggregating Monte-Carlo rollouts of future scenarios in a tree structure. During each rollout, a single state particle is recursively propagated from the root node to the leaves of the tree. It adaptively trades off between actions that lead to unexplored areas of the tree and actions that lead to rewarding areas of the tree search by utilizing UCT [Auer et al., 2002]. The guarantees on the provided solution by POMCP are asymptotic, implying that the quality of the solution remains unknown within any finite time frame.

Vadim Indelman

Department of Aerospace Engineering Technion - Israel Institute of Technology vadim.indelman@technion.ac.il

bo

Simplified tree

Txlxa

Oz|x

Oz1x

Deterministicguarantees

TxIxa

Txlxa

IVt-VtI≤∈

Oz|x

Ozx

Another notable approximate solver, Anytime Regularized DESPOT (AR-DESPOT) [Somani et al., 2013, Ye et al., 2017] is derived from Regularized DESPOT, which holds theoretical guarantees for the solution quality with respect to its optimal value. Similar to POMCP, AR-DESPOT performs forward search and propagates a single particle from the root node down to its leaves. It relies on branch-and-bound approach in the forward search, and utilizes dynamic programming techniques to update the value function estimate at each node. In contrast to POMCP, Regularized DESPOT offers a probabilistic lower bound on the value function obtained at the root node, providing a theoretical appeal by measuring its proximity to the optimal policy.

While the primary focus of this paper is on discrete POMDP planning, it is essential to acknowledge recent advancements in POMDP planning that encompass both discrete and continuous observation spaces. Few notable approaches include POMCPOW [Sunberg and Kochenderfer, 2018], LABECOP [Hoerger and Kurniawati, 2021] and AdaOPS [Wu et al., 2021], which leverage explicit use of observation models. These algorithms employ importance sampling mechanisms to weigh each state sample based on its likelihood value, which is assumed to be known. Although these methods have exhibited promising performance in practical scenarios, they currently lack formal guarantees. To address this gap, [Lim et al., 2020, 2022] introduced a simplified solver aimed at bridging the theoretical gap between the empirical success of these algorithms and the absence of theoretical guarantees for continuous observation spaces. In [Lim et al., 2022], probabilistic guarantees were derived for the simplified solver concerning its proximity to the optimal value function, thus contributing to a more comprehensive understanding of POMDP planning in both discrete and continuous settings.

In this paper, we focus on deriving deterministic guarantees for POMDPs with discrete state and observation spaces. Unlike existing black-box sampling mechanisms employed in algorithms such as [Sunberg and Kochenderfer, 2018, Hoerger and Kurniawati, 2021, Wu et al., 2021], our approach assumes access not only to the observation model but also to the transition model and the prior. By leveraging this additional information, we develop a novel algorithm that utilizes a subset of the observation space, enabling the computation of deterministic bounds with respect to the optimal policy at any belief node within the constructed tree. Furthermore, we demonstrate how these deterministic bounds can be integrated with state-of-the-art algorithms, including those that sample subsets of both the state and observation spaces. In particular, we provide experimental results showcasing the use of our bounds incorporated into the AR-DESPOT algorithm.

In this paper, our main contributions are as follows. First, we derive deterministic upper and lower bounds for a POMDP problem by considering a subset of the observation space at each node along the tree under a fixed policy. Next, we extend these bounds to cover scenarios where both the state and observation spaces are restricted to subsets, enhancing the applicability of our bounds in practical settings. Based on the derived bounds, we illustrate how to incorporate the bounds into a general structure of common state-of-the-art algorithms. Specifically, in the experimental section, we present results demonstrating the integration of our theoretical guarantees with the AR-DESPOT algorithm, yielding certified solutions with deterministic bounds. Last, we perform simulations to demonstrate the use of our derived bounds in practice, further validating their relevance in POMDP planning.

2 Preliminaries

A finite horizon POMDP M is defined as a tuple 〈X , A , Z , T, O, R〉 , where X , A , and Z represent a discrete state, action, and observation spaces, respectively. The transition density function T ( x t , a t , x t +1 ) glyph[defines] P ( x t +1 | x t , a t ) defines the probability of transitioning from state x t ∈ X to state x t +1 ∈ X by taking action a t ∈ A . The observation density function O ( x t , z t ) glyph[defines] P ( z t | x t ) expresses the probability of receiving observation z t ∈ Z from state x t ∈ X .

Given the limited information provided by observations, the true state of the agent is uncertain and a probability distribution function over the state space, also known as a belief, is maintained. The belief depends on the entire history of actions and observations, and is denoted H t glyph[defines] { z 1: t , a 0: t -1 } . We also define the propagated history as H -t glyph[defines] { z 1: t -1 , a 0: t -1 } . At each time step t , the belief is updated by applying Bayes' rule using the transition and observation models, given the previous action a t -1 and the current observation z t , b ( x t ) = η t P ( z t | x t ) ∑ x t -1 ∈X P ( x t | x t -1 , a t -1 ) b ( x t -1 ) , where η t denotes a normalization constant and b t glyph[defines] P ( x t | H t ) denotes the belief at time t. The updated belief, b t , is sometimes referred to as the posterior belief, or simply the posterior. We will use these interchangeably throughout the paper.

A policy function a t = π t ( b t ) determines the action to be taken at time step t , based on the current belief b t and time t . In the rest of the paper we write π t ≡ π t ( b t ) for conciseness. The reward is defined as an expectation over a state-dependent function, r ( b t , a t ) = E x ∼ b t [ r x ( x, a t )] , and is bounded by -R max ≤ r x ( x, a t ) ≤ R max . The value function for a policy π over a finite horizon T is defined as the expected cumulative reward received by executing π and can be computed using the Bellman update equation,

The action-value function is defined by executing action a t and then following policy π . The optimal value function may be computed using Bellman's principle of optimality,

The goal of the agent is to find the optimal policy π ∗ that maximizes the value function.

3 Mathematical Analysis

Typically, it is infeasible to fully expand a Partially Observable Markov Decision Process (POMDP) tree due to the extensive computational resources and time required. To address this challenge, we propose two approaches. In the first approach, we propose a solver that selectively chooses a subset of the observations to branch from, while maintaining a full posterior belief at each node. This allows us to derive a hypothetical algorithm that directly uses our suggested deterministic bounds to choose which actions to take while exploring the tree. As in some scenarios computing a complete posterior belief may be too expensive, we suggest a second method that in addition to branching only a subset of the observations, selectively chooses a subset of the states at each encountered belief. Using the deterministic bounds at the root node for each action value allows us to certify the performance of a given policy at the root of the planning tree. Moreover, given a scenario in which an action exists whose lower bound surpasses all other actions' upper bound, the algorithm can identify the optimal action and stop further exploration. In contrast, existing state-of-the-art algorithms either do not provide any guarantee on the solution quality (e.g. POMCP Silver and Veness [2010]) or merely provide probabilistic guarantee on the solution (e.g. DESPOT Somani et al. [2013]). In the following section, we show how to use the deterministic bounds in conjunction with state-of-the-art algorithms to obtain performance guarantees. 1

Both of the presented approaches diverge from many existing algorithms that rely on black-box prior, transition, and observation models. Instead, our method directly utilizes state and observation probability values to evaluate both the value function and the associated bounds. In return, we offer

anytime deterministic guarantees on the value function for the derived policy and establish bounds on its deviation from the value function of the optimal policy.

In this section we provide a mathematical quantification of the impact of using a solver that only considers a small subset of the theoretical tree branches and a subset of states within each node. We begin by defining a simplified POMDP, which is a reduced complexity version of the original POMDP that abstracts or ignores certain states and observations. We then establish a connection between the simplified value function and its theoretical counterpart. Finally, we demonstrate a relationship between the simplified value function, obtained by following the best policy for the simplified problem, and the theoretically optimal value function.

We begin with general definitions of a simplified prior, transition and observation models,

where ¯ X ( H -t +1 ) ⊆ X and ¯ Z ( H t ) ⊆ Z may be chosen arbitrarily, e.g. by sampling or choosing a fixed subset a-priori, as the derivations of the bounds are independent of the subset choice. Note that the simplified prior, transition and observation models are unnormalized and thus do not represent a valid distribution function. For the rest of the sequel we will drop the explicit dependence on the history, and denote ¯ X ( H -t +1 ) ≡ ¯ X , ¯ Z ( H t ) ≡ ¯ Z . Using the simplified models, we define the simplified belief ¯ b t +1 . The simplified belief is updated using the simplified belief update equation, which is a modification of the standard belief update equation that replaces the usual models with the simplified ones. More precisely, glyph[negationslash]

where ¯ b t +1 ( x ) ≡ ¯ P ( x t +1 | H t +1 ) , and ¯ P ( z t +1 | H -t +1 ) glyph[defines] ¯ P ( z t +1 | x t +1 ) ∑ x t ¯ P ( x t +1 | x t , π t ) ¯ P ( x t | H t ) . Note that a simplified belief cannot depend on an observation that is not part of the simplified observation set and is considered undefined. Last we define a simplified value function,

where the simplified expectation operator, ¯ E [ · ] , is taken with respect to the unnormalized likelihood ¯ P ( z t +1 | H -t +1 ) .

3.1 Simplified observation space

We first analyze the performance guarantees of a simplified observation space, while assuming a complete belief update at each belief state, i.e., ¯ X ≡ X . The following theorem describes the guarantees of the observation-simplified value function with respect to its theoretical value,

Theorem 1. Let b t belief state at time t , and T be the last time step of the POMDP. Let V π ( b t ) be the theoretical value function by following a policy π , and let ¯ V π ( b t ) be the simplified value function, as defined in (18) , by following the same policy. Then, for any policy π , the difference between the theoretical and simplified value functions is bounded as follows,

where we use a subscript of ( z ) in glyph[epsilon1] π z ( b t ) to denote observation-only simplification. Importantly, the bound only contains terms which depend on observations that are within the simplified space, z ∈ ¯ Z . This is an essential property of the bound, as it is a value that can easily be calculated during the planning process and provides a certification of the policy quality at any given node along the tree. Furthermore, it is apparent from (20) that as the number of observations included in the simplified set, ¯ Z , increases, the value of glyph[epsilon1] π z ( b t ) consequently diminishes,

leading to a convergence towards the theoretical value function, i.e. glyph[epsilon1] π z ( b t ) → 0 .

Theorem 3 provides both lower and upper bounds for the theoretical value function, assuming a fixed policy. Using this theorem, we can derive upper and lower bounds for any policy, including the optimal one. This is achieved by applying the Bellman optimality operator to the upper bound in a repeated manner, instead of the estimated value function; In the context of tree search algorithms, our algorithm explores only a subset of the decision tree due to pruned observations. However, at every belief node encountered during this exploration, all potential actions are expanded. The action-value of these expanded actions is bounded using the Upper Deterministic Bound, which we now define as

where the action-dependent bound on the value difference, glyph[epsilon1] π z ( b t , a t ) , is the bound of taking action a t in belief b t and following policy π thereafter,

In the event that no subsequent observations are chosen for a given history, the value of ¯ Q π ( b t , a t ) simplifies to the immediate reward plus an upper bound for any subsequent policy, given by R max · ( T -t -1) .

Using UDB we define the action selection criteria according to

Moreover, the optimal value function can be bounded as follows,

Lemma 1. The optimal value function can be bounded as

where the policy π is determined according to Bellman optimality over the UDB, i.e.

Corollary 1.1. By utilizing Lemma 1 and the exploration criteria defined in (65) , an increasing number of explored belief nodes guarantees convergence to the optimal value function.

Notably, UDB does not require a complete recovery of the posterior branches to yield an optimal policy. Each action-value is bounded by a specific lower and upper bound, which can be represented as an interval enclosing the theoretical value. When the bound intervals of two candidate actions do not overlap, one can clearly discern which action is suboptimal, rendering its subtree redundant for further exploration. This distinction sets UDB apart from current state-of-the-art online POMDP algorithms. In those methods, any finite-time stopping condition fails to ensure optimality since the bounds used are either heuristic or probabilistic in nature.

3.2 Simplified State and Observation Spaces

In certain scenarios, the complete evaluation of posterior beliefs during the planning stage may pose significant computational challenges. To tackle this issue, we propose the use of a simplified state space in addition to the simplified observation space considered thus far. Specifically, we derive deterministic guarantees of the value function that allow for the selection of a subset from both the states and observations.

Theorem 2. Let b 0 and ¯ b 0 be the theoretical and simplified belief states, respectively, at time t = 0 , and T be the last time step of the POMDP. Let V π ( b 0 ) be the theoretical value function by following a policy π , and let ¯ V π ( ¯ b 0 ) be the simplified value function by following the same policy, as defined in (18) . Then, for any policy π , the difference between the theoretical and simplified value functions is bounded as follows,

Similar to Theorem 3, glyph[epsilon1] π x,z ( b 0 ) only contains probability values of instances from the simplified sets. However, this bound accounts for both the states and observations that are within the simplified spaces, x, z ∈ ¯ X , ¯ Z , which makes it a viable to compute at planning time. In contrast to Theorem 3, this bound can only be calculated at the root since it relies on the knowledge of the actual probability value of the prior, b 0 ( x ) , for states x ∈ ¯ X , which are only available at the root. Furthermore, since now the belief density function at different belief nodes is not required, we demonstrate in the next section that a belief update can be avoided completely, which makes it suitable to attach guarantees to state-of-the-art algorithms.

The process of maintaining an upper bound for action-value function follows similarly to the one presented in the observation-only simplification subsection 3.1,

In glyph[epsilon1] π x,z ( b t , a t ) , the subscript of ( x, z ) differentiates the bound of the stateand observationsimplification from the observation-only simplification bound.

Computing the bounds in practice is straightforward and can be done iteratively, which fits many online planning algorithms. As it requires the summation over probability values for every considered trajectory, x 0 , a 0 , x 1 , z 1 , ..., a t -1 , x t , z t , as shown in equations (17) and (11). In the subsequent section, we provide further details and illustrate how the bounds derived from the state-observation simplification can be used to derive deterministic guarantees on policies obtained from existing state-of-the-art algorithms.

4 Methods

In this section we aim to describe how to fit our bounds to a general structure algorithm, named ALGORITHM - A , which serves as an abstraction to many existing algorithms. To compute the deterministic bounds, we utilize Bellman's update and optimality criteria. This approach naturally fits dynamic programming approaches such as DESPOT [Ye et al., 2017] and AdaOPS [Wu et al., 2021]. However, it may also be attached with algorithms that rely on Monte-Carlo estimation, such as POMCP [Silver and Veness, 2010], by viewing the search tree as a policy tree.

While the analysis presented in section 3 is general and independent of the selection mechanism of the states or observations, we focus on sampling as a way to choose the simplified states at each belief node and the observations to branch from. Furthermore, the selection of the subspaces ¯ X , ¯ Z need not be fixed, and may change over the course of time, similar to state-of-the-art algorithms, such as Hoerger and Kurniawati [2021], Silver and Veness [2010], Somani et al. [2013], Sunberg and Kochenderfer [2018], Wu et al. [2021]. Alternative selection methods may also be feasible, as sampling from the correct distribution is not required for the bounds to hold. Importantly, attaching our bounds to arbitrary exploration mechanism, such as in POMCP or DESPOT, leverages the derivations and convergence guarantees shown in 3.2. Clearly, the deterministic bounds remain valid, but a convergence analysis depends on the characteristics of the specific ALGORITHM -A being used.

ALGORITHM -A is outlined in algorithm 1. For the clarity of exposition, we assume the following; at each iteration a single state particle is propagated from the root node to the leaf (line 2 of function SEARCH). The selection of the next state and observations are done by sampling from the observation and transition models (line 5), and each iteration ends with the full horizon of the POMDP (lines 2). However, none of these are a restriction of our approach and may be replaced with arbitrary number of particles, arbitrary state and observation selection mechanism and a single or multiple expansions of new belief nodes at each iteration.

To compute the UDB value, we require both the state trajectory, denoted as τ , and its probability value, P τ . We use the state trajectory as a mechanism to avoid duplicate summation of an already accounted for probability value and is utilized to ascertain its uniqueness at a belief node. The probability value, P τ , is the likelihood of visiting a trajectory τ = { x 0 , a 0 , x 1 , z 1 , . . . , a t -1 , x t , z t } and is calculated as the product of the prior, transition and observation likelihoods (line 6). If a trajectory was not seen before in a belief node, its reward value is multiplied by the trajectory likelihood, shown in 3. Each node maintains a cumulative sum of the likelihoods of all visited trajectories. This is then being

used to compute the upper and lower bounds, shown in lines 2. The bounds are computed in lines 1 represent the loss of holding only a subset of the sates in node ha from the set in node h , plus the loss of having only a subset of the future states and observations, where V max represent the maximum possible value function. A simple bound on the value function may be V max = R max · ( D -d -1) , but other more sophisticated bounds are also possible, as well as different values for lower and upper bounds.

The time complexity for each posterior node, primarily depends on the specific algorithm being used. In the case of dynamic programming methods, such as DESPOT and AdaOPS, there is a negligible added computational complexity detailed below. In the case of Monte Carlo methods, such as POMCP, the computational complexity is O ( |A| ) attributed mainly to the action-selection, while our approach adds another linear time complexity term, making it O ( |A| + | ¯ Z| ) due to the summation over the simplified observation space. During each iteration of the algorithm, an "IF" statement is used to determine whether a specific trajectory has already been encountered at the current node. This verification process can potentially result in an added linear complexity of O ( D ) , where D represents the planning horizon. However, this overhead can be circumvented by assigning a unique ID value to each trajectory at the previous step and subsequently checking whether a pair, comprising the ID value and the new state, has already been visited. This approach reduces the overhead to an average time complexity of O (1) by utilizing hash maps efficiently.

5 Experiments

In this section, we present the experimental results obtained by integrating deterministic bounds into a state-of-the-art algorithms namely, AR-DESPOT Somani et al. [2013] and POMCP Silver and Veness [2010] as a baseline. The primary objective of these experiments is to demonstrate the validity of our derived bounds, as presented in Theorem 2, and the corresponding algorithm outlined in Algorithm 1.

The implementation of our algorithm was carried out using the Julia programming language and evaluated through the Julia POMDPs package, provided by Egorov et al. [2017]. Although the library primarily focused on infinite horizon problems, we made the required modifications to extend its capabilities to accommodate finitehorizon POMDPs. The experiments were conducted on a computing platform consisting of an Intel(R) Core(TM) i7-7700 processor with 8 CPUs operating at 3.60GHz and 15.6 GHz. The

300

Lower Bound

Optimal Value

200

function

100

0

Value

-100

-200

-300

0.0

0.5

1.0

1.5

Planning Time [s]

selection of hyper-parameters for the POMCP and AR-DESPOT solvers, and further details about the POMDPs used for our experiments are detailed in the appendix.

In Figure 2, we demonstrate the empirical convergence of the deterministic bounds in relation to planning time. For these experiments, we focused on a toy example, Tiger POMDP Kaelbling et al. [1998]. By conducting an exhaustive search and computing a full posterior belief for each node, we derived the optimal value, depicted as a dashed green line in figure 2. The graph presents the convergence of the bounds calculated with Deterministically-Bounded AR-DESPOT (DB-DESPOT), to the optimal value. The mean values of the upper and lower bounds across 100 simulations are plotted, accompanied by the standard deviation, for time increments of ∆ t = 0 . 1 seconds.

Algorithms that provide probabilistic guarantees have a non-zero probability of taking a suboptimal action regardless of the planning time. In contrast, when given sufficient planning time, the deterministic bounds can certify the optimal action once the lower bound of an immediate action exceeds the upper bounds of all other actions. In our experiments, we observed that although the baseline algorithms, AR-DESPOT and POMCP, and the deterministically bounded algorithms, DB-DESPOT

and DB-POMCP, had access to the same tree and samples, AR-DESPOT and POMCP occasionally made incorrect decisions, resulting in degraded performance.

We evaluated the performance of both algorithms on different POMDPs, including the Tiger POMDP, Discrete Light Dark Sunberg and Kochenderfer [2018] and Baby POMDP. The corresponding results are summarized in Table 1. After each planning session the calculated best action is executed, the belief is updated according to the captured observation and a new planning session is invoked. The table reports the empirical mean and std of the cumulative rewards obtained by executing the calculated best actions based on 100 runs.

In the POMDPs with manageable state, action and observation sizes, both the DB-DESPOT and DB-POMCP algorithms frequently identified the optimal action within a 1-second time budget. It's important to highlight that these algorithms employ stochastic methods for belief tree exploration as in the baseline algorithms. Consequently, the outcomes between DB-DESPOT and DB-POMCP can vary, and there is no guarantee of consistently identifying the optimal action in every constrained-time planning session. An optimal action is determined when a lower bound of a given action is higher than any other upper bound; notably, this does not necessitates a construction of the entire planning tree and can be used as an early stopping mechanism for the optimal action. In the larger Laser Tag POMDP Somani et al. [2013], the DB-DESPOT and DB-POMCP did not outperform AR-DESPOT and POMCP. This discrepancy occurred because the planning time was insufficient to find an optimal action due to larger state, action and observation spaces of Laser Tag POMDP. While our proposed algorithms shows promise in more compact POMDPs, their performance in larger-scale problems, like the Laser Tag POMDP, especially when constrained by external time budgets, merits further investigation.

6 Conclusions

In this work, we presented a novel methodology aimed at offering anytime, deterministic guarantees for approximate POMDP solvers. These solvers strategically leverage a subset of the state and observation spaces to alleviate the computational overhead. Our key proposition elucidates a linkage between the optimal value function, which is inherently computationally intensive, and a more tractable approximation frequently employed in contemporary algorithms. In the first part of the paper, we derived the theoretical relationship between the use of a selective subset of observations in a planning tree. One contribution of this work is the formulation of an upper deterministic bound (UDB) which governs exploration within the belief space, and is theoretically guaranteed to converge to the optimal value. This approach, however, depends on a complete belief update at each node of the tree, a requirement that can be computationally infeasible in many practical POMDPs. In the second part, we address this computational challenge, by extending our derivations to account for both a subset of the observations and states. This extension increases the feasibility of our approach and allows it to be incorporated into existing state-of-the-art algorithms. We have outlined how our methodology can be integrated within these algorithms. To illustrate the practical utility of our derivations, we applied them to certify and improve the solution of AR-DESPOT and POMCP algorithms, a state-of-the-art solvers for POMDPs.

6.1 Limitations and future work

While the application of our deterministic bounds within the context of the AR-DESPOT algorithm showcased its potential, there are several challenges that need to be addressed in future work. Firstly, the convergence rate of our proposed method remains an area for further exploration. A detailed theoretical analysis on this aspect could provide valuable insights into the practicality of our approach

for larger, more complex domains. Secondly, the current use of the UDB is predominantly restricted to problems characterized by low state dimensionality. Extending the UDB to handle higher state dimensionality is a significant challenge due to the increased computational complexity involved. Finally, we consider the integration of the UDB with simplified state and observation spaces to be a promising future direction. This could potentially lead to a more comprehensive and efficient strategy for handling larger POMDPs, thereby further enhancing the applicability of our deterministic approach.

Acknowledgments

This work was supported by the Israel Science Foundation (ISF) and by US NSF/US-Israel BSF.

References

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Maxim Egorov, Zachary N. Sunberg, Edward Balaban, Tim A. Wheeler, Jayesh K. Gupta, and Mykel J. Kochenderfer. POMDPs.jl: A framework for sequential decision making under uncertainty. Journal of Machine Learning Research , 18(26):1-5, 2017. URL http://jmlr.org/papers/ v18/16-300.html .

Marcus Hoerger and Hanna Kurniawati. An on-line pomdp solver for continuous observation spaces. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , pages 7643-7649. IEEE, 2021.

P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence , 101(1):99-134, 1998.

Michael H. Lim, Claire Tomlin, and Zachary N. Sunberg. Sparse tree search optimality guarantees in pomdps with continuous observation spaces. In Intl. Joint Conf. on AI (IJCAI) , pages 4135-4142, 7 2020.

Michael H Lim, Tyler J Becker, Mykel J Kochenderfer, Claire J Tomlin, and Zachary N Sunberg. Generalized optimality guarantees for solving continuous observation pomdps through particle belief mdp approximation. arXiv preprint arXiv:2210.05015 , 2022.

David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in Neural Information Processing Systems (NIPS) , pages 2164-2172, 2010.

Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. In NIPS , volume 13, pages 1772-1780, 2013.

Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 28, 2018.

Chenyang Wu, Guoyu Yang, Zongzhang Zhang, Yang Yu, Dong Li, Wulong Liu, and Jianye Hao. Adaptive online packing-guided search for pomdps. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems (NIPS) , volume 34, pages 28419-28430. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ ef41d488755367316f04fc0e0e9dc9fc-Paper.pdf .

Nan Ye, Adhiraj Somani, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. JAIR , 58:231-266, 2017.

Appendices

7 Mathematical Analysis

We start by restating the definition of the simplified value function,

7.1 Theorem 1

Theorem 3. Let b t belief state at time t , and T be the last time step of the POMDP. Let V π ( b t ) be the theoretical value function by following a policy π , and let ¯ V π ( b t ) be the simplified value function, as defined in (18) , by following the same policy. Then, for any policy π , the difference between the theoretical and simplified value functions is bounded as follows,

Proof. For notational convenience, we derive the bounds for the value function by denoting the prior belief as b 0 ,

applying the belief update equation,

which applies similarly to the simplified value function,

We begin the derivation by focusing on a single time step, t , and later generalize to the complete value function.

where the second transition is due to triangle inequality, the third transition is equality by the construction, i.e. using the simplified observation models imply that the difference is nonnegative. We add and subtract, followed by rearranging terms,

applying Holder's inequality,

following similar steps recursively, where,

plugging the term in (37) to (36) and expanding the terms results in the desired bound,

which concludes our derivation.

7.2 Lemma 1

Lemma 2. The optimal value function can be bounded as

where the policy π is determined according to Bellman optimality over the UDB, i.e.

Proof. In the following, we prove by induction that applying the Bellman optimality operator on upper bounds to the value function in finite-horizon POMDPs will result in an upper bound on the optimal value function. The notations are the same as the ones presented in the main body of the paper. We restate some of the definitions from the paper for convenience.

The policy π t ( b t ) determined by applying Bellman optimality at belief b t , i.e.,

As it will be needed in the following proof, we also define the value of a belief which includes in its history at least one observation out of the simplified set, e.g. H t = { a 0 , z 1 , . . . , z k / ∈ Z , . . . , z t } as being equal to zero. Explicitly,

We also use the following simple bound,

Base case ( t = T ) - At the final time step T , for each belief we set the value function to be equal to the reward value at that belief state, b T and taking the action that maximizes the immediate reward,

Finally, applying similar steps for every time step t ∈ [1 , T ] results in,

which provides an upper bound for the optimal value function for the final time step, V glyph[star] T ( b T ) ≤ UDB π ( b T ) .

Induction hypothesis - Assume that for a given time step, t , for all belief states the following holds,

Induction step - We will show that the hypothesis holds for time step t -1 . By the induction hypothesis,

For the following transition, we make use of lemma 3,

Therefore, under the induction hypothesis, Q glyph[star] t -1 ( b t -1 , a t -1 ) ≤ UDB π ( b t -1 , a t -1 ) . Taking the maximum over all actions a t ,

which completes the induction step and the required proof.

Lemma 3. Let b t denote a belief state and π t a policy at time t . Let ¯ P ( z t | x t ) be the simplified observation model which represents the likelihood of observing z t given x t . Then, the following terms are equivalent,

Proof.

focusing on the second summand,

thus,

by marginalizing over x t -1 ,

canceling out the denominator,

it is left to show that E z t | b t -1 ,a t -1 [ V π t ( b t ) ] = E z t | b t -1 ,a t -1 [ V π t ( b t ) ] . By the definition of a value function of a belief not included in the simplified set, we have that,

which concludes the derivation.

7.3 Corollary 1.1

We restate the definition of UDB exploration criteria,

Corollary 3.1. Using Lemma 2 and the exploration criteria defined in (65) guarantees convergence to the optimal value function.

Proof. Let us define a sequence of bounds, UDB π n ( b t ) and a corresponding difference value between UDB n and the simplified value function,

where n ∈ [0 , |Z| ] corresponds to the number of unique observation instances within the simplified observation set, Z n , and |Z| denotes the cardinality of the complete observation space. Additionally, for the clarity of the proof and notations, assume that by construction the simplified set is chosen such that Z n ( H t ) ≡ Z n remains identical for all time steps t and history sequences, H t given n . By the definition of glyph[epsilon1] π n,z ( b t ) ,

we have that glyph[epsilon1] π n,z ( b t ) → 0 as n →|Z| , since

as more unique observation elements are added to the simplified observation space, Z n , eventually recovering the entire support of the discrete observation distribution.

From lemma 2 we have that, for all n ∈ [0 , |Z| ] the following holds,

Additionally, from theorem 3 we have that,

for any policy π and subset Z n ⊆ Z , thus,

Since glyph[epsilon1] π n,z ( b t ) → 0 as n →|Z| , and |Z| is finite, it is guaranteed that UDB π n ( b t ) n →|Z| - - - - → V π ∗ ( b t ) which completes our proof.

Moreover, depending on the algorithm implementation, the number of iterations can be finite (e.g. by directly choosing actions and observations to minimize the bound). A stopping criteria can also be verified by calculating the difference between the upper and lower bounds. The optimal solution is obtained once the upper bound equals the lower bound.

8 Experiments

8.1 POMDP scenarios

We begin with a brief description of the Partially Observable Markov Decision Process (POMDP) scenarios implemented for the experiments. each scenario was bounded by a finite number of time steps used for every episode, where each action taken by the agent led to a decrement in the number of time steps left. After the allowable time steps ended, the simulation was reset to its initial state.

8.1.1 Tiger POMDP

The Tiger is a classic POMDP problem Kaelbling et al. [1998], involves an agent making decisions between two doors, one concealing a tiger and the other a reward. The agent needs to choose among three actions, either open each one of the doors or listen to receive an observation about the tiger position. In our experiments, the POMDP was limited horizon of 5 steps. The problem consists of 3 actions, 2 observations and 2 states.

8.1.2 Discrete Light Dark

Is an adaptation from Sunberg and Kochenderfer [2018]. In this setting the agent needs to travel on a 1D grid to reach a target location. The grid is divided into a dark region, which offers noisy observations, and a light region, which offers accurate localization observations. The agent receives a penalty for every step and a reward for reaching the target location. The key challenge is to balance between information gathering by traveling towards the light area, and moving towards the goal region.

8.1.3 Laser Tag POMDP

In the Laser Tag problem, Somani et al. [2013], an agent has to navigate through a grid world, shoot and tag opponents by using a laser gun. The main goal is to tag as many opponents as possible within a given time frame. The grid is segmented into various sections that have varying visibility, characterized by obstacles that block the line of sight, and open areas. There are five possible actions,

moving in four cardinal directions (North, South, East, West) and shooting the laser. The observation space cardinality is |Z| ≈ 1 . 5 × 10 6 , which is described as a discretized normal distribution and reflect the distance measured by the laser. The states reflect the agent's current position and the opponents' positions. The agent receives a reward for tagging an opponent and a penalty for every movement, encouraging the agent to make strategic moves and shots.

8.1.4 Baby POMDP

The Baby POMDP is a classic problem that represents the scenario of a baby and a caregiver. The agent, playing the role of the caregiver, needs to infer the baby's needs based on its state, which can be either crying or quiet. The states in this problem represent the baby's needs, which could be hunger, discomfort or no need. The agent has three actions to choose from: feeding, changing the diaper, or doing nothing. The observations are binary, either the baby is crying or not. The crying observation does not uniquely identify the baby's state, as the baby may cry due to hunger or discomfort, which makes this a partially observable problem. The agent receives a reward when it correctly addresses the baby's needs and a penalty when the wrong action is taken.

8.2 Hyperparameters

The hyperparameters for both DB-DESPOT and AR-DESPOT algorithms were selected through a grid search. We explored an array of parameters for AR-DESPOT, choosing the highest-performing configuration. Specifically, the hyperparameter K was varied across { 10 , 50 , 500 , 5000 } , while λ was evaluated at { 0 , 0 . 01 , 0 . 1 } . Similarly, DB-POMCP and POMCP were examined three different values for the exploration-exploitation weight, c = { 0 . 1 , 1 . 0 , 10 . 0 } multiplied by V max , which denotes an upper bound for the value function.

For the initialization of the upper and lower bounds used by the algorithms, we used the maximal reward, multiplied by the remaining time steps of the episode, R max · ( T t -1) .

Finally, we provide our algorithm implementation in https://github.com/moranbar/Online-POMDPPlanning-with-Anytime-Deterministic-Guarantees .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical bounds for POMDP planning algorithms and derive a new planning algorithms. The method applies to POMDP with discrete actions and observations spaces where the transition and observation distributions are known. \n\nThe derivation of the bound relies on considering a POMDP with only a subset of observations and states with unnormalized distributions for the transition and the observation. \n\nThe authors proceed to compute the difference in value for this simplified POMDP and the original problem for a given policy and then extend the derivation to the optimal policy in the case of limited observations. This step should address the fact that tree based solver have limited branching in the observations. \n\nThen they update the bounds to address subsets of states. This part addresses the challenge that the belief update requires summing over all possible states which can be expensive. \n\nThe theoretical results are used to improved upon existing tree-based POMDP planners and the resulting algorithm is evaluated on well-known benchmark problems.\n\nSTRENGTHS:\nThey are able to leverage the explicit transition model in a tree-based POMDP solver. They derive bounds using a creative idea of a simplified POMDP model ignoring the observations that are not in the planning tree.\n\nThe theoretical bounds seem easier to use than previous probabilistic bounds in AR-DESPOT. However, the computational tractability of the bounds seem to be very reliable on the sampling of the states.\n\nThey show competitive performance on standard POMDP problems against a state of the art algorithm.\n\nThe main approach is quite easy to follow and mostly well motivated.\n\nWEAKNESSES:\nSoundness / intuitive justification of the choice of the simplified model. Although the mathematical derivation of the bound seem correct, it would be nice to further motivate this choice. \n\nThe notation is sometimes confusing and could use extra clarification (see questions for specific examples). Equation (6), none of the distributions are normalized, yet the simplified belief is normalized.\n\nIf my understanding is correct, given a POMDP planning tree, and a subset of states per node, the bounds can be calculated and are deterministic guarantees. However, the construction of the tree itself is still probabilistic since sampling is used at two places: sampling the observations, and sampling the subset of states to use. I think it was a bit misleading initially, the bounds are deterministic, but the performance of the algorithm will still be probabilistic. The authors could clarify that aspect, and comment on whether the convergence properties change or would be the same as the underlying algorithm using the bounds. \n\nThe paragraph line 215 was a very clear explanation of how to perform the calculation of the bound and could come sooner.\n\nThere are some inaccuracies in the complexity discussion: “the Bellman update is linear in terms of time complexity”. This statement does not make sense.\n\nIt would have been very strong if the bounds had been added to POMCP or POMCPOW as well to show that the theoretical results indeed applies to different algorithms.\n\nI did not understand the discussion about why DB-DESPOT did not perform better on the Laser tag problem. \n\nIn the supplementary material, it wouldn’t hurt to add a few extra steps between 4 and 7 to make it easier on the reader. Specifically between step (6) and (7).\n\n\nMinor typos and formatting:\n- In the background in the belief equation, take $P(z_t \\mid x_t)$ out of the sum for easier mapping with Equation (6)\n- Equation (6) should be $b_{t+1}(x_{t+1})$\n- Equation (28) in supplementary material seems wrong.\n- L208 A is outlined.\n- L220 seeing->seen\n\nQUESTIONS:\n- Does $\\bar{P}(x_t | H_t) = \\bar{b_t}(x_t)$ ?\n\n- What is $H_t^-$? The superscript – is not explained I think.\n\n- Why normalizing the simplified belief? \n\n- Can you clarify the dependency between the bound and the future observations? Why $\\epsilon_z$?\n\n- How is the subset of state determined to derive the bound in theorem 2?\n\n- For theorem 2, does it have the property that if the subset of states tends towards the full state space then the bound goes to 0?\n\n- What time horizon is used to compute the optimal value function in figure 2?\n\n- What do you mean by “non overlapping bounds between actions at the decision time”?\n\n- Is the computation of the bound more expensive than one Bellman backup?\n\nLIMITATIONS:\nUsing black-box models for transition and observations can be easier to implement in practice. \n\nThe authors could be a bit more up front in the beginning of the paper about the finite horizon aspect, although it might not matter much in practice since the tree-based method have limited depth anyway.\n\nThe author mention the limitation regarding the computational complexity, however it was not very clear to me if this was a limitation of their method specifically or a general statement since they also mention that the added complexity is negligible compared to AR-DESPOT.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

comments: SUMMARY:
The paper derives deterministic upper and lower bounds for POMDP. It also proposes a relationship between the computationally intensive optimal value function and a more accessible, simplified approximation.

STRENGTHS:
This paper provides a strong theoretical analysis

WEAKNESSES:
The empirical results are limited

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online planning in partially observable Markov decision processes (POMDP). POMDP is known to be a difficult problem, and this paper focuses on how to get anytime deterministic guarantees. This paper proposes to consider a simplified version of the problem by using only a subset of observation space and a subset of state space. They prove the gap between the value functions of an algorithm $\\pi$ in the original problem and in the simplified problem. The gap mainly depends on how many states or observations are dropped in the simplified problem.\n\nSTRENGTHS:\n1. This paper studies an important and difficult problem, online planning in POMDP.\n\n2. This paper provides an idea to simplified the problem by reducing the state space and observation space, which seems interesting to me.\n\nWEAKNESSES:\n1. The nontriviality and contribution of the results seem limited. Theorem 1 and Theorem 2 basically shows that the value functions of the algorithm $\\pi$ in the original problem and in the simplified problem depends on how many states or observations are dropped in the subset that is used. This seems trivial to me. The relations shown in Theorem 1 and Theorem 2 seem also trivial. The second term in the bracket basically shows the probability that the chosen observations can be received, and then the gap between the value functions depends on the probability of the other observations that cannot be received, i.e., one minus the previous probability. Moreover, is the selection of the subset of states and observations more important and interesting, compared to showing the impact on the gap of value functions of dropped states and observations?\n\n2. The presentation needs to be improved. See my questions below. Moreover, according to the statement in line 88, the policy considered is a deterministic policy, why the random policy is not considered? Around line 105, some intuition about why \"This will allow us to certify the performance of a given policy at the root of the planning tree\" will be easier for me to understand. According to line 121, the subsets could be chosen arbitrarily. But then, how could you guarantee that the probabilities in equations (3), (4) and (5) are well-defined?\n\nQUESTIONS:\n1. What is $R_{max}$ in equation (9) and (16)? It seems not defined in the paper.\n\n2. Why is there an algorithm 1? It seems not mentioned or used in the paper.\n\n3. What is the difference between $\\epsilon^{\\pi}(b_t)$ in equation (9) and $\\epsilon_z^{\\pi}(b_t,a_t)$ defined in equation (11)? Is $\\epsilon^{\\pi}(b_t)$ just a special case of $\\epsilon_z^{\\pi}(b_t,a_t)$ when the action $a_t = \\pi_t$?\n\nLIMITATIONS:\nSee weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission develops a method for POMDP planning based on a notion of limiting the states and observations that have been and will be encountered during the tree search.  Two cases are developed, one where only observations are limited, and one where both states and observations are limited.\n\nIn both cases, the theory develops the notion of a value associated with this limited setting, and shows that the limited value has a well-defined absolute-error bound w.r.t. the real `unlimited` value function.  Such bounds are then used to choose actions in an UCB-like fashion, which results in the final planning method.\n\nThe simplified POMDP formalism is quite interesting, and explained clearly enough.  However, it might benefit from aditional discusssion about the assumptions that the limited observation and state sets need to satisfy, e.g., it seems like the limites state set must contain at least one state that is supported by the non-limited initial belief.  It also seems like the observation and state sets may not always be chosen freely and independently, because if the limited observation set allows for limited histories whose beliefs **only** support states that don't belong to the limited state set, then the limited version of belief updates shown in equation (6) breaks with a division by 0.\n\nSTRENGTHS:\nThe topic addressed by the authors is interesting and novel, and a theoretically driven approach is highly appreciated.  With significant improvements in the presentation and evaluation, this work will become much stronger.\n\nWEAKNESSES:\nThe inconsistency of the presentation and notation is a significant drawback to comprehension. The first few sections of the document are well written and clear, however the last few sections seem rushed and with a significant drop in discussion and analysis.  The general impression I get is that the writing of the final parts was rushed due to lack of time.  The mathematical formalisms are for the most part clear, but there are some instances of typos, inconsistent notation, and undefined terms that should be addressed.  I invite the authors to carefully review the math to make sure these issues are addresses.  The algorithm description and discussion in section (4) also may use improvements and clarifications.\n\nThe exact significance of the theoretical erorr bounds for any given limited set Z or X is unclear, as it seems like they may be as large as the real values themselves.  The main theorems of this paper seem to only guarantee optimality in the limit as the limited sets Z and X grow to cover the respective original sets.\n\nThe evaluation setting seems a bit questionable and confusing;  It seems like the main advantage of the proposed method would be in tackling complicated POMDPs that have many possible states and observations.  Instead, the evaluation is performed on the Tiger problem, which contains only 2 possible states and 2 possible observations.  The tiger problem contains complexities that are related to the estimation of values in the highly stochastic and uncertain setting, but in terms of state and observation spaces, it is truly minimal, to the point that it's unclear how a method that is designed around a notion of \"ignoring\" certain states and observations could possibly work well or even at all.  Any clarification would be greatly appreciated.  In addition, a broader discussion about sizes of these problems, and how they related to the possibility of limiting the observation and state spaces would greatly help make the evaluation section more convincing.\n\nIn addition to the above, it seems like the evaluation could be performed on larger problems, since for tiger, discrete-light and baby, the authors claim an optimal action could be found within 1 second.  What kind of problems could be solved as well, if e.g., 10 minutes were allowed?\n\nThe submission title refers to \"anytime deterministic guarantees\", but aside from a single claim that this is achieved, no additional discusssion of the meaning of this term is provided.\n\nQUESTIONS:\nIn addition to any possible question that has sneaked in the previous forms, I have additional questions\n\n\n- in line 156, the first clause of the sentence seems incomplete \"In the event that no subsequent chosen observations [...], the value of Q etc\";  what condition do the subsequence chosen observations have satisfy?\n\n- in corollary 1.1, it's not very clear what are the conditions for the convergence to work out.  I assume this is the convergence as \\bar Z -> Z.  is this right?\n\n- in line 233, it is said that the complexity attributed to the Bellman upate is linear due to the max and expectation operators.  Since the operators are nested and not independent, shouldn't the complexity be quadratic?\n\n# Minor notes and comments about notation\n\n- although clear enough from context and formalization, it might be worthwhile to make a explicit comment about the fact that deterministic policies are implied by the formalization.\n\n- the expectation in equation (1) should be conditioned on b_t.\n\n- the notation in line 89 \"\\pi_t \\equiv \\pi_t(b_t)\" is a bit confusing;  you're now using \\pi_t to refer not just to the policy, but also to the action determined by the policy.  why not use a_t instead, as that's how a_t was already appropriately defined?\n\n- in equation (7), is the simplified expectation suppsed to be of the next simplified belief \\bar V(b_{t+1}) ?  If so (as it seems), then the expectation should be conditioned on the current belief.\n\n- in equation (7), is it correct that the expectation  is not conditioned on anything?  it seems like that might be correct just due to a combination of the deterministic actions and how simplified expectation is defined, but a clarification would be helpful.  Arguably, the notation of simplified expectation is confusing, and lines (7-8) would be simpler if an explicit sum is used directly.\n\n- in equation (8), it seems like \\bar b(z_{t+1}) is trying to denote the updated belief after integrating the current belief with the new observation z_{t+1}, but this notation was not introduced, and seems in conflict with the notation of b(x) denoting the probabiliy of a state according to the belief.  I think overloading the symbol b this way is fine, as long as both uses have been previously defined appropriately.\n\n- in equation (9), R_max is not defined.  Supposedly this is either the maximal absolute reward achievable by a reachable state-action pair, or the maximal absolute reward achievable by a reachable belief-action pair.  Either way, it should be defined earlier in the preliminaries section.\n\n- in the preliminaries section, the pomdp is defined in terms of R, which supposedly is the reward function, but is never defined.  Later in the section, rewards are instead expressed as r(b, a) for beliefs and r_x(x, a) for states.  In later sections, state rewards are only identified as r(x, a), according to their inputs.  This is fine, but the notation should remain consistent.\n\n- the bound term defined in equation (11) includes the subscript z, in contrast to the one defined in equation (9).  What is the significance of this notation difference?  Similarly, the bound term defined in equation (17) inclues the subscript \"x,z\".  Including the initial bound denoted with the subscript 0, there are not 4 different notations used, which can be confusing, so a clearer explanation would help.  Anything as simple as explicitly describing that e_0 is the initial bound term, e(b) is the belief-only bound term, e_z(b, a) is the belief-action bound term using simplified observations only, and e_{x,z} is the belief-action bound term using simplified states and observations.\n\n- in many equations, the notation for expectation (and limited expectations) appears to be inconsistent, as sometimes the expectation conditions on an action and sometimes it doesn't.  It appears like this may simply be the case the conditional action is shown explicitly when it is not the one which would be chosen deterministically by the policy \\pi, while it is implied when it is the action chosen by the deterministic policy \\pi, but it is not clear whether this is really the case.  I urge the authors to standardize their notation as to avoid confusion and maximize comprehension for the reader.\n\n- Algorithm (1) is never referred to in the text, or described.  Further, both algorithm (1) and (2) have the same name `Algorithm-A`.  Please describe the contents of both.\n\nLIMITATIONS:\nAs noted by the authors themselves, their proposed method had additional requirements compared to other similar methods, specifically the ability to not just simulate/sample states and observations, but also to compute the respective probabilities.  Although this does not automatically disqualify the proposed method, its validity, and the authors' contributions, it is a significant requirement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}

comments: SUMMARY:
The paper introduces novel contributions in the form of deterministic bounds for discrete state space POMDPs and an algorithm to effectively implement these bounds in existing algorithms. The authors' key innovation lies in the identification of a specific subset within the observation and state space.

Through experimentation, the results demonstrate that incorporating these bounds leads to notable performance improvements in the majority of cases.

STRENGTHS:
The paper's presentation is commendable, offering clarity, and the derivations appear to be accurate. The inclusion of pseudo-code facilitates easy comprehension of the algorithm.

The presence of theoretical guarantees adds value to the research, instilling confidence in the proposed approach.

WEAKNESSES:
The authors' assumptions in this paper include a discrete state space and a known transition and observation model, which may not always align with real-world practicality.

Certain notations concerning reward functions are found to be inadequately defined, requiring further clarification.

Regarding the experimental results, the substantial gap observed on the Discrete Light Dark scenario is mentioned but lacks in-depth discussion, leaving room for further elaboration.

QUESTIONS:
See weaknesses

LIMITATIONS:
The limitation is addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28