SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moran Barenboim
Technion Autonomous Systems Program (TASP)
moranbar@campus.technion.ac.il
Vadim Indelman
Department of Aerospace Engineering
vadim.indelman@technion.ac.il
Autonomous agents operating in real-world scenarios frequently encounter uncertainty and make decisions based on incomplete information. Planning under uncertainty can be mathematically formalized using partially observable Markov decision processes (POMDPs). However, finding an optimal plan for POMDPs can be computationally expensive and is feasible only for small tasks. In recent years, approximate algorithms, such as tree search and sample-based methodologies, have emerged as state-of-the-art POMDP solvers for larger problems. Despite their effectiveness, these algorithms offer only probabilistic and often asymptotic guarantees toward the optimal solution due to their dependence on sampling. To address these limitations, we derive a deterministic relationship between a simplified solution that is easier to obtain and the theoretically optimal one. First, we derive bounds for selecting a subset of the observations to branch from while computing a complete belief at each posterior node. Then, since a complete belief update may be computationally demanding, we extend the bounds to support reduction of both the state and the observation spaces. We demonstrate how our guarantees can be integrated with existing state-of-the-art solvers that sample a subset of states and observations. As a result, the returned solution holds deterministic bounds relative to the optimal policy. Lastly, we substantiate our findings with supporting experimental results.
Partially Observable Markov Decision Processes (POMDPs) serve as a comprehensive mathematical framework for addressing uncertain sequential decision-making problems. Despite their applicability, most problems framed as POMDPs struggle to achieve optimal solutions, largely due to factors such as large state spaces and an extensive range of potential future scenarios. The latter tends to grow exponentially with the horizon, rendering the solution process computationally prohibitive. The advent of approximate online, tree-based solvers has expanded the capacity of POMDPs, enabling them to tackle larger problems by providing a more scalable approach to problem-solving.
A prominent search algorithm addressing the challenges posed by large state and observation spaces in POMDPs is POMCP [Silver and Veness, 2010]. POMCP is a forward search algorithm which handles the large state and observation spaces by aggregating Monte-Carlo rollouts of future scenarios in a tree structure. During each rollout, a single state particle is recursively propagated from the root node to the leaves of the tree. It adaptively trades off between actions that lead to unexplored areas of the tree and actions that lead to rewarding areas of the tree search by utilizing UCT [Auer et al., 2002]. The guarantees on the provided solution by POMCP are asymptotic, implying that the quality of the solution remains unknown within any finite time frame.
Figure 1: The figure depicts two search trees: a complete tree (left) that considers all states and observations at each planning step, and a simplified tree (right) that incorporates only a subset of states and observations, linked to simplified models. Our methodology establishes a deterministic link between these two trees.
Another notable approximate solver, Anytime Regularized DESPOT (AR-DESPOT) [Somani et al., 2013, Ye et al., 2017] is derived from Regularized DESPOT, which holds theoretical guarantees for the solution quality with respect to its optimal value. Similar to POMCP, AR-DESPOT performs forward search and propagates a single particle from the root node down to its leaves. It relies on branch-and-bound approach in the forward search, and utilizes dynamic programming techniques to update the value function estimate at each node. In contrast to POMCP, Regularized DESPOT offers a probabilistic lower bound on the value function obtained at the root node, providing a theoretical appeal by measuring its proximity to the optimal policy.
While the primary focus of this paper is on discrete POMDP planning, it is essential to acknowledge recent advancements in POMDP planning that encompass both discrete and continuous observation spaces. Few notable approaches include POMCPOw [Sunberg and Kochenderfer, 2018], LABECOP [Hoerger and Kurniawati, 2021] and AdaOPS [Wu et al., 2021], which leverage explicit use of observation models. These algorithms employ importance sampling mechanisms to weigh each state sample based on its likelihood value, which is assumed to be known. Although these methods have exhibited promising performance in practical scenarios, they currently lack formal guarantees. To address this gap, [Lim et al., 2020, 2022] introduced a simplified solver aimed at bridging the theoretical gap between the empirical success of these algorithms and the absence of theoretical guarantees for continuous observation spaces. In [Lim et al., 2022], probabilistic guarantees were derived for the simplified solver concerning its proximity to the optimal value function, thus contributing to a more comprehensive understanding of POMDP planning in both discrete and continuous settings.
In this paper, we focus on deriving deterministic guarantees for POMDPs with discrete state and observation spaces. Unlike existing black-box sampling mechanisms employed in algorithms such as [Sunberg and Kochenderfer, 2018, Hoerger and Kurniawati, 2021, Wu et al., 2021], our approach assumes access not only to the observation model but also to the transition model and the prior. By leveraging this additional information, we develop a novel algorithm that utilizes a subset of the observation space, enabling the computation of deterministic bounds with respect to the optimal policy at any belief node within the constructed tree. Furthermore, we demonstrate how these deterministic bounds can be integrated with state-of-the-art algorithms, including those that sample subsets of both the state and observation spaces. In particular, we provide experimental results showcasing the use of our bounds incorporated into the AR-DESPOT algorithm.
In this paper, our main contributions are as follows. First, we derive deterministic upper and lower bounds for a POMDP problem by considering a subset of the observation space at each node along the tree under a fixed policy. Next, we extend these bounds to cover scenarios where both the state and observation spaces are restricted to subsets, enhancing the applicability of our bounds in practical settings. Based on the derived bounds, we illustrate how to incorporate the bounds into a general structure of common state-of-the-art algorithms. Specifically, in the experimental section, we present results demonstrating the integration of our theoretical guarantees with the AR-DESPOT algorithm, yielding certified solutions with deterministic bounds. Last, we perform simulations to demonstrate the use of our derived bounds in practice, further validating their relevance in POMDP planning.
A finite horizon POMDP  $M$  is defined as a tuple  $\langle \mathcal{X},\mathcal{A},\mathcal{Z},T,O,\mathcal{R}\rangle$ , where  $\mathcal{X}$ ,  $\mathcal{A}$ , and  $\mathcal{Z}$  represent a discrete state, action, and observation spaces, respectively. The transition density function  $T(x_{t},a_{t},x_{t + 1})\triangleq \mathbb{P}(x_{t + 1}|x_{t},a_{t})$  defines the probability of transitioning from state  $x_{t}\in \mathcal{X}$  to state  $x_{t + 1}\in \mathcal{X}$  by taking action  $a_{t}\in \mathcal{A}$ . The observation density function  $O(x_{t},z_{t})\triangleq \mathbb{P}(z_{t}|x_{t})$  expresses the probability of receiving observation  $z_{t}\in \mathcal{Z}$  from state  $x_{t}\in \mathcal{X}$ .
Given the limited information provided by observations, the true state of the agent is uncertain and a probability distribution function over the state space, also known as a belief, is maintained. The belief depends on the entire history of actions and observations, and is denoted  $H_{t} \triangleq \{z_{1:t}, a_{0:t-1}\}$ . We also define the propagated history as  $H_{t}^{-} \triangleq \{z_{1:t-1}, a_{0:t-1}\}$ . At each time step  $t$ , the belief is updated by applying Bayes' rule using the transition and observation models, given the previous action  $a_{t-1}$  and the current observation  $z_{t}$ ,  $b(x_{t}) = \eta_{t} \mathbb{P}(z_{t} | x_{t}) \sum_{x_{t-1} \in \mathcal{X}} \mathbb{P}(x_{t} | x_{t-1}, a_{t-1}) b(x_{t-1})$ , where  $\eta_{t}$  denotes a normalization constant and  $b_{t} \triangleq \mathbb{P}(x_{t} | H_{t})$  denotes the belief at time  $t$ . The updated belief,  $b_{t}$ , is sometimes referred to as the posterior belief, or simply the posterior. We will use these interchangeably throughout the paper.
A policy function  $a_{t} = \pi_{t}(b_{t})$  determines the action to be taken at time step  $t$ , based on the current belief  $b_{t}$  and time  $t$ . In the rest of the paper we write  $\pi_t\equiv \pi_t(b_t)$  for conciseness. The reward is defined as an expectation over a state-dependent function,  $r(b_{t},a_{t}) = \mathbb{E}_{x\sim b_{t}}[r_{x}(x,a_{t})]$ , and is bounded by  $-\mathcal{R}_{\mathrm{max}}\leq r_x(x,a_t)\leq \mathcal{R}_{\mathrm{max}}$ . The value function for a policy  $\pi$  over a finite horizon  $\mathcal{T}$  is defined as the expected cumulative reward received by executing  $\pi$  and can be computed using the Bellman update equation,
$$
V _ {t} ^ {\pi} \left(b _ {t}\right) = r \left(b _ {t}, \pi_ {t}\right) + \underset {z _ {t + 1: T}} {\mathbb {E}} \left[ \sum_ {\tau = t + 1} ^ {T} r \left(b _ {\tau}, \pi_ {\tau}\right) \right]. \tag {1}
$$
The action-value function is defined by executing action  $a_{t}$  and then following policy  $\pi$ . The optimal value function may be computed using Bellman's principle of optimality,
$$
V _ {t} ^ {\pi^ {*}} \left(b _ {t}\right) = \max  _ {a _ {t}} \left\{r \left(b _ {t}, a _ {t}\right) + \underset {z _ {t + 1} \mid a _ {t}, b _ {t}} {\mathbb {E}} \left[ V _ {t + 1} ^ {\pi^ {*}} \left(b _ {t + 1}\right) \right] \right\}. \tag {2}
$$
The goal of the agent is to find the optimal policy  $\pi^{*}$  that maximizes the value function.
3 Mathematical Analysis
Typically, it is infeasible to fully expand a Partially Observable Markov Decision Process (POMDP) tree due to the extensive computational resources and time required. To address this challenge, we propose two approaches. In the first approach, we propose a solver that selectively chooses a subset of the observations to branch from, while maintaining a full posterior belief at each node. This allows us to derive a hypothetical algorithm that directly uses our suggested deterministic bounds to choose which actions to take while exploring the tree. As in some scenarios computing a complete posterior belief may be too expensive, we suggest a second method that in addition to branching only a subset of the observations, selectively chooses a subset of the states at each encountered belief. Using the deterministic bounds at the root node for each action value allows us to certify the performance of a given policy at the root of the planning tree. Moreover, given a scenario in which an action exists whose lower bound surpasses all other actions' upper bound, the algorithm can identify the optimal action and stop further exploration. In contrast, existing state-of-the-art algorithms either do not provide any guarantee on the solution quality (e.g. POMCP Silver and Veness [2010]) or merely provide probabilistic guarantee on the solution (e.g. DESPOT Somani et al. [2013]). In the following section, we show how to use the deterministic bounds in conjunction with state-of-the-art algorithms to obtain performance guarantees.
Both of the presented approaches diverge from many existing algorithms that rely on black-box prior, transition, and observation models. Instead, our method directly utilizes state and observation probability values to evaluate both the value function and the associated bounds. In return, we offer
<sup>1</sup>All proofs and derivations are deferred to the supplementary file.
anytime deterministic guarantees on the value function for the derived policy and establish bounds on its deviation from the value function of the optimal policy.
In this section we provide a mathematical quantification of the impact of using a solver that only considers a small subset of the theoretical tree branches and a subset of states within each node. We begin by defining a simplified POMDP, which is a reduced complexity version of the original POMDP that abstracts or ignores certain states and observations. We then establish a connection between the simplified value function and its theoretical counterpart. Finally, we demonstrate a relationship between the simplified value function, obtained by following the best policy for the simplified problem, and the theoretically optimal value function.
We begin with general definitions of a simplified prior, transition and observation models,
$$
\bar {b} _ {0} (x) \triangleq \left\{ \begin{array}{l l} b _ {0} (x) & , x \in \bar {\mathcal {X}} _ {0} \\ 0 & , o t h e r w i s e \end{array} \right. \tag {3}
$$
$$
\bar {\mathbb {P}} \left(x _ {t + 1} \mid x _ {t}, a _ {t}\right) \triangleq \left\{ \begin{array}{l l} \mathbb {P} \left(x _ {t + 1} \mid x _ {t}, a _ {t}\right) & , x _ {t + 1} \in \bar {\mathcal {X}} \left(H _ {t + 1} ^ {-}\right) \\ 0 & , o t h e r w i s e \end{array} \right. \tag {4}
$$
$$
\bar {\mathbb {P}} \left(z _ {t} \mid x _ {t}\right) \triangleq \left\{ \begin{array}{l l} \mathbb {P} \left(z _ {t} \mid x _ {t}\right) & , z _ {t} \in \tilde {\mathcal {Z}} \left(H _ {t}\right) \\ 0 & , o t h e r w i s e \end{array} \right. \tag {5}
$$
where  $\bar{\mathcal{X}}(H_{t+1}^-) \subseteq \mathcal{X}$  and  $\bar{\mathcal{Z}}(H_t) \subseteq \mathcal{Z}$  may be chosen arbitrarily, e.g. by sampling or choosing a fixed subset a-priori, as the derivations of the bounds are independent of the subset choice. Note that the simplified prior, transition and observation models are unnormalized and thus do not represent a valid distribution function. For the rest of the sequel we will drop the explicit dependence on the history, and denote  $\bar{\mathcal{X}}(H_{t+1}^-) \equiv \bar{\mathcal{X}}, \bar{\mathcal{Z}}(H_t) \equiv \bar{\mathcal{Z}}$ . Using the simplified models, we define the simplified belief  $\bar{b}_{t+1}$ . The simplified belief is updated using the simplified belief update equation, which is a modification of the standard belief update equation that replaces the usual models with the simplified ones. More precisely,
$$
\bar {b} _ {t + 1} \left(x _ {t + 1}\right) \triangleq \left\{ \begin{array}{l l} \frac {\bar {\mathbb {P}} \left(z _ {t + 1} \mid x _ {t + 1}\right) \sum_ {x _ {t}} \bar {\mathbb {P}} \left(x _ {t + 1} \mid x _ {t} , \pi_ {t}\right) \bar {\mathbb {P}} \left(x _ {t} \mid H _ {t}\right)}{\bar {\mathbb {P}} \left(z _ {t + 1} \mid H _ {t + 1} ^ {-}\right)} & , \bar {\mathbb {P}} \left(z _ {t + 1} \mid H _ {t + 1} ^ {-}\right) \neq 0 \\ 0 & , o t h e r w i s e \end{array} \right. \tag {6}
$$
where  $\bar{b}_{t+1}(x) \equiv \bar{\mathbb{P}}(x_{t+1} \mid H_{t+1})$ , and  $\bar{\mathbb{P}}(z_{t+1} \mid H_{t+1}^-) \triangleq \bar{\mathbb{P}}(z_{t+1} \mid x_{t+1}) \sum_{x_t} \bar{\mathbb{P}}(x_{t+1} \mid x_t, \pi_t) \bar{\mathbb{P}}(x_t \mid H_t)$ . Note that a simplified belief cannot depend on an observation that is not part of the simplified observation set and is considered undefined. Last we define a simplified value function,
$$
\begin{array}{l} \bar {V} ^ {\pi} \left(\bar {b} _ {t}\right) \triangleq r \left(\bar {b} _ {t}, \pi_ {t}\right) + \bar {\mathbb {E}} \left[ \bar {V} \left(b _ {t}\right) \right] (7) \\ = \sum_ {x _ {t}} \bar {b} \left(x _ {t}\right) r \left(x _ {t}, \pi_ {t}\right) + \sum_ {z _ {t}} \bar {\mathbb {P}} \left(z _ {t + 1} \mid H _ {t + 1} ^ {-}\right) \bar {V} (\bar {b} \left(z _ {t + 1}\right)), (8) \\ \end{array}
$$
where the simplified expectation operator,  $\bar{\mathbb{E}} [\cdot ]$ , is taken with respect to the unnormalized likelihood  $\bar{\mathbb{P}} (z_{t + 1}\mid H_{t + 1}^{-})$
3.1 Simplified observation space
We first analyze the performance guarantees of a simplified observation space, while assuming a complete belief update at each belief state, i.e.,  $\tilde{\mathcal{X}}\equiv \mathcal{X}$ . The following theorem describes the guarantees of the observation-simplified value function with respect to its theoretical value,
Theorem 1. Let  $b_{t}$  belief state at time  $t$ , and  $T$  be the last time step of the POMDP. Let  $V^{\pi}(b_{t})$  be the theoretical value function by following a policy  $\pi$ , and let  $\bar{V}^{\pi}(b_{t})$  be the simplified value function, as defined in (18), by following the same policy. Then, for any policy  $\pi$ , the difference between the theoretical and simplified value functions is bounded as follows,
$$
\left| V ^ {\pi} \left(b _ {t}\right) - \bar {V} ^ {\pi} \left(b _ {t}\right) \right| \leq \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} \left[ 1 - \sum_ {z _ {t + 1: \tau}} \sum_ {x _ {t: \tau}} b \left(x _ {t}\right) \prod_ {k = t + 1} ^ {\tau} \overline {{\mathbb {P}}} \left(z _ {k} \mid x _ {k}\right) \mathbb {P} \left(x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}\right) \right] \triangleq \epsilon_ {z} ^ {\pi} (b _ {t}), \tag {9}
$$
where we use a subscript of  $(z)$  in  $\epsilon_z^\pi (b_t)$  to denote observation-only simplification. Importantly, the bound only contains terms which depend on observations that are within the simplified space,  $z\in \bar{\mathcal{Z}}$ . This is an essential property of the bound, as it is a value that can easily be calculated during the planning process and provides a certification of the policy quality at any given node along the tree. Furthermore, it is apparent from (20) that as the number of observations included in the simplified set,  $\bar{\mathcal{Z}}$ , increases, the value of  $\epsilon_z^\pi (b_t)$  consequently diminishes,
$$
\sum_ {z _ {1: \tau}} \sum_ {x _ {0: \tau}} b (x _ {0}) \prod_ {k = 1} ^ {\tau} \overline {{\mathbb {P}}} (z _ {k} \mid x _ {k}) \mathbb {P} (x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) \xrightarrow {\bar {\mathcal {Z}} \to \mathcal {Z}} 1
$$
leading to a convergence towards the theoretical value function, i.e.  $\epsilon_z^\pi (b_t)\to 0$
Theorem 3 provides both lower and upper bounds for the theoretical value function, assuming a fixed policy. Using this theorem, we can derive upper and lower bounds for any policy, including the optimal one. This is achieved by applying the Bellman optimality operator to the upper bound in a repeated manner, instead of the estimated value function; In the context of tree search algorithms, our algorithm explores only a subset of the decision tree due to pruned observations. However, at every belief node encountered during this exploration, all potential actions are expanded. The action-value of these expanded actions is bounded using the Upper Deterministic Bound, which we now define as
$$
\mathrm {U D B} ^ {\pi} \left(b _ {t}, a _ {t}\right) \triangleq \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) = r \left(b _ {t}, a _ {t}\right) + \bar {\mathbb {E}} _ {z _ {t + 1}} \left[ \bar {V} ^ {\pi} \left(b _ {t + 1}\right) \right] + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right), \tag {10}
$$
where the action-dependent bound on the value difference,  $\epsilon_z^\pi (b_t,a_t)$ , is the bound of taking action  $a_{t}$  in belief  $b_{t}$  and following policy  $\pi$  thereafter,
$$
\epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \triangleq \mathcal {R} _ {\max } \sum_ {\tau = t + 1} ^ {T} [ 1 - \sum_ {z _ {t + 1}; \tau} \sum_ {x _ {t}; \tau} b (x _ {t}) \overline {{\mathbb {P}}} \left(z _ {t + 1} \mid x _ {t + 1}\right) \mathbb {P} \left(x _ {t + 1} \mid x _ {t}, a _ {t}\right). \tag {11}
$$
$$
\prod_ {k = t + 2} ^ {\tau} \overline {{\mathbb {P}}} (z _ {k} \mid x _ {k}) \mathbb {P} (x _ {k} \mid x _ {k - 1}, \pi_ {k - 1}) ].
$$
In the event that no subsequent observations are chosen for a given history, the value of  $\bar{Q}^{\pi}(b_t,a_t)$  simplifies to the immediate reward plus an upper bound for any subsequent policy, given by  $\mathcal{R}_{\mathrm{max}}\cdot (T - t - 1)$ .
Using UDB we define the action selection criteria according to
$$
a _ {t} = \arg \max  _ {a _ {t} \in \mathcal {A}} \left[ \mathrm {U D B} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right] = \arg \max  _ {a _ {t} \in \mathcal {A}} \left[ \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right]. \tag {12}
$$
Moreover, the optimal value function can be bounded as follows,
Lemma 1. The optimal value function can be bounded as
$$
V ^ {\pi *} \left(b _ {t}\right) \leq \mathrm {U D B} ^ {\pi} \left(b _ {t}\right), \tag {13}
$$
where the policy  $\pi$  is determined according to Bellman optimality over the UDB, i.e.
$$
\begin{array}{l} \mathrm {U D B} ^ {\pi} \left(b _ {t}\right) \triangleq \max  _ {a _ {t} \in \mathcal {A}} \left[ \bar {Q} ^ {\pi} \left(b _ {t}, a _ {t}\right) + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right] (14) \\ = \max  _ {a _ {t} \in \mathcal {A}} \left[ r \left(b _ {t}, a _ {t}\right) + \bar {\mathbb {E}} _ {z _ {t + 1} \mid b _ {t}, a _ {t}} \left[ \bar {V} ^ {\pi} \left(b _ {t + 1}\right) \right] + \epsilon_ {z} ^ {\pi} \left(b _ {t}, a _ {t}\right) \right]. (15) \\ \end{array}
$$
Corollary 1.1. By utilizing Lemma 1 and the exploration criteria defined in (65), an increasing number of explored belief nodes guarantees convergence to the optimal value function.
Notably, UDB does not require a complete recovery of the posterior branches to yield an optimal policy. Each action-value is bounded by a specific lower and upper bound, which can be represented as an interval enclosing the theoretical value. When the bound intervals of two candidate actions do not overlap, one can clearly discern which action is suboptimal, rendering its subtree redundant for further exploration. This distinction sets UDB apart from current state-of-the-art online POMDP algorithms. In those methods, any finite-time stopping condition fails to ensure optimality since the bounds used are either heuristic or probabilistic in nature.
3.2 Simplified State and Observation Spaces
In certain scenarios, the complete evaluation of posterior beliefs during the planning stage may pose significant computational challenges. To tackle this issue, we propose the use of a simplified state space in addition to the simplified observation space considered thus far. Specifically, we derive deterministic guarantees of the value function that allow for the selection of a subset from both the states and observations.
Theorem 2. Let  $b_{0}$  and  $\bar{b}_{0}$  be the theoretical and simplified belief states, respectively, at time  $t = 0$ , and  $T$  be the last time step of the POMDP. Let  $V^{\pi}(b_{0})$  be the theoretical value function by following a policy  $\pi$ , and let  $\bar{V}^{\pi}(\bar{b}_{0})$  be the simplified value function by following the same policy, as defined in (18). Then, for any policy  $\pi$ , the difference between the theoretical and simplified value functions is bounded as follows,
$$
\left| V ^ {\pi} \left(b _ {0}\right) - \bar {V} ^ {\pi} \left(\bar {b} _ {0}\right) \right| \leq \mathcal {R} _ {\max } \left[ 1 - \sum_ {x} \bar {b} _ {0} (x) \right] + \mathcal {R} _ {\max } \sum_ {\tau = 1} ^ {T} \left[ 1 - \overline {{\mathbb {E}}} _ {z _ {1: \tau}} \sum_ {x} \bar {b} _ {\tau} (x) \right] \triangleq \epsilon_ {x, z} ^ {\pi} \left(b _ {0}\right), \tag {16}
$$