SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Edward Kim
School of Computing Australian National University Canberra, Australia edward.kim@anu.edu.au
Yohan Karunanayake
School of Computing Australian National University Canberra, Australia yohan.karunanayake@anu.edu.au
Hanna Kurniawati
School of Computing Australian National University Canberra, Australia hanna.kurniawati@anu.edu.au
Making good decisions in partially observable and non-deterministic scenarios is a crucial capability for robots. A Partially Observable Markov Decision Process (POMDP) is a general framework for the above problem. Despite advances in POMDP solving, problems with long planning horizons and evolving environments remain difficult to solve even by the best approximate solvers today. To alleviate this difficulty, we propose a slightly modified POMDP problem, called a Reference-Based POMDP, where the objective is to balance between maximizing the expected total reward and being close to a given reference (stochastic) policy. The optimal policy of a Reference-Based POMDP can be computed via iterative expectations using the given reference policy, thereby avoiding exhaustive enumeration of actions at each belief node of the search tree. We demonstrate theoretically that the standard POMDP under stochastic policies is related to the Reference-Based POMDP. To demonstrate the feasibility of exploiting the formulation, we present a basic algorithm REFSOLVER. Results from experiments on long-horizon navigation problems indicate that this basic algorithm substantially outperforms POMCP.
Computing motion strategies that are robust to uncertainty is a fundamental problem in robotics. A robot often needs to strategically plan in an environment where it only has partial knowledge about itself and the state of the world due to actuator disturbances, limited knowledge about the environment, and imperfect sensory information. Such planning problems can generally be framed as a Partially Observable Markov Decision Process (POMDP) [7, 22], which is a general mathematical framework for planning in partially observable and non-deterministic scenarios. POMDPs explicitly model partial observability using a probability distribution over the state space, called a belief, and compute the best strategy with respect to beliefs. Finding the exact solution to a POMDP problem is computationally intractable in the worst case [15, 17] for two main reasons. Firstly, the size of the belief space grows exponentially with respect to the size of the state space. Secondly, the number of action-observation histories grows exponentially with respect to the planning horizon. These challenges are known as the curse of dimensionality and the curse of history respectively. To deal with these challenges, many practical solvers have been proposed in the past few decades [9]. Sampling-based approximate solvers [12, 21, 29] have made good progress on both curses by employing particle filters to maintain a belief search tree and judiciously exploring a subset of histories in this search tree.
Notwithstanding these advances, some partially observed problems, such as those involving long planning horizons and progressively evolving environments remain relatively difficult to solve by even the best approximate POMDP solvers today. In particular, all of the aforementioned solvers struggle with a planning horizon of more than 15 steps primarily because, to infer the best action, they rely on exhaustive enumeration of actions at each belief node in the search tree.
To alleviate the above difficulties, and hence further improve POMDP solving capabilities, we propose the idea of a Reference-Based POMDP. A Reference-Based POMDP is a modification of the standard POMDP where, in addition to maximising total rewards, an agent's objective function is penalised for deviating too far from a given stochastic reference policy. This approach leads to an interesting interpretation where the agent must trade off between two possibly competing objectives. The agent should respect the given initial stochastic policy unless deviating substantially from it leads to much higher rewards.
A key outcome of this formulation is that solving a Reference-Based POMDP does not require exhaustive enumeration of the actions at each belief node. Rather, optimisation can be performed analytically because: (i) the space of policies is relaxed to include stochastic policies; and (ii) the form of the penalty is represented as the KL divergence from the reference policy. Under suitable transformations, this leads to a linear Bellman backup (Theorem 3.1) where the sampling distribution is the reference policy itself. As the reference policy is known a priori, the above insight facilitates algorithms where the actions can be sampled (as opposed to enumerated) at each step which, in turn, can be efficiently computed using Monte Carlo methods.
This work can be viewed as an extension of a body of literature related to Linearly Solvable Optimal Control[24, 26, 27, 1, 19, 28, 3] in fully observable environments (see Section 7 for a brief summary) to partially observable ones. While it is true that a POMDP can be viewed as an MDP in the belief space [7], extending the aforementioned works to partially observable domains is non-trivial because of the infinite and continuous nature of the belief space and the difficulty of computing belief updates. To the authors' knowledge, the approaches outlined above have not yet been extended to partially observable domains.
The paper is organised as follows. Section 2 recounts the basic setup of a POMDP. Section 3 introduces the Reference-Based POMDP and outlines the relevant theory. Section 4 demonstrates that the standard POMDP can be related to the Reference-Based POMDP via an embedding. Section 5 presents REFsOLVER a preliminary approximate solver for the Reference-Based POMDP. Experimental results on long-horizon 2D and 3D navigation problems are presented in Section 6 and indicate that our solver can be employed to substantially outperform POMCP [21]. Note that the problems we have presented, are not trivial due to the long planning horizon and the challenges of partial observability and transition uncertainty. Finally, we outline the relevant literature in Section 7 and summarise the contributions, limitations and future directions of the paper in Section 8.
2 POMDP Preliminaries
In this paper, we focus on an infinite-horizon formulation of the POMDP. Formally, a standard POMDP is completely specified by a tuple  $\langle S, \mathcal{A}, \mathcal{O}, \mathcal{Z}, \mathcal{T}, R, \gamma \rangle$ . Here,  $\mathcal{S}$  denotes the set of all possible states of the agent and its environment,  $\mathcal{A}$  denotes the set of all possible actions the agent can perform, and  $\mathcal{O}$  denotes all the possible observations that the agent can perceive. Although our transformation could be applied to POMDPs with continuous state, action, and observation spaces, for simplicity, in this paper, we will assume that  $\mathcal{S}$ ,  $\mathcal{A}$  and  $\mathcal{O}$  are all discrete.
At each time step, the agent occupies a state  $s \in S$  and executes an action  $a \in \mathcal{A}$ , after which its state transitions to a new state  $s' \in S$ . The outcome of this execution is non-deterministic, and this uncertainty is represented in the Markovian transition function  $\mathcal{T}$  as a conditional probability function  $\mathcal{T}(s'|s,a)$ . At the new state  $s'$ , the agent perceives an observation  $o \in \mathcal{O}$ . Uncertainty in the observation perceived – having undertaken action  $a \in \mathcal{A}$  and transitioned to state  $s'$  – is described by the conditional probability function  $\mathcal{Z}(o|s',a)$ , referred to as the observation function. The agent's action incurs a reward, which is defined by a bounded real-valued function  $R: S \times \mathcal{A} \to \mathbb{R}$ . The parameter  $\gamma \in (0,1)$  represents the discount factor.
As the agent does not know its true state, at each time step, the agent maintains a belief about its state. A belief, denoted as  $b$ , is a probability distribution over the state space  $S$ . The set of all possible
beliefs forms a belief space of  $(|S| - 1)$ -dimensional simplex, which we denote as  $\mathcal{B}$ . The belief the agent maintains is updated via Bayesian inference using the transition and observation models  $\mathcal{Z}$  and  $\mathcal{T}$ , respectively. The exact belief update is completely determined by an action-observation pair. Formally, if  $b' = \tau(b, a, o)$  denotes the agent's next belief after taking action  $a$  and perceiving observation  $o$ , then  $b'$  is defined as:
$$
b ^ {\prime} \left(s ^ {\prime}\right) = \eta \mathcal {Z} (o \mid s ^ {\prime}, a) \sum_ {s \in \mathcal {S}} \mathcal {T} \left(s ^ {\prime} \mid s, a\right) b (s) \tag {1}
$$
where  $\eta$  is a normalising factor. The solution to a POMDP problem is an optimal policy  $\pi^{*}:\mathcal{B}\to \mathcal{A}$  that maximises the value function, defined as:
$$
V ^ {*} (b) = \max  _ {a \in \mathcal {A}} \left[ R (b, a) + \gamma \sum_ {o \in \mathcal {O}} P (o \mid a, b) V ^ {*} (\tau (b, a, o)) \right] \tag {2}
$$
$$
R (b, a) := \sum_ {s} R (s, a) b (s) \tag {3}
$$
is the expected reward under the belief  $b$ . The notation  $P(o \mid a, b)$  is the probability the agent perceives an observation  $o \in \mathcal{O}$  having performed the action  $a \in \mathcal{A}$  under the belief  $b$ , and is defined as
$$
P (o \mid a, b) := \sum_ {s ^ {\prime} \in \mathcal {S}} \mathcal {Z} (o \mid s ^ {\prime}, a) \left(\sum_ {s \in \mathcal {S}} \mathcal {T} \left(s ^ {\prime} \mid s, a\right) b (s)\right). \tag {4}
$$
Computing the exact solution  $\pi^{*}$  and optimal value function  $V^{*}$  is intractable [14, 15, 17] in general, and therefore state-of-the-art solvers approximate solutions via sampling [9].
3 Reference-Based POMDP
We now extend the analytical approach for the Linearly Solvable MDP (see Appendix Section A.1) to the POMDP. To this end, we define a Reference-Based POMDP, which is specified by the tuple  $\langle S, \mathcal{A}, \mathcal{O}, \mathcal{Z}, \mathcal{T}, R, \gamma, \bar{U} \rangle$ . Similar to the standard POMDP,  $\mathcal{S}$ ,  $\mathcal{A}$ , and  $\mathcal{O}$  refer to the state, action, and observation spaces respectively. We denote by  $\mathcal{Z}$ ,  $\mathcal{T}$ , and  $R$  the observation, transition, and reward functions, respectively, while  $\gamma \in (0, 1)$  denotes the discount factor.
Assumption 3.1. A reference belief-to-belief transition probability:
$$
\bar {U} (\cdot , \cdot \mid b) \in \Delta (\mathcal {A} \times \mathcal {O}) \tag {5}
$$
is given for all  $b \in \mathcal{B}$ .
This reference transition is assumed by the problem and could be constructed in multiple ways. At the most abstract the level this could be given in a top-down fashion (e.g. an offline policy that has been generated for a similar version of a POMDP problem). On the other hand, it could also be constructed from the ground up. For example, one could take a fully observed policy  $\pi^{\mathrm{FO}}: S \to \mathcal{A}$  and infer a reference stochastic action  $\bar{\pi}(\cdot | b)$  according to
$$
\bar {\pi} (a \mid b) := \eta \left[ \alpha \sum_ {s \in \mathcal {S}} I _ {\{\pi^ {\mathrm {F O}} (s) = a \}} b (s) + (1 - \alpha) \rho (a \mid b) \right] \tag {6}
$$
where  $\rho$  is some default distribution for sampling actions and the parameter  $\alpha \in (0,1)$  indicates the saturation level of the fully observed policy. The exact choice of fully observed policy depends on the context. Examples might include: an MDP policy for the fully observed version of the problem, a policy induced by feasible paths generated by motion planners, a policy induced by all source shortest path in the case of a navigation problem, and even a policy representing the passive dynamics of a system at any given state. Regardless of how the reference stochastic action is constructed, in a ground-up construction, the reference belief-to-belief transition probability will take on the more specific form:
$$
\bar {U} (a, o \mid b) := P (o \mid a, b) \bar {\pi} (a \mid b) \tag {7}
$$
where  $P(o\mid a,b)$  is given by (4).
Inspired by Linearly-Solvable MDPs (Section A.1), a Reference-Based POMDP asserts that, for any belief  $b \in \mathcal{B}$ , we can realise "belief-to-belief" transition probabilities  $U(\cdot, \cdot | b) \in \Delta(\mathcal{A} \times \mathcal{O})$  of
our choosing. Note that this is an ideal transition probability which may not actually be realisable in practice even by a stochastic policy (see Section 3.3 and Section 8 for further commentary). To prohibit arbitrary transitions, at any belief  $b \in \mathcal{B}$ , we will further restrict our choice of distribution  $U$  to one that satisfies the property
$$
\bar {U} (a, o \mid b) = 0 \Longrightarrow U (a, o \mid b) = 0 \quad \forall a \in \mathcal {A}, o \in \mathcal {O}.
$$
Denote the class of such distributions by  $\mathcal{U}(b) \subset \Delta(\mathcal{A} \times \mathcal{O})$  which will form our admissible reference transition probabilities (which can be viewed as admissible controls) from a given belief. Then,  $\mathcal{U} := \bigcup_{b \in \mathcal{B}} \mathcal{U}(b)$  is the class of all admissible reference transition probabilities. Similar to (25), we penalise choices of  $U(\cdot, \cdot | b)$  that represent large deviations from  $\bar{U}(\cdot, \cdot | b)$  using KL divergence: KL  $(U(\cdot, \cdot | b) \| \bar{U}(\cdot, \cdot | b))$ .
Now, the above reference transition probability affects the definition of the reward function too. At the most abstract level, the reward function  $R: \mathcal{B} \times \mathcal{A} \to \mathbb{R}$  is parameterised by belief-action pairs. Hence, the expected immediate cost of taking a stochastic belief-to-belief transition governed by  $U(\cdot, \cdot | b) \in \mathcal{U}(b)$  is given by
$$
R (b, U) := \sum_ {a, o} R (b, a) U (a, o \mid b) \tag {8}
$$
where again  $R(b, a) \coloneqq \sum_{s \in S} R(s, a)b(s)$ . The solution to a Reference-Based POMDP problem is then an optimal stochastic policy  $\pi^{*}: \mathcal{B} \to \mathcal{U}$  that maximises:
$$
\begin{array}{l} \mathcal {V} ^ {*} (b) = \sup  _ {U (\cdot , \cdot | b) \in \mathscr {U} (b)} \left(R (b, U) - \operatorname {K L} \left(U (\cdot , \cdot | b) \| \bar {U} (\cdot , \cdot | b)\right) \right. \\ + \gamma \sum_ {a, o} U (a, o \mid b) \mathcal {V} ^ {*} (\tau (b, a, o)) \Big). \tag {9} \\ \end{array}
$$
3.1 Existence and Uniqueness of the Solution
One might ask if the reformulation (9) is satisfied uniquely by some function  $\mathcal{V}:\mathcal{B}\to \mathbb{R}$ . The following lemma ensures that there exists a unique (up to  $\| \cdot \|_{\infty}$ -equivalence) bounded function  $\mathcal{V}:\mathcal{B}\rightarrow \mathbb{R}$  satisfying the Bellman equation. Its proof is deferred to Appendix A.2.
Lemma 3.1. Consider the Banach space  $\mathbb{B}(\mathcal{B})$  of bounded functions  $\mathcal{V}:\mathcal{B}\to \mathbb{R}$  equipped with the supremum norm  $\| \cdot \|_{\infty}$ . Let  $\Phi :\mathbb{B}(\mathcal{B})\to \mathbb{B}(\mathcal{B})$  be a self-mapping defined by the Reference-Based POMDP backup operator described by equation (9). Then, for  $\gamma \in (0,1)$ , the mapping  $\Phi$  is a contraction and
$$
\left\| \Phi \mathcal {V} _ {1} - \Phi \mathcal {V} _ {2} \right\| _ {\infty} \leq \gamma \left\| \mathcal {V} _ {1} - \mathcal {V} _ {2} \right\| _ {\infty}. \tag {10}
$$
Hence, by the Banach fixed point theorem [8], the solution to the Bellman equation exists uniquely in this space whenever  $\gamma \in (0,1)$ .
3.2 Transforming the Value Function
Here, we show that under an appropriate transformation, the value function of a Reference-Based POMDP (9) can also be linearised. Its proof is given in Appendix A.3.
Theorem 3.1. Define the transformation  $\mathcal{W}(b) \coloneqq e^{\mathcal{V}(b)}$  for any  $b \in \mathcal{B}$ . The value function (9) of the Reference-Based POMDP  $\langle S, \mathcal{A}, \mathcal{O}, \mathcal{Z}, \mathcal{T}, R, \gamma, \tilde{U} \rangle$  is equivalent to
$$
\mathcal {W} (b) = \sum_ {a, o} \bar {U} (a, o \mid b) e ^ {R (b, a)} \mathcal {W} ^ {\gamma} (\tau (b, a, o)). \tag {11}
$$
The solution  $\mathcal{W}^*$  to equation (11) exists and is unique. And, the optimal stochastic "belief-to-belief" transition to the Bellman equation (9) is given by
$$
U ^ {*} (a, o \mid b) = \frac {\bar {U} (a , o \mid b) e ^ {R (b , a)} \mathcal {W} ^ {* \gamma} (\tau (b , a , o))}{\mathcal {D} [ \mathcal {W} ^ {* \gamma} ] (b)} \tag {12}
$$
where  $\mathcal{D}[\mathcal{W}^{*\gamma}](b) \coloneqq \sum_{\hat{a}, \hat{o}} \bar{U}(\hat{a}, \hat{o} \mid b) e^{R(b, \hat{a})} \mathcal{W}^{*\gamma}\left(\tau(b, \hat{a}, \hat{o})\right)$  is a normaliser.
Intuitively, the solution  $\mathcal{W}^*$  represents the desirability of a given belief (or history of action-observation sequences) where, (12) represents distorting the reference dynamics to higher desirability scores. Moreover, Theorem 3.1 shows that we can converge to  $\mathcal{W}^*$  by iterating the transformed Reference-Based POMDP backup operator (11). Lemma 3.1 indicates that the speed of convergence depends on the size of  $\gamma$  with faster convergence as  $\gamma$  tends to 0. Moreover, Theorem 3.1 also implies that once  $\mathcal{W}^*$  is computed,  $U^{*}$  can be computed exactly via (12).
Unlike the linearly solvable MDP, exactly computing  $\mathcal{W}^*$  is still intractable in practice due to the continuity of the belief space and the cost of belief estimation. However, if we leverage the advances made by sampling-based POMDP solvers and assume that the belief space is sufficiently represented by a finite number of beliefs, then this quantity can be approximated. Of course, finding such a set of beliefs is in itself difficult. Nevertheless, the fact that the optimal value function of a Reference-Based POMDP is equivalent to (11) implies that the optimal value function of Reference-Based POMDP can be computed by purely computing expectations under the reference transition probabilities recursively, which can be approximated efficiently (e.g. via Monte Carlo approximation).
3.3 Recovering the Stochastic Actions
While restricting the agent's transition probabilities to  $\mathcal{U}(b)$  limits the agent's level of control,  $U^{*}$  as given in (12) is still an ideal transition probability and may therefore not be realisable under a stochastic action in general. In other words, one cannot guarantee the existence of a stochastic action  $\pi(a|b)$  such that  $U(a,o|b) := P(o|a,b)$ $\pi(a|b) = U^{*}(a,o|b)$  for every  $a\in \mathcal{A}$  and  $o\in \mathcal{O}$ .
Nevertheless, we can still find a  $\pi(a|b)$  such that  $U$  and  $U^{*}$  are close in some sense. A suitable way to formulate the problem is to minimise the KL divergence between  $U^{*}$  and  $U$ . That is, for each  $b \in \mathcal{B}$ , we minimise
$$
\sum_ {a, o} P (o \mid a, b) \pi (a \mid b) \log \left[ \frac {P (o \mid a , b) \pi (a \mid b)}{U ^ {*} (a , o \mid b)} \right] \tag {13}
$$
over all stochastic actions  $\pi (\cdot |b)\in \Delta (\mathcal{A})$ . Solving the constrained minimisation problem using Lagrange multipliers yields the following result. See Appendix A.4 for its proof.
Proposition 3.1. The minimiser of (13) is
$$
\pi^ {*} (a \mid b) = \frac {\exp [ - \Pi (a \mid b) ]}{\sum_ {\hat {a} \in \mathcal {A}} \exp [ - \Pi (\hat {a} \mid b) ]} \quad \forall a \in \mathcal {A} \tag {14}
$$
$$
\Pi (a \mid b) := \sum_ {o} P (o \mid a, b) \log \left[ \frac {P (o \mid a , b)}{U ^ {*} (a , o \mid b)} \right]. \tag {15}
$$
4 Relating the Reference-Based POMDP and the Standard POMDP
In A.1, we recount a procedure for embedding the standard MDP  $\langle S, \mathcal{A}, \mathcal{T}, R, \gamma \rangle$  inside the reformulated MDP  $\langle S, p, \rho, \gamma \rangle$ . The key advantage of this is to apply the more efficient machinery of the reformulation to quickly determine good approximate solutions to the standard MDP. Furthermore, it establishes a theoretical basis to relate the two formulations. Here, we present an analogous embedding for the POMDP. Proofs are deferred to Section A for conciseness.
Suppose an initial belief  $b_{0} \in \mathcal{B}$  is given and let  $\mathcal{R}_{b_0}$  be the set of all reachable beliefs from  $b_{0}$ . Let  $\mathbb{B}(\mathcal{R}_{b_0})$  denote the set of all bounded functions from  $\mathcal{R}_{b_0}$  to  $\mathbb{R}$ . Consider the family of Reference-Based POMDPs with purely belief-dependent rewards  $\rho \in \mathbb{B}(\mathcal{R}_{b_0})$ .
$$
\{\langle \mathcal {S}, \mathcal {A}, \mathcal {O}, \mathcal {Z}, \mathcal {T}, \rho , \gamma , \bar {U} \rangle \} _ {\rho \in \mathbb {B} \left(\mathcal {R} _ {b _ {0}}\right), \bar {U} \in \mathcal {U}} \tag {16}
$$
and a standard POMDP of the form
$$
\langle \mathcal {S}, \mathcal {A}, \mathcal {O}, \mathcal {Z}, \mathcal {T}, R, \gamma \rangle . \tag {17}
$$
For fixed  $(\rho, \bar{U}) \in \mathbb{B}(\mathcal{R}_{b_0}) \times \mathcal{U}$ , their respective Bellman equations are:
$$
\begin{array}{l} \mathcal {V} (b) = \sup  _ {U (\cdot , \cdot | b) \in \mathscr {U} (b)} \left(\rho (b) - \operatorname {K L} \left(U (\cdot , \cdot | b) \| \bar {U} (\cdot , \cdot | b)\right) \right. \\ + \gamma \sum_ {a, o} U (a, o \mid b) \mathcal {V} (\tau (b, a, o)) \bigg) \tag {18} \\ \end{array}
$$
$$
V (b) = \max  _ {a \in \mathcal {A}} \left[ R (b, a) + \gamma \sum_ {o \in \mathcal {O}} P (o \mid a, b) V (\tau (b, a, o)) \right] \tag {19}
$$
for all  $b\in \mathcal{R}_{b_0}$
Definition 4.1. Let  $\mathbb{B}(\mathcal{R}_{b_0})$  be the set of all bounded functions from  $\mathcal{R}_{b_0}$  to  $\mathbb{R}$ . We say that the pair  $(\rho, \bar{U}) \in \mathbb{B}(\mathcal{R}_{b_0}) \times \mathcal{U}$  is an embedding of the standard POMDP (17) in the Reference-Based POMDP (16) with purely belief-dependent rewards if, for each  $\hat{a} \in \mathcal{A}$  and  $b \in \mathcal{B}$  there exists a  $U^{\hat{a}}(\cdot, \cdot | b) \in \mathcal{U}$  such that:
$$
\sum_ {a \in \mathcal {A}} U ^ {\hat {a}} (a, o \mid b) = P (o \mid \hat {a}, b) \tag {20}
$$
$$
R (b, \hat {a}) = \rho (b) - \mathrm {K L} \left(U ^ {\hat {a}} (\cdot , \cdot | b) \| \bar {U} (\cdot , \cdot | b)\right). \tag {21}
$$
Proposition 4.1. Suppose  $\rho \in \mathbb{B}(\mathcal{R}_{b_0})$  is such that
$$
\rho (b) = \log \left[ \sum_ {\hat {a} \in \mathcal {A}} e ^ {R (b, \hat {a})} \right] \tag {22}
$$
holds where  $R(b, \hat{a}) \coloneqq \sum_{s \in S} R(s, \hat{a}) b(s)$ . Then, if
$$
\bar {U} (a, o \mid b) := P (o \mid a, b) \frac {e ^ {R (b , a)}}{\sum_ {\hat {a} \in \mathcal {A}} e ^ {R (b , \hat {a})}} \quad \forall b \in \mathcal {B} \tag {23}
$$
the pair  $(\rho, \bar{U}) \in \mathbb{B}(\mathcal{R}_{b_0}) \times \mathcal{U}$  is an embedding of the standard POMDP in the Reference-Based POMDP with purely belief-dependent rewards.
5 An Online Planning Algorithm
We have developed a preliminary on-line solver REFSOLVER for a Reference-Based POMDP to demonstrate some of the advantages of the formulation presented in Section 3. Pseudo code for an implemented algorithm where the reference policy is constructed using a fully-observed policy  $\pi^{\mathrm{FO}}$  is presented in the Appendix Section A.6 Algorithm 1. Note that although our description of REFSOLVER is very specific in terms of the reference policy and reference stochastic action, they can readily be replaced with other types of policies and induced reference stochastic actions.