SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to many state-of-the-art on-line POMDP solvers (e.g., [21, 29]), REFSOLVER constructs a search tree of action-observation histories and uses particles to represent the beliefs. The key insight of REFSOLVER, however, is to exploit the additional information afforded by the fully observed policy to limit exploring every possible action at each history node. Instead, as each backup of the transformed value (9) is essentially an expectation under the reference policy, actions are sampled using the reference policy and the backup is estimated via a straightforward Monte Carlo method. This allows sparser sampling and simulations to resemble a depth-first search traversing to a prespecified depth before rolling out the leaf nodes using the fully observed policy. Hence, rather than expanding all the actions as POMCP [21], ABT[12] and DESPOT [29] do, REFSOLVER constructs a deeper tree thereby exploiting more information about the long-term behaviour of the system.
To allow for REFsOLvER to also search forward for actions that are not supported by the fully observed policy, some additional noise  $\rho$  can be added according to (6) so that  $(1 - \alpha)$  can be interpreted as an exploration constant. In our implementation the noise  $\rho$  is a uniform distribution over the actions. The parameter  $(1 - \alpha)$  can thus be interpreted as REFsOLvER's belief that the fully observed policy can be trusted.
We tested REFsOLVER against the implementation of POMCP [21] in the pomdp.py library [4] in two different  $60 \times 60$  long-horizon grid-world navigation problems. As REFsOLVER is written in Python, the implementation of POMCP was de-Cythonised for a fair comparison.
<sup>1</sup> Note that  $\mathcal{R}_{b_0}$  is common to both formulations as their observation and transition models are the same.
We did not compare with DESPOT [29] because the problem scenarios require long horizons and the planning horizon that DESPOT can perform is much lower than POMCP because of its requirement to construct a tree of size  $|\mathcal{A}|^d K^d$  where  $d$  is the depth of the DESPOT (this difficulty has also been observed in [6]). Given the long horizon nature of the problems, one may question comparison with solvers specifically designed for long horizon problems (as summarised in [9]). Most, if not all, of such solvers rely on reducing the effective planning horizon by constructing macro actions or sub-policies. Since REFSOLVER does not change the overall structure of action and observation spaces and transition and observation functions, it is trivial to apply most techniques that reduce the effective planning horizon to Reference-Based POMDP solvers (including REFSOLVER), which in turn will further improve the performance of REFSOLVER too.
All experiments were performed on a desktop computer with an 8 Core Intel Xeon Silver 4110 Processor and 128GB DDR4 RAM.
6.1 Problem Scenarios and Testing Parameters
6.1.1 Navigation1: Long Horizon
The environment is a  $60 \times 60$  static gridworld populated with obstacles, danger zones, and landmarks, as illustrated in Figure 1a. Details of the actions, observations and reward are presented in Section A.7.1.
POMCP's rollout policy and REFSOLVER's fully observed policy are the  $\mathrm{A}^*$  policy for the fully observed and deterministic version of the problem. POMCP was run with an exploration constant of 300 and maximum depth of 180 - this maximum depth is the upper bound for (tree depth + rollout steps). For REFSOLVER, the maximum tree depth was 90, the maximum rollout depth was 180 and  $\alpha = 0.5$ . Both methods were executed for a maximum of 180 steps.
6.1.2 Navigation2: Slightly Perturbed Environment
The environment is a  $60 \times 60$  gridworld populated with obstacles and landmarks. The robot is provided with an initial environment map that was slightly different from the environment used during execution. Details of the scenario (actions, observations, and reward) are presented in Section A.7.2.
For testing purposes, we generated 64 different initial maps with randomly placed obstacles and landmarks. For each initial map, we randomly generate four slight modifications of the initial map by adding obstacles at random. An illustration of an initial environment map and a corresponding environment used during execution are illustrated in Figure 1b and 1c.
Both the POMCP rollout policy and REFsOLVER's fully observed policy are the  $\mathbf{A}^*$  policy for the fully observed and deterministic version of the robot operating in the initial environment map. However, the dynamics, observation, and reward that the robot experiences and receives during execution follow the deformed map, which is different from the initial map.
POMCP was run with an exploration constant of 300 and maximum depth of 60 - this maximum depth is the upper bound for (tree depth + rollout steps). For REFSOLVER, the maximum tree depth was 30, the maximum rollout depth was 60 and  $\alpha = 0.5$ .
6.1.3 Navigation3: 3D Navigation Environment
Navigation3 is a 3D environment where a drone has to fly over uneven terrain. The drone starts at the location illustrated in Figure 1d and needs to reach the red goal region, which is located behind the hill. State and observation spaces are continuous, while actions are discrete. See Appendix Section A.7.3 for details.
The rollout and fully observed policy recommend the action that minimises the Euclidean distance between the current state and goal location with no consideration of obstructing terrain. POMCP was run with an exploration constant of 400 and a maximum depth of 150 (upper bound for tree depth + rollout depth). For REFSOLVER, the maximum tree depth was 150, the maximum rollout depth was 200, and  $\alpha = 0.4$ . Both methods were executed for a maximum of 200 steps.
6.2 Performance Comparison
To evaluate the performance on REFSOLVER, we ran each method on each problem scenario 256 times. For each scenario, each method was given 30 seconds planning time per step. The initial policy generation used by POMCP for rollout and by REFSOLVER for the 2D experiments took an average of 45 minutes for each scenario. We only need to compute this initial policy once for Navigation1 and as many as the number of initial maps for Navigation2.
For Navigation3, the initial policy is generated on-line and takes a small fraction of the planning time per step. The initial policy is derived from a simple shortest path distance policy that assumes the environment is totally empty, allowing it to be computed fast. Note that this way of generating the initial policy substantially alleviates the
computational cost, especially for complex problems. The quality of such an initial policy is of course relatively low, as indicated by the red-line in Figure 2. However, our results indicate that REFSOLVER is able to iteratively deform the initial policy to generate a good strategy for the environment the robot operates in. Note that the trace of the deformed policy indicates that the agent is aware of the potential collision danger quite early, even though it requires more than 100 steps lookahead.
Table 1 compares the success rates and the average total discounted reward (with a  $95\%$  confidence interval) achieved by POMCP and REFSOLVER, respectively. To better understand the performance improvement with respect to the planning time, we also ran Navigation2 with different planning times per step. The results of 256 runs are presented in Table 2. The results indicate that in all instances and for all planning times, the basic implementation of REFSOLVER was able to reach the goal with a significantly greater likelihood and average total discounted reward.
One may be discouraged by the success rate presented. However, all the problem scenarios presented here have a horizon of over 60 steps, which remains a tremendous challenge for many general approximate on-line POMDP solvers today, as indicated by the performance of POMCP.
7.1 MDPs with KL Divergence Minimisation and Reinforcement Learning
The idea of using KL divergence minimisation in fully observable environments goes back to a series of works on Linearly Solvable MDPs² [24, 25, 26, 2] who used it to efficiently approximate solutions to certain classes of (fully-controllable) MDPs. A learning-based algorithm (Z-Learning) was also proposed for problems where the model was not available [24]. Azar et al. [1] extended the approach to general MDPs with stochastic policies and introduced Dynamic Policy Programming which converges asymptotically to the optimal (stochastic) policy of the MDP. The authors also demonstrated that the formulation could be suitably adapted using basis functions to deal with large-scale (continuous) state-action problems. A unifying perspective on this approach was then offered by Rawlik et al. [19] who demonstrated that the stochastic control formulation can be related to approximate inference control in trajectory optimisation [28] and provided further experimental validation for a Z-Learning-like algorithm. Finally, we note that the literature on KL divergence minimisation is closely related to Maximum Entropy RL where some recent theoretical developments [3] have demonstrated robustness to the model (i.e. dynamics and rewards) in fully observable domains.
See Section A.1 for a brief summary of the theory.
As far as the authors are aware, all of the above papers have no parallel in partially-observable environments as of yet. As such, this paper can be seen as first step to extending the above results to such environments.
7.2 Sampling-Based Approximate POMDP Solvers
Methods for finding exact solutions to POMDPs [16] are impractical for realistic robotics problems due to their computational complexity. In recent decades, sampling-based approximate methods have significantly scaled up the capabilities of both online and offline POMDP solvers [18, 11, 21, 29, 12] (see [9] for a comprehensive survey). A major constraint in all state-of-the-art methods is that they rely on a dynamic programming paradigm which necessitates enumeration of actions at every belief node of the search tree. As such, existing solvers cannot effectively deal with the curse of history because they do not have sufficient computational resources to perform a sufficiently long look-ahead.
To deal with this problem, methods that abstract the problem using macro-actions have been proposed [23, 5, 13, 10] where the agent only plans with respect to judiciously chosen sequences of actions, called macro actions, or sub-policies. They reduce the effective planning horizon, but can be sensitive to the class of macro-actions or sub-policy chosen. In contrast, we propose to alleviate the issue via a slight reformulation of the POMDP, which allows numerical computation to infer a close to optimal policy via sparse sampling of both the action and observation spaces. Moreover, existing techniques that reduce the effective planning horizon can also be applied to Reference-Based POMDPs, which will further improve its ability to compute good solutions to POMDPs with long planning horizons.
8 Discussion and Further Work
We have introduced the Reference-Based POMDP, which is a modification of the standard POMDP. This modified formulation enables the problem of computing the optimal policy to be analytically transformed into a problem of computing expectations, which can be approximated without exhaustive enumeration of the actions. We showed that, under mild assumptions, the standard POMDP can be related to a Reference-Based POMDP via an embedding (Proposition 4.1). We also presented a preliminary solver REFSOLVER which solves a Reference-Based POMDP. Initial performance comparisons on 2D and 3D navigation problems indicate that this preliminary method is promising and can be employed to substantially outperform state-of-the-art solvers in long-horizon problems.
By introducing KL minimisation with respect to reference probabilities, Reference-Based POMDPs are likely to be suitable for problems with dynamically changing POMDP models with long planning horizons where one needs to deform the policy being followed without making abrupt changes to the policy. We also see applications of this kind of framework in the context of artificial agents that are guided by a human to make responsible decisions while also judiciously transgressing guidance in the case of significant human error.
One limitation of this paper is that the formulation of a Reference-Based POMDP asserts that arbitrarily "belief-to-belief" transitions can be realised which is a remnant of following the approach set out in Todorov [24] too closely. This assumption can be relaxed by stating the Bellman equation with respect to stochastic policies over actions in a manner similar to Azar et al. [1] rather than abstract belief-to-belief transitions. Such an approach yields very similar analytical results and avoids the need to recover approximate stochastic actions as set out in Section 3.3. We expect to set out these results in a subsequent paper.
The above limitation is just one of the many possible directions for expanding this work. We hope this paper opens new avenues to further scale the capability of POMDP solvers.
This work is done in collaboration with Safran Electronics & Defense Australia Pty Ltd and Safran Group under the ARC Linkage project LP200301612.
See [19, 1] for further commentary.
[20] S. M. Ross. Applied Probability Models with Optimization Applications. Dover, New York, 1970.
[21] D. Silver and J. Veness. Monte-Carlo planning in large POMDPs. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[22] R. D. Smallwood and E. J. Sondik. The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21(5):1071-1088, 1973.
[23] G. Theocharous and L. Kaelbling. Approximate planning in POMDPs with macro-actions. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003.
[24] E. Todorov. Linearly-solvable Markov decision problems. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
[25] E. Todorov. Efficient computation of optimal actions. Proceedings of the National Academy of Sciences, 106(28):11478-11483, 2009.
[26] E. Todorov. Eigenfunction approximation methods for linearly-solvable optimal control problems. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 161-168. IEEE, 2009.
[27] E. Todorov and W. Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005 American Control Conference, 2005, pages 300-306, 2005.
[28] M. Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th International Conference on Machine Learning, 2009.
[29] N. Ye, A. Somani, D. Hsu, and W. S. Lee. DESPOT: Online POMDP planning with regularization. Journal of Artificial Intelligence Research, 58:231-266, 2017.
A.1 Background on Linearly Solvable MDP
Since the Reference-Based POMDP expands the Linearly Solvable (fully observed) MDPs[24, 25, 26] to POMDPs, for completeness, here, we summarise Linearly Solvable MDPs.
A standard infinite horizon MDP is specified by tuple  $\langle S, \mathcal{A}, \mathcal{T}, R, \gamma \rangle$ , where  $\mathcal{S}$  and  $\mathcal{A}$  are the state and action spaces,  $\mathcal{T}(s, a, s')$  is the conditional probability function that specifies the probability the agent arrives at state  $s' \in S$  after performing action  $a \in \mathcal{A}$  at state  $s \in S$ ,  $R$  is the reward function, and  $\gamma \in (0, 1)$  is the discount factor. The solution to an MDP problem is an optimal policy  $\pi^{*}: \mathcal{S} \to \mathcal{A}$  that maximises the value function:
$$
V ^ {*} (s) = \max  _ {a \in \mathcal {A}} \left[ R (s, a) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} \mathcal {T} (s, a, s ^ {\prime}) V ^ {*} \left(s ^ {\prime}\right) \right] \tag {24}
$$
The works in [24, 25, 26] consider a class of MDPs where, the state space  $S$  is finite and for any states  $s, s' \in S$ , there exists a one-step (not necessarily time-homogeneous) transition probability  $p(s' \mid s)$  representing the passive dynamics of the system. They propose a new formulation of MDPs, called Linearly Solvable MDPs, to be specified by  $\langle S, p, r, \gamma \rangle$ , where  $r: S \to \mathbb{R}$  is the reward function. A solution to the Linearly Solvable MDP is a stochastic state-to-state transition probability  $u(\cdot \mid s)$  that maximises:
$$
v (s) = \sup  _ {u (\cdot | s) \in \mathcal {U} _ {p} (s)} \left(r (s) - \operatorname {K L} \left(u (\cdot | s) \| p (\cdot | s)\right) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} u \left(s ^ {\prime} | s\right) v \left(s ^ {\prime}\right)\right) \tag {25}
$$
where  $\mathcal{U}_p(s)$  is the set of admissible controls. An admissible control  $u(\cdot |s)$  is one that prohibits state transitions that are not feasible under the passive dynamics  $p(\cdot |s)$ .
Now, suppose  $w(s) \coloneqq e^{v(s)}$  for any  $s \in S$ , then (25) is equivalent to
$$
w (s) = e ^ {r (s)} \sum_ {s ^ {\prime} \in \mathcal {S}} p \left(s ^ {\prime} \mid s\right) w ^ {\gamma} \left(s ^ {\prime}\right). \tag {26}
$$
Moreover, the solution  $w^{*}$  to the above equation exists and is unique. The optimal stochastic transition to the equation (25) is given by
$$
u ^ {*} (\cdot \mid s) = \frac {p (\cdot \mid s) w ^ {* \gamma} (\cdot)}{D [ w ^ {* \gamma} ] (s)}. \tag {27}
$$
where  $D[w^{*\gamma}](s) \coloneqq \sum_{s' \in S} p(s' | s) w^{*\gamma}(s')$  is a normaliser. Intuitively, one can view  $w^*$  as the desirability score, so that (27) represents distorting the passive dynamics to transition dynamics that favour transitioning to states with higher desirability scores. Of course,  $w^*$  is not known a priori but it can be determined by iterating the Bellman backup operator given by (26). This computation essentially reduces to taking expectations under the reference dynamics, which can be computed faster than searching for the optimal value function in (24) directly.
A standard MDP can be embedded in a linearly solvable MDP. This implies that, for a given standard MDP problem  $\langle S, \mathcal{A}, \mathcal{T}, R, \gamma \rangle$ , one can embed it as an instance of a linearly solvable MDP, use the above efficient machinery to determine the solution to the linearly solvable MDP  $u^{*}(\cdot | s)$ , and then choose the symbolic action  $a^{*} \in \mathcal{A}$  such that  $\mathcal{T}(s' | s, a^{*})$  is as close as possible to  $u^{*}(\cdot | s)$ . Empirical results in [25] indicate that there is a close correspondence between the optimal value of the embedded standard MDP  $\langle S, \mathcal{A}, \mathcal{T}, R, \gamma \rangle$  and the optimal value of the linearly solvable MDP.
Step 1. We first need to verify that a maximiser to the supremum in (9) exists. To this end, define  $\mathcal{W}(b)\coloneqq e^{\mathcal{V}(b)}$  for any  $b\in \mathcal{B}$  and notice that the terms inside the supremum in the RHS of equation
(9) can be rewritten as
$$
\begin{array}{l} \sum_ {a, o} U (a, o \mid b) \left[ R (b, a) - \log \left\{\frac {U (a , o \mid b)}{\bar {U} (a , o \mid b)} \right\} \right. \\ \left. + \gamma \sum_ {a, o} \mathcal {V} (\tau (b, a, o)) \right] \\ = - \sum_ {a, o} U (a, o \mid b) \left[ \log \left\{\frac {U (a , o \mid b)}{\bar {U} (a , o \mid b) e ^ {R (b , a)} \mathcal {W} ^ {\gamma} (\tau (b , a , o))} \right\} \right] \\ = - \sum_ {a, o} U (a, o \mid b) \left[ \log \left\{\frac {U (a , o \mid b) \mathcal {D} [ \mathcal {W} ^ {\gamma} ] (b)}{\bar {U} (a , o \mid b) e ^ {R (b , a)} \mathcal {W} ^ {\gamma} (\tau (b , a , o))} \right\} \right. \\ \left. - \log \left\{\mathcal {D} \left[ \mathcal {W} ^ {\gamma} \right] (b) \right\} \right] \\ = - \operatorname {K L} \left(U (\cdot , \cdot | b) \| \frac {\bar {U} (\cdot , \cdot | b) e ^ {R (b , a)} \mathcal {W} ^ {\gamma} (\tau (b , a , o))}{\mathcal {D} [ \mathcal {W} ^ {\gamma} ] (b) ]}\right) \\ + \log \left\{\mathcal {D} \left[ \mathcal {W} ^ {\gamma} \right] (b) \right\} \tag {28} \\ \end{array}
$$
where  $\mathcal{D}[\mathcal{W}^{\gamma}](b) := \sum_{a,o} \bar{U}(a,o|b)e^{R(b,a)}\mathcal{W}^{\gamma}\big(\tau(b,a,o)\big)$  is a normalising factor. Only the KL divergence term in the last line above depends on  $U$ . We know that the KL divergence is minimised when its two component distributions are identical. That is, when
$$
U ^ {*} (a, o \mid b) = \frac {\bar {U} (a , o \mid b) e ^ {R (b , a)} \mathcal {W} ^ {\gamma} (\tau (b , a , o))}{\mathcal {D} [ \mathcal {W} ^ {\gamma} ] (b)}. \tag {29}
$$
It is clear that  $U^{*}$  belongs to the space  $\mathcal{U}(b)$  since  $\bar{U}(a, o|b) = 0$  implies that  $U^{*}(a, o|b) = 0$  too. Therefore, we conclude that the supremum is attained and that  $U^{*}$  is the maximiser.
Step 2. Now, we can essentially repeat the classical argument from Ross [20] (see e.g. Theorem 6.5). Namely, let  $\Phi : \mathbb{B}(\mathcal{B}) \to \mathbb{B}(\mathcal{B})$  be the Bellman backup operator
$$
\Phi \mathcal {V} (b) := \sup  _ {U \in \mathscr {U} (b)} \left(R (b, U) - \operatorname {K L} (U \| \bar {U}) + \gamma \mathbb {E} _ {U} [ \mathcal {V} (\tau , \cdot , \cdot) ]\right) \quad \forall b \in \mathcal {B} \tag {30}
$$
where, for brevity, we write
$$
\operatorname {K L} (U \| \bar {U}) := \operatorname {K L} \left(U (\cdot , \cdot | b) \| \bar {U} (\cdot , \cdot | b)\right) \tag {31}
$$
$$
\mathbb {E} _ {U} \left[ \mathcal {V} (\tau , \cdot , \cdot)\right) := \sum_ {a, o} U (a, o \mid b) \mathcal {V} (\tau (b, a, o)). \tag {32}
$$
We want to show that  $\Phi$  is a contraction. For any  $b\in \mathcal{B}$  and any  $\nu_{1},\nu_{2}\in \mathbb{B}(\mathcal{B})$
$$
\begin{array}{l} (\Phi \mathcal {V} _ {1}) (b) - (\Phi \mathcal {V} _ {2}) (b) \\ = \sup  _ {U \in \mathcal {U} (b)} \left(R (b, U) - \operatorname {K L} (U \| \bar {U}) + \gamma \mathbb {E} _ {U} \left[ \mathcal {V} _ {1} (\tau , \cdot , \cdot) \right]\right) \\ - \sup  _ {\tilde {U} \in \mathcal {U} (b)} \left(R (b, \tilde {U}) - \mathrm {K L} (\tilde {U} \| \bar {U}) + \gamma \mathbb {E} _ {\tilde {U}} \left[ \mathcal {V} _ {2} (\tau , \cdot , \cdot) \right]\right) \\ \leq \left(R (b, U ^ {*}) - \operatorname {K L} \left(U ^ {*} \| \bar {U}\right) + \gamma \mathbb {E} _ {U ^ {*}} \left[ \mathcal {V} _ {1} (\tau , \cdot , \cdot) \right]\right) \\ \left. - \left(R (b, U ^ {*}) - \operatorname {K L} \left(U ^ {*} \| \bar {U}\right) + \gamma \mathbb {E} _ {U ^ {*}} \left[ \mathcal {V} _ {2} (\tau , \cdot , \cdot)\right) \right]\right) \\ = \gamma \sum_ {a, o} U ^ {*} (a, o \mid b) \left[ \mathcal {V} _ {1} (\tau (b, a, o)) - \mathcal {V} _ {2} (\tau (b, a, o)) \right] \\ \leq \gamma \| \mathcal {V} _ {1} - \mathcal {V} _ {2} \| _ {\infty} \tag {33} \\ \end{array}
$$
where  $U^{*}$  is the maximiser of
$$
R (b, U) - \operatorname {K L} (U \| \bar {U}) + \gamma \mathbb {E} _ {U} \left[ \mathcal {V} _ {1} (\tau , \cdot , \cdot)\right). \tag {34}
$$
Reversing the roles of  $\mathcal{V}_1$  and  $\mathcal{V}_2$  and using the fact that  $b\in \mathcal{B}$  is arbitrary, we conclude that
$$
\left\| \Phi \mathcal {V} _ {1} - \Phi \mathcal {V} _ {2} \right\| _ {\infty} \leq \gamma \left\| \mathcal {V} _ {1} - \mathcal {V} _ {2} \right\| _ {\infty}. \tag {35}
$$
Since we assumed that  $\gamma \in (0,1)$ , we conclude that  $\Phi$  is a contraction.
A.3 Proof of Theorem 3.1
Repeating the argument in Step 1 of A.2, we see that the Bellman equation (9) reduces to
$$
\mathcal {V} (b) = \log [ \mathcal {D} [ w ^ {\gamma} ] (b) ] = \log [ \sum_ {a, o} \bar {U} (a, o \mid b) e ^ {R (b, a)} \mathcal {W} ^ {\gamma} (\tau (b, a, o)) ] \tag {36}
$$