Planning in Continuous State Spaces with Parametric POMDPs

This paper presents a novel approach to efficiently applying the POMDP formulation to problems with continuous states, typical in robotics. Continuous distributions over state-space are stored in parametric form, using a vector of sufficient statistics. The value function over continuous sufficient-statistic space is represented using a grid-based approximation, and Dynamic Programming is used to estimate this value function. The algorithm is applied to a simulated robot navigation problem. Traditional MDP-based approaches to planning are well-studied and effective in solving problems where perfect knowledge of the agent’s state is available. Unfortunately, they are less effective in problems where the agent’s state is uncertain, a condition which prevails in many real-world problems. Partially-Observable Markov Decision Processes (POMDPs) are promising as a method for dealing with this uncertainty. Instead of considering the agent’s state to be perfectly known, POMDPs operate on the agent’s belief, which is a probabilistic distribution over possible states of the world. The POMDP is essentially transformed into an MDP, using the belief distribution as the known state. POMDPs are often cited as potential solutions for robot navigation problems, with the motivation that localization is imperfect and MDP-based approaches do not account for this uncertainty. The POMDP solution explicitly models the robot’s position uncertainty, making decisions based on the probabilistic distribution over pose space. This naturally imparts the useful property that the robot will trade off actions that move the robot towards its goal with actions that reduce the robot’s uncertainty, in a principled way. POMDP solution methods can be divided into two approaches: policy-based methods and value-based methods [1]. This paper deals with the latter. The majority of mainstream value-based POMDP research has focussed on the discrete case, requiring that the state space be a finite set of cells. Robot navigation, however, is a fundamentally continuous problem that is poorly represented in the discrete domain unless the discritization is sufficiently fine. POMDP solution methods have problems with fine discretizations because the computational complexity increases rapidly with the size of the state space. This paper presents an approach to solving robot-navigation POMDP problems in a continuous statespace. By constraining distributions over state space to a parametric family, points in the infinite-dimensional continuous belief space can be represented by finite vectors of sufficient statistics. For an appropriate choice of parametric family and MDP model, the parametric form of beliefs is closed under the transition and observation functions. Under these conditions value iteration can be performed efficiently, directly in the space of sufficient statistics. Since the value function is not likely to be piecewise-linear and convex (PWLC) in sufficient-statistic space, a grid-based approximation is used to solve the POMDP. Section 2 discusses related approaches, and Section 3 formulates the DP equations on which the POMDP solution is founded. Section 4 provides a brief overview of POMDP representations based on