Solving POMDPs from Both Sides : Growing Dual Parsimonious Bounds

Partially Observable Markov Decision Processes, or POMDPs, are useful for representing a variety of decision problems; unfortunately, solving for an optimal policy is computational intractable in general. In this paper, we present a set of novel search techniques for solving POMDPs approximately. We build on previous heuristic search and point-based algorithms, but improve upon them in several ways: we introduce an efficient method for approximating the convex hull of the upper bound, we expose embedded finite Markov structure in each bound, and we show how to prune aggressively while still maintaining convergence. The net result is a more targeted growth of the bound representations, leading to lower overall runtime and storage. We synthesize these contributions into a novel algorithm, which we call AAA-POMDP (Appropriately Acronymmed Algorithm). We describe its theoretical properties, including computational efficiency, and examine its performance on on standard benchmark problems from the literature. On these problems, our algorithms exhibit competitive or superior performance when compared to previous methods. Partially Observable Markov Decision Processes (POMDPs) provide an elegant and general framework for representing decision problems with uncertain state and action outcomes (Sondik 1971) (Kaelbling, Littman, and Cassandra 1998), but unfortunately finding optimal (or -optimal) policies is computationally intractable, in general (Papadimitriou and Tsisiklis 1987). Recent advances have relied on using a finite set of belief points and expanding that set during the search (Pineau, Gordon, and Thrun 2003) (Smith and Simmons 2004) (Ji et al. 2007) (Spaan and Vlassis 2005). This approach helps in two ways: it reduces the beliefs considered from a continuous set to a finite set, a more computationally manageable task; and it provides intelligent heuristics to focus the policy search. Historically, POMDP solvers have maintained a set of hyperplanes whose convex hull is a lower bound for the value function. These algorithms add new hyperplanes via Bellman backups and prune unnecessary hyperplanes. The lower-bounding value of a belief is its projection onto the lower-bounding convex hull, which can be computed efficiently by maximizing over its projection onto each hyperCopyright c © 2008, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. plane (a dot product calculation). HSVI (Smith and Simmons 2004) also maintains an upper bound, represented as the convex hull of a set of points; it adds new points by looking ahead via action and observation heuristics. The upper-bounding value of a belief is its projection onto the upper-bounding convex hull. Unfortunately, this value cannot be computed efficiently for non-vertices (it requires solving a linear program), which is especially problematic since nearly all HSVI subroutines rely on computing this value. HSVI2 uses a faster but less tight approximation to the upper-bounding convex hull. POMDP policy iteration (Sondik 1971) (Hansen 1998) (Ji et al. 2007) is an alternative to performing Bellman backups on hyperplanes. It represents a policy as a finite-state controller (FSC); for a given initial controller state and POMDP world state, the FSC induces a Markov chain whose value can be computed efficiently. The algorithm proceeds by evaluating the chains for each controller state and world state, adding new controller states that will result in improved values, and pruning old controller states that have no predecessors. There are two aspects of this algorithm that chafe. Firstly, controller states are not modified once they’ve been added, even if it becomes clear that a different action would be optimal; instead, the algorithm waits until new states are added that allow pruning of the old state. Secondly, the algorithm only prunes controller states when it can guarantee that doing so will not reduce the value function anywhere, even if the the reduction would only occur in a non-reachable portion of the belief space. Our contributions in this paper are threefold: 1. We present a novel method of maintaining the upper bound — to our knowledge, the first efficient, convergent point-based method to do so. By doing so, we address the inefficiency of maintaining an upper bound (the same issue that afflicts HSVI). 2. We show that POMDP policy iteration is in fact solving a finite-state MDP in a restricted policy class. We show how to construct the MDPs, for both upper and lower bounds, and how to solve them efficiently. This technique addresses the first stated problem with policy iteration. 3. We propose an improved pruning criterion that prunes more aggressively but still guarantees convergence, thereby solving the second stated problem with policy it-