Hierarchical Solution of Large Markov Decision Processes

Our goal is to solve a large class of very large Markov decision processes (MDPs), necessarily sacrificing optimality for feasibility. It is widely believed that hierarchical decomposition is the key to solving very large planning problems. Several approaches have shown that, given an appropriate hierarchical decomposition in advance, it can indeed provide considerable speed-up in solving an MDP (Parr 1998; Hauskrecht et al. 1998; Lane and Kaelbling 2002) or doing reinforcement learning (Parr and Russell 1997; Sutton, Precup, and Singh 1999; Dietterich 1998). Finding an appropriate hierarchy has proved challenging, however. There are recent approaches to learning hierarchies (McGovern and Barto 2001; Simsek, Wolfe, and Barto 2005; Mehta et al. 2008); they operate over a relatively long timescale as the agent has to learn the world dynamics in the process, making this problem substantially more difficult. The expectation is that the work associated with learning the hierarchy will pay off over the course of solving several related problems in the same or similar domains. Our goal in this work is somewhat different. We would like to solve a single large MDP very quickly. As the size of the MDP increases, exact solution becomes intractable, so we expect only to find an approximate solution. If we are to use hierarchy to solve this problem, we must be able to construct and solve the hierarchical model in less time than it would have taken to simply solve the original flat model. Bakker et al. (Bakker, Zivkovic, and Krose 2005) presented an approach to this problem, which works well on two-dimensional navigation problems but does not seem to generalize well to other types of domains, sometimes failing to find any strategy, even a suboptimal one, for an achievable goal. Jonsson and Barto’s VISA algorithm (Jonsson and Barto 2006) addresses the problem by taking as input a factored MDP model described as a DBN, and doing causal analysis to find a good hierarchical decomposition. The work of Maggioni and Mahadevan (Mahadevan 2008) finds a multi-scale basis for representing value functions in MDPs and other problems; this basis can serve as an effective representation for solving the MDP efficiently, but is not hierarchical in the same sense as the other methods discussed here. Typical approaches to solving hierarchical MDPs work bottom-up. But the problem is that because it is not yet known what high-level actions will be selected, the lowlevel problems must be solved many times with many different possible high-level objectives. We take advantage of a quick bottom-up pass based on a deterministic approximation of expected costs to move from one state to another, and use that to derive a policy from the top down, which avoids solving low-level MDPs for multiple objectives. The resulting policy may be suboptimal but it is guaranteed to reach a goal state in any problem in which it is reachable under the optimal policy. We begin by describing our conception of a hierarchical model of an MDP and how we can create and solve this model for enumerated-states MDPs. We will then briefly discuss our current work on factored MDPs.