Decision-Theoretic Planning with a Possible Models Action Formalism

This paper extends classical action formalisms so as to account for uncertainty about the state of the world and about the effects of actions. We show how this formalism can be combined with decision theory in order to guide the search for reaction plans and to estimate their quality, thereby providing a basis for anytime ~ planning under uncertainty. This paper deals with the combination of principles from the three following topics, the perspective being the sound engineering of planners coping with uncertainty and time pressure. Action formalisms. In classical planning, action formalisms play the crucial role of providing a symbolic and concise representation of the underlying huge state space. Their ultimate goal is to specify the reasoning about prerequisites and consequences of actions that an ideally rational agent should perform. For sake of efficiency, most of the action formalisms used for planning (which are mostly based on STRIPS) assume that the agent has prefect knowledge about the current state of the world and about the effects of actions. More powerful action formalisms exist, which enable reasoning under uncertainty, e.g., [1], [2]. However, it is not clear how to turn them into efficient planners. In particular the requirements of correctness and completeness, with respect to the underlying formalism, of classical plans/planners turn out to be overly strong, and conditional planning creates additional complexity. Reaction plans provide a way of coping with uncertainty at execution time and ensure real-time behavior. They encode goal-directed behavior not only for single planning problem but for whole parts of the problem domain. Reaction plans are suitable for execution in undeterministic domains, but on the other hand, they are mostly generated by planners using restricted action formalisms and/or assuming that the world is deterministic [12]. It remains then difficult to explicitly reason under uncertainty and to decide which reactions a system should rationally cache. Decision theory offers planning a normative model for reasoning under uncertainty, when the actions only have a probabilistic influence on the world. Numerical utilities enable a richer expression of planning objectives. *who thanks Joachim Hertzberg, William Shoaff and Moti Schneider for their participation to this work. The criterion of correctness of classical plans/planners can be replaced by quality measures, which allows reasoning about tradeoffs involving the gain provided by the various plan outcomes, their probability, and the time available to generate a plan. Algorithms like policy iteration [8] are well-suited to the anytime generation and improvement of reaction plans. Recently, methods based on the policy iteration algorithm have been proposed which do not need to explore the whole state space at once, but restrict the planner’s attention to judiciously chosen larger and larger sets of situations [3]. However, these algorithms do not use symbolic action formalisms, and they must still work on a low-level representation of the state space as a stochastic automaton. Obviously, these fields are complementary, and it is useful to combine them in order to build anytime uncertainty planners. However, due to the non-exhaustive list of problems mentioned in each of the above paragraphs, it is not yet clear what a judicious combination looks like and how much effort is needed to achieve it. Our work has mainly focused on building a model of actions and plans recasting the different fields mentioned into a common framework for planning under uncertainty and time pressure. This paper should be viewed as a summary of the results concerning this work, which are presented in more detail in [13, 14]. The model has the following main features. The action formalism, which blends the possible models approach in [1] with Nilsson’s probabilistic logic [11], copes with the frame and ramification problems. It handles incomplete information about world states, as well as context-dependent and stochastic effects of actions. On this basis, reaction plans are generated, which specify different courses of actions reflecting the domain uncertainty. They are represented as deterministic automata for both planning and execution. Plans do not need to cover all the situations that might be encountered at execution, but achieve an intermediate degree of robustness between triangle tables [6] and universal plans [12]. When taking into account the probabilities provided by the action formalism, plans can be considered as absorbing Markov chains, and results of decision theory can be used to estimate their quality. The quality estimation can be updated during execution, because additional knowledge about the followed path of the plan becomes available, reducing uncertainty about what may happen during the rest of this execution. 265 From: AAAI Technical Report SS-94-06. Compilation copyright © 1994, AAAI (www.aaai.org). All rights reserved. At the planning level, the quality measure and the probability information can be used to cope with the complexity 0f reaction planning by guiding the search for a reaction plan. The probability information is useful with respect to focusing the planner on the situations it is likely to encounter. The quality estimation is useful with respect to pruning undesirable plans, and can also enable meta-reasoning in order know when to stop planning 1. The framework is suitable to the design of anytime algorithms that plan for the most probable evolution of the environment first. Since the quality measure can be updated during execution, one can also design on-line planning algorithms which improve the plan parts that will be of most use in the rest of the execution. Section 1 describes the action formalism, Section 2 presents the representation of reaction plans we use, Section 3 explains how the quality of these plans can be evaluated, building on Markov chains and decision theory, and sketches how anytime algorithms can be developed within this framework. Section 4 concludes. 1 The Action Formalism