Newton, M. A. H. and Levine, J. (2007) Evolving macro-actions for planning. In: Proceedings of the Workshop on AI Planning and Learning

Domain re-engineering through macro-actions (i.e. macros) provides one potential avenue for research into learning for planning. However, most existing work learns macros that are reusable plan fragments and so observable from planner behaviours online or plan characteristics offline. Also, there are learning methods that learn macros from domain analysis. Nevertheless, most of these methods explore restricted macro spaces and exploit specific features of planners or domains. But, the learning examples, especially that are used to acquire previous experiences, might not cover many aspects of the system, or might not always reflect that better choices have been made during the search. Moreover, any specific properties are not likely to be common with many planners or domains. This paper presents an offline evolutionary method that learns macros for arbitrary planners and domains. Our method explores a wider macro space and learns macros that are somehow not observable from the examples. Our method also represents a generalised macro learning framework as it does not discover or utilise any specific structural properties of planners or domains. Introduction Planning has achieved significant progress in recent years from planning competitions. The focus of planning research, however, lies mostly on developing planning technologies while the impact of problem formulation on its solution process remains overlooked. Re-engineering a domain by utilising knowledge acquired for a planner paves the way for further research in this direction. Macro-actions, when represented as additional actions, are one relatively convenient way by which to convey such knowledge and achieve domain enhancements. Within current limits of the Planning Domain Definition Language (PDDL), any knowledge can be conveyed only by additional actions and practically only in STRIPS and FLUENTS subsets of the PDDL. A macro-action, or macro, is a group of actions selected for application at one time like a single action. So, one application of a macro leads to planning of several steps at a time. Macros could represent plan fragments that are found with enormous search effort or are frequently used. Macros could capture local search to find better successor nodes especially when the immediate search neighbourhood is not good. Macros could affect neighbourhood evaluation and Copyright c © 2007, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. thus take the search in a different direction. Consequently, a goal could be reached quickly and problems that are unsolvable could become solvable. When macros are added into a domain as additional actions, no planner modification is needed; also, the reachability of a problem is not affected. But they cause more preprocessing time and incur an extra overhead for the planners adding more branches in the search tree. However, the latter problem is minimised due to the use of a technique called helpful action pruning (Hoffmann & Nebel 2001) by many recent planners. Motivations. Most existing work learns macros from domain analysis or reusable plan fragments (see the section describing related work). Only macros that encapsulate static domain properties can be learnt from domain analysis. The reusable plan fragments can be captured from online planner behaviours or offline plan characteristics. Although our other work in (Newton et al. 2007) showed various macros that improve performance could be learnt successfully from plans using no specific structural knowledge, most existing work somehow exploits particular domain or planner properties. However, during search, the planners often get overwhelmed by the numerous grounded actions available at each node. In these situations, they adopt pruning strategies that help them select next steps very quickly. While making such cursory moves, the chance of not considering all potentially good options remains high. Such hasty decisions thus do not always reflect the better choices available. Furthermore, the example problems, used for macro acquisition, might not cover many aspects of the system as what problems make a good collection is not easily addressed. Nevertheless, most of the existing methods explore macro spaces that are, in effect, restricted. Therefore, the main motivation for this work is to explore the macros that are normally not considered and so not observable. An evolutionary algorithm with its rich operator collection and search direction could help much in this regard. Another motivation of this work is to maintain the generality about planners and domains that our aforementioned work obtained. This is because any specific properties are not likely to be common with a wider range of planners or domains. By solvability we mean, using the original domain, whether the planner can solve the problem within given resource (e.g. time, memory, etc.) limits. Whether the goal of a problem can be attained in a given context is discussed under the term reachability. Contribution. This paper presents an offline evolutionary macro learning method that works with arbitrary planners and domains. Given a planner, a domain, and a number of example problems, our method learns and suggests individual macros that are to be included permanently in the domain as additional actions. The main highlight however is this method can learn macros that are not observable from examples; the reasons could be they are not normally considered and/or the examples do not cover the aspects. This work successfully shows that such macros are also useful and in many cases, are suggested, based on our evaluation, over the observable macros. Another highlight is this method, unlike existing work, does not exploit any specific structural properties of planners or domains. The desirable aspects of our method are due to the use of an evolutionary approach as our learning technique and plans as the source of constituent actions. On one hand, plans invariably reflect successful choices of actions by the planner and so bear inherently the characteristics of the planner or the domain especially that led to the solution. Evolutionary algorithms, on the other hand, are automatic learning methods that not only can capture observable features of a system using no explicitly specific knowledge about it, but also can evolve other inherent features. We have achieved convincing results with several planners and domains. Overview. For the sake of convenience, macros are represented both as sequences of parameterised (and so generalised) constituent actions and as resultant actions built up by regression of the actions in the sequences. While building macros from scratch, or lifting from plans, or evolving from other macros, plans of smaller problems, called seeding problems, are used as the source of constituent actions. A rich collection of genetic operators is used to explore a wider macro space which includes observable macros as well. The evaluation of macros is done against other larger but solvable problems, called ranking problems. These problems are, however, not too large because they are attempted to be solved for every macro. Also, they are not too small because time gains cannot be measured properly for smaller problems. The fitness function is based on a weighted average of the time gains while solving the ranking problems with the macro augmented domain and the original domain. After the learning is accomplished, yet another set of more difficult problems (which might include unsolvable instances), called testing problems, are used to demonstrate the performance of the selected individual macros. The rest of the paper is organised as follows: the next three sections discuss evolutionary algorithms, related work, and an evolutionary approach of learning macros, the last two sections discuss our experiments and conclusion. Evolutionary Algorithms An evolutionary algorithm keeps a population of good individuals, generates a new population from the current one using a given set of genetic operators. It then replaces inferior current individuals by superior new individuals (if any) to get a better current population, which is again used to repeat By problem size or difficulty level we mean, the time required by the given planner to solve the problem with the original domain. the process until the termination condition is met. In a particular problem context, an individual is taken for a solution (macro in our case); which means evolutionary algorithms are an optimisation based multi-point search on the solution space. Moreover, newly generated individuals are other possible solutions in the neighbourhood of the currently kept solutions and a richer collection of genetic operators explore more possible solutions. The requirements of an evolutionary algorithm are a suitable encoding of the individuals, a method to seed the initial population, definitions of the genetic operators to generate new individuals from the current population, and a method to evaluate individuals across the populations. Note that, by satisfying such requirements, the specific knowledge, we give, is actually generic in planning and by no way specific to a planner or a domain. Evolutionary algorithms have produced promising results in learning control knowledge for domains and some success in generating plans. EvoCK (Aler, Borrajo, & Isasi 2001) evolved heuristics generated by HAMLET (Borrajo & Veloso 1997) for PRODIGY4.0 (Veloso et al. 1995) and outperformed both of them. L2Plan (Levine & Humphreys 2003) evolved control knowledge or policies that outperformed hand-coded policies. Spector, using evolutionary algorithms, managed to achieve plans for small problems having a range of initial and goal states (Spector 1994). SINERGY (Muslea 1998) could only solve problems with specific initial and goal states. GenPlan (Westerberg & Levine 2000) showed that evolutionary algorithms can generate