strong planning under partial observability rarely planning domains are fully observable for this reason the ability to deal with partial observability is one of the most important challenges in planning in this paper we tackle the problem of strong planning under partial observability in nondeterministic domains find a conditional plan that will result in a successful state regardless of multiple initial states nondeterministic action effects and partial observability we make the following contributions first we formally define the problem of strong planning within a general framework for modeling partially observable planning domains second we propose an effective planning algorithm based on and or search in the space of beliefs we prove that our algorithm always terminates and is correct and complete in order to achieve additional effectiveness we leverage on a symbolic bdd based representation for the domain and propose several search strategies we provide a thorough experimental evaluation of our approach based on a wide selection of benchmarks we compare the performance of the proposed search strategies and identify a uniform winner that combines heuristic distance measures with mechanisms that reduce runtime uncertainty then we compare our planner mbp with other state of the art systems mbp is able to outperform its competitor systems often by orders of magnitude

complexity of stability and controllability of elementary hybrid systems in this paper we consider simple classes of nonlinear systems and prove that basic questions related to their stability and controllability are either undecidable or computationally intractable np hard as a special case we consider a class of hybrid systems in which the state space is partitioned into two halfspaces and the dynamics in each halfspace correspond to a different linear system

thompson sampling for stochastic control the continuous parameter case recently thompson sampling has been shown to achieve good theoretical performance guarantees for stochastic control problems with parameter uncertainty when the state control and parameter spaces are all finite much less is known however about the performance of thompson sampling when applied to continuous or more general spaces which constitutes an important class of problems in practice in this paper we study thompson sampling when applied to a broad class of average cost stochastic control problems where the state control and parameter spaces are all general measurable spaces the main contributions of our paper are establishing theoretical performance guarantees for thompson sampling as measured by first expected posterior sampling error and second average per period regret

partial information state based optimization of partially observable markov decision processes and the separation principle we propose a partial information state based approach to the optimization of the long run average performance in a partially observable markov decision process pomdp in this approach the information history is summarized at least partially by a or a few statistic s not necessary sufficient called a partial information state and actions depend on the partial information state rather than system states we first propose the single policy based comparison principle under which we derive an hjb type of optimality equation and policy iteration for the optimal policy in the partial information state based policy space we then introduce the q sufficient statistics and show that if the partial information state is q sufficient then the optimal policy in the partial information state based policy space is optimal in the space of all feasible information state based policies we show that with some further conditions the well known separation principle holds the results are obtained by applying the direct comparison based approach initially developed for discrete event dynamic systems

exploiting symmetries for single and multi agent partially observable stochastic domains while partially observable markov decision processes pomdps and their multi agent extension partially observable stochastic games posgs provide a natural and systematic approach to modeling sequential decision making problems under uncertainty the computational complexity with which the solutions are computed is known to be prohibitively expensive in this paper we show how such high computational resource requirements can be alleviated through the use of symmetries present in the problem the problem of finding the symmetries can be cast as a graph automorphism ga problem on a graphical representation of the problem we demonstrate how such symmetries can be exploited in order to speed up the solution computation and provide computational complexity results

constrained discounted markov decision processes and hamiltonian cycles this paper establishes new links between stochastic and discrete optimization we consider the following three problems for discrete time markov decision processes with finite states and action sets i find an optimal deterministic policy for a discounted problem with constraints ii find an optimal stationary policy for a weighted discounted problem with constraints iii find an optimal deterministic policy for a weighted discounted problem with constraints we formulate mathematical programs for problems i iii and show that the hamiltonian cycle problem is a special case of each of these problems therefore problems i iii are np hard we also provide new mathematical programming formulations for the hamiltonian cycle and traveling salesman problems

a strongly polynomial algorithm for controlled queues we consider the problem of computing optimal policies of finite state finite action markov decision processes mdps a reduction to a continuum of constrained mdps cmdps is presented such that the optimal policies for these cmdps constitute a path in a graph defined over the deterministic policies this path contains in particular an optimal policy of the original mdp we present an algorithm based on this new approach that finds this path and thus an optimal policy in the general case this path might be exponentially long in the number of states and actions we prove that the length of this path is polynomial if the mdp satisfies a coupling property thus we obtain a strongly polynomial algorithm for mdp s that satisfies the coupling property we prove that discrete time versions of controlled m m 1 queues induce mdp s that satisfy the coupling property the only previously known polynomial algorithm for controlled m m 1 queues in the expected average cost model is based on linear programming and is not known to be strongly polynomial our algorithm works both for the discounted and expected average cost models and the running time does not depend on the discount factor

temporal constraint networks abstract this paper extends network based methods of constraint satisfaction to include continuous variables thus providing a framework for processing temporal constraints in this framework called temporal constraint satisfaction problem tcsp variables represent time points and temporal information is represented by a set of unary and binary constraints each specifying a set of permitted intervals the unique feature of this framework lies in permitting the processing of metric information namely assessments of time differences between events we present algorithms for performing the following reasoning tasks finding all feasible times that a given event can occur finding all possible relationships between two given events and generating one or more scenarios consistent with the information provided we distinguish between simple temporal problems stps and general temporal problems the former admitting at most one interval constraint on any pair of time points we show that the stp which subsumes the major part of vilain and kautz s point algebra can be solved in polynomial time for general tcsps we present a decomposition scheme that performs the three reasoning tasks considered and introduce a variety of techniques for improving its efficiency we also study the applicability of path consistency algorithms as preprocessing of temporal problems demonstrate their termination and bound their complexities

an optimal one way multigrid algorithm for discrete time stochastic control the numerical solution of discrete time stationary infinite horizon discounted stochastic control problems is considered for the case where the state space is continuous and the problem is to be solved approximately within a desired accuracy after a discussion of problem discretization the authors introduce a multigrid version of the successive approximation algorithm that proceeds one way from coarse to fine grids and analyze its computational requirements as a function of the desired accuracy and of the discount factor they also study the effects of a certain mixing ergodicity condition on the algorithm s performance it is shown that the one way multigrid algorithm improves upon the complexity of its single grid variant and is in a certain sense optimal

planning as constraint satisfaction solving the planning graph by compiling it into csp the idea of synthesizing bounded length plans by compiling planning problems into a combinatorial substrate and solving the resulting encodings has become quite popular in recent years most work to date has however concentrated on compilation to satisfiability sat theories and integer linear programming ilp in this paper we will show that csp is a better substrate for the compilation approach compared to both sat and ilp we describe gp csp a system that does planning by automatically converting graphplan s planning graph into a csp encoding and solving it using standard csp solvers our comprehensive empirical evaluation of gp csp demonstrates that it is superior to both the blackbox system which compiles planning graphs into sat encodings and an ilp based planner in a wide range of planning domains our results show that csp encodings outperform sat encodings in terms of both space and time requirements in various problems the space reduction is particularly important as it makes gp csp less susceptible to the memory blow up associated with sat compilation methods the paper also discusses various techniques in setting up the csp encodings planning specific improvements to csp solvers and strategies for variable and value selection heuristics for solving the csp encodings of different types of planning problems copyright 2001 elsevier science b v

on probabilistic timed automata we propose a model of probabilistic timed automaton which substitutes for the nondeterminism of an ordinary timed automaton a new one directly drawn from markov decision processes by means of actions which provide a probabilistic distribution over transitions using buchi acceptance conditions timed automata can refer timing properties as during every open time interval of length 1 at least one message is delivered a policy is a mechanism which solves the non determinism by choosing for each finite run an action and the time moment of the next transition step implied by this action we prove that given a probabilistic timed automaton a there exists a markov memoryless policy which maximizes the probability p of the set of accepting runs realized by this policy this policy as well as the maximal value of p are computable in polytime in the size of the region automaton of a this result provides an algorithm of model checking for properties like there is a policy which realizes a correct behavior of the system with probability at least p

on the logic of causal explanation abstract the mccain turner semantics of causal rules is based on a fixpoint construction similar to the one found in the definition of default logic in the special case when the heads of the rules are literals it can be equivalently expressed by a translation from sets of rules into sets of propositional formulas we define a translation from causal logic into classical logic that characterizes the semantics of arbitrary causal rules without any restrictions on their syntactic form this translation suggests a way to extend the mccain turner logic to nonpropositional causal theories

planning graph as the basis for deriving heuristics for plan synthesis by state space and csp search most recent strides in scaling up planning have centered around two competing themesdisjunctive planners exemplified by graphplan and heuristic state search planners exemplified by unpop hsp and hsp r in this paper we present a novel approach for successfully harnessing the advantages of the two competing paradigms to develop planners that are significantly more powerful than either of the approaches specifically we show that the polynomial time planning graph structure that the graphplan builds provides a rich substrate for deriving a family of highly effective heuristics for guiding state space search as well as csp style search the main leverage provided by the planning graph structure is a systematic and graded way to take subgoal interactions into account in designing state space heuristics for state space search we develop several families of heuristics some aimed at search speed and others at optimality of solutions and analyze many approaches for improving the cost quality tradeoffs offered by these heuristics our normalized empirical comparisons show that our heuristics handily outperform the existing state space heuristics for csp style search we describe a novel way of using the planning graph structure to derive highly effective variable and value ordering heuristics we show that these heuristics can be used to improve graphplan s own backward search significantly to demonstrate the effectiveness of our approach vis a vis the state of the art in plan synthesis we present altalt a planner literally cobbled together from the implementations of graphplan and state search style planners using our theory we evaluate altalt on the suite of problems used in the recent aips 2000 planning competition the results place altalt in the top tier of the competition plannersoutperforming both graphplan based and heuristic search based planners

empirical dynamic programming we propose empirical dynamic programming algorithms for markov decision processes in these algorithms the exact expectation in the bellman operator in classical value iteration is replaced by an empirical estimate to get empirical value iteration evi policy evaluation and policy improvement in classical policy iteration are also replaced by simulation to get empirical policy iteration epi thus these empirical dynamic programming algorithms involve iteration of a random operator the empirical bellman operator we introduce notions of probabilistic fixed points for such random monotone operators we develop a stochastic dominance framework for convergence analysis of such operators we then use this to give sample complexity bounds for both evi and epi we then provide various variations and extensions to asynchronous empirical dynamic programming the minimax empirical dynamic program and show how this can also be used to solve the dynamic newsvendor problem preliminary experimental results

l algebre de hopf et le groupe de galois motiviques d un corps de caracteristique nulle ii this is the second article of a series of two aiming at constructing and studying motivic galois groups in the context of triangulated motives these motivic galois groups were constructed in the first article and their algebras of regular functions were described concretely in terms of differential forms or algebraic cycles in the present article we gather important complements to the first one in the first part of this article we describe the link between the motivic galois group and the usual galois group of a subfield of in the second part we develop the basis of a theory of ramification for the motivic galois groups by constructing motivic versions of the decomposition and inertia groups associated to a geometric valuation we also introduce the relative motivic galois group of an extension k k which measures the difference between the motivic galois groups of k and k the latter is more accessible than its absolute counterpart in effect we show that it is a quotient of the pro algebraic completion of the topological fundamental pro group of a the pro variety homk k at least when the extension k k is of finite type

probabilistic planning with clear preferences on missing information for many real world problems environments at the time of planning are only partially known for example robots often have to navigate partially known terrains planes often have to be scheduled under changing weather conditions and car route finders often have to figure out paths with only partial knowledge of traffic congestions while general decision theoretic planning that takes into account the uncertainty about the environment is hard to scale to large problems many such problems exhibit a special property one can clearly identify beforehand the best called clearly preferred values for the variables that represent the unknowns in the environment for example in the robot navigation problem it is always preferred to find out that an initially unknown location is traversable rather than not in the plane scheduling problem it is always preferred for the weather to remain a good flying weather and in route finding problem it is always preferred for the road of interest to be clear of traffic it turns out that the existence of the clear preferences can be used to construct an efficient planner called ppcp probabilistic planning with clear preferences that solves these planning problems by running a series of deterministic low dimensional a like searches in this paper we formally define the notion of clear preferences on missing information present the ppcp algorithm together with its extensive theoretical analysis describe several useful extensions and optimizations of the algorithm and demonstrate the usefulness of ppcp on several applications in robotics the theoretical analysis shows that once converged the plan returned by ppcp is guaranteed to be optimal under certain conditions the experimental analysis shows that running a series of fast low dimensional searches turns out to be much faster than solving the full problem at once since memory requirements are much lower and deterministic searches are orders of magnitude faster than probabilistic planning

weak strong and strong cyclic planning via symbolic model checking planning in nondeterministic domains yields both conceptual and practical difficulties from the conceptual point of view different notions of planning problems can be devised for instance a plan might either guarantee goal achievement or just have some chances of success from the practical point of view the problem is to devise algorithms that can effectively deal with large state spaces in this paper we tackle planning in nondeterministic domains by addressing conceptual and practical problems we formally characterize different planning problems where solutions have a chance of success weak planning are guaranteed to achieve the goal strong planning or achieve the goal with iterative trial and error strategies strong cyclic planning in strong cyclic planning all the executions associated with the solution plan always have a possibility of terminating and when they do they are guaranteed to achieve the goal we present planning algorithms for these problem classes and prove that they are correct and complete we implement the algorithms in the mbp planner by using symbolic model checking techniques we show that our approach is practical with an extensive experimental evaluation mbp compares positively with state of the art planners both in terms of expressiveness and in terms of performance

robust control of partially observable failing systems this paper is concerned with optimal maintenance decision making in the presence of model misspecification specifically we are interested in the situation where the decision maker fears that a nominal bayesian model may be miss specified or unrealistic and would like to find policies that work well even when the underlying model is flawed to this end we formulate a robust dynamic optimization model for condition based maintenance in which the decision maker explicitly accounts for distrust in the nominal bayesian model by solving a worst case problem against a second agent nature who has the ability to alter the underlying model distributions in an adversarial manner the primary focus of our analysis is on establishing structural properties and insights that hold in the face of model miss specification in particular we prove i an explicit characterization of nature s optimal response through an analysis of the robust dynamic programming equation ii convexity results for both the robust va

search algorithms under different kinds of heuristics a comparative study three heuristic search algorithms called algorithms a b and c are presented their performance with the admissibility condition relaxed is compared using the following two criteria i number of node expansions and ii cost of solution found first a general comparison is made in this process some variations and extensions of c are also considered subsequently two types of heuristic estimates called proper and path dependent are defined and the algorithms are reexamined it is shown that on the whole a nilsson s algorithm and b martelli s algorithm are inferior to c which is a slightly modified version of b 7 references

representing action indeterminacy and ramifications abstract we define and study a high level language for describing actions more expressive than the action language a introduced by gelfond and lifschitz the new language ar allows us to describe actions with indirect effects ramifications nondeterministic actions and actions that may be impossible to execute it has symbols for nonpropositional fluents and for the fluents that are exempt from the commonsense law of inertia temporal projection problems specified using the language ar can be represented as nested abnormality theories based on the situation calculus

riemann roch for homotopy invariant k theory and gysin morphisms abstract we prove the riemann roch theorem for homotopy invariant k theory and projective local complete intersection morphisms between finite dimensional noetherian schemes without smoothness assumptions we also prove a new riemann roch theorem for the relative cohomology of a morphism in order to do so we construct and characterize new gysin morphisms for regular immersions between cohomologies represented by spectra examples include homotopy invariant k theory motivic cohomology their arithmetic counterparts real absolute hodge and deligne beilinson cohomology rigid syntomic cohomology mixed weil cohomologies and we use this construction to prove a motivic version of the riemann roch

solving factored mdps using non homogeneous partitions we present an algorithm for aggregating states in solving large mdps represented as factored mdps using search by successive refinement in the space of non homogeneous partitions homogeneity is defined in terms of stochastic bisimulation and reward equivalence within blocks of a partition since homogeneous partitions that define equivalent reduced state space mdps can have a large number of blocks we relax the requirement of homogeneity the algorithm constructs approximate aggregate mdps from non homogeneous partitions solves the aggregate mdps exactly and then uses the resulting value functions as part of a heuristic in refining the current best nonhomogeneous partition we outline the theory motivating the use of this heuristic and present empirical results in addition to investigating more exhaustive local search methods we explore the use of techniques derived from research on discretizing continuous state spaces finally we compare the results from our algorithms which search in the space of non homogeneous partitions with exact and approximate algorithms which represent homogeneous and approximately homogeneous partitions as decision trees or algebraic decision diagrams

on the complexity of planning for agent teams and its implications for single agent planning if the complexity of planning for a single agent is described by some function f of the input how much more difficult is it to plan for a team of n cooperating agents if these agents are completely independent we can simply solve n single agent problems scaling linearly with the number of agents but if all the agents interact tightly we really need to solve a single problem that is n times larger which could be exponentially in n harder to solve is a more general characterization possible to formulate this question precisely we minimally extend the standard strips model to describe multi agent planning problems then we identify two problem parameters that help us answer our question the first parameter is independent of the precise task the multi agent system should plan for and it captures the structure of the possible direct interactions between the agents via the tree width of a graph induced by the team the second parameter is task dependent and it captures the minimal number of interactions by the most interacting agent in the team that is needed to solve the problem we show that multi agent planning problems can be solved in time exponential only in these parameters thus when these parameters are bounded the complexity scales only polynomially in the size of the agent team these results also have direct implications for the single agent case by casting single agent planning tasks as multi agent planning tasks we can devise novel methods for decomposition based planning for single agents we analyze one such method and use the techniques developed to provide some of the strongest tractability results for classical single agent planning to date

partially observable markov decision processes with imprecise parameters this study extends the framework of partially observable markov decision processes pomdps to allow their parameters i e the probability values in the state transition functions and the observation functions to be imprecisely specified it is shown that this extension can reduce the computational costs associated with the solution of these problems first the new framework pomdps with imprecise parameters pomdpips is formulated we consider 1 the interval case in which each parameter is imprecisely specified by an interval that indicates possible values of the parameter and 2 the point set case in which each probability distribution is imprecisely specified by a set of possible distributions second a new optimality criterion for pomdpips is introduced as in pomdps the criterion is to regard a policy i e an action selection rule as optimal if it maximizes the expected total reward the expected total reward however cannot be calculated precisely in pomdpips because of the parameter imprecision instead we estimate the total reward by adopting arbitrary second order beliefs i e beliefs in the imprecisely specified state transition functions and observation functions although there are many possible choices for these second order beliefs we regard a policy as optimal as long as there is at least one of such choices with which the policy maximizes the total reward thus there can be multiple optimal policies for a pomdpip we regard these policies as equally optimal and aim at obtaining one of them by appropriately choosing which second order beliefs to use in estimating the total reward computational costs incurred in obtaining such an optimal policy can be reduced significantly we provide an exact solution algorithm for pomdpips that does this efficiently third the performance of such an optimal policy as well as the computational complexity of the algorithm are analyzed theoretically last empirical studies show that our algorithm quickly obtains satisfactory policies to many pomdpips

partially observable markov decision processes a geometric technique and analysis this paper presents a novel framework for studying partially observable markov decision processes pomdps with finite state action observation sets and discounted rewards the new framework is solely based on future reward vectors associated with future policies which is more parsimonious than the traditional framework based on belief vectors it reveals the connection between the pomdp problem and two computational geometry problems i e finding the vertices of a convex hull and finding the minkowski sum of convex polytopes which can help solve the pomdp problem more efficiently the new framework can clarify some existing algorithms over both finite and infinite horizons and shed new light on them it also facilitates the comparison of pomdps with respect to their degree of observability as a useful structural result

an algorithm for probabilistic planning abstract we define the probabilistic planning problem in terms of a probability distribution over initial world states a boolean combination of propositions representing the goal a probability threshold and actions whose effects depend on the execution time state of the world and on random chance adopting a probabilistic model complicates the definition of plan success instead of demanding a plan that provably achieves the goal we seek plans whose probability of success exceeds the threshold in this paper we present buridan an implemented least commitment planner that solves problems of this form we prove that the algorithm is both sound and complete we then explore buridan s efficiency by contrasting four algorithms for plan evaluation using a combination of analytic methods and empirical experiments we also describe the interplay between generating plans and evaluating them and discuss the role of search control in probabilistic planning

quickest detection in multiple on off processes we consider the quickest detection of idle periods in multiple on off processes at each time only one process can be observed and the observations are random realizations drawn from two different distributions depending on the current state on or off of the chosen process the objective is to catch an idle period in any of the on off processes as quickly as possible subject to a reliability constraint we show that this problem presents a fresh twist to the classic problem of quickest change detection that considers only one stochastic process a bayesian formulation of the problem is developed for both infinite and finite number of processes based on the theory of partially observable markov decision process pomdp while a general pomdp is pspace hard we show that the optimal decision rule has a simple threshold structure for the infinite case for the finite case basic properties of the optimal decision rule are established and a low complexity threshold policy is proposed which converges to the optimal decision rule for the infinite case as the number of processes increases this problem finds applications in spectrum sensing in cognitive radio networks where a secondary user searches for idle channels in the spectrum

admissible heuristic search in and or graphs abstract and or graphs play a major role in heuristic problem solving martelli and montanari 1973 have developed an elegant algorithm called hs for heuristic search in and or graphs which they have used for converting decision tables to programs it has so long been thought that hs yields minimal cost solution graphs only if the heuristic satisfies the so called consistency condition it is shown here that the requirement that the heuristic be consistent can be relaxed to the one that the heuristic be merely admissible this should encourage wider use of hs in applications

optimal nonblocking directed control of discrete event systems for the control of discrete event systems the notion of directed control refines that of supervisory control a directed controller is one that selects at most one controllable event to be enabled at any state without disabling any uncontrollable event which is in fact how a discrete event control is implemented in contrast a supervisory controller computes a maximal allowable set of controllable events at each state leaving undecided exactly which controllable event should be enabled in previous works we developed a framework for the computation of optimal directed controllers and a polynomial synthesis algorithm for acyclic plants in this paper we present a novel synthesis approach for general plants i e plants with or without cycles thus providing a complete solution to the optimal directed control problem the complexity of the approach remains polynomial in the size of plant

on the relations between intelligent backtracking and failure driven explanation based learning in constraint satisfaction and planning abstract the ideas of intelligent backtracking ib and explanation based learning ebl have developed independently in the constraint satisfaction planning machine learning and problem solving communities the variety of approaches developed for ib and ebl in the various communities have hither to been incomparable in this paper i formalize and unify these ideas under the task independent framework of refinement search which can model the search strategies used in both planning and constraint satisfaction problems csps i show that both ib and ebl depend upon the common theory of explanation analysis which involves explaining search failures and regressing them to higher levels of the search tree my comprehensive analysis shows that most of the differences between the csp and planning approaches to ebl and ib revolve around different solutions to a how the failure explanations are computed b how they are contextualized contextualization involves deciding whether or not to keep the flaw description and the description of the violated problem constraints and c how the storage of explanations is managed the differences themselves can be understood in terms of the differences between planning and csp problems as instantiations of refinement search this unified understanding is expected to support a greater cross fertilization of ideas among csp planning and ebl communities

the simplex and policy iteration methods are strongly polynomial for the markov decision problem with a fixed discount rate we prove that the classic policy iteration method howard r a 1960 dynamic programming and markov processes mit cambridge and the original simplex method with the most negative reduced cost pivoting rule of dantzig are strongly polynomial time algorithms for solving the markov decision problem mdp with a fixed discount rate furthermore the computational complexity of the policy iteration and simplex methods is superior to that of the only known strongly polynomial time interior point algorithm ye y 2005 a new complexity result on solving the markov decision problem math oper res 30 3 733 749 for solving this problem the result is surprising because the simplex method with the same pivoting rule was shown to be exponential for solving a general linear programming problem klee v g j minty 1972 how good is the simplex method technical report o shisha ed inequalities iii academic press new york the simplex method with the smallest index pivoting rule was shown to be exponential for solving an mdp regardless of discount rates melekopoglou m a condon 1994 on the complexity of the policy improvement algorithm for markov decision processes informs j comput 6 2 188 192 and the policy iteration method was recently shown to be exponential for solving undiscounted mdps under the average cost criterion we also extend the result to solving mdps with transient substochastic transition matrices whose spectral radii are uniformly below one

conformant plans and beyond principles and complexity conformant planning is used to refer to planning for unobservable problems whose solutions like classical planning are linear sequences of operators called linear plans the term conformant is automatically associated with both the unobservable planning model and with linear plans mainly because the only possible solutions for unobservable problems are linear plans in this paper we show that linear plans are not only meaningful for unobservable problems but also for partially observable problems in such case the execution of a linear plan generates observations from the environment which must be collected by the agent during the execution of the plan and used at the end in order to determine whether the goal had been achieved or not this is the typical case in problems of diagnosis in which all the actions are knowledge gathering actions thus there are substantial differences about linear plans for the case of unobservable or fully observable problems and for the case of partially observable problems while linear plans for the former model must conform with properties in state space linear plans for partially observable problems must conform with properties in belief space this differences surface when the problems are allowed to express epistemic goals and conditions using modal logic and place the plan existence decision problem in different complexity classes linear plans is one extreme point in a discrete spectrum of solution forms for planning problems the other extreme point is contingent plans in which there is a branch point for every possible observation at each time step and thus the number of branch points is not bounded a priori in the middle of the spectrum there are plans with a bounded number of branch points thus linear plans are plans with zero branch points and contingent plans are plans with unbounded number of branch points in this work we lay down foundations and principles for the general treatment of linear plans and plans of bounded branching and provide exact complexity results for novel decision problems we also show that linear plans for partially observable problems are not only of theoretical interest since some challenging real life problems can be dealt with them

structured threshold policies for dynamic sensor scheduling a partially observed markov decision process approach we consider the optimal sensor scheduling problem formulated as a partially observed markov decision process pomdp due to operational constraints at each time instant the scheduler can dynamically select one out of a finite number of sensors and record a noisy measurement of an underlying markov chain the aim is to compute the optimal measurement scheduling policy so as to minimize a cost function comprising of estimation errors and measurement costs the formulation results in a nonstandard pomdp that is nonlinear in the information state we give sufficient conditions on the cost function dynamics of the markov chain and observation probabilities so that the optimal scheduling policy has a threshold structure with respect to a monotone likelihood ratio mlr ordering as a result the computational complexity of implementing the optimal scheduling policy is inexpensive we then present stochastic approximation algorithms for estimating the best linear mlr order threshold policy

sat based planning in complex domains concurrency constraints and nondeterminism planning as satisfiability is a very efficient technique for classical planning i e for planning domains in which both the effects of actions and the initial state are completely specified in this paper we present c sat a sat based procedure capable of dealing with planning domains having incomplete information about the initial state and whose underlying transition system is specified using the highly expressive action language c thus c sat allows for planning in domains involving i actions which can be executed concurrently ii ramification and qualification constraints affecting the effects of actions and iii nondeterminism in the initial state and in the effects of actions we first prove the correctness and the completeness of c sat discuss some optimizations and then we present c plan a system based on c sat c plan works on any c planning problem but some optimizations have not been fully implemented yet nevertheless the experimental analysis shows that sat based approaches to planning with incomplete information are viable at least in the case of problems with a high degree of parallelism

probabilistic ω automata probabilistic automata are variants of nondeterministic automata over infinite words where all choices are resolved by probabilistic distributions acceptance of a run for an infinite input word can be defined using traditional acceptance criteria for automata such as buchi rabin or streett conditions the accepted language of a probabilistic automata is then defined by imposing a constraint on the probability measure of the accepting runs in this paper we study a series of fundamental properties of probabilistic automata with three different language semantics 1 the probable semantics that requires positive acceptance probability 2 the almost sure semantics that requires acceptance with probability 1 and 3 the threshold semantics that relies on an additional parameter i r0 1l that specifies a lower probability bound for the acceptance probability we provide a comparison of probabilistic automata under these three semantics and nondeterministic automata concerning expressiveness and efficiency furthermore we address closure properties under the boolean operators union intersection and complementation and algorithmic aspects such as checking emptiness or language containment

the simplex method is strongly polynomial for deterministic markov decision processes we prove that the simplex method with the highest gain most negative reduced cost pivoting rule converges in strongly polynomial time for deterministic markov decision processes mdps regardless of the discount factor for a deterministic mdp with n states and m actions we prove the simplex method runs in o n3 m2 log2 n iterations if the discount factor is uniform and o n5 m3 log2 n iterations if each action has a distinct discount factor previously the simplex method was known to run in polynomial time only for discounted mdps where the discount was bounded away from 1

point based value iteration for continuous pomdps we propose a novel approach to optimize partially observable markov decisions processes pomdps defined on continuous spaces to date most algorithms for model based pomdps are restricted to discrete states actions and observations but many real world problems such as for instance robot navigation are naturally defined on continuous spaces in this work we demonstrate that the value function for continuous pomdps is convex in the beliefs over continuous state spaces and piecewise linear convex for the particular case of discrete observations and actions but still continuous states we also demonstrate that continuous bellman backups are contracting and isotonic ensuring the monotonic convergence of value iteration algorithms relying on those properties we extend the algorithm originally developed for discrete pomdps to work in continuous state spaces by representing the observation transition and reward models using gaussian mixtures and the beliefs using gaussian mixtures or particle sets with these representations the integrals that appear in the bellman backup can be computed in closed form and therefore the algorithm is computationally feasible finally we further extend to deal with continuous action and observation sets by designing effective sampling approaches

the complexity of decentralized control of markov decision processes we consider decentralized control of markov decision processes and give complexity bounds on the worst case running time for algorithms that find optimal solutions generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described for even two agents the finite horizon problems corresponding to both of these models are hard for nondeterministic exponential time these complexity results illustrate a fundamental difference between centralized and decentralized control of markov decision processes in contrast to the problems involving centralized control the problems we considerprovably do not admit polynomial time algorithms furthermore assuming exp nexp the problems require superexponential time to solve in the worst case

applicability conditions for plans with loops computability results and algorithms the utility of including loops in plans has been long recognized by the planning community loops in a plan help increase both its applicability and the compactness of its representation however progress in finding such plans has been limited largely due to lack of methods for reasoning about the correctness and safety properties of loops of actions we present novel algorithms for determining the applicability and progress made by a general class of loops of actions these methods can be used for directing the search for plans with loops towards greater applicability while guaranteeing termination as well as in post processing of computed plans to precisely characterize their applicability experimental results demonstrate the efficiency of these algorithms we also discuss the factors which can make the problem of determining applicability conditions for plans with loops incomputable

myopic bounds for optimal policy of pomdps an extension of lovejoy s structural results this paper provides a relaxation of the sufficient conditions and an extension of the structural results for partially observed markov decision processes pomdps obtained by lovejoy in 1987 sufficient conditions are provided so that the optimal policy can be upper and lower bounded by judiciously chosen myopic policies these myopic policy bounds are constructed to maximize the volume of belief states where they coincide with the optimal policy numerical examples illustrate these myopic bounds for both continuous and discrete observation sets

cost sensitive feature acquisition and classification there are many sensing challenges for which one must balance the effectiveness of a given measurement with the associated sensing cost for example when performing a diagnosis a doctor must balance the cost and benefit of a given test measurement and the decision to stop sensing stop performing tests must account for the risk to the patient and doctor malpractice for a given diagnosis based on observed data this motivates a cost sensitive classification problem in which the features sensing results are not given a priori the algorithm determines which features to acquire next as well as when to stop sensing and make a classification decision based on previous observations accounting for the costs of various types of errors as well as the rewards of being correct we formally define the cost sensitive classification problem and solve it via a partially observable markov decision process pomdp while the pomdp constitutes an intuitively appealing formulation the intrinsic properties of classification tasks resist application of it to this problem we circumvent the difficulties of the pomdp via a myopic approach with an adaptive stopping criterion linked to the standard pomdp the myopic algorithm is computationally feasible easily handles continuous features and seamlessly avoids repeated actions experiments with several benchmark data sets show that the proposed method yields state of the art performance and importantly our method uses only a small fraction of the features that are generally used in competitive approaches

bayesian sequential detection with phase distributed change time and nonlinear penalty a pomdp lattice programming approach we show that the optimal decision policy for several types of bayesian sequential detection problems has a threshold switching curve structure on the space of posterior distributions this is established by using lattice programming and stochastic orders in a partially observed markov decision process pomdp framework a stochastic gradient algorithm is presented to estimate the optimal linear approximation to this threshold curve we illustrate these results by first considering quickest time detection with phase type distributed change time and a variance stopping penalty then it is proved that the threshold switching curve also arises in several other bayesian decision problems such as quickest transient detection exponential delay risk sensitive penalties stopping time problems in social learning and multi agent scheduling in a changing world using blackwell dominance it is shown that for dynamic decision making problems the optimal decision policy is lower bounded by a myopic policy finally it is shown how the achievable cost of the optimal decision policy varies with change time distribution by imposing a partial order on transition matrices

enhancement schemes for constraint processing backjumping learning and cutset decomposition abstract researchers in the areas of constraint satisfaction problems logic programming and truth maintenance systems have suggested various schemes for enhancing the performance of the backtracking algorithm this paper defines and compares the performance of three such schemes backjumping learning and cycle cutset the backjumping and the cycle cutset methods work best when the constraint graph is sparse while the learning scheme mostly benefits problem instances with dense constraint graphs an integrated strategy is described which utilizes the distinct advantages of each scheme experiments show that in hard problems the average improvement realized by the integrated scheme is 20 25 higher than any of the individual schemes

algebras and modules in monoidal model categories in recent years the theory of structured ring spectra formerly known as a infty and e infty ring spectra has been simplified by the discovery of categories of spectra with strictly associative and commutative smash products now a ring spectrum can simply be defined as a monoid with respect to the smash product in one of these new categories of spectra in this paper we provide a general method for constructing model category structures for categories of ring algebra and module spectra this provides the necessary input for obtaining model categories of symmetric ring spectra functors with smash product gamma rings and diagram ring spectra algebraic examples to which our methods apply include the stable module category over the group algebra of a finite group and unbounded chain complexes over a differential graded algebra 1991 mathematics subject classification primary 55u35 secondary 18d10

a mean variance optimization problem for discounted markov decision processes in this paper we consider a mean variance optimization problem for markov decision processes mdps over the set of deterministic stationary policies different from the usual formulation in mdps we aim to obtain the mean variance optimal policy that minimizes the variance over a set of all policies with a given expected reward for continuous time mdps with the discounted criterion and finite state and action spaces we prove that the mean variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and markov properties then we show that a mean variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations we also address related issues such as a mutual fund theorem and illustrate our results with an example

thompson sampling for stochastic control the finite parameter case in this paper we apply thompson sampling to a class of average reward stochastic control problems with parameter uncertainty specifically we study an average reward stochastic control problem over an infinite horizon in which both the reward and state transition distributions are parameterized by an unknown parameter taking values in a finite space the main result of this paper is a proof showing that thompson sampling achieves a worst case average per period regret of o t 1 which is asymptotically optimal

lao a heuristic search algorithm that finds solutions with loops classic heuristic search algorithms can find solutions that take the form of a simple path a a tree or an acyclic graph ao in this paper we describe a novel generalization of heuristic search called lao that can find solutions with loops we show that lao can be used to solve markov decision problems and that it shares the advantage heuristic search has over dynamic programming for other classes of problems given a start state it can find an optimal solution without evaluating the entire state space 2001 elsevier science b v all rights reserved

optimal eviction policies for stochastic address traces the eviction problem for memory hierarchies is studied for the hidden markov reference model hmrm of the memory trace showing how miss minimization can be naturally formulated in the optimal control setting in addition to the traditional version assuming a buffer of fixed capacity a relaxed version is also considered in which buffer occupancy can vary and its average is constrained resorting to multiobjective optimization viewing occupancy as a cost rather than as a constraint the optimal eviction policy is obtained by composing solutions for the individual addressable items this approach is then specialized to the least recently used stack model lrusm a type of hmrm often considered for traces which includes v 1 parameters where v is the size of the virtual space a gain optimal policy for any target average occupancy is obtained which i is computable in time o v from the model parameters ii is optimal also for the fixed capacity case and iii is characterized in terms of priorities with the name of least profit rate lpr policy an o logc upper bound being c the buffer capacity is derived for the ratio between the expected miss rate of lpr and that of opt the optimal off line policy the upper bound is tightened to o 1 under reasonable constraints on the lrusm parameters using the stack distance framework an algorithm is developed to compute the number of misses incurred by lpr on a given input trace simultaneously for all buffer capacities in time o logv per access finally some results are provided for miss minimization over a finite horizon and over an infinite horizon under bias optimality a criterion more stringent than gain optimality

complexity of finite horizon markov decision process problems controlled stochastic systems occur in science engineering manufacturing social sciences and many other cntexts if the systems is modeled as a markov decision process mdp and will run ad infinitum the optimal control policy can be computed in polynomial time using linear programming the problems considered here assume that the time that the process will run is finite and based on the size of the input there are mny factors that compound the complexity of computing the optimal policy for instance there are many factors that compound the complexity of this computation for instance if the controller does not have complete information about the state of the system or if the system is represented in some very succint manner the optimal policy is provably not computable in time polynomial in the size of the input we analyze the computational complexity of evaluating policies and of determining whether a sufficiently good policy exists for a mdp based on a number of confounding factors including the observability of the system state the succinctness of the representation the type of policy even the number of actions relative to the number of states in almost every case we show that the decision problem is complete for some known complexity class some of these results are familiar from work by papadimitriou and tsitsiklis and others but some such as our pl completeness proofs are surprising we include proofs of completeness for natural problems in the as yet little studied classes np pp

modules over motivic cohomology abstract for fields of characteristic zero we show that the homotopy category of modules over the motivic ring spectrum representing motivic cohomology is equivalent to voevodsky s big category of motives the proof makes use of some highly structured models for motivic stable homotopy theory motivic spanier whitehead duality the homotopy theories of motivic functors and of motivic spaces with transfers as introduced from ground up in this paper working with rational coefficients we extend the equivalence for fields of characteristic zero to all perfect fields by employing the techniques of alterations and homotopy purity in motivic homotopy theory

on the complexity of partially observed markov decision processes abstract in the paper we consider the complexity of constructing optimal policies strategies for some type of partially observed markov decision processes this particular case of the classical problem deals with finite stationary processes and can be represented as constructing optimal strategies to reach target vertices from a starting vertex in a graph with colored vertices and probabilistic deviations from an edge chosen to follow the colors of the visited vertices is the only information available to a strategy the complexity of markov decision in the case of perfect information bijective coloring of vertices is known and briefly surveyed at the beginning of the paper for the unobservable case all the colors are equal we give an improvement of the result of papadimitriou and tsitsiklis namely we show that the problem of constructing even a very weak approximation to an optimal strategy is np hard our main results concern the case of a fixed bound on the multiplicity of coloring that is a case of partially observed processes where some upper bound on the unobservability is supposed we show that the problem of finding an optimal strategy is still np hard but polytime approximations are possible some relations of our results to the max word problem are also indicated

how to schedule measurements of a noisy markov chain in decision making a decision maker records measurements of a finite state markov chain corrupted by noise the goal is to decide when the markov chain hits a specific target state the decision maker can choose from a finite set of sampling intervals to pick the next time to look at the markov chain the aim is to optimize an objective comprising of false alarm delay cost and cumulative measurement sampling cost taking more frequent measurements yields accurate estimates but incurs a higher measurement cost making an erroneous decision too soon incurs a false alarm penalty waiting too long to declare the target state incurs a delay penalty what is the optimal sequential strategy for the decision maker this paper shows that under reasonable conditions the optimal strategy has the following intuitive structure when the bayesian estimate posterior distribution of the markov chain is away from the target state look less frequently while if the posterior is close to the target state look more frequently bounds are derived for the optimal strategy also the achievable optimal cost of the sequential detector as a function of transition dynamics and observation distribution is analyzed the sensitivity of the optimal achievable cost to parameter and strategy variations is bounded in terms of the kullback divergence also structural results are obtained for joint optimal sampling and measurement control active sensing

optimal cost almost sure reachability in pomdps we consider partially observable markov decision processes pomdps with a set of target states and an integer cost associated with every transition the optimization objective we study asks to minimize the expected total cost of reaching a state in the target set while ensuring that the target set is reached almost surely with probability 1 we show that for integer costs approximating the optimal cost is undecidable for positive costs our results are as follows i we establish matching lower and upper bounds for the optimal cost both double exponential in the pomdp state space size ii we show that the problem of approximating the optimal cost is decidable and present approximation algorithms developing on the existing algorithms for pomdps with finite horizon objectives while the worst case running time of our algorithm is double exponential we also present efficient stopping criteria for the algorithm and show experimentally that it performs well in many examples of interest

a new representation and associated algorithms for generalized planning constructing plans that can handle multiple problem instances is a longstanding open problem in ai we present a framework for generalized planning that captures the notion of algorithm like plans and unifies various approaches developed for addressing this problem using this framework and building on the tvla system for static analysis of programs we develop a novel approach for computing generalizations of classical plans by identifying sequences of actions that will make measurable progress when placed in a loop in a wide class of problems that we characterize formally in the paper these methods allow us to find generalized plans with loops for solving problem instances of unbounded sizes and also to determine the correctness and applicability of the computed generalized plans we demonstrate the scope and scalability of the proposed approach on a wide range of planning problems

a generic approach to planning in the presence of incomplete information theory and implementation abstract this paper proposes a generic approach to planning in the presence of incomplete information the approach builds on an abstract notion of a belief state representation along with an associated set of basic operations these operations facilitate the development of a sound and complete transition function for reasoning about effects of actions in the presence of incomplete information and a set of abstract algorithms for planning the paper demonstrates how the abstract definitions and algorithms can be instantiated in three concrete representations minimal dnf minimal cnf and prime implicates resulting in three highly competitive conformant planners dnf cnf and pip the paper relates the notion of a representation to that of ordered binary decision diagrams a well known belief state representation employed by many conformant planners and several target compilation languages that have been presented in the literature the paper also includes an experimental evaluation of the planners dnf cnf and pip and proposes a new set of conformant planning benchmarks that are challenging for state of the art conformant planners

the complexity of markov decision processes we investigate the complexity of the classical problem of optimal policy computation in markov decision processes all three variants of the problem finite horizon infinite horizon discounted and infinite horizon average cost were known to be solvable in polynomial time by dynamic programming finite horizon problems linear programming or successive approximation techniques infinite horizon we show that they are complete for p and therefore most likely cannot be solved by highly parallel algorithms we also show that in contrast the deterministic cases of all three problems can be solved very fast in parallel the version with partially observed states is shown to be pspace complete and thus even less likely to be solved in polynomial time than the np complete problems in fact we show that most likely it is not possible to have an efficient on line implementation involving polynomial time on line computations and memory of an optimal policy even if an arbitrary amount of precomputation is allowed finally the variant of the problem in which there are no observations is shown to be np complete

planning and acting in partially observable stochastic domains in this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains we begin by introducing the theory of markov decision processes mdps and partially observable mdps pomdps we then outline a novel algorithm for solving pomdps off line and show how in some cases a finite memory controller can be extracted from the solution to a pomdp we conclude with a discussion of how our approach relates to previous work the complexity of finding exact solutions to pomdps and of some possibilities for finding approximate solutions

on the complexity of vlsi implementations and graph representations of boolean functions with application to integer multiplication lower bound results on boolean function complexity under two different models are discussed the first is an abstraction of tradeoffs between chip area and speed in very large scale integrated vlsi circuits the second is the ordered binary decision diagram obdd representation used as a data structure for symbolically representing and manipulating boolean functions the lower bounds demonstrate the fundamental limitations of vlsi as an implementation medium and that of the obdd as a data structure it is shown that the same technique used to prove that any vlsi implementation of a single output boolean function has area time complexity at sup 2 omega n sup 2 also proves that any obdd representation of the function has omega c sup n vertices for some c 1 but that the converse is not true an integer multiplier for word size n with outputs numbered 0 least significant through 2n 1 most significant is described for the boolean function representing either output i 1 or output 2n i 1 where 1

on the computability of solomonoff induction and aixi abstract how could we solve the machine learning and the artificial intelligence problem if we had infinite computation solomonoff induction and the reinforcement learning agent aixi are proposed answers to this question both are known to be incomputable we quantify this using the arithmetical hierarchy and prove upper and in most cases corresponding lower bounds for incomputability moreover we show that aixi is not limit computable thus it cannot be approximated using finite computation however there are limit computable e optimal approximations to aixi we also derive computability bounds for knowledge seeking agents and give a limit computable weakly asymptotically optimal reinforcement learning agent

state set branching leveraging bdds for heuristic search in this article we present a framework called state set branching that combines symbolic search based on reduced ordered binary decision diagrams bdds with best first search such as a and greedy best first search the framework relies on an extension of these algorithms from expanding a single state in each iteration to expanding a set of states we prove that it is generally sound and optimal for two a implementations and show how a new bdd technique called branching partitioning can be used to efficiently expand sets of states the framework is general it applies to any heuristic function evaluation function and transition cost function defined over a finite domain moreover branching partitioning applies to both disjunctive and conjunctive transition relation partitioning an extensive experimental evaluation of the two a implementations proves state set branching to be a powerful framework the algorithms outperform the ordinary a algorithm in almost all domains in addition they can improve the complexity of a exponentially and often dominate both a and blind bdd based search by several orders of magnitude moreover they have substantially better performance than bdda the currently most efficient bdd based implementation of a

mixed weil cohomologies abstract we define for a regular scheme s and a given field of characteristic zero k the notion of k linear mixed weil cohomology on smooth s schemes by a simple set of properties mainly nisnevich descent homotopy invariance stability which means that the cohomology of g m behaves correctly and kunneth formula we prove that any mixed weil cohomology defined on smooth s schemes induces a symmetric monoidal realization of some suitable triangulated category of motives over s to the derived category of the field k this implies a finiteness theorem and a poincare duality theorem for such a cohomology with respect to smooth and projective s schemes which can be extended to smooth s schemes when s is the spectrum of a perfect field this formalism also provides a convenient tool to understand the comparison of such cohomology theories

conformant planning via symbolic model checking and heuristic search in this paper we tackle the problem of conformant planning find a sequence of actions that guarantees goal achievement regardless of an uncertain initial condition and of nondeterministic action effects our approach set in the framework of search in the belief space is based on two main intuitions first symbolic model checking techniques binary decision diagrams in particular can be used to represent and expand the search space and to provide an efficient computational platform second driving the search solely with reachability information is not effective and the notion of knowledge has to be taken explicitly into account we make the following contributions first we thoroughly analyze reachability heuristics we formally prove their admissibility we show how they can be implemented by means of symbolic techniques and experimentally evaluate them second we analyze the limitations of reachability heuristics and propose more informed yet admissible heuristics based on a formal notion of knowledge third we present a practical conformant planning algorithm where the search is explicitly based on the notion of target knowledge i e the amount of information that has to be available in order to reach the goal the search alternates between the acquire knowledge mode where actions have the precise purpose of gathering necessary information and the reach goal mode where actions are directed towards the goal finally we provide a thorough experimental comparison between several conformant planners our approach is sometimes able to outperform the competitor systems by orders of magnitude

on the undecidability of probabilistic planning and related stochastic optimization problems automated planning the problem of how an agent achieves a goal given a repertoire of actions is one of the foundational and most widely studied problems in the ai literature the original formulation of the problem makes strong assumptions regarding the agent s knowledge and control over the world namely that its information is complete and correct and that the results of its actions are deterministic and known recent research in planning under uncertainty has endeavored te relax these assumptions providing formal and computation models wherein the agent has incomplete or noisy information about the world and has noisy sensors and effectors this research has mainly taken one of two approaches extend the classical planning paradigm to a semantics that admits uncertainty or adopt another framework for approaching the problem most commonly the markov decision process mdp model this paper presents a complexity analysis of planning under uncertainty it begins with the probabilistic classical planning problem showing that problem to be formally undecidable this fundamental result is then applied to a broad class of stochastic optimization problems in brief any problem statement where the agent a operates over an infinite or indefinite time horizon and b has available only probabilistic information about the system s state undecidability is established for policy existence problems for partially observable infinite horizon markov decision processes under discounted and undiscounted total reward models average reward models and state avoidance models the results also apply to corresponding approximation problems with undiscounted objective functions the paper answers a significant open question raised by papadimitriou and tsitsiklis math oper res 12 3 1987 441 450 about the complexity of infinite horizon pomdps

an efficient algorithm for searching implicit and or graphs with cycles abstract we present an efficient ao like algorithm that handles cyclic graphs without neither unfolding the cycles nor looping through them its top down search strategy is based on mahanti and bagchi s cf j acm 32 1985 28 whereas its bottom up revision process is inspired in chakrabarti s rev artificial intelligence 65 1994 329 however important modifications have been introduced in both algorithms to attain a true integration and gain efficiency proofs of correctness and completeness are included up to our knowledge the resulting algorithm called cfc rev is the most efficient one available for this problem

the maximum flow problem is log space complete for p the space complexity of the maximum flow problem is investigated it is shown that the problem is log space complete for deterministic polynomial time thus the maximum flow problem probably has no algorithm which needs only o logk n storage space for any constant k another consequence is that there is probably no fast parallel algorithm for the maximum flow problem

understanding planning with incomplete information and sensing pks is the framework for planning with incomplete information and sensing recently introduced by bacchus and petrick proc kr 98 pp 432 443 the fact that pks generalizes strips to domains with incomplete information and sensing opens up the possibility of proposing it as a reference for comparisons with other formalisms that approach the problem from different perspectives to this end we first provide a formal semantics for pks then analyze and extend it the formal definition of the extended pks entails the identification of a number of properties of this planning framework in particular we prove that for any finite instance of the pks planning problem the reachable states are finite on the basis of this result we propose an improved planning algorithm that is not only sound as the one proposed by petrick and bacchus proc aips 02 pp 212 221 but also complete we extend pks to include conditional plans with cycles and introduce the distinction between different classes of solutions strong strong cyclic weak acyclic and weak cyclic in contrast with current belief we prove that some weak acyclic solutions are more likely to succeed for a limited execution than some strong cyclic solutions revealing the lack of a method for judging the quality of different solutions finally we introduce a quality measure for solutions of any class and a quantitative method for comparing them

polynomial time computation of strong and n present value optimal policies in markov decision chains this paper studies the problem of finding a stationary strong present value optimal and more generally an n present value optimal policy for an infinite horizon stationary finite state action substochastic markov decision chain a policy is strong present value optimal if it has maximum present value for all small positive interest rates a policy is n present value optimal if its present value falls below the maximum present value for all small positive by o n 1 the importance of stationary n present value optimal policies arises from the following known facts the set of such policies diminishes with n and for n s is precisely the set of stationary strong present value optimal policies for 0 n s stationary n present value optimal policies are nearly strong present value optimal and are of independent interest the best algorithms for finding stationary strong present value optimal policies find stationary n present value policies for n 1 s in that order this paper decomposes th

the complexity of optimal small policies we investigate the complexity of problems concerned with partially observable markov decision processes which are run for a finite number of steps under small policies the calculation of the expected sum of rewards of a process under a small policy is shown to be complete for the complexity class pp a class which lies intermediate between np and pspace optimal small policy computation is shown to be complete for nppp the latter contrasts results of papadimitriou and tsitsiklis papadimitriou and tsitsiklis 1987 who showed that this problem is pspace complete if no assumptions about the representability of the policy are made and that it is p complete for fully observable processes

simulation based uniform value function estimates of markov decision processes the value function of a markov decision process mdp assigns to each policy its expected discounted reward this expected reward can be estimated as the empirical average of the reward over many independent simulation runs we derive bounds on the number of runs needed for the uniform convergence of the empirical average to the expected reward for a class of policies in terms of the vapnik chervonenkis or p dimension of the policy class further we show through a counterexample that whether we get uniform convergence or not for an mdp depends on the simulation method used uniform convergence results are also obtained for the average reward case for partially observed markov decision processes and can be easily extended to markov games the results can be viewed as a contribution to empirical process theory and as an extension of the probably approximately correct pac learning theory for partially observable mdps and markov games

conformant planning via heuristic forward search a new approach conformant planning is the task of generating plans given uncertainty about the initial state and action effects and without any sensing capabilities during plan execution the plan should be successful regardless of which particular initial world we start from it is well known that conformant planning can be transformed into a search problem in belief space the space whose elements are sets of possible worlds we introduce a new representation of that search space replacing the need to store sets of possible worlds with a need to reason about the effects of action sequences the reasoning is done by implication tests on propositional formulas in conjunctive normal form cnf that capture the action sequence semantics based on this approach we extend the classical heuristic forward search planning system ff to the conformant setting the key to this extension is an appropriate extension of the relaxation that underlies ff s heuristic function and of ff s machinery for solving relaxed planning problems the extended machinery includes a stronger form of the cnf implication tests that we use to reason about the effects of action sequences our experimental evaluation shows the resulting planning system to be superior to the state of the art conformant planners mbp kacmbp and gpt in a variety of benchmark domains

task decomposition on abstract states for planning under nondeterminism although several approaches have been developed for planning in nondeterministic domains solving large planning problems is still quite difficult in this work we present a new planning algorithm called yoyo for solving planning problems in fully observable nondeterministic domains yoyo combines an htn based mechanism for constraining its search and a binary decision diagram bdd representation for reasoning about sets of states and state transitions we provide correctness theorems for yoyo and an experimental comparison of it with mbp and nd shop2 the two previously best algorithms for planning in nondeterministic domains in our experiments yoyo could easily deal with problem sizes that neither mbp nor nd shop2 could scale up to and could solve problems about 100 to 1000 times faster than mbp and nd shop2

planning as satisfiability parallel plans and algorithms for plan search we address two aspects of constructing plans efficiently by means of satisfiability testing efficient encoding of the problem of existence of plans of a given number t of time points in the propositional logic and strategies for finding plans given these formulae for different values of t for the first problem we consider three semantics for plans with parallel operator application in order to make the search for plans more efficient the standard semantics requires that parallel operators are independent and can therefore be executed in any order we consider a more relaxed definition of parallel plans which was first proposed by dimopoulos et al as well as a normal form for parallel plans that requires every operator to be executed as early as possible we formalize the semantics of parallel plans emerging in this setting and present translations of these semantics into the propositional logic the sizes of the translations are asymptotically optimal each of the semantics is constructed in such a way that there is a plan following the semantics exactly when there is a sequential plan and moreover the existence of a parallel plan implies the existence of a sequential plan with as many operators as in the parallel one for the second problem we consider strategies based on testing the satisfiability of several formulae representing plans of n time steps for several values of n concurrently by several processes we show that big efficiency gains can be obtained in comparison to the standard strategy of sequentially testing the satisfiability of formulae for an increasing number of time steps

and or graph heuristic search methods two new marking algorithms for and or graphs called cf and cs are presented for admissible heuristics cs is not needed and cf is shown to be preferable to the marking algorithms of martelli and montanari when the heuristic is not admissible the analysis is carried out with the help of the notion of the first and second discriminants of an and or graph it is proved that in this case cf can be followed by cs to get optimal solutions provided the sumcost criterion is used and the first discriminant equals the second estimates of time and storage requirements are given other cost measures such as maxcost are also considered and a number of interesting open problems are enumerated

approximation of action theories and its application to conformant planning this paper describes our methodology for building conformant planners which is based on recent advances in the theory of action and change and answer set programming the development of a planner for a given dynamic domain starts with encoding the knowledge about fluents and actions of the domain as an action theory d of some action language our choice in this paper is al an action language with dynamic and static causal laws and executability conditions an action theory d of al defines a transition diagram t d containing all the possible trajectories of the domain a transition belongs to t d iff the execution of the action a in the state s may move the domain to the state s the second step in the planner development consists in finding a deterministic transition diagram t l p d such that nodes of t l p d are partial states of d its arcs are labeled by actions and a path in t l p d from an initial partial state d 0 to a partial state satisfying the goal d f corresponds to a conformant plan for d 0 and d f in t d the transition diagram t l p d is called an approximation of t d we claim that a concise description of an approximation of t d can often be given by a logic program p d under the answer sets semantics moreover complex initial situations and constraints on plans can be also expressed by logic programming rules and included in p d if this is possible then the problem of finding a parallel or sequential conformant plan can be reduced to computing answer sets of p d this can be done by general purpose answer set solvers if plans are sequential and long then this method can be too time consuming in this case p d is used as a specification for a procedural graph searching conformant planning algorithm the paper illustrates this methodology by building several conformant planners which work for domains with complex relationship between the fluents the efficiency of the planners is experimentally evaluated on a number of new and old benchmarks in addition we show that for a subclass of action theories of al our planners are complete i e if in t l p d we cannot get from d 0 to a state satisfying the goal d f then there is no conformant plan for d 0 and d f in t d