strong planning under partial observability rarely planning domains are fully observable for this reason the ability to deal with partial observability is one of the most important challenges in planning in this paper we tackle the problem of strong planning under partial observability in nondeterministic domains find a conditional plan that will result in a successful state regardless of multiple initial states nondeterministic action effects and partial observability we make the following contributions first we formally define the problem of strong planning within a general framework for modeling partially observable planning domains second we propose an effective planning algorithm based on and or search in the space of beliefs we prove that our algorithm always terminates and is correct and complete in order to achieve additional effectiveness we leverage on a symbolic bdd based representation for the domain and propose several search strategies we provide a thorough experimental evaluation of our approach based on a wide selection of benchmarks we compare the performance of the proposed search strategies and identify a uniform winner that combines heuristic distance measures with mechanisms that reduce runtime uncertainty then we compare our planner mbp with other state of the art systems mbp is able to outperform its competitor systems often by orders of magnitude

an algorithm for coarsening unstructured meshes we develop and analyze a procedure for creating a hierarchical basis of continuous piecewise linear polynomials on an arbitrary unstructured nonuniform triangular mesh using these hierarchical basis functions we are able to define and analyze corresponding iterative methods for solving the linear systems arising from finite element discretizations of elliptic partial differential equations we show that such iterative methods perform as well as those developed for the usual case of structured locally refined meshes in particular we show that the generalized condition numbers for such iterative methods are of order j 2 where j is the number of hierarchical basis levels

a weakly overlapping domain decomposition preconditioner for the finite element solution of elliptic partial differential equations we present a new two level additive schwarz domain decomposition preconditioner which is appropriate for use in the parallel finite element solution of elliptic partial differential equations pdes as with most parallel domain decomposition methods each processor may be assigned one or more subdomains and the preconditioner is such that the processors are able to solve their own subproblem s concurrently the novel feature of the technique proposed here is that it requires just a single layer of overlap in the elements which make up each subdomain at each level of refinement and it is shown that this amount of overlap is sufficient to yield an optimal preconditioner some numerical experiments posed in both two and three space dimensions are included to confirm that the condition number when using the new preconditioner is indeed independent of the level of mesh refinement on the test problems considered

w symmetries of ito stochastic differential equations we discuss w symmetries of ito stochastic differential equations introduced in a recent paper by gaeta and spadaro j math phys 58 053503 2017 in particular we discuss the general form of acceptable generators for continuous lie point w symmetry arguing that they are related to the linear conformal group and how w symmetries can be used in the integration of ito stochastic equations along kozlov theory for standard deterministic or random symmetries it turns out that this requires in general considering more general classes of stochastic equations than just ito ones we discuss w symmetries of ito stochastic differential equations introduced in a recent paper by gaeta and spadaro j math phys 58 053503 2017 in particular we discuss the general form of acceptable generators for continuous lie point w symmetry arguing that they are related to the linear conformal group and how w symmetries can be used in the integration of ito stochastic equations along kozlov theory for standard deterministic or random symmetries it turns out that this requires in general considering more general classes of stochastic equations than just ito ones

algebraic multigrid amg experiences and comparisons algebraic multigrid methods are designed for the solution of sparse linear systems of equations using multigrid principles in contrast to standard multigrid methods amg does not take advantage of the origin of a particular system of equations at hand nor does it exploit any underlying geometrical situation fully automatically and based solely on algebraic information contained in the given matrix amg constructs a sequence of grids and corresponding operators a special amg algorithm will be presented for a wide range of problems including certain problems which do not have a continuous background this algorithm yields an iterative method which exhibits a convergence behavior typical for multigrid methods

the complexity of mean payoff games on graphs abstract we study the complexity of finding the values and optimal strategies of mean payoff games on graphs a family of perfect information games introduced by ehrenfeucht and mycielski and considered by gurvich karzanov and khachiyan we describe a pseudo polynomial time algorithm for the solution of such games the decision problem for which is in np conp finally we describe a polynomial reduction from mean payoff games to the simple stochastic games studied by condon these games are also known to be in np conp but no polynomial or pseudo polynomial time algorithm is known for them

nonmyopic multiaspect sensing with partially observable markov decision processes we consider the problem of sensing a concealed or distant target by interrogation from multiple sensors situated on a single platform the available actions that may be taken are selection of the next relative target platform orientation and the next sensor to be deployed the target is modeled in terms of a set of states each state representing a contiguous set of target sensor orientations over which the scattering physics is relatively stationary the sequence of states sampled at multiple target sensor orientations may be modeled as a markov process the sensor only has access to the scattered fields without knowledge of the particular state being sampled and therefore the problem is modeled as a partially observable markov decision process pomdp the pomdp yields a policy in which the belief state at any point is mapped to a corresponding action the nonmyopic policy is compared to an approximate myopic approach with example results presented for measured underwater acoustic scattering data

random lie point symmetries of stochastic differential equations we study the invariance of stochastic differential equations under random diffeomorphisms and establish the determining equations for random lie point symmetries of stochastic differential equations both in ito and in stratonovich forms we also discuss relations with previous results in the literature

exploiting symmetries for single and multi agent partially observable stochastic domains while partially observable markov decision processes pomdps and their multi agent extension partially observable stochastic games posgs provide a natural and systematic approach to modeling sequential decision making problems under uncertainty the computational complexity with which the solutions are computed is known to be prohibitively expensive in this paper we show how such high computational resource requirements can be alleviated through the use of symmetries present in the problem the problem of finding the symmetries can be cast as a graph automorphism ga problem on a graphical representation of the problem we demonstrate how such symmetries can be exploited in order to speed up the solution computation and provide computational complexity results

the group classification of a scalar stochastic differential equation lie point group classification of a scalar stochastic differential equation sde with one dimensional brownian motion is presented the admitted symmetry group can be zero one two or three dimensional if an equation admits a three dimensional symmetry group it can be transformed into the equation of brownian motion by a change of variables with non random time transformation such equations can be integrated by quadratures there are described drift coefficients for which sdes with constant diffusion coefficients admit two and three dimensional symmetry groups

the hierarchical basis multigrid method we derive and analyze the hierarchical basis multigrid method for solving discretizations of self adjoint elliptic boundary value problems using piecewise linear triangular finite elements the method is analyzed as a block symmetric gau seidel iteration with inner iterations but it is strongly related to 2 level methods to the standard multigridv cycle and to earlier jacobi like hierarchical basis methods the method is very robust and has a nearly optimal convergence rate and work estimate it is especially well suited to difficult problems with rough solutions discretized using highly nonuniform adaptively refined meshes

a review of algebraic multigrid abstract since the early 1990s there has been a strongly increasing demand for more efficient methods to solve large sparse unstructured linear systems of equations for practically relevant problem sizes classical one level methods had already reached their limits and new hierarchical algorithms had to be developed in order to allow an efficient solution of even larger problems this paper gives a review of the first hierarchical and purely matrix based approach algebraic multigrid amg amg can directly be applied for instance to efficiently solve various types of elliptic partial differential equations discretized on unstructured meshes both in 2d and 3d since amg does not make use of any geometric information it is a plug in solver which can even be applied to problems without any geometric background provided that the underlying matrix has certain properties

the incomplete factorization multigraph algorithm we present a new family of multigraph algorithms ilu mg based upon an incomplete sparse matrix factorization using a particular ordering and allowing a limited amount of fill in while much of the motivation for multigraph comes from multigrid ideas ilu mg is distinctly different from algebraic multilevel methods the graph of the sparse matrix a is recursively coarsened by eliminating vertices using a graph model similar to gaussian elimination incomplete factorizations are obtained by allowing only the fill in generated by the vertex parents associated with each vertex multigraph is numerically compared with algebraic multigrid on some examples arising from discretizations of partial differential equations on unstructured grids

a remark on symmetry of stochastic dynamical systems and their conserved quantities the symmetry properties of stochastic dynamical systems described by a stochastic differential equation of stratonovich type and related conserved quantities are discussed extending previous results by misawa new conserved quantities are given by applying symmetry operators to known conserved quantities some detailed examples are presented

computing rank dependent utility in graphical models for sequential decision problems this paper is devoted to automated sequential decision in ai more precisely we focus here on the rank dependent utility rdu model this model is able to encompass rational decision behaviors that the expected utility model cannot accommodate however the non linearity of rdu makes it difficult to compute an rdu optimal strategy in sequential decision problems this has considerably slowed the use of rdu in operational contexts in this paper we are interested in providing new algorithmic solutions to compute an rdu optimal strategy in graphical models specifically we present algorithms for solving decision tree models and influence diagram models of sequential decision problems for decision tree models we propose a mixed integer programming formulation that is valid for a subclass of rdu models corresponding to risk seeking behaviors this formulation reduces to a linear program when mixed strategies are considered in the general case i e when there is no particular assumption on the parameters of rdu we propose a branch and bound procedure to compute an rdu optimal strategy among the pure ones after highlighting the difficulties induced by the use of rdu in influence diagram models we show how this latter procedure can be extended to optimize rdu in an influence diagram finally we provide empirical evaluations of all the presented algorithms

lie point symmetries and stochastic differential equations we discuss lie point symmetries of stochastic ordinary differential equations and the interrelations between these and analogous symmetries of the associated fokker planck equation for the probability measure

black box multigrid one major problem with the multigrid method has been that each new grid configuration has required a major programming effort to develop a code that specifically handles that grid configuration such a penalty is not required for methods like sor iccg etc in these methods one need only specify the matrix problem no matter what the grid configuration in this paper we investigate such a situation for the multigrid method the end result is code boxmg in which one need only specify the logically rectangular positive definite matrix problem boxmg does everything else necessary to set up the auxilliary coarser problems to achieve a multigrid solution

event based optimization of markov systems recent research indicates that markov decision processes mdps and perturbation analysis pa based optimization can be derived easily from two fundamental performance sensitivity formulas with this sensitivity point of view an event based optimization approach including event based sensitivity analysis and event based policy iteration was proposed via an example by x r cao discrete event dyn syst theory appl vol 15 pp 169 197 2005 this approach utilizes the special feature of a system and illustrates how the potentials can be aggregated using the special feature the approach applies to many practical problems that do not fit well the standard mdp formulation this note provides a mathematical formulation and proves the main results for this approach

bayesian policy gradient and actor critic algorithms policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate many conventional policy gradient methods use monte carlo techniques to estimate this gradient the policy is improved by adjusting the parameters in the direction of the gradient estimate since monte carlo methods tend to have high variance a large number of samples is required to attain accurate estimates resulting in slow convergence in this paper we first propose a bayesian framework for policy gradient based on modeling the policy gradient as a gaussian process this reduces the number of samples needed to obtain accurate gradient estimates moreover estimates of the natural gradient as well as a measure of the uncertainty in the gradient estimates namely the gradient covariance are provided at little extra cost since the proposed bayesian framework considers system trajectories as its basic observable unit it does not require the dynamics within trajectories to be of any particular form and thus can be easily extended to partially observable problems on the downside it cannot take advantage of the markov property when the system is markovian to address this issue we proceed to supplement our bayesian policy gradient framework with a new actor critic learning model in which a bayesian class of non parametric critics based on gaussian process temporal difference learning is used such critics model the action value function as a gaussian process allowing bayes rule to be used in computing the posterior distribution over action value functions conditioned on the observed data appropriate choices of the policy parameterization and of the prior covariance kernel between action values allow us to obtain closed form expressions for the posterior distribution of the gradient of the expected return with respect to the policy parameters we perform detailed experimental comparisons of the proposed bayesian policy gradient and actor critic algorithms with classic monte carlo based policy gradient methods as well as with each other on a number of reinforcement learning problems

planning under time constraints in stochastic domains we provide a method based on the theory of markov decision processes for efficient planning in stochastic domains goals are encoded as reward functions expressing the desirability of each world state the planner must find a policy mapping from states to actions that maximizes future rewards standard goals of achievement as well as goals of maintenance and prioritized combinations of goals can be specified in this way an optimal policy can be found using existing methods but these methods require time at best polynomial in the number of states in the domain where the number of states is exponential in the number of propositions or state variables by using information about the starting state the reward function and the transition probabilities of the domain we restrict the planner s attention to a set of world states that are likely to be encountered in satisfying the goal using this restricted set of states the planner can generate more or less complete plans depending on the time it has available our approach employs several iterative refinement routines for solving different aspects of the decision making problem we describe the meta level control problem of em deliberation scheduling allocating computational resources to these routines we provide different models corresponding to optimization problems that capture the different circumstances and computational strategies for decision making under time constraints we consider em precursor models in which all decision making is performed prior to execution and em recurrent models in which decision making is performed in parallel with execution accounting for the states observed during execution and anticipating future states we describe experimental results for both the precursor and recurrent problems that demonstrate planning times that grow slowly as a function of domain size

multiresolution approximate inverse preconditioners we introduce a new preconditioner for elliptic pdes on unstructured meshes using a wavelet inspired basis we compress the inverse of the matrix allowing an effective sparse approximate inverse by solving the sparsity vs accuracy conflict the key issue in this compression is to use second generation wavelets which can be adapted to the unstructured mesh the true boundary conditions and even the pde coefficients we also show how this gives a new perspective on multiresolution algorithms such as multigrid interpreting the new preconditioner as a variation on node nested multigrid in particular we hope the new preconditioner will combine the best of both worlds fast convergence when multilevel methods can succeed but with robust performance for more difficult problems the rest of the paper discusses the core issues for the preconditioner ordering and construction of a factored approximate inverse in the multiresolution basis robust interpolation on unstructured meshes automatic mesh coarsening and purely algebraic alternatives some exploratory numerical experiments suggest the superiority of the new basis over the standard basis for several tough problems including discontinuous anisotropic coefficients strong convection and indefinite reaction problems on unstructured meshes with scalability like hierarchical basis methods achieved

partially observable markov decision processes with imprecise parameters this study extends the framework of partially observable markov decision processes pomdps to allow their parameters i e the probability values in the state transition functions and the observation functions to be imprecisely specified it is shown that this extension can reduce the computational costs associated with the solution of these problems first the new framework pomdps with imprecise parameters pomdpips is formulated we consider 1 the interval case in which each parameter is imprecisely specified by an interval that indicates possible values of the parameter and 2 the point set case in which each probability distribution is imprecisely specified by a set of possible distributions second a new optimality criterion for pomdpips is introduced as in pomdps the criterion is to regard a policy i e an action selection rule as optimal if it maximizes the expected total reward the expected total reward however cannot be calculated precisely in pomdpips because of the parameter imprecision instead we estimate the total reward by adopting arbitrary second order beliefs i e beliefs in the imprecisely specified state transition functions and observation functions although there are many possible choices for these second order beliefs we regard a policy as optimal as long as there is at least one of such choices with which the policy maximizes the total reward thus there can be multiple optimal policies for a pomdpip we regard these policies as equally optimal and aim at obtaining one of them by appropriately choosing which second order beliefs to use in estimating the total reward computational costs incurred in obtaining such an optimal policy can be reduced significantly we provide an exact solution algorithm for pomdpips that does this efficiently third the performance of such an optimal policy as well as the computational complexity of the algorithm are analyzed theoretically last empirical studies show that our algorithm quickly obtains satisfactory policies to many pomdpips

integration of stochastic ordinary differential equations from a symmetry standpoint we derive an algorithm for calculating lie point symmetries of systems of stochastic ordinary differential equations sodes of any order from this algorithm the following facts emerge symmetries of a sode do not in general form a lie algebra the determining equations for ito s equation are in general stochastic linear partial differential equations whereas for sodes of order n 2 the determining equations are linear deterministic partial differential equations that form an overdetemined system which is solvable by classical methods for scalar second order sodes we provide a complete classification of equations admitting finite dimensional symmetry lie algebras this classification is applied to the integration of scalar second order sodes in general a sode admitting a two dimensional symmetry algebra is not integrable by quadratures although it is reducible to a homogeneous ito equation in particular a scalar second order sode admitting a two dimensional symmetry algebra with connected operators is linearizable we also characterize integrable scalar second order sodes admitting three dimensional symmetry algebras finally we show that a sode can admit maximally a zero one two three or four dimensional lie algebra

partially observable markov decision processes a geometric technique and analysis this paper presents a novel framework for studying partially observable markov decision processes pomdps with finite state action observation sets and discounted rewards the new framework is solely based on future reward vectors associated with future policies which is more parsimonious than the traditional framework based on belief vectors it reveals the connection between the pomdp problem and two computational geometry problems i e finding the vertices of a convex hull and finding the minkowski sum of convex polytopes which can help solve the pomdp problem more efficiently the new framework can clarify some existing algorithms over both finite and infinite horizons and shed new light on them it also facilitates the comparison of pomdps with respect to their degree of observability as a useful structural result

an algorithm for probabilistic planning abstract we define the probabilistic planning problem in terms of a probability distribution over initial world states a boolean combination of propositions representing the goal a probability threshold and actions whose effects depend on the execution time state of the world and on random chance adopting a probabilistic model complicates the definition of plan success instead of demanding a plan that provably achieves the goal we seek plans whose probability of success exceeds the threshold in this paper we present buridan an implemented least commitment planner that solves problems of this form we prove that the algorithm is both sound and complete we then explore buridan s efficiency by contrasting four algorithms for plan evaluation using a combination of analytic methods and empirical experiments we also describe the interplay between generating plans and evaluating them and discuss the role of search control in probabilistic planning

model based reinforcement learning for partially observable games with sampling based state estimation games constitute a challenging domain of reinforcement learning rl for acquiring strategies because many of them include multiple players and many unobservable variables in a large state space the difficulty of solving such realistic multiagent problems with partial observability arises mainly from the fact that the computational cost for the estimation and prediction in the whole state space including unobservable variables is too heavy to overcome this intractability and enable an agent to learn in an unknown environment an effective approximation method is required with explicit learning of the environmental model we present a model based rl scheme for large scale multiagent problems with partial observability and apply it to a card game hearts this game is a well defined example of an imperfect information game and can be approximately formulated as a partially observable markov decision process pomdp for a single learning agent to reduce the computational cost we use a sampling technique in which the heavy integration required for the estimation and prediction can be approximated by a plausible number of samples computer simulation results show that our method is effective in solving such a difficult partially observable multiagent problem

on the synthesis of optimal schedulers in discrete event control problems with multiple goals this paper deals with a new type of optimal control for discrete event systems our control problem extends the theory of r sengupta and s lafortune siam j control optim 36 1998 pp 488 541 that is characterized by the presence of uncontrollable events the notion of occurrence and control costs for events and a worst case objective function a significant difference with r sengupta and s lafortune siam j control optim 36 1998 pp 488 541 is that our aim is to make the system evolve through a set of multiple goals one by one with no order necessarily prespecified whereas the previous theory only deals with a single goal our solution approach is divided into two steps in the first step we use the optimal control theory in r sengupta and s lafortune siam j control optim 36 1998 pp 488 541 to synthesize individual controllers for each goal in the second step we develop the solution of another optimal control problem namely how to modify if necessary and piece together or schedule all of the controllers built in the first step in order to visit each of the goals with the least total cost we solve this problem by defining the notion of a scheduler and then by mapping the problem of finding an optimal scheduler to an instance of the well known traveling salesman problem tsp e l lawler j k lenstra a h g rinooy kan and d b shmoys the traveling salesman problem john wiley 1985 we finally suggest various strategies to reduce the complexity of the tsp resolution while still preserving global optimality

algebraic multilevel preconditioning methods ii a recursive way of constructing preconditioning matrices for the stiffness matrix in the discretization of selfadjoint second order elliptic boundary value problems is proposed it is based on a sequence of nested finite element spaces with the usual nodal basis functions using a nodeordering corresponding to the nested meshes the finite element stiffness matrix is recursively split up into two level block structures and is factored approximately in such a way that any successive schur complement is replaced approximated by a matrix defined recursively and thereform only implicitely given to solve a system with this matrix we need to perform a fixed number v of iterations on the preceding level using as an iteration matrix the preconditioning matrix already defined on that level it is shown that by a proper choice of iteration parameters it suffices to use v left 1 gamma 2 right tfrac 1 2 iterations for the so constructedv foldv cycle wherev 2 corresponds to aw cycle preconditioning matrices to be spectrally equivalent to the stiffness matrix the conditions involve only the constant in the strengthened c b s inequality for the corresponding two level hierarchical basis function spaces and are therefore independent of the regularity of the solution for instance if we use successive uniform refinements of the meshes the method is of optimal order of computational complexity if gamma 2 tfrac 8 9 under reasonable assumptions of the finite element mesh the condition numbers turn out to be so small that there are in practice few reasons to use an accelerated iterative method like the conjugate gradient method for instance

weighted matchings for preconditioning symmetric indefinite linear systems maximum weight matchings have become an important tool for solving highly indefinite unsymmetric linear systems especially in direct solvers in this study we investigate the benefit of reorderings and scalings based on symmetrized maximum weight matchings as a preprocessing step for incomplete mathrm ldl t factorizations the reorderings are constructed such that the matched entries form 1 times 1 or 2 times 2 diagonal blocks in order to increase the diagonal dominance of the system during the incomplete factorization only tridiagonal pivoting is used we report results for this approach and comparisons with other solution methods for a diverse set of symmetric indefinite matrices ranging from nonlinear elasticity to interior point optimization

conformant plans and beyond principles and complexity conformant planning is used to refer to planning for unobservable problems whose solutions like classical planning are linear sequences of operators called linear plans the term conformant is automatically associated with both the unobservable planning model and with linear plans mainly because the only possible solutions for unobservable problems are linear plans in this paper we show that linear plans are not only meaningful for unobservable problems but also for partially observable problems in such case the execution of a linear plan generates observations from the environment which must be collected by the agent during the execution of the plan and used at the end in order to determine whether the goal had been achieved or not this is the typical case in problems of diagnosis in which all the actions are knowledge gathering actions thus there are substantial differences about linear plans for the case of unobservable or fully observable problems and for the case of partially observable problems while linear plans for the former model must conform with properties in state space linear plans for partially observable problems must conform with properties in belief space this differences surface when the problems are allowed to express epistemic goals and conditions using modal logic and place the plan existence decision problem in different complexity classes linear plans is one extreme point in a discrete spectrum of solution forms for planning problems the other extreme point is contingent plans in which there is a branch point for every possible observation at each time step and thus the number of branch points is not bounded a priori in the middle of the spectrum there are plans with a bounded number of branch points thus linear plans are plans with zero branch points and contingent plans are plans with unbounded number of branch points in this work we lay down foundations and principles for the general treatment of linear plans and plans of bounded branching and provide exact complexity results for novel decision problems we also show that linear plans for partially observable problems are not only of theoretical interest since some challenging real life problems can be dealt with them

preconditioning techniques for large linear systems a survey this article surveys preconditioning techniques for the iterative solution of large linear systems with a focus on algebraic methods suitable for general sparse matrices covered topics include progress in incomplete factorization methods sparse approximate inverses reorderings parallelization issues and block and multilevel extensions some of the challenges ahead are also discussed an extensive bibliography completes the paper

new conserved quantities derived from symmetry for stochastic dynamical systems recently the author has proposed an elementary theory of conserved quantities and symmetry for stochastic dynamical systems described by stochastic differential equations of stratonovich type within the framework a new method for deriving conserved quantities from symmetry is developed in a similar manner to that of hojman s work 1992 which gives a new conservation law constructed without using either lagrangians or hamiltonians for deterministic dynamical systems some examples of the conserved quantities obtained through the new method are given

point based value iteration for continuous pomdps we propose a novel approach to optimize partially observable markov decisions processes pomdps defined on continuous spaces to date most algorithms for model based pomdps are restricted to discrete states actions and observations but many real world problems such as for instance robot navigation are naturally defined on continuous spaces in this work we demonstrate that the value function for continuous pomdps is convex in the beliefs over continuous state spaces and piecewise linear convex for the particular case of discrete observations and actions but still continuous states we also demonstrate that continuous bellman backups are contracting and isotonic ensuring the monotonic convergence of value iteration algorithms relying on those properties we extend the algorithm originally developed for discrete pomdps to work in continuous state spaces by representing the observation transition and reward models using gaussian mixtures and the beliefs using gaussian mixtures or particle sets with these representations the integrals that appear in the bellman backup can be computed in closed form and therefore the algorithm is computationally feasible finally we further extend to deal with continuous action and observation sets by designing effective sampling approaches

the complexity of stochastic games abstract we consider the complexity of stochastic games simple games of chance played by two players we show that the problem of deciding which player has the greatest chance of winning the game is in the class np co np

multi task reinforcement learning in partially observable stochastic environments we consider the problem of multi task reinforcement learning mtrl in multiple partially observable stochastic environments we introduce the regionalized policy representation rpr to characterize the agent s behavior in each environment the rpr is a parametric model of the conditional distribution over current actions given the history of past actions and observations the agent s choice of actions is directly based on this conditional distribution without an intervening model to characterize the environment itself we propose off policy batch algorithms to learn the parameters of the rprs using episodic data collected when following a behavior policy and show their linkage to policy iteration we employ the dirichlet process as a nonparametric prior over the rprs across multiple environments the intrinsic clustering property of the dirichlet process imposes sharing of episodes among similar environments which effectively reduces the number of episodes required for learning a good policy in each environment when data sharing is appropriate the number of distinct rprs and the associated clusters the sharing patterns are automatically discovered by exploiting the episodic data as well as the nonparametric nature of the dirichlet process we demonstrate the effectiveness of the proposed rpr as well as the rpr based mtrl framework on various problems including grid world navigation and multi aspect target classification the experimental results show that the rpr is a competitive reinforcement learning algorithm in partially observable domains and the mtrl consistently achieves better performance than single task reinforcement learning

solving partially observable problems with inaccurate psr models modeling dynamical systems is a commonly used technique to solve partially observable problems in the artificial intelligence field predictive state representations psrs have been proposed as an alternative to partially observable markov decision processes pomdps to model dynamical systems although pomdps and psrs provide general frameworks to solve partially observable problems they rely heavily on a known and accurate model of the environment however in real world applications it is extremely difficult to build an accurate model in this paper we propose an algorithm to solve partially observable problems using an inaccurate psr model which is learned from samples the proposed algorithm can also improve the accuracy of the learned model given the inaccurate psr model the psr state is identified firstly then the traditional markov decision processes mdp techniques are used to solve the partially observable problem furthermore the learned model which may get off track as often happens when the model is learned from samples can be reset the effectiveness of our proposed algorithm is demonstrated based on a standard set of pomdp test problems

the complexity of decentralized control of markov decision processes we consider decentralized control of markov decision processes and give complexity bounds on the worst case running time for algorithms that find optimal solutions generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described for even two agents the finite horizon problems corresponding to both of these models are hard for nondeterministic exponential time these complexity results illustrate a fundamental difference between centralized and decentralized control of markov decision processes in contrast to the problems involving centralized control the problems we considerprovably do not admit polynomial time algorithms furthermore assuming exp nexp the problems require superexponential time to solve in the worst case

state of the art a survey of partially observable markov decision processes theory models and algorithms this paper surveys models and algorithms dealing with partially observable markov decision processes a partially observable markov decision process pomdp is a generalization of a markov decision process which permits uncertainty regarding the state of a markov process and allows for state information acquisition a general framework for finite state and action pomdp s is presented next there is a brief discussion of the development of pomdp s and their relationship with other decision processes a wide range of models in such areas as quality control machine maintenance internal auditing learning and optimal stopping are discussed within the pomdp framework lastly algorithms for computing optimal solutions to pomdp s are presented

learning where to attend with deep architectures for image tracking we discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data motivated by theories of perception the model consists of two interacting pathways identity and control intended to mirror the what and where pathways in neuroscience models the identity pathway models object appearance and performs classification using deep factored restricted boltzmann machines at each point in time the observations consist of foveated images with decaying resolution toward the periphery of the gaze the control pathway models the location orientation scale and speed of the attended object the posterior distribution of these states is estimated with particle filtering deeper in the control pathway we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty unlike in our previous work we introduce gaze selection strategies that operate in the presence of partial information and on a continuous action space we show that a straightforward extension of the existing approach to the partial information setting results in poor performance and we propose an alternative method based on modeling the reward surface as a gaussian process this approach gives good performance in the presence of partial information and allows us to expand the action space from a small discrete set of fixation points to a continuous domain

myopic bounds for optimal policy of pomdps an extension of lovejoy s structural results this paper provides a relaxation of the sufficient conditions and an extension of the structural results for partially observed markov decision processes pomdps obtained by lovejoy in 1987 sufficient conditions are provided so that the optimal policy can be upper and lower bounded by judiciously chosen myopic policies these myopic policy bounds are constructed to maximize the volume of belief states where they coincide with the optimal policy numerical examples illustrate these myopic bounds for both continuous and discrete observation sets

cost sensitive feature acquisition and classification there are many sensing challenges for which one must balance the effectiveness of a given measurement with the associated sensing cost for example when performing a diagnosis a doctor must balance the cost and benefit of a given test measurement and the decision to stop sensing stop performing tests must account for the risk to the patient and doctor malpractice for a given diagnosis based on observed data this motivates a cost sensitive classification problem in which the features sensing results are not given a priori the algorithm determines which features to acquire next as well as when to stop sensing and make a classification decision based on previous observations accounting for the costs of various types of errors as well as the rewards of being correct we formally define the cost sensitive classification problem and solve it via a partially observable markov decision process pomdp while the pomdp constitutes an intuitively appealing formulation the intrinsic properties of classification tasks resist application of it to this problem we circumvent the difficulties of the pomdp via a myopic approach with an adaptive stopping criterion linked to the standard pomdp the myopic algorithm is computationally feasible easily handles continuous features and seamlessly avoids repeated actions experiments with several benchmark data sets show that the proposed method yields state of the art performance and importantly our method uses only a small fraction of the features that are generally used in competitive approaches

an algebraic multilevel multigraph algorithm we describe an algebraic multilevel multigraph algorithm many of the multilevel components are generalizations of algorithms originally applied to general sparse gaussian elimination indeed general sparse gaussian elimination with minimum degree ordering is a limiting case of our algorithm our goal is to develop a procedure which has the robustness and simplicity of use of sparse direct methods yet offers the opportunity to obtain the optimal or near optimal complexity typical of classical multigrid methods

the potential structure of sample paths and performance sensitivities of markov systems we study the structure of sample paths of markov systems by using performance potentials as the fundamental units with a sample path based approach we show that performance sensitivity formulas performance gradients and performance differences of markov systems can be constructed intuitively by first principles with performance potentials or equivalently perturbation realization factors as building blocks in particular we derive sensitivity formulas for two markov chains with possibly different state spaces the proposed approach can be used to obtain flexibly the sensitivity formulas for a wide range of problems including those with partial information these formulas are the basis for performance optimization of discrete event dynamic systems including perturbation analysis markov decision processes and reinforcement learning the approach thus provides insight on on line learning and performance optimization and opens up new research directions sample path based algorithms can be developed

testing probabilistic equivalence through reinforcement learning checking if a given system implementation respects its specification is often done by proving that the two are equivalent the equivalence is chosen in particular for its computability and of course for its meaning that is for its adequacy with what is observable from the two systems implementation and specification trace equivalence is easily testable decidable from interaction but often considered too weak in contrast bisimulation is accepted as the canonical equivalence for interaction but it is not testable richer than an equivalence is a form of distance it is zero between equivalent systems and it provides an estimation of their difference if the systems are not equivalent our main contribution is to define such a distance in a context where 1 the two systems to be compared have a stochastic behavior 2 the model of one of them e g the implementation is unknown hence our only knowledge is obtained by interacting with it 3 consequently the target equivalence observed when distance is zero must be testable to overcome the problem that the model is unknown we use a reinforcement learning approach that provides powerful stochastic algorithms that only need to interact with the model our second main contribution is a new family of testable equivalences called k moment the weakest of them 1 moment equivalence is trace equivalence as k grows k moment equivalences become finer all remaining as well as their limit weaker than bisimulation we propose a framework to define and test a bigger class of testable equivalences test observation equivalences toes and we show how they can be made coarser or not by tuning some parameters

the optimal search for a moving target when the search path is constrained a search is conducted for a target moving in discrete time among a finite number of cells according to a known markov process the searcher must choose one cell in which to search in each time period the set of cells available for search depends upon the cell chosen in the last time period the problem is to find a searcher path i e a sequence of search cells that maximizes the probability of detecting the target in a fixed number of time periods we formulate the problem as a partially observable markov decision process and present a finite time horizon pomdp solution technique which is simpler than the standard linear programming methods

lie point symmetries of stratonovich stochastic differential equations this paper considers lie point symmetries of stochastic differential equations sdes in stratonovich form first we derive the determining equations of the random symmetries which correspond to lie group transformations acting in the space of the independent variable time the dependent variables and the brownian motion then the determining equations of the random symmetries are used to obtain the determining equations of the deterministic symmetries which correspond to lie group transformations acting in the space of the independent variable and dependent variables in the case of the deterministic symmetries the transformation of the brownian motion is induced by the random time change it is proved that the corresponding ito and stratonovich sdes admit the same random symmetries and the same deterministic symmetries

using convex switching techniques for partially observable decision processes we present and examine a novel method for obtaining solutions to specific discrete time optimal control problems our approach is based on linear state dynamics and convexity assumptions commonly satisfied in practical applications we show that the important class of optimal switching problems under partial observation is covered by our methodology and we exploit specific model features to achieve simple algorithmic form of a numerical solution

relaxing dynamic programming the idea of dynamic programming is general and very simple but the curse of dimensionality is often prohibitive and restricts the fields of application this paper introduces a method to reduce the complexity by relaxing the demand for optimality the distance from optimality is kept within prespecified bounds and the size of the bounds determines the computational complexity several computational examples are considered the first is optimal switching between linear systems with application to design of a dc dc voltage converter the second is optimal control of a linear system with piecewise linear cost with application to stock order control finally the method is applied to a partially observable markov decision problem pomdp

optimal cost almost sure reachability in pomdps we consider partially observable markov decision processes pomdps with a set of target states and an integer cost associated with every transition the optimization objective we study asks to minimize the expected total cost of reaching a state in the target set while ensuring that the target set is reached almost surely with probability 1 we show that for integer costs approximating the optimal cost is undecidable for positive costs our results are as follows i we establish matching lower and upper bounds for the optimal cost both double exponential in the pomdp state space size ii we show that the problem of approximating the optimal cost is decidable and present approximation algorithms developing on the existing algorithms for pomdps with finite horizon objectives while the worst case running time of our algorithm is double exponential we also present efficient stopping criteria for the algorithm and show experimentally that it performs well in many examples of interest

symmetries of systems of stochastic differential equations with diffusion matrices of full rank lie point symmetries of a system of stochastic differential equations sdes with diffusion matrices of full rank are considered it is proved that the maximal dimension of a symmetry group admitted by a system of n sdes is n 2 in addition such systems cannot admit symmetry operators whose coefficients are proportional to a nonconstant coefficient of proportionality these results are applied to compute the lie group classification of a system of two sdes the classification is obtained with the help of non equivalent realizations of real lie algebras by fiber preserving vector fields in 1 2 variables possibilities of using symmetries for integration of sdes by quadratures are discussed

approximation algorithms for restless bandit problems the restless bandit problem is one of the most well studied generalizations of the celebrated stochastic multi armed bandit mab problem in decision theory in its ultimate generality the restless bandit problem is known to be pspace hard to approximate to any nontrivial factor and little progress has been made on this problem despite its significance in modeling activity allocation under uncertainty in this article we consider the feedback mab problem where the reward obtained by playing each of n independent arms varies according to an underlying on off markov process whose exact state is only revealed when the arm is played the goal is to design a policy for playing the arms in order to maximize the infinite horizon time average expected reward this problem is also an instance of a partially observable markov decision process pomdp and is widely studied in wireless scheduling and unmanned aerial vehicle uav routing unlike the stochastic mab problem the feedback mab problem does not admit to greedy index based optimal policies we develop a novel duality based algorithmic technique that yields a surprisingly simple and intuitive 2 approximate greedy policy to this problem we show that both in terms of approximation factor and computational efficiency our policy is closely related to the whittle index which is widely used for its simplicity and efficiency of computation subsequently we define a multi state generalization that we term monotone bandits which remains subclass of the restless bandit problem we show that our policy remains a 2 approximation in this setting and further our technique is robust enough to incorporate various side constraints such as blocking plays switching costs and even models where determining the state of an arm is a separate operation from playing it our technique is also of independent interest for other restless bandit problems and we provide an example in nonpreemptive machine replenishment interestingly in this case our policy provides a constant factor guarantee whereas the whittle index is provably polynomially worse by presenting the first o 1 approximations for nontrivial instances of restless bandits as well as of pomdps our work initiates the study of approximation algorithms in both these contexts

planning and acting in partially observable stochastic domains in this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains we begin by introducing the theory of markov decision processes mdps and partially observable mdps pomdps we then outline a novel algorithm for solving pomdps off line and show how in some cases a finite memory controller can be extracted from the solution to a pomdp we conclude with a discussion of how our approach relates to previous work the complexity of finding exact solutions to pomdps and of some possibilities for finding approximate solutions

a bayesian approach for learning and planning in partially observable markov decision processes bayesian learning methods have recently been shown to provide an elegant solution to the exploration exploitation trade off in reinforcement learning however most investigations of bayesian reinforcement learning to date focus on the standard markov decision processes mdps the primary focus of this paper is to extend these ideas to the case of partially observable domains by introducing the bayes adaptive partially observable markov decision processes this new framework can be used to simultaneously 1 learn a model of the pomdp domain through interaction with the environment 2 track the state of the system under partial observability and 3 plan near optimal sequences of actions an important contribution of this paper is to provide theoretical results showing how the model can be finitely approximated while preserving good learning performance we present approximate algorithms for belief tracking and planning in this model as well as empirical results that illustrate how the model estimate and agent s return improve as a function of experience

a fenchel moreau rockafellar type theorem on the kantorovich wasserstein space with applications in partially observable markov decision processes abstract by using the fact that the space of all probability measures with finite support can be completed in two different fashions one generating the arens eells space and another generating the kantorovich wasserstein wasserstein 1 space and by exploiting the duality relationship between the arens eells space with the space of lipschitz functions we provide a dual representation of fenchel moreau rockafellar type for proper convex functionals on wasserstein 1 we retrieve dual transportation inequalities as a corollary and we provide examples where the theorem can be used to easily prove dual expressions like the celebrated donsker varadhan variational formula finally our result allows to write convex functions as the supremum over all linear functions that are generated by roots of its conjugate dual something that we apply to the field of partially observable markov decision processes pomdps to approximate the value function of a given pomdp by iterating level sets this extends the method used in smallwood and sondik 1973 20 for finite state spaces to the case were the state space is a polish metric space

a two state partially observable markov decision process with three actions abstract a process can be in either a stable or an unstable state interchangeably the true state is unobservable and can only be inferred from observations three actions are available continue with the process con repair the process for a certain fee bring the process to the stable state rep and obtain the state of the process for a cost ins the objective is to maximize the expected discounted value of the total future profits we formulate the problem as a discrete time partially observable markov decision process pomdp we show that the expected profit function is convex and strictly increasing and that the optimal policy has either one or two control limits also we show that dominance in expectation the expected revenue is larger in the stable state than in the unstable state suffices for a control limit structure

random lie symmetries of ito stochastic differential equations lie point symmetries of stochastic differential equations sdes which include transformations depending on the brownian motion are considered the corresponding lie group transformations acting in the space of the independent variable time the dependent variables and the brownian motion preserve the differential form of sdes and the properties of the brownian motion such symmetries are called random symmetries the focus of the paper is a definition of the random symmetries and their properties it is shown that the random symmetries form a lie algebra we also investigate relations between the random symmetries and the conserved quantities of sdes for several stochastic differential equations we find their random symmetries and compare them with the deterministic symmetries which correspond to lie group transformations in the space of the independent and dependent variables

construction of approximation spaces for reinforcement learning linear reinforcement learning rl algorithms like least squares temporal difference learning lstd require basis functions that span approximation spaces of potential value functions this article investigates methods to construct these bases from samples we hypothesize that an ideal approximation spaces should encode diffusion distances and that slow feature analysis sfa constructs such spaces to validate our hypothesis we provide theoretical statements about the lstd value approximation error and induced metric of approximation spaces constructed by sfa and the state of the art methods krylov bases and proto value functions pvf in particular we prove that sfa minimizes the average over all tasks in the same environment bound on the above approximation error compared to other methods sfa is very sensitive to sampling and can sometimes fail to encode the whole state space we derive a novel importance sampling modification to compensate for this effect finally the lstd and least squares policy iteration lspi performance of approximation spaces constructed by krylov bases pvf sfa and pca is compared in benchmark tasks and a visual robot navigation experiment both in a realistic simulation and with a robot the results support our hypothesis and suggest that i sfa provides subspace invariant features for mdps with self adjoint transition operators which allows strong guarantees on the approximation error ii the modified sfa algorithm is best suited for lspi in both discrete and continuous state spaces and iii approximation spaces encoding diffusion distances facilitate lspi performance

symmetry classification and exact solutions of the one dimensional fokker planck equation with arbitrary coefficients of drift and diffusion symmetry properties of the one dimensional fokker planck equations with arbitrary coefficients of drift and diffusion are investigated it is proved that the group symmetry of these equations can be one two four or six parametric and corresponding criteria are obtained the changes of the variables reducing fokker planck equations to the heat equation and schrodinger equation with certain potentials are determined using the substructure of the invariance algebra of the rayleigh process equation we obtain the differential invariants and construct some classes of the exact solutions of this equation

contingent planning under uncertainty via stochastic satisfiability we describe a new planning technique that efficiently solves probabilistic propositional contingent planning problems by converting them into instances of stochastic satisfiability ssat and solving these problems instead we make fundamental contributions in two areas the solution of ssat problems and the solution of stochastic planning problems this is the first work extending the planning as satisfiability paradigm to stochastic domains our planner zander can solve arbitrary goal oriented finite horizon partially observable markov decision processes pomdps an empirical study comparing zander to seven other leading planners shows that its performance is competitive on a range of problems

a grid based multilevel incomplete lu factorization preconditioning technique for general sparse matrices we design a grid based multilevel incomplete lu preconditioner gilum for solving general sparse matrices this preconditioner combines a high accuracy ilu factorization with an algebraic multilevel recursive reduction the gilum preconditioner is a compliment to the domain based multilevel block ilut preconditioner a major difference between these two preconditioners is the way that the coarse level nodes are chosen the approach of gilum is analogous to that of algebraic multigrid method the gilum approach avoids some controversial issues in algebraic multigrid method such as how to construct the interlevel transfer operators and how to compute the coarse level operator numerical experiments are conducted to compare gilum with other ilu preconditioners

robust multigrid methods for nonsmooth coefficient elliptic linear systems we survey the literature on robust multigrid methods which have been developed in recent years for solving second order elliptic pdes with nonsmooth coefficients we highlight the key ideas of designing robust multigrid methods which are able to recover the usual multigrid efficiency for nonsmooth coefficient pdes on structured or unstructured grids in particular we shall describe various approaches for constructing the interpolation and the smoothing operators and the coarse grid points selections

multilevel ilu decomposition in this paper the multilevel ilu mlilu decomposition is introduced during an incomplete gaussian elimination process new matrix entries are generated such that a special ordering strategy yields distinct levels on these levels some smoothing steps are computed the mlilu decomposition exists and the corresponding iterative scheme converges for all symmetric and positive definite matrices convergence rates independent of the number of unknowns are shown numerically for several examples many numerical experiments including unsymmetric and anisotropic problems problems with jumping coefficients as well as realistic problems are presented they indicate a very robust convergence behavior of the mlilu method

optimal stochastic switching under convexity assumptions we address a method of approximate calculation of optimal control policy applicable to a particular class of stochastic control problems whose stochastic dynamics exhibit a certain convexity preserving property problems of this type appear in many applications and encompass important examples arising in the area of optimal stopping and in the framework of control based on partial observations utilizing this specific structure we suggest a numerical method which enjoys a number of desirable properties in particular we work out a remarkably strong method for calculation of the value function within our numerically well trackable approach we show a convergence to the value function of the original problem uniformly on compact sets this issue can be of great advantage particularly for high dimensional control problem where the only competing methods from least squares monte carlo family are able to serve merely l p convergence under several restrictions since the presented algorithm is simpl

multifrontal incomplete factorization for indefinite and complex symmetric systems a new class of preconditioners based on the adaptive threshold incomplete multifrontal factorization for indefinite and complex symmetric systems is developed numerical experiments consisting of the 3d helmholtz equations fluid structure interaction and localization problems demonstrate the excellent performance of the preconditioner copyright 2001 john wiley sons ltd

solution procedures for partially observed markov decision processes we present three algorithms to solve the infinite horizon expected discounted total reward partially observed markov decision process pomdp each algorithm integrates a successive approximations algorithm for the pomdp due to a smallwood and e sondik with an appropriately generalized numerical technique that has been shown to reduce cpu time until convergence for the completely observed case the first technique is reward revision the second technique is reward revision integrated with modified policy iteration the third is a standard extrapolation a numerical study indicates the potentially significant computational value of these algorithms

posterior weighted reinforcement learning with state uncertainty reinforcement learning models generally assume that a stimulus is presented that allows a learner to unambiguously identify the state of nature and the reward received is drawn from a distribution that depends on that state however in any natural environment the stimulus is noisy when there is state uncertainty it is no longer immediately obvious how to perform reinforcement learning since the observed reward cannot be unambiguously allocated to a state of the environment this letter addresses the problem of incorporating state uncertainty in reinforcement learning models we show that simply ignoring the uncertainty and allocating the reward to the most likely state of the environment results in incorrect value estimates furthermore using only the information that is available before observing the reward also results in incorrect estimates we therefore introduce a new technique posterior weighted reinforcement learning in which the estimates of state probabilities are updated according to the observed rewards e g if a learner observes a reward usually associated with a particular state this state becomes more likely we show analytically that this modified algorithm can converge to correct reward estimates and confirm this with numerical experiments the algorithm is shown to be a variant of the expectation maximization algorithm allowing rigorous convergence analyses to be carried out a possible neural implementation of the algorithm in the cortico basal ganglia thalamic network is presented and experimental predictions of our model are discussed

efficient learning and planning with compressed predictive states predictive state representations psrs offer an expressive framework for modelling partially observable systems by compactly representing systems as functions of observable quantities the psr learning approach avoids using local minima prone expectation maximization and instead employs a globally optimal moment based algorithm moreover since psrs do not require a predetermined latent state structure as an input they offer an attractive framework for model based reinforcement learning when agents must plan without a priori access to a system model unfortunately the expressiveness of psrs comes with significant computational cost and this cost is a major factor inhibiting the use of psrs in applications in order to alleviate this shortcoming we introduce the notion of compressed psrs cpsrs the cpsr learning approach combines recent advancements in dimensionality reduction incremental matrix decomposition and compressed sensing we show how this approach provides a principled avenue for learning accurate approximations of psrs drastically reducing the computational costs associated with learning while also providing effective regularization going further we propose a planning framework which exploits these learned models and we show that this approach facilitates model learning and planning in large complex partially observable domains a task that is infeasible without the principled use of compression

on maximal lie point symmetry groups admitted by scalar stochastic differential equations it is proved that the lie point symmetry group admitted by a scalar stochastic differential equation sde of order n 3 is at most n 2 dimensional this result supplements those for first and second order sdes maximal dimensionality can be used to carry out lie group classifications lie point group classification of the third order sdes is provided

links between multiplicity automata observable operator models and predictive state representations a unified learning framework stochastic multiplicity automata sma are weighted finite automata that generalize probabilistic automata they have been used in the context of probabilistic grammatical inference observable operator models ooms are a generalization of hidden markov models which in turn are models for discrete valued stochastic processes and are used ubiquitously in the context of speech recognition and bio sequence modeling predictive state representations psrs extend ooms to stochastic input output systems and are employed in the context of agent modeling and planning we present sma ooms and psrs under the common framework of sequential systems which are an algebraic characterization of multiplicity automata and examine the precise relationships between them furthermore we establish a unified approach to learning such models from data many of the learning algorithms that have been proposed can be understood as variations of this basic learning scheme and several turn out to be closely related to each other or even equivalent