on a direct method for the solution of nearly uncoupled markov chains this note is concerned with the accuracy of the solution of nearly uncoupled markov chains by a direct method based on the lu decomposition it is shown that plain gaussian elimination may fail in the presence of rounding errors a modification of gaussian elimination with diagonal pivoting and correction of small pivots is proposed and analyzed it is shown that the accuracy of the solution is affected by two condition numbers associated with aggregation and the coupling respectively

on the lp formulation in measure spaces of optimal control problems for jump diffusions abstract in this short note we formulate a infinite horizon stochastic optimal control problem for jump diffusions of ito levy type as a lp problem in a measure space and prove that the optimal value functions of both problems coincide the main tools are the dual formulation of the lp primal problem which is strongly connected to the notion of sub solution of the partial integro differential equation of hamilton jacobi bellman type associated with the optimal control problem and the krylov regularization method for viscosity solutions

sensitivity of the stationary distribution vector for an ergodic markov chain stationary distribution vectors p sup infinity for markov chains with associated transition matrices t are important in the analysis of many models in the mathematical sciences such as queuing networks input output economic models and compartmental tracer analysis models the purpose of this paper is to provide insight into the sensitivity of p sup infinity to perturbations in the transition probabilities of t and to understand some of the difficulties in computing an accurate p sup infinity the group inverse a sup number of i t is shown to be of fundamental importance in understanding sensitivity or conditioning of p sup infinity the main result shows that if there is a state that is accessible from every other state and the corresponding column of t has no small off diagonal elements then p sup infinity cannot be sensitive to small perturbations in t ecological examples are given a new stable algorithm for calculating a sup number is described

nonlinear optimal control via occupation measures and lmi relaxations we consider the class of nonlinear optimal control problems ocps with polynomial data i e the differential equation state and control constraints and cost are all described by polynomials and more generally for ocps with smooth data in addition state constraints as well as state and or action constraints are allowed we provide a simple hierarchy of lmi linear matrix inequality relaxations whose optimal values form a nondecreasing sequence of lower bounds on the optimal value under some convexity assumptions the sequence converges to the optimal value of the ocp preliminary results show that good approximations are obtained with few moments

an lp approach to dynamic programming principles for stochastic control problems with state constraints abstract we study a class of nonlinear stochastic control problems with semicontinuous cost and state constraints using a linear programming lp approach first we provide a primal linearized problem stated on an appropriate space of probability measures with support contained in the set of constraints this space is completely characterized by the coefficients of the control system second we prove a semigroup property for this set of probability measures appearing in the definition of the primal value function this leads to dynamic programming principles for control problems under state constraints with general bounded costs a further linearized dpp is obtained for lower semicontinuous costs

computing moments of the exit time distribution for markov processes by linear programming we provide a new approach to the numerical computation of moments of the exit time distribution of markov processes the method relies on a linear programming formulation of a process exiting from a bounded domain the lp formulation characterizes the evolution of the process through the moments of the induced occupation measure and naturally provides upper and lower bounds for the exact values of the moments the conditions the moments have to satisfy are derived directly from the generator of the markov process and are not based on some approximation of the process excellent software is readily available because the computations involve finite dimensional linear programs

on a representation of the limit occupational measures set of a control system with applications to singularly perturbed control systems a representation of the limit occupational measures set of a control system in terms of the vector function defining the system s dynamics is established applications in averaging of singularly perturbed control systems are demonstrated

on a class of entire matrix function equations abstract this paper treats a class of linear entire matrix function equations which appear naturally in the study of krei n orthogonal matrix functions necessary and sufficient conditions for solvability are given in terms of root functions of the coefficients

multiscale singularly perturbed control systems limit occupational measures sets and averaging an averaging technique for nonlinear multiscale singularly perturbed control systems is developed issues concerning the existence and structure of limit occupational measures sets generated by such systems are discussed general results are illustrated with special cases

occupation measures for controlled markov processes characterization and optimality for controlled markov processes taking values in a polish space control problems with ergodic cost infinite horizon discounted cost and finite horizon cost are studied each is posed as a convex optimization problem wherein one tries to minimize a linear functional on a closed convex set of appropriately defined occupation measures for the problem these are characterized as solutions of a linear equation asssociated with the problem this characterization is used to establish the existence of optimal markov controls the dual convex optimization problem is also studied

perturbation results for nearly uncoupled markov chains with applications to iterative methods the standard perturbation theory for linear equations states that nearly uncoupled markov chains numcs are very sensitive to small changes in the elements indeed some algorithms such as standard gaussian elimination will obtain poor results for such problems a structured perturbation theory is given that shows that numcs usually lead to well conditioned problems it is shown that with appropriate stopping criteria iterative aggregation disaggregation algorithms will achieve these structured error bounds a variant of gaussian elimination due to grassman taksar and heyman was recently shown by o cinneide to achieve such bounds

on a class of matrix polynomial equations abstract this paper deals with a class of matrix polynomial equations which appears in a natural way in the study of szeg krei n orthogonal matrix polynomials necessary and sufficient conditions for solvability are given in terms of left and right root functions of the coefficients the main result is a discrete version of the main theorem in 3 which concerns entire matrix functions

regenerative analysis and steady state distributions for markov chains we apply regenerative theory to derive certain relations between steady state probabilities of a markov chain these relations are then used to develop a numerical algorithm to find these probabilities the algorithm is a modification of the gauss jordan method in which all elements used in numerical computations are nonnegative as a consequence the algorithm is numerically stable

derivatives and perturbations of eigenvectors for a matrix bf a z whose entries are complex valued functions of a complex variable z results are presented concerning derivatives of an eigenvector bf x z of bf a z associated with a simple eigenvalue lambda z when bf x z is restricted to satisfy a constraint of the form sigma bf x z 1 where sigma is a rather arbitrary scaling function the differentiation formulas lead to a new approach for analyzing the sensitivity of an eigenvector under small perturbations in the underlying matrix finally an application is given which concerns the analysis of a finite markov chain subject to perturbations in the transition probabilities

on a rayleigh ritz refinement technique for nearly uncoupled stochastic matrices abstract an analysis is presented for the convergence of an iterative technique for computing the dominant left eigenvector of a nearly uncoupled nearly completely decomposable stochastic matrix the technique combines the power method and rayleigh ritz refinement to force rapid convergence to the stationary probability vector of the associated markov chain

mayer and optimal stopping stochastic control problems with discontinuous cost we study two classes of stochastic control problems with semicontinuous cost the mayer problem and optimal stopping for controlled diffusions the value functions are introduced via linear optimization problems on appropriate sets of probability measures these sets of constraints are described deterministically with respect to the coefficient functions both the lower and upper semicontinuous cases are considered the value function is shown to be a generalized viscosity solution of the associated hjb system respectively of some variational inequality dual formulations are given as well as the relations between the primal and dual value functions under classical convexity assumptions we prove the equivalence between the linearized mayer problem and the standard weak control formulation counter examples are given for the general framework

exact slow fast decomposition of a class of non linear singularly perturbed optimal control problems via invariant manifolds we study a hamilton jacobi partial differential equation arising in an optimal control problem for an affine non linear singularly perturbed system this equation is solvable iff there exists a special invariant manifold of the corresponding hamiltonian system we obtain exact slow fast decomposition of the hamiltonian system and of the special invariant manifold into slow and fast components we get sufficient conditions for the solvability of the hamiltonian jacobi equation in terms of the reduced order slow submanifold or in the hyperbolic case in terms of a reduced order slow riccati equation on the basis of this decomposition we construct asymptotic expansions of the optimal state feedback optimal trajectory and optimal open loop control in powers of a small parameter

linear programming approach to deterministic long run average problems of optimal control we establish that deterministic long run average problems of optimal control are asymptotically equivalent to infinite dimensional linear programming problems lpps and we establish that these lpps can be approximated by finite dimensional lpps the solutions of which can be used for construction of the optimal controls general results are illustrated with numerical examples

a geometrical characterization of multidimensional hausdorff polytopes with applications to exit time problems we present a formula for the corner points of the multidimensional hausdorff polytopes and show how this result can be used to improve linear programming models for computing e g moments of exit time distributions of diffusion processes specifically we compute the mean exit time of two dimensional brownian motion from the unit square as well as higher moments of the exit time of time space brownian motion i e the two dimensional process composed of a one dimensional wiener process and the time component from a rectangle the corner point formula is complemented by a convergence result which provides the analytical underpinning of the numerical method that we use

a direct projection method for markov chains a direct method based on oblique projections is adapted to compute the stationary distribution vector of a finite markov chain the algorithm can also be used to compute the group inverse of the corresponding generator matrix it is shown how to update the stationary vector and other quantities of interest when one row of the transition probability matrix is modified a gth like variant that appears to compute the stationary probabilities to high relative accuracy is developed

singular perturbations in ergodic control of diffusions ergodic control of a nondegenerate diffusion with two time scales is studied in the limiting case as the time scale separation increases to infinity it is shown that the limit problem is another ergodic control problem for the slow time scale component alone with its dynamics averaged over the controlled invariant probability measures for the fast component these measures in turn can be treated as the effective control variable

on sets of occupational measures generated by a deterministic control system on an infinite time horizon abstract we give a representation for the closed convex hull of the set of discounted occupational measures generated by control state trajectories of a deterministic control system we also investigate the limit behavior of the latter when the discount factor tends to zero and compare it with the limit behavior of the long run time average occupational measures set the novelty of our results is in that we allow the control set dependence on the state variables that make the results to be applicable to differential inclusions

linear programming formulation for optimal stopping problems optimal stopping problems for continuous time markov processes are shown to be equivalent to infinite dimensional linear programs over a space of pairs of measures under very general conditions the measures involved represent the joint distribution of the stopping time and stopping location and the occupation measure of the process until it is stopped these measures satisfy an identity for each function in the domain of the generator which is sufficient to characterize the stochastic process finite dimensional linear programs obtained using markov chain approximations are solved in two examples to illustrate the numerical accuracy of the linear programming formulation

convex computation of the maximum controlled invariant set for polynomial control systems we characterize the maximum controlled invariant mci set for discrete as well as continuous time nonlinear dynamical systems as the solution of an infinite dimensional linear programming problem for systems with polynomial dynamics and compact semialgebraic state and control constraints we describe a hierarchy of finite dimensional linear matrix inequality lmi relaxations whose optimal values converge to the volume of the mci set dual to these lmi relaxations are sum of squares sos problems providing a converging sequence of outer approximations to the mci set the approach is simple and readily applicable in the sense that the approximations are the outcome of a single semidefinite program with no additional input apart from the problem description a number of numerical examples illustrate the approach

convex duality approach to the optimal control of diffusions an alternative to the usual dynamic programming approach to the optimal control of markov processes is considered it is based on duality of convex analysis the control problem is embedded in a convex mathematical programming problem on a space of measures the dual problem is to find the supremum of smooth subsolutions to the hamilton jacobi bellman equation

characterization of the optimal trajectories for the averaged dynamics associated to singularly perturbed control systems abstract the aim of this paper is to study singularly perturbed control systems firstly we provide linearized formulation version for the calculus of the value function associated with the averaged dynamics secondly we obtain necessary and sufficient conditions in order to identify the optimal trajectory of the averaged system

average cost markov decision processes optimality conditions we are concerned in this paper with discrete time markov decision processes mdps with borel state and action spaces x and a respectively and the long run expected average cost criterion when x is a denumerable set man y necessary and or sufficient conditions for the existence of optimal control policies are known however when x is a borel space i e a borel subset of a complete separable metric space most of the available results impose on the mdp very restrictive topological conditions e g compactness and or strong recurrence assumptions such as doeblin s condition see e g 4 9 12 and their references another related work is 7 where we have studied mdps from the viewpoint of the recurrence or ergodicity properties of the state process ln the present paper however we are concerned with the existence of average optimal policies by looking at static optimization problems see condition cs in section 4 related in sorne cases equivalent to the existence of a bounded solution to the so called optimality equation see c4 in section 4 these optimization problems are dual in the sense that under appropriate conditions the existence of an optimal solution to one of the problems implies existence of an optimal

on the rate of convergence of finite difference approximations for bellmans equations with variable coefficients the estimates presented here for parabolic bellman s equations with variable coefficients extend the ones earlier obtained for constant coefficients

stationary distributions and mean first passage times of perturbed markov chains stationary distributions of perturbed finite irreducible discrete time markov chains are intimately connected with the behaviour of associated mean first passage times this interconnection is explored through the use of generalized matrix inverses some interesting qualitative results regarding the nature of the relative and absolute changes to the stationary probabilities are obtained together with some improved bounds ams classification 15a51 60j10 60j20 65f20 65f35

pathwise stochastic optimal control this paper approaches optimal control problems for discrete time controlled markov processes by representing the value of the problem in a dual lagrangian form the value is expressed as an infimum over a family of lagrangian martingales of an expectation of a pathwise supremum of the objective adjusted by the lagrangian martingale term this representation opens up the possibility of numerical methods based on monte carlo simulation which may be advantageous in high dimensional problems or in problems with complicated constraints

a contribution to the theory of condition in this paper the use of the condition number of a problem as defined by rice in 1966 is discussed for the eigenvalue eigenvector and linear least squares problems either condition numbers according to various norms are determined or lower and upper bounds for them are derived

blockwise perturbation theory for nearly uncoupled markov chains and its application abstract let p be the transition matrix of a nearly uncoupled markov chain the states can be grouped into aggregates such that p has the block form p p ij i j 1 k where p ii is square and p ij is small for i j let t be the stationary distribution partitioned conformally as t 1 t k t in this paper we bound the relative error in each aggregate distribution i t caused by small relative perturbations in p ij the error bounds demonstrate that nearly uncoupled markov chains usually lead to well conditioned problems in the sense of blockwise relative error as an application we show that with appropriate stopping criteria iterative aggregation disaggregation algorithms will achieve such structured backward errors and compute each aggregate distribution with high relative accuracy

a criterion for invariant measures of markov processes using the martingale formulation for markov processes introduced by stroock and varadhan we develop a criterion for checking if a measure happens to be invariant

linear programming approach to deterministic infinite horizon optimal control problems with discounting we investigate relationships between the deterministic infinite time horizon optimal control problem with discounting in which the state trajectories remain in a given compact set y and a certain infinite dimensional linear programming idlp problem we introduce the problem dual with respect to this idlp problem and obtain some duality results we construct necessary and sufficient optimality conditions for the optimal control problem under consideration and we give a characterization of the viability kernel of y we also indicate how one can use finite dimensional approximations of the idlp problem and its dual for construction of near optimal feedback controls the construction is illustrated with a numerical example

duality in linear programming problems related to deterministic long run average problems of optimal control it has been established recently that under mild conditions deterministic long run average problems of optimal control are asymptotically equivalent to infinite dimensional linear programming problems lpps and that these lpps can be approximated by finite dimensional lpps in this paper we introduce the corresponding infinite and finite dimensional dual problems and study duality relationships we also investigate the possibility of using solutions of finite dimensional lpps and their duals for numerical construction of the optimal controls in periodic optimization problems the construction is illustrated with a numerical example

suboptimization of singularly perturbed control systems a technique different from the boundary layer method is developed to deal with singularly perturbed optimal control problems the technique is applicable in particular in the case when the optima

averaging method for discontinuous mayer s problem of singularly perturbed control systems abstract in the present paper we prove that the optimal cost of a mayer s problem for a control system with singular perturbation does converge to the optimal cost of a mayer s problem associated with control system obtained by averaging the main novelty of our result lies on the fact that we did not require any continuity assumption on the final cost

limit occupational measures set for a control system and averaging of singularly perturbed control systems abstract we discuss conditions for the existence of the limit occupational measures set for a control system we approximate slow components of the trajectories of a singularly perturbed control system by the solutions of a differential inclusion the differential inclusion is obtained via averaging the slow subsystem over measures from the limit occupational measures set constructed for the associated system describing the fast motions with frozen slow ones

weak convergence to a markov process the martingale approach in this article we obtain some sufficient conditions for weak convergence of a sequence of processes xn tox whenx arises as a solution to a well posed martingale problem these conditions are tailored for application to the case when the state space for the processesxn x is infinite dimensional the usefulness of these conditions is illustrated by deriving donsker s invariance principle for hilbert space valued random variables also continuous dependence of hilbert space valued diffusions on diffusion and drift coefficients is proved

factored forms for solutions of ax xb c and x axb c in companion matrices abstract the main concern of this paper is linear matrix equations with block companion matrix coefficients it is shown that general matrix equations ax xb c and x axb c can be transformed to equations whose coefficients are block companion matrices c l x xc m diag i 0 0 and x c l xc m diag i 0 0 respectively where l and cm stand for the first and second block companion matrices of some monic r r matrix polynomials l si s 1j 0 jlj and m ti t7minus 1j 0 jmj the solution of the equat with block companion coefficients is reduced to solving vector equations sx where the matrix s is r2l r2l l max s t and enjoys some symmetry properties