uniqueness and stability of optimal policies of finite state markov decision processes in this paper we consider infinite horizon discrete time optimal control of markov decision processes mdps with finite state spaces and compact action sets we restrict attention to unicost mdps which form a class that contains all the weakly communicating mdps the unicost mdps are characterized as those mdps for which there exists a solution to the single optimality equation we address the problem of uniqueness and stability of minimizing markov actions our main result asserts that when we endow the set of unicost mdps with a certain natural metric under which it is complete then the class of mdps with essentially unique and stable minimizing markov actions contains the intersection of countably many open dense sets hence is itself dense thus the property of having essentially unique and stable minimizing markov actions is generic for unicost mdps

spectral synthesis for coadjoint orbits of nilpotent lie groups we determine the space of primary ideals in the group algebra l 1 g of a connected nilpotent lie group by identifying for every pi in widehat g the family mathcal i pi of primary ideals with hull pi with the family of invariant subspaces of a certain finite dimensional sub space mathcal p q pi of the space of polynomials mathcal p g on g

functional characterization for average cost markov decision processes with doeblin s conditions abstract this paper is a sequel to kurano 1 2 in which average cost markov decision process mdps with compact state and action spaces have been considered under the hypothesis of doeblin and the existence of optimal stationary policies has been shown the objective of this paper is to give the functional characterization that is we derive optimality inequalities for mdps under the hypothesis of doeblin and prove the validity of them under some reasonable conditions

a call routing problem with service level constraints we consider a queueing system commonly found in inbound telephone call centers that processes two types of work type h jobs arrive at rate h are processed at rate i i h and are served first come first served within class aservice level constraint of the form e delay a orp delay i limits the delay in queue that these jobs may face an infinite backlog of type l jobs awaits processing at rate i l and there is no service level constraint on this type of work a pool ofc identical servers processes all jobs and a system controller must maximize the rate at which type l jobs are processed subject to the service level constraint placed on the type h work we formulate the problem as a constrained average cost markov decision process and determine the structure of effective routing policies when the expected service times of the two classes are the same these policies are globally optimal and the computation time required to find the optimal policy is about that required to calculate the normalizing constant for a simplem m c system when the expected service times of the two classes differ the policies are optimal within the class of priority policies and the determination of optimal policy parameters can be determined through the solution of a linear program with o c3 variables and o c2 constraints

multiplicative markov decision chains previous treatments of multiplicative markov decision chains eg bellman bellman r 1957 dynamic programming princeton university press princeton new jersey mandl mandl p 1967 an iterative method for maximizing the characteristic root of positive matrices rev roumaine math pures appl xii 1317 1322 and howard and matheson howard r a matheson j e 1972 risk sensitive markov decision processes management sci 8 356 369 restricted attention to stationary policies and assumed that all transition matrices are irreducible and aperiodic they also used a first term optimality criterion namely maximizing the spectral radius of the associated transition matrix we give a constructive proof of the existence of optimal policies among all policies under new cumulative average optimality criteria which are more sensitive than the maximization of the spectral radius the algorithm for finding an optimal policy first searches for a stationary policy with a nonnilpotent transition matrix provided such a rule exists otherwise the method still finds an optimal policy though in this case the set of optimal policies usually does not contain a stationary policy if a stationary policy with a nonnilpotent transition matrix exists then we develop a policy improvement algorithm which finds a stationary optimal policy

a note on controlled diffusions on line with time averaged cost existence of stable optimal markov controls is established for a class of cost functions for controlled one dimensional diffusions with time averaged cost

a variance minimization problem for a markov decision process abstract in the steady state of a discrete time markov decision process we consider the problem to find an optimal randomized policy that minimizes the variance of the reward in a transition among the policies which give the mean not less than a specified value the problem is solved by introducing a parametric markov decision process with average cost criterion it is shown that there exists an optimal policy which is a mixture of at most two pure policies as an application the toymaker s problem is discussed

action time sharing policies for ergodic control of markov chains ergodic control for discrete time controlled markov chains with a locally compact state space and a compact action space is considered under suitable stability irreducibility and feller continuity conditions a flexible family of controls called action time sharing ats policies associated with a given continuous stationary markov control is introduced it is shown that the long term average cost for such a control policy for a broad range of one stage cost functions is the same as that for the associated stationary markov policy in addition ats policies are well suited for a range of estimation information collection and adaptive control goals to illustrate the possibilities we present two examples the first demonstrates a construction of an ats policy that leads to consistent estimators for unknown model parameters while producing the desired long term average cost value the second example considers a setting where the target stationary markov control q is not known but there are sampling schemes available that allow for consistent estimation of q we construct an ats policy which uses dynamic estimators for q for control decisions and show that the associated cost coincides with that for the unknown markov control q

matrix semigroups with constant spectral radius abstract multiplicative matrix semigroups with constant spectral radius c s r are studied and applied to several problems of algebra combinatorics functional equations and dynamical systems we show that all such semigroups are characterized by means of irreducible ones each irreducible c s r semigroup defines walks on euclidean sphere all its nonsingular elements are similar in the same basis to orthogonal we classify all nonnegative c s r semigroups and arbitrary low dimensional semigroups for higher dimensions we describe five classes and leave an open problem on completeness of that list the problem of algorithmic recognition of c s r property is proved to be polynomially solvable for irreducible semigroups and undecidable for reducible ones

a convex analytic framework for ergodic control of semi markov processes the ergodic control problem for semi markov processes is reformulated as an optimization problem over the set of suitably defined ergodic occupation measures this set is shown to be closed and convex with its extreme points corresponding to stationary strategies this leads to the existence of optimal stationary strategies under additional hypotheses a pathwise analysis of the joint empirical occupation measures of the state and control processes shows that this optimality is in the strong i e almost sure sense

denumerable constrained markov decision processes and finite approximations the purpose of this paper is two fold first to establish the theory of discounted constrained markov decision processes with a countable state and action spaces with general multi chain structure second to introduce finite approximation methods we define the occupation measures and obtain properties of the set of all achievable occupation measures under the different admissible policies we establish the optimality of stationary policies for the constrained control problem and obtain an lp with a countable number of decision variables through which stationary optimal policies are computed since for such an lp one cannot expect to find an optimal solution in a finite number of operations we present two schemes for finite approximations and establish the convergence of optimal values and policies for both the discounted and the expected average cost with unbounded cost sometimes it turns out to be easier to solve the problem with infinite state space than the problem with finite yet large state space based on the optimal policy for the problem with infinite state space we construct policies which are almost optimal for the problem with truncated state space this method is applied to obtain an e optimal policy for a problem of optimal priority assignment under constraints for a system of k finite queues

on stationary strategies in borel dynamic programming we consider a discrete time markov decision model with borel state and action spaces it is proved that the supremum of the expected total rewards under all randomized stationary strategies is equal to the supremum of these rewards under all nonrandomized stationary strategies

controlled markov chains and stochastic networks controlled markov chains with average cost criterion and with special cost and transition structures are studied existence of optimal stationary strategies is established for the average cost criterion corresponding dynamic programming equations are derived a stochastic network problem that includes interconnected queues as a special case is described and studied within this framework

continuity of optimal values and solutions for control of markov chains with constraints we consider in this paper constrained markov decision processes this type of control model has many applications in telecommunications and other fields e altman and a shwartz ieee trans automat control 34 1989 pp 1089 1102 e a feinberg and m i reiman probab engrg inform sci 8 1994 pp 463 489 a hordijk and f spieksma adv in appl probab 21 1989 pp 409 431 a lazar ieee trans automat control 28 1983 pp 1001 1007 p nain and k w ross ieee trans automat control 31 1986 pp 883 888 k w ross and b chen ieee trans automat control 33 1988 pp 261 267 we address the issue of the convergence of the value and optimal policies of the problem with discounted costs to the ones for the problem with expected average cost we consider the general multichain ergodic structure we present two stability results in this paper we establish the continuity of optimal values and solutions of as well as some type of robustness of some suboptimal solutions in the discount factor our proof relies on same general theory on continuity of values and solutions in convex optimization that relies on well known notions of gamma convergence

an algorithm to identify and compute average optimal policies in multichain markov decision processes this paper concerns discrete time finite state multichain mdps with compact action sets the optimality criterion is long run average cost simple examples illustrate that optimal stationary markov policies do not always exist we establish the existence of e optimal policies that are stationary markovian and develop an algorithm that computes these approximate optimal policies we establish a necessary and sufficient condition for the existence of an optimal policy that is stationary markovian and in case that such an optimal policy exists the algorithm computes it

on the empirical state action frequencies in markov decision processes under general policies we consider the empirical state action frequencies and the empirical reward in weakly communicating finite state markov decision processes under general policies we define a certain polytope and establish that every element of this polytope is the limit of the empirical frequency vector under some policy in a strong sense furthermore we show that the probability of exceeding a given distance between the empirical frequency vector and the polytope decays exponentially with time under every policy we provide similar results for vector valued empirical rewards

the average cost of markov chains subject to total variation distance uncertainty abstract this paper addresses the problem of controlling a markov chain so as to minimize the long run expected average cost per unit time when the invariant distribution is unknown but we know it belongs to a given uncertain set the mathematical model used to describe this set is the total variation distance uncertainty we show that the equilibrium control policy which yields higher probability to the states with low cost and lower probability to the states with the high cost is an optimal control policy that minimizes the average cost recognition of such a policy may be of value in practical situations with constraints consistent to those studied here when the invariant distribution is uncertain and deriving online an optimal control policy is required

markov decision processes with a minimum variance criterion abstract we consider the optimization of the variance of the sum of costs as well as that of an average expected cost in markov decision processes with unbounded cost in case of general state and action space we find the stationary policy which makes the average variance as small as possible in the class of policies which are e optimal in an average expected cost

discrete time controlled markov processes with average cost criterion a survey this work is a survey of the average cost control problem for discrete time markov processes the authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades the exposition ranges from finite to borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies the authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography the authors have also identified several important questions that are still open to investigation

denumerable undiscounted semi markov decision processes with unbounded rewards this paper establishes the existence of a solution to the optimality equations in undis counted semi markov decision models with countable state space under conditions generalizing the hitherto obtained results in particular we merely require the existence of a finite set of states in which every pair of states can reach each other via some stationary policy instead of the traditional and restrictive assumption that every stationary policy has a single irreducible set of states a replacement model and an inventory model illustrate why this extension is essential our approach differs fundamentally from classical approaches we convert the optimality equations into a form suitable for the application of a fixed point theorem

multichain markov decision processes with a sample path constraint a decomposition approach we consider finite state finite action markov decision processes which accumulate both a reward and a cost at each decision epoch we study the problem of finding a policy that maximizes the expected long run average reward subject to the constraint that the long run average cost be no greater than a given value with probability one we establish that if there exists a policy that meets the constraint then there exists an e optimal stationary policy furthermore an algorithm is outlined to locate the e optimal stationary policy the proof of the result hinges on a decomposition of the state space into maximal recurrent classes and a set of transient states

value iteration in a class of communicating markov decision chains with the average cost criterion markov decision processes with denumerable state space and discrete time parameter are considered the performance index of a control policy is the lim sup expected average cost criterion and the the main structural restrictions on the model are the following i under the action of any stationary policy the state space is a communicating class ii the cost function has an almost monotone or penalized structure v s borkar siam j control optim 21 1983 pp 652 666 22 1984 pp 965 978 and iii some stationary policy induces an ergodic chain with finite average cost in this context it is shown that the value iteration scheme can be used to construct convergent approximations of a solution to the optimality equation as well as a sequence of stationary policies whose limit points are optimal

a gel fand type spectral radius formula and stability of linear constrained switching systems abstract using ergodic theory in this paper we present a gel fand type spectral radius formula which states that the joint spectral radius is equal to the generalized spectral radius for a matrix multiplicative semigroup s restricted to a subset that need not carry the algebraic structure of s this generalizes the berger wang formula using it as a tool we study the absolute exponential stability of a linear switched system driven by a compact subshift of the one sided markov shift associated to s

adaptive control of constrained markov chains adaptive control of finite state markov chains is discussed the optimal performance is characterized through the minimization of a long run average cost functional subject to constraints on several other such functionals under mild structural and feasibility conditions two explicit adaptive control policies are exhibited for the case where the transition probabilities are unknown the policies are optimal under the constrained optimization criterion they rely on a powerful estimation scheme which provides consistent estimators for the transition probabilities this scheme is of independent interest as it provides strong consistency under a large number of adaptive schemes and is independent of any identifiability conditions as an application an optimal adaptive policy is derived for a system of k competing queues with countable state space for which the constrained criteria arise naturally in the context of communication networks

optimality of mixed policies for average continuous time markov decision processes with constraints this article concerns the average criteria for continuous time markov decision processes with n constraints we show the following a every extreme point of the space of performance vectors corresponding to the set of stable measures is generated by a deterministic stationary policy and b there exists a mixed optimal policy where the mixture is over no more than n 1 deterministic stationary policies

constrained undiscounted stochastic dynamic programming in this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function the state space and action sets are assumed to be finite constraints which are linear functions of the state action frequencies are allowed in the general multichain case an optimal policy will be a randomized nonstationary policy an algorithm to compute such an optimal policy is presented furthermore sufficient conditions for optimal policies to be stationary are derived there are many applications for constrained undiscounted stochastic dynamic programming e g in multiple objective markovian decision models

a note on the convergence rate of the value iteration scheme in controlled markov chains this work is concerned with controlled markov chains with bounded costs assuming that the transition probabilities satisfy a simultaneous doeblin condition it is shown that schweitzer s transformation on the transition law yields a strong ergodicity condition that implies that the solution to the average cost optimality equation can be approximated at a geometric rate via the value iteration scheme

constrained discounted dynamic programming this paper deals with constrained optimization of markov decision processes with a countable state space compact action sets continuous transition probabilities and upper semicontinuous reward functions the objective is to maximize the expected total discounted reward for one reward function under several inequality constraints on similar criteria with other reward functions suppose a feasible policy exists for a problem with m constraints we prove two results on the existence and structure of optimal policies first we show that there exists a randomized stationary optimal policy which requires at most m actions more than a nonrandomized stationary one this result is known for several particular cases second we prove that there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is at most m iii the total number of actions that are added by nonstationarity is at most m we also establish pareto optimality of policies from the two classes described above for multi criteria problems we describe an algorithm to compute optimal policies with properties i iii for constrained problems the policies that satisfy properties i iii have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints in contrast a randomized stationary policy may require an infinite number of randomizations over time

sets of matrices all infinite products of which converge an infinite product iit lmi of matrices converges on the right if limi m mi exists a set z ai i l of n x n matrices is called an rcp set rightconvergent product set if all infinite products with each element drawn from z converge such sets of matrices arise in constructing self similar objects like von koch s snowflake curve in various interpolation schemes in constructing wavelets of compact support and in studying nonhomogeneous markov chains this paper gives necessary conditions and also some sufficient conditions for a set x to be an rcp set these are conditions on the eigenvalues and left eigenspaces of matrices in 2 and finite products of these matrices necessary and sufficient conditions are given for a finite set z to be an rcp set having a limit function m d rit lad where d d d which is a continuous function on the space of all sequences d with the sequence topology finite rcp sets of column stochastic matrices are completely characterized some results are given on the problem of algorithmically deciding if a given set x is an rcp set

a convex analytic approach to markov decision processes this paper develops a new framework for the study of markov decision processes in which the control problem is viewed as an optimization problem on the set of canonically induced measures on the trajectory space of the joint state and control process this set is shown to be compact convex one then associates with each of the usual cost criteria infinite horizon discounted cost finite horizon control up to an exit time a naturally defined occupation measure such that the cost is an integral of some function with respect to this measure these measures are shown to form a compact convex set whose extreme points are characterized classical results about existence of optimal strategies are recovered from this and several applications to multicriteria and constrained optimization problems are briefly indicated

another set of conditions for average optimality in markov control processes abstract the existence of an average cost optimal stationary policy in a countable state markov decision chain is shown under assumptions weaker than those given by sennott 1989 this treatment is a modification of that given by hu 1992 and is related to conditions of hordijk 1977 an example is given for which the new axiom set holds whereas the axiom set of sennott 1989 fails to hold

on the solvability of bellman s functional equations for markov renewal programming abstract the functional equations of undiscounted stationary infinite horizon markov renewal programming are shown to possess a solution by an elementary application of the leray schauder fixed point theorem

martingale limit theorem and its application to an ergodic controlled markov chain abstract slightly different sufficient conditions for the martingale limit theorem of loeve 1978 are studied for an adapted sequence of random variables xi f i i 1 as an application of this form of the martingale limit theorem existence of an optimal control for an ergodic controlled markov chain with long run average cost criterion is established under a weaker condition than borkar s

achieving target state action frequencies in multichain average reward markov decision processes in this paper we address a basic problem that arises naturally in average reward markov decision processes with constraints and or nonstandard payoff criteria given a feasible state action frequency vector the target construct a policy whose state action frequencies match those of the target vector while it is well known that the solution to this problem cannot in general be found in the space of stationary randomized policies we construct a solution that has ultimately stationary structure it consists of two stationary policies where the first one is used initially and then the switch to the second one is made at a certain random switching time the computational effort required to construct this solution is minimal we also show that our problem can always be solved by a stationary policy if the original mdp is extended by adding certain states and actions the solution in the original mdp is obtained by mapping the solution in the extended mdp back to the original process

on the second optimality equation for semi markov decision models for a semi markov decision model with average return the validity of the second optimality equation is shown in the nonmodified form where the actions run through the set of all admissible actions rather than through the set of maximum points conserving actions for the first optimal equation as a consequence the existence of a strongly optimal stationary policy is shown the results seem to be known only for finite state finite action models whereas here countable state compact action models with unbounded rewards are considered

corrigendum addendum to sets of matrices all infinite products of which converge abstract this corrigendum addendum supplies corrected statements and proofs of some results in our paper appearing in linear algebra appl 161 1992 227 263 these results concern special kinds of bounded semigroups of matrices it also reports on progress on the topics of this paper made in the last eight years

stability and singular perturbations in constrained markov decision problems constrained markov decision problems cmdps with the average cost criterion and a single ergodic chain or the discounted cost with a general multichain structure are considered conditions for stability of the optimal value and control to changes of the parameters of the problem such as immediate costs transition probabilities and the discount factor are established singular constrained problems for which the optimal value and controls exhibit discontinuities are studied

ergodic control of multidimensional diffusions 1 the existence results the existence of optimal stable markov relaxed controls for the ergodic control of multidimensional diffusions is established by direct probabilistic methods based on a characterization of a s limit sets of empirical measures the optimality of the above is established in the strong i e almost sure sense among all admissible controls under very general conditions

a brouwer fixed point mapping approach to communicating markov decision processes abstract the solvability of the functional equations of communicating markov decision processes is demonstrated by an elementary application of the brouwer fixed point mapping theorem the technical assumptions include finite state space and finite or compact action spaces

average sensitive and blackwell optimal policies in denumerable markov decisionchains with unbounded rewards in this paper we consider a discrete time markov decision chain with a denumerable state space and compact action sets and we assume that for all states the rewards and transition probabilities depend continuously on the actions the first objective of this paper is to develop an analysis for average optimality without assuming a special markov chain structure in doing so we present a set of conditions guaranteeing average optimality which are automatically fulfilled in the finite state and action model the second objective is to study simultaneously average and discount optimality as veinott veinott a f jr 1969 on discrete dynamic programming with sensitive discount optimality criteria ann math statist 40 1635 1660 did for the finite state and action model we investigate the concepts of n discount and blackwell optimality in the denumerable state space using a laurent series expansion for the discounted rewards under the same condition as for average optimality we establish solutions to the n discount optimality equations for every n

extremum problems with total variation distance and their applications the aim of this paper is to investigate extremum problems with pay off being the total variation distance metric defined on the space of probability measures subject to linear functional constraints on the space of probability measures and vice versa that is with the roles of total variation metric and linear functional interchanged utilizing concepts from signed measures the extremum probability measures of such problems are obtained in closed form by identifying the partition of the support set and the mass of these extremum measures on the partition the results are derived for abstract spaces specifically complete separable metric spaces known as polish spaces while the high level ideas are also discussed for denumerable spaces endowed with the discrete topology these extremum problems often arise in many areas such as approximating a family of probability distributions by a given probability distribution maximizing or minimizing entropy subject to total variation distance metric constraints quantifying uncertainty of probability distributions by total variation distance metric stochastic minimax control and in many problems of information decision theory and minimax theory

a note on the vanishing interest rate approach in average markov decision chains with continuous and bounded costs abstract this work considers denumerable state markov decision processes with discrete time parameter the performance of a control policy is measured by the lim sup expected average cost criterion the action sets are compact metric and the cost function is continuous and bounded within this framework necessary and sufficient conditions are given so that the vanishing interest rate vir method also known as the vanishing discount effect approach yields an average optimal stationary policy

the generalized spectral radius and extremal norms the generalized spectral radius also known under the name of joint spectral radius or after taking logarithms maximal lyapunov exponent of a discrete inclusion is examined we present a new proof for a result of barabanov which states that for irreducible sets of matrices an extremal norm always exists this approach lends itself easily to the analysis of further properties of the generalized spectral radius we prove that the generalized spectral radius is locally lipschitz continuous on the space of compact irreducible sets of matrices and show a strict monotonicity property of the generalized spectral radius sufficient conditions for the existence of extremal norms are obtained

control of markov chains with long run average cost criterion the dynamic programming equations the long run average cost control problem for discrete time markov chains on a countable state space is studied in a very general framework necessary and sufficient conditions for optimality in terms of the dynamic programming equations are given when an optimal stable stationary strategy is known to exist e g for the situations studied in stochastic differential systems stochastic control theory and applications ima vol math app 10 springer verlag new york berlin 1988 pp 57 77 a characterization of the desired solution of the dynamic programming equations is given in a special case also included is a novel convex analytic argument for deducing the existence of an optimal stable stationary strategy when that of a randomized one is known

growth optimality for branching markov decision chains this paper considers a multiplicative process called branching markov decision chains in which the output at the end of the nth period equals the product of n nonnegative matrices chosen at the beginning of periods 1 n respectively times a positive fixed terminal reward vector it is assumed that the above transition matrices are drawn out of a finite set of matrices given in product form i e the rows of the matrices can be selected independently out of finite sets of nonnegative row vectors for each coordinate s we define the geometric and algebraic growth rates respectively of the sth coordinate of the stream of output these growth rates are defined so that the magnitude of the corresponding sequence is of the order nnk where is the geometric growth rate and k is the algebraic growth rate the main result of this paper is the constructive establishment of the existence of a transition matrix whose repeated use will guarantee for each coordinate the achievement of the best geometri

on minimum cost per unit time control of markov chains the minimum cost per unit time control problem is studied for a class of markov chains that though important in applications does not fit the conventional framework for this problem existence of optimal stationary strategies and necessary and sufficient conditions for optimality are established

irreducible semigroups with multiplicative spectral radius abstract irreducible multiplicative semigroups of finite dimensional linear operators with some additional properties are studied each of the properties submultiplicativity of the spectral radius its multiplicativity or is constancy as well as a certain property called the rota condition implies that every member of such a semigroup is except for a scalar coefficient similar to the direct sum of an isometry and a nilpotent operator

markov decision problems and state action frequencies consider a controlled markov chain with countable state and action spaces basic quantities that determine the values of average cost functionals are identified under some regularity conditions these turn out to be a collection of numbers one for each state action pair describing for each state the relative number of uses of each action these conditional frequencies which are defined pathwise are shown to determine the state action frequencies that in the finite case are known to determine the costs this is extended to the countable case allowing for unbounded costs the space of frequencies is shown to be compact and convex and the extreme points are identified with stationary deterministic policies conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given these problems include the standard markov decision process as well as constrained optimization both in terms of average cost functionals and variability sensitive optimization an application to a queueing problem is given where these results imply the existence and explicit computation of optimal policies in constrained optimization problems the pathwise definition of the conditional frequencies implies that their values can be controlled directly moreover they depend only on the limiting behavior of the control this has immediate application to adaptive control of markov chains including adaptive control under constraints

on the hilbert schmidt semi norms ofl 1 of a nilpotent lie group in 2 appears the convolution product o f f with f on the non abelian group g combined with the abelian fourier transform on r a consequence of this mixture of non abelian with abelian operations is the fact that the integrand f f o exp need not be nonnegative this prohibits the extension of 2 to general ll functions it makes it also difficult to give estimations of ih f la s in terms o f i 7 it would be much better if one could replace 2 by

optimal controls for diffusion in rd a min max max min formula for the minimal cost growth rate we study the infinite horizon optimal control problem for a system whose dynamics are described by the stochastic differential equation dx wu u dt 4x da x0 given 1 1 let e be the o algebra generated by j jo g s g r then an admissible control is an t adapted process in r which satisfies the constraint u e wx t 2 0 in the last relation x u x is a measurable set valued mapping from rd to r to every control corresponds the cost flow

variability sensitive markov decision processes considered are time average markov decision processes mdps with finite state and action spaces two definitions of variability are introduced namely the expected time average variability and time average expected variability the two criteria are in general different although they can both be employed to penalize for variance in the stream of rewards for communicating mdps we construct a randomized stationary policy that is e optimal for both criteria the policy is optimal and pure for a specific variability function for general multichain mdps a state space decomposition leads to a similar result for the expected time average variability we also consider the problem of the decision maker choosing the initial state along with the policy

guaranteed performance regions in markovian systems with competing decision makers a decision maker is facing a dynamic system which is being controlled by himself herself as well as by other decision makers he she considers a vector of performance measures acceptable performance is defined through a set in the space of performance vectors whether this decision maker can guarantee a time averaged performance vector which approaches this desired set is considered for the worst case scenario in which other decision makers may for selfish reasons try to exclude his her vector from the desired set for a controlled markov model of the system a sufficient condition for approachability is given and appropriate control strategies are constructed under certain recurrence conditions a complete characterization of approachability is then provided for convex sets the mathematical formulation leads to a theory of approachability for stochastic games with vector payoffs a simple queuing example illustrates this approach

the generalized spectral radius is strictly increasing using a result linking convexity and irreducibility of matrix sets it is shown that the generalized spectral radius of a compact set of matrices is a strictly increasing function of the set in a very natural sense as an application some consequences of this property in the area of time varying stability radii are discussed in particular using the implicit function theorem sufficient conditions for lipschitz continuity are derived an example is presented of a linearly increasing family of matrix polytopes for which the proximal subgradient of the generalized spectral radius at a certain polytope contains 0 so that the implicit function theorem is not applicable in all cases

overtaking and almost sure optimality for infinite horizon markov decision processes we consider infinite horizon optimal control of markov chains on complete metric spaces we employ the overtaking optimality criterion which is either applied to the expected cost flow or to the individual sample paths yielding almost sure optimality results we use the existence of a solution pair to the optimality equation l x to establish and characterize optimal strategies for finite state spaces we derive sufficient as well as necessary conditions for overtaking optimality

optimal kullback leibler aggregation via spectral theory of markov chains this paper is concerned with model reduction for complex markov chain models the kullback leibler divergence rate is employed as a metric to measure the difference between the markov model and its approximation for a certain relaxation of the bi partition model reduction problem the solution is shown to be characterized by an associated eigenvalue problem the form of the eigenvalue problem is closely related to the markov spectral theory for model reduction this result is the basis of a heuristic proposed for the m ary partition problem resulting in a practical recursive algorithm the results are illustrated with examples

time sharing policies for controlled markov chains we propose a class of nonstationary policies called policy time sharing pts which possesses several desirable properties for problems where the criteria are of the average cost type an optimal policy exists within this class the computation of optimal policies is straightforward and the implementation of this policy is easy while in the finite state case stationary policies are also known to share these properties the new policies are much more flexible in the sense that they can be applied to solve adaptive problems and they suggest new ways to incorporate the particular structure of the problem at hand into the derivation of optimal policies in addition they provide insight into the pathwise structure of controlled markov chains to use pts policies one alternates between the use of several stationary deterministic policies switching when reaching some predetermined state in some countable state cases optimal solutions of the pts type are available and easy to compute whereas optimal stationary policies are not available examples that illustrate the last point and the usefulness of the new approach are discussed involving constrained optimization problems with countable state space or compact action space

variance penalized markov decision processes we consider a markov decision process with both the expected limiting average and the discounted total return criteria appropriately modified to include a penalty for the variability in the stream of rewards in both cases we formulate appropriate nonlinear programs in the space of state action frequencies averaged or discounted whose optimal solutions are shown to be related to the optimal policies in the corresponding variance penalized mdp the analysis of one of the discounted cases is facilitated by the introduction of a cartesian product of two independent mdps

sample path average optimality for markov control processes the authors consider a markov control process with borel state and actions spaces unbounded costs and under the long run sample path average cost criterion they prove that under very weak assumptions on the transition law and a moment assumption for the one step cost there exists a stationary policy with invariant probability distribution v that is sample path average cost optimal for v almost all initial states in addition every expected average cost optimal stationary policy is in fact liminf sample path average cost optimal and strongly expected average cost optimal

on matrix semigroups bounded above and below abstract an irreducible norm closed semigroup of complex matrices is simultaneously similar to a semigroup of partial isometries if and only if a the norms of all nonzero members of it are uniformly bounded above and below and b its idempotents commute this is a generalization of the well known result on bounded groups

the existence of minimum pair of state and policy for markov decision processes under the hypothesis of doeblin this paper studies the average cost markov decision process with compact state and action spaces and bounded lower semicontinuous cost functions following the idea of borkar s excellent papers siamj control optim 21 1983 pp 652 666 22 1984 pp 965 978 the general case where irreducibility is not assumed is considered under the hypothesis of doeblin and the existence of a minimum pair of state and policy which attains the infimum of the average expected cost over all initial states and policies is established further it is proved that under additional weak conditions there exists an optimal stationary policy in the usual sense

strong 1 optimal stationary policies in denumerable markov decision processes abstract usual conditions for existence of stationary average optimal policies in denumerable mdps with general bounded rewards are shown to be also sufficient for strong 1 optimality moreover we prove that all limit points of discounted optimal stationary policies when the discount factor goes to 1 are strong 1 optimal

bounded semigroups of matrices in this note are proved two conjectures of daubechies and lagarias the first asserts that if z is a bounded set of matrices such that all left infinite products converge then 8 generates a bounded semigroup the second asserts the equality of two differently defined joint spectral radii for a bounded set of matrices one definition involves the conventional spectral radius and one involves the operator norm

on the convergence of policy iteration in finite state undiscounted markov decision processes the unichain case we study the convergence of policy iteration for the undiscounted finite state discrete time markov decision problem with compact action space and unichain transition structure using a newton method type representation for policy iteration we establish the existence of a solution to the optimality equation we show that to find an average optimal policy it is sufficient to solve the optimality equation on the recurrent set of the maximizing policy under the additional assumption of a unique maximizing policy at each stage of the policy iteration procedure we show that the iterates are convergent and the resulting policy is blackwell optimal