statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

convergence of controlled models and finite state approximation for discounted continuous time markov decision processes with constraints abstract in this paper we consider the convergence of a sequence m n of the models of discounted continuous time constrained markov decision processes mdp to the limit one denoted by m for the models with denumerable states and unbounded transition rates under reasonably mild conditions we prove that the constrained optimal policies and the optimal values of m n converge to those of m respectively using a technique of occupation measures as an application of the convergence result developed here we show that an optimal policy and the optimal value for countable state continuous time mdp can be approximated by those of finite state continuous time mdp finally we further illustrate such finite state approximation by solving numerically a controlled birth and death system and also give the corresponding error bound of the approximation

constrained optimization for average cost continuous time markov decision processes to the best of our knowledge this note is a first study on a multiconstrained optimization problem for average cost continuous time markov decision processes in polish spaces the optimality criterion is the long run expected average cost and the constraints are imposed on similar average costs we give suitable conditions under which the existence of a constrained optimal stationary policy is shown by introducing occupation measures and using a martingale technique also we illustrate our conditions with controlled potlach processes

on reducing a constrained gradual impulsive control problem for a jump markov model to a model with gradual control only in this paper we consider a gradual impulsive control problem for continuous time markov decision processes ctmdps with total cost criteria and constraints we develop a simple and useful method

constrained markov decision models with weighted discounted rewards this paper deals with constrained optimization of markov decision processes both objective function and constraints are sums of standard discounted rewards but each with a different discount factor such models arise e g in production and in applications involving multiple time scales we prove that it a feasible policy exists then there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is bounded by the number of constraints optimality of such policies for multi criteria problems is also established these new policies have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints this result is new even for constrained optimization with a single discount factor where the optimality of randomized stationary policies is known however a randomized stationary policy may req

optimal risk probability for first passage models in semi markov decision processes this paper studies the risk minimization problem in semi markov decision processes with denumerable states the criterion to be optimized is the risk probability or risk function that a first passage time to some target set doesn t exceed a threshold value we first characterize such risk functions and the corresponding optimal value function and prove that the optimal value function satisfies the optimality equation by using a successive approximation technique then we present some properties of optimal policies and further give conditions for the existence of optimal policies in addition a value iteration algorithm and a policy improvement method for obtaining respectively the optimal value function and optimal policies are developed finally two examples are given to illustrate the value iteration procedure and essential characterization of the risk function

universal formats for nonlinear ordinary differential systems it is shown that very general nonlinear ordinary differential systems embracing all that arise in practice may first be brought down to polynomial systems where the nonlinearities occur only as polynomials in the dependent variables by introducing suitable new variables into the original system second that polynomial systems are reducible to riccati systems where the nonlinearities are quadratic at most third that riccati systems may be brought to elemental universal formats containing purely quadratic terms with simple arrays of coefficients that are all zero or unity the elemental systems have representations as novel types of matrix riccati equations different starting systems and their associated riccati systems differ from one another at the final elemental level in order and in initial data but not in format

nonatomic total rewards markov decision processes with multiple criteria we consider a markov decision process with an uncountable state space for which the vector performance functional has the form of expected total rewards under the single condition that initial distribution and transition probabilities are nonatomic we prove that the performance space coincides with that generated by nonrandomized markov policies we also provide conditions for the existence of optimal policies when the goal is to maximize one component of the performance vector subject to inequality constraints on other components we illustrate our results with examples of production and financial problems

existence without explicit compactness in stochastic dynamic programming a new kind of existence result for stochastic dynamic programming without explicit compactness conditions is obtained by working with a new coercivity condition for the reward functions

on ws convergence of product measures a number of fundamental results centered around extensions of prohorov s theorem is proven for the ws topology for measures on a product space these results contribute to the foundations of stochastic decision theory they also subsume the principal results of young measure theory which only considers product measures with a fixed common marginal specializations yield the criterion for relative ws compactness of scha l 1975 the refined characterizations of ws convergence of galda ano and truffert 1997 1998 and a new version of fatou s lemma in several dimensions in a separate nonsequential development a generalization is given of the relative ws compactness criterion of jacod and ma min 1981 new applications are given to the existence of optimal equilibrium distributions over player action pairs in game theory and the existence of most optimistic scenarios in stochastic decision theory

finite horizon semi markov decision processes with application to maintenance systems abstract this paper investigates finite horizon semi markov decision processes with denumerable states the optimality is over the class of all randomized history dependent policies which include states and also planning horizons and the cost rate function is assumed to be bounded below under suitable conditions we show that the value function is a minimum nonnegative solution to the optimality equation and there exists an optimal policy moreover we develop an effective algorithm for computing optimal policies derive some properties of optimal policies and in addition illustrate our main results with a maintenance system

optimal interventions in countable jump markov processes in this paper an intervention refers to an immediate change of the state of the system between interventions the continuous time jump markov process is uncontrollable with natural jump intensities the multicriteria control problem for such a model is considered and the constrained version is investigated with the help of the lagrange multipliers technique all of the theory is illustrated by an example of the optimal control of epidemic with carriers

best uniform rational approximation of on 0 1 a strong error estimate for the uniform rational approximation of x on 0 1 is given and its proof is sketched let e nn x 0 1 denote the minimal approximation error in the uniform norm then it is shown that formule holds true for each 0

optimality of mixed policies for average continuous time markov decision processes with constraints this article concerns the average criteria for continuous time markov decision processes with n constraints we show the following a every extreme point of the space of performance vectors corresponding to the set of stable measures is generated by a deterministic stationary policy and b there exists a mixed optimal policy where the mixture is over no more than n 1 deterministic stationary policies

moduli of smoothness and best approximation of functions with singularities error estimates for approximation of functions 0 x 1 x i 2 x x exp ia x a 0 0 a r are given let e f b lp denote the error of approximation of f by elements from b in the lp metric then it is shown that for polynomial approximation e i pn lp a a n 1p 1 holds true for 1 p 0 i 2 the similar estimates are also valid for the errors of approximation by entire functions of exponential type trigonometric polynomials and periodic splines the proofs are based on exact estimates of the moduli of smoothness of i and a general stechkin type theorem

average continuous control of piecewise deterministic markov processes this paper deals with the long run average continuous control problem of piecewise deterministic markov processes pdmps taking values in a general borel space and with compact action space depending on the state variable the control variable acts on the jump rate and transition measure of the pdmp and the running and boundary costs are assumed to be positive but not necessarily bounded our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete time optimality equation related to the embedded markov chain given by the postjump location of the pdmp our second main result guarantees the existence of a feedback measurable selector for the discrete time optimality equation by establishing a connection between this equation and an integro differential equation our final main result is to obtain some sufficient conditions for the existence of a solution for a discrete time optimality inequality and an ordinary optimal feedback control for the long run average cost using the so called vanishing discount approach two examples are presented illustrating the possible applications of the results developed in the paper

on the first passage g mean variance optimality for discounted continuous time markov decision processes this paper considers the discounted continuous time markov decision processes mdps in borel spaces and with unbounded transition rates the discount factors are allowed to depend on states and actions main attention is concentrated on the set f g of stationary policies attaining a given mean performance g up to the first passage of the continuous time mdp to an arbitrarily fixed target set under suitable conditions we prove the existence of a g mean variance optimal policy that minimizes the first passage variance over the set f g using a transformation technique and also give the value iteration and policy iteration algorithms for computing the g variance value function and a g mean variance optimal policy respectively two examples are analytically solved to demonstrate the application of our results

brief paper towards the optimal control of markov chains with constraints an optimal control problem with constraints is considered on a finite interval for a non stationary markov chain with a finite state space the constraints are given as a set of inequalities the optimal solution existence is proved under a natural assumption that the set of admissible controls is non empty the stochastic control problem is reduced to a deterministic one and it is shown that the optimal solution satisfies the maximum principle moreover it can be chosen within a class of markov controls on the basis of this result an approach to the numerical solution is proposed and its implementation is illustrated by examples

constrained discounted dynamic programming this paper deals with constrained optimization of markov decision processes with a countable state space compact action sets continuous transition probabilities and upper semicontinuous reward functions the objective is to maximize the expected total discounted reward for one reward function under several inequality constraints on similar criteria with other reward functions suppose a feasible policy exists for a problem with m constraints we prove two results on the existence and structure of optimal policies first we show that there exists a randomized stationary optimal policy which requires at most m actions more than a nonrandomized stationary one this result is known for several particular cases second we prove that there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is at most m iii the total number of actions that are added by nonstationarity is at most m we also establish pareto optimality of policies from the two classes described above for multi criteria problems we describe an algorithm to compute optimal policies with properties i iii for constrained problems the policies that satisfy properties i iii have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints in contrast a randomized stationary policy may require an infinite number of randomizations over time

a linear programming formulation for constrained discounted continuous control for piecewise deterministic markov processes this paper deals with the constrained discounted control of piecewise deterministic markov process pdmps in general borel spaces the control variable acts on the jump rate and transition measure and the goal is to minimize the total expected discounted cost composed of positive running and boundary costs while satisfying some constraints also in this form the basic idea is by using the special features of the pdmps to re write the problem via an embedded discrete time markov chain associated to the pdmp and re formulate the problem as an infinite dimensional linear programming lp problem via the occupation measures associated to the discrete time process it is important to stress however that our new discrete time problem is not in the same framework of a general constrained discrete time markov decision process and due to that some conditions are required to get the equivalence between the continuous time problem and the lp formulation we provide in the sequel sufficient conditions for the solvability of the associated lp problem based on a generalization of theorem 4 1 in 8 in appendix a we present the proof of this generalization which we believe is of interest on its own the paper is concluded with some examples to illustrate the obtained results

randomized and relaxed strategies in continuous time markov decision processes one of the goals of this article is to describe a wide class of control strategies which includes the traditional relaxed strategies as well as the so called randomized strategies which appeared earlier only in the framework of semi markov decision processes if the objective is the total expected cost up to the accumulation of jumps then without loss of generality one can consider only markov relaxed strategies under a simple condition the markov randomized strategies are also sufficient an example shows that the mentioned condition is important finally without any conditions the class of so called poisson related strategies is also sufficient in the optimization problems all the results are applicable to the discounted model they may be useful also for the case of long run average cost

optimal finite difference grids and rational approximations of the square root i elliptic problems the main objective of this paper is optimization of second order finite difference schemes for elliptic equations in particular for equations with singular solutions and exterior problems a model problem corresponding to the laplace equation on a semi infinite strip is considered the boundary impedance neumann to dirichlet map is computed as the square root of an operator using the standard three point finite difference scheme with optimally chosen variable steps the finite difference approximation of the boundary impedance for data of given smoothness is the problem of rational approximation of the square root on the operator s spectrum we have implemented zolotarev s optimal rational approx imant obtained in terms of elliptic functions we have also found that a geometrical progression of the grid steps with optimally chosen parameters is almost as good as the optimal approximant for bounded operators it increases from second to exponential the convergence order of the finite difference impedance with the convergence rate proportional to the inverse of the logarithm of the condition number for the case of unbounded operators in sobolev spaces associated with elliptic equations the error decays as the exponential of the square root of the mesh dimension as an example we numerically compute the green function on the boundary for the laplace equation some features of the optimal grid obtained for the laplace equation remain valid for more general elliptic problems with variable coefficients 2000 john wiley sons inc

the compactness of a policy space in dynamic programming via an extension theorem for carathe odory functions the background for this paper is a dynamic programming model with a borel state space and compact action sets a new simple proof of the compactness of a space of measures corresponding to randomized stationary policies is given in a topology induced by caratheodory functions on the state action space for this purpose an extension theorem for caratheodory functions is proved which is a new version of the tietze s extension theorem for continuous functions

scientific computation on some mathematical problems abstract our goal is to survey using three examples how high precision computations have stimulated mathematical research in the areas of polynomial and rational approximation theory the first example will be the 1 9 conjecture in rational approximation theory here high precision computations gave strong evidence that this conjecture is false gonchar and rakhmanov have given an exact solution of this conjecture the second example will be the 8 conjecture in rational approximation theory in this case high precision computations and the use of the richardson extrapolation method led to this conjecture stahl has proved that this conjecture and its generalization are true the final example will be the bernstein conjecture in polynomial approximation theory and its generalization

recurrent and feedforward polynomial modeling of coupled time series we present two methods for the prediction of coupled time series the first one is based on modeling the series by a dynamic system with a polynomial format this method can be formulated in terms of learning in a recurrent network for which we give a computationally effective algorithm the second method is a purely feedforward network procedure whose architecture derives from the recurrence relations for the derivatives of the trajectories of a ricatti format dynamic system it can also be used for the modeling of discrete series in terms of nonlinear mappings both methods have been tested successfully against chaotic series

continuous time discounted jump markov decision processes a discrete event approach this paper introduces and develops a new approach to the theory of continuous time jump markov decision processes ctjmdp this approach reduces discounted ctjmdps to discounted semi markov decision processes smdps and eventually to discrete time markov decision processes mdps the reduction is based on the equivalence of strategies that change actions between jumps and the randomized strategies that change actions only at jump epochs this holds both for one criterion problems and for multiple objective problems with constraints in particular this paper introduces the theory for multiple objective problems with expected total discounted rewards and constraints if a problem is feasible there exist three types of optimal policies i nonrandomized switching stationary policies ii randomized stationary policies for the ctjmdp and iii randomized stationary policies for the corresponding smdp with exponentially distributed sojourn times and these policies can be implemented as randomized strategies in the ctjmdp

blackwell optimality in borelian continuous in action markov decision processes we prove the existence of stationary blackwell optimal policies in markov decision processes with a borel state space compact action sets and continuous in action and bounded transition densities and rewards satisfying a simultaneous doeblin type condition the proof is based on a compactification of the randomized stationary policy space in a weak strong topology on the continuity of laurent coefficients of the discounted rewards in this topology and on a lexicographical policy improvement until now similar results were obtained for the models with a denumerable state space or with a borel state space and finite action sets

constrained and unconstrained optimal discounted control of piecewise deterministic markov processes the main goal of this paper is to study the infinite horizon expected discounted continuous time optimal control problem of piecewise deterministic markov processes with the control acting continuously on the jump intensity lambda and on the transition measure q of the process but not on the deterministic flow phi the contributions of the paper are for the unconstrained as well as the constrained cases the set of admissible control strategies is assumed to be formed by policies possibly randomized and depending on the history of the process taking values in a set valued action space for the unconstrained case we provide sufficient conditions based on the three local characteristics of the process phi lambda q and the semicontinuity properties of the set valued action space to guarantee the existence and uniqueness of the integro differential optimality equation the so called bellman hamilton jacobi equation as well as the existence of an optimal and delta optimal as well

splitting randomized stationary policies in total reward markov decision processes this paper studies a discrete time total reward markov decision process mdp with a given initial state distribution a randomized stationary policy can be split on a given set of states if the occupancy measure of this policy can be expressed as a convex combination of the occupancy measures of stationary policies each selecting deterministic actions on the given set and coinciding with the original stationary policy outside of this set for a stationary policy necessary and sufficient conditions are provided for splitting it at a single state as well as sufficient conditions for splitting it on the whole state space these results are applied to constrained mdps the results are refined for absorbing including discounted mdps with finite state and actions spaces in particular this paper provides an efficient algorithm that presents the occupancy measure of a given policy as a convex combination of the occupancy measures of finitely many stationary deterministic policies this algorithm generates the splitting policies in a way that each pair of consecutive policies differs at exactly one state the results are applied to constrained problems to efficiently compute an optimal policy by computing and splitting a stationary optimal policy

realizable strategies in continuous time markov decision processes for the borel model of the continuous time markov decision process we introduce a wide class of control strategies in a particular case such strategies transform to the standard relaxed strategies intensively studied in the last decade in another special case if one restricts to another special subclass of the general strategies the model transforms to the semi markov decision process further we show that the relaxed strategies are not realizable for the constrained optimal control problem with total expected costs we describe the sufficient class of realizable strategies the so called poisson related strategies finally we show that for solving the formulated optimal control problems one can use all the tools developed earlier for the classical discrete time markov decision processes

extreme points of sets of randomized strategies in constrained optimization and control problems this paper concerns the existence and characterization of optimal randomized strategies for some constrained optimization and control problems we first present a characterization of the extreme points of a set of randomized strategies that satisfy n moment like constraints conditions are given under which those extreme points are randomizations of at most n 1 deterministic strategies this result is then applied to obtain the existence and characterization of optimal strategies for a class of deterministic allocation like optimization problems and their young relaxations similar results are obtained for constrained markov control processes in borel spaces