a convex analytic approach to risk aware markov decision processes in classical markov decision process mdp theory we search for a policy that say minimizes the expected infinite horizon discounted cost expectation is of course a risk neutral measure which does not suffice in many applications particularly in finance we replace the expectation with a general risk functional and call such models risk aware mdp models we consider minimization of such risk functionals in two cases the expected utility framework and conditional value at risk a popular coherent risk measure later we consider risk aware mdps wherein the risk is expressed in the constraints this includes stochastic dominance constraints and the classical chance constrained optimization problems in each case we develop a convex analytic approach to solve such risk aware mdps in most cases we show that the problem can be formulated as an infinite dimensional linear program lp in occupation measures when we augment the state space we provide a discretization method and finite approximati

a new approach for aggregating edge points into line segments abstract a technique is presented for aggregating edge points into polylines which can effectively represent object contours the technique is based on a new formulation of hough transform ht for detection of line segments the space requirement of the ht is brought down by considering a different parameterization of straight lines in this method the process of edge linking and boundary approximation are combined into a single algorithm consequently the scheme is computationally more efficient than the classical boundary approximation techniques which require a separate edge linking algorithm for preparing their input the linked list of edge points experimental results presented here highlight the effectiveness of this method for approximating object boundaries of polygonal as well as curved shapes present in the images of complex multi object scenes even in the presence of noise further the parallel algorithm proposed herein for implementation of this technique makes it an ideal choice for real time applications

optimal control of hidden markov models with binary observations a hidden markov model hmm consisting of a controlled markov chain and a binary output is considered the binary output reports the occurrence of a certain state versus any other state in the markov chain the transition matrix of the markov chain depends on a control vector which is to be chosen to minimize a discounted infinite horizon cost function the information provided for decision making is the entire history of the binary output which presents an incomplete knowledge of the markov chain and as a consequence leads to a stochastic control problem with partial observations the solution to such a problem is a control policy consisting of two components an estimator to derive the posterior probability distribution of the markov chain from the observation set and a nonlinear control law to map this estimate into the control vector this paper introduces a nonlinear state space representation for the estimator develops analytical expressions for the control law and presents numerical methods for efficient computation of the optimal control application of these results to optimal management of a class of inventory systems is discussed and the performance of optimal control for this application is examined by numerical simulations

controlled diffusions with constraints controlled diffusion problems with classical cost structures ergodic discounted etc are considered when finitely many additional cost functionals of the same type are required to lie in prescribed intervals existence of an optimal markov control is proved by convex analytic methods

a movable pair of tetrahedra when we draw all the diagonals of all the faces of a cube we obtain the edges of two regular tetrahedra keeping one of these fixed we move the other under the condition that the originally intersecting edges of the two tetrahedra should still remain coplanar i e intersect are parallel or coincide we determine all such finite motions these will be seen to constitute one dimensional and two dimensional smooth manifolds we also deal with the infinitesimal degrees of freedom of the motions of our mechanism in several positions the number of infinitesimal degrees of freedom is not one or two but is three this is connected with the bifurcation phenomenon of the solutions

a call routing problem with service level constraints we consider a queueing system commonly found in inbound telephone call centers that processes two types of work type h jobs arrive at rate h are processed at rate i i h and are served first come first served within class aservice level constraint of the form e delay a orp delay i limits the delay in queue that these jobs may face an infinite backlog of type l jobs awaits processing at rate i l and there is no service level constraint on this type of work a pool ofc identical servers processes all jobs and a system controller must maximize the rate at which type l jobs are processed subject to the service level constraint placed on the type h work we formulate the problem as a constrained average cost markov decision process and determine the structure of effective routing policies when the expected service times of the two classes are the same these policies are globally optimal and the computation time required to find the optimal policy is about that required to calculate the normalizing constant for a simplem m c system when the expected service times of the two classes differ the policies are optimal within the class of priority policies and the determination of optimal policy parameters can be determined through the solution of a linear program with o c3 variables and o c2 constraints

resource constrained management of heterogeneous assets with stochastic deterioration we consider a collection of heterogeneous assets which exhibit independent stochastic behavior but are economically interdependent due to resource constraints on management decisions we represent the collection of assets as a network of nonhomogeneous markov decision processes linked by side constraints to facilitate a procedure for obtaining nonrandomized decision policies we express the resource limitations as integrated chance constraints and relax them into the objective function within a lagrangian penalty term we utilize subgradient optimization to establish upper bounds and a greedy randomized lagrangian repair heuristic to obtain feasible solutions we empirically validate the tightness of these a posteriori upper and lower bounds with computational experiments on a pair of applications for pavement maintenance we examine the effect that the reward structure of each constituent mdp has on the maintenance policy in the presence of budget constraints for equipment replacement we consider constraints on two resource types

a variance minimization problem for a markov decision process abstract in the steady state of a discrete time markov decision process we consider the problem to find an optimal randomized policy that minimizes the variance of the reward in a transition among the policies which give the mean not less than a specified value the problem is solved by introducing a parametric markov decision process with average cost criterion it is shown that there exists an optimal policy which is a mixture of at most two pure policies as an application the toymaker s problem is discussed

a convex analytic framework for ergodic control of semi markov processes the ergodic control problem for semi markov processes is reformulated as an optimization problem over the set of suitably defined ergodic occupation measures this set is shown to be closed and convex with its extreme points corresponding to stationary strategies this leads to the existence of optimal stationary strategies under additional hypotheses a pathwise analysis of the joint empirical occupation measures of the state and control processes shows that this optimality is in the strong i e almost sure sense

denumerable constrained markov decision processes and finite approximations the purpose of this paper is two fold first to establish the theory of discounted constrained markov decision processes with a countable state and action spaces with general multi chain structure second to introduce finite approximation methods we define the occupation measures and obtain properties of the set of all achievable occupation measures under the different admissible policies we establish the optimality of stationary policies for the constrained control problem and obtain an lp with a countable number of decision variables through which stationary optimal policies are computed since for such an lp one cannot expect to find an optimal solution in a finite number of operations we present two schemes for finite approximations and establish the convergence of optimal values and policies for both the discounted and the expected average cost with unbounded cost sometimes it turns out to be easier to solve the problem with infinite state space than the problem with finite yet large state space based on the optimal policy for the problem with infinite state space we construct policies which are almost optimal for the problem with truncated state space this method is applied to obtain an e optimal policy for a problem of optimal priority assignment under constraints for a system of k finite queues

constrained markov decision models with weighted discounted rewards this paper deals with constrained optimization of markov decision processes both objective function and constraints are sums of standard discounted rewards but each with a different discount factor such models arise e g in production and in applications involving multiple time scales we prove that it a feasible policy exists then there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is bounded by the number of constraints optimality of such policies for multi criteria problems is also established these new policies have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints this result is new even for constrained optimization with a single discount factor where the optimality of randomized stationary policies is known however a randomized stationary policy may req

on stationary strategies in borel dynamic programming we consider a discrete time markov decision model with borel state and action spaces it is proved that the supremum of the expected total rewards under all randomized stationary strategies is equal to the supremum of these rewards under all nonrandomized stationary strategies

controlled markov chains and stochastic networks controlled markov chains with average cost criterion and with special cost and transition structures are studied existence of optimal stationary strategies is established for the average cost criterion corresponding dynamic programming equations are derived a stochastic network problem that includes interconnected queues as a special case is described and studied within this framework

continuity of optimal values and solutions for control of markov chains with constraints we consider in this paper constrained markov decision processes this type of control model has many applications in telecommunications and other fields e altman and a shwartz ieee trans automat control 34 1989 pp 1089 1102 e a feinberg and m i reiman probab engrg inform sci 8 1994 pp 463 489 a hordijk and f spieksma adv in appl probab 21 1989 pp 409 431 a lazar ieee trans automat control 28 1983 pp 1001 1007 p nain and k w ross ieee trans automat control 31 1986 pp 883 888 k w ross and b chen ieee trans automat control 33 1988 pp 261 267 we address the issue of the convergence of the value and optimal policies of the problem with discounted costs to the ones for the problem with expected average cost we consider the general multichain ergodic structure we present two stability results in this paper we establish the continuity of optimal values and solutions of as well as some type of robustness of some suboptimal solutions in the discount factor our proof relies on same general theory on continuity of values and solutions in convex optimization that relies on well known notions of gamma convergence

computation of kinematic paths and bifurcation points this article deals with the kinematic simulation of movable structures that go through special configurations of kinematic bifurcation as they move a series of algorithms are developed for structures that can be modelled using pin jointed bars and that admit a single parameter motion these algorithms are able to detect and locate any bifurcation points that exist along the path of the structure and at each bifurcation point can determine all possible motions of the structure the theory behind the algorithms is explained and the analysis of a simple example is discussed in detail then a simplified version of the particular problem that had motivated this work the simulation of the folding and deployment of a thin membrane structure forming a solar sail is analysed for the particular cases that are considered it is found that the entire process is inextensional but a detailed study of the simulation results shows that in more general cases it is likely that stretching or wrinkling will occur

simplex algorithm for countable state discounted markov decision processes we consider discounted markov decision processes mdps with countably infinite state spaces finite action spaces and unbounded rewards typical examples of such mdps are inventory management and queueing control problems in which there is no specific limit on the size of inventory or queue existing solution methods obtain a sequence of policies that converges to optimality in value but may not improve monotonically ie a policy in the sequence may be worse than preceding policies our proposed approach considers countably infinite linear programming cilp formulations of the mdps a cilp is defined as a linear program lp with countably infinite numbers of variables and constraints under standard assumptions for analyzing mdps with countably infinite state spaces and unbounded rewards we extend the major theoretical extreme point and duality results to the resulting cilps under additional mild assumptions which are satisfied by several applications of interest we present a simplex type algor

piecewise linear approximation of plane curves by perimeter optimization abstract a method for the piecewise linear approximation of a plane curve is described an approximate curve is obtained by choosing a certain number of points from a set of sampled points of the original curve a point choice function that represents the relation between the original points and the chosen points is formulated the approximation is performed by the dynamic programming principle searching for the optimal point choice function that attains the minimal error about arc length of the curve evaluation of the global approximate error provides efficient curve approximations irrespective of shape complexity number of sampled points and irregularity of sampling interval

markov decision processes with a minimum variance criterion abstract we consider the optimization of the variance of the sum of costs as well as that of an average expected cost in markov decision processes with unbounded cost in case of general state and action space we find the stationary policy which makes the average variance as small as possible in the class of policies which are e optimal in an average expected cost

non parametric dominant point detection abstract a new method for detecting dominant points is presented it does not require any input parameter and the dominant points obtained by this method remain relatively the same even when the object curve is scaled or rotated in this method for each boundary point a support region is assigned to the point based on its local properties each point is then smoothed by a gaussian filter with a width proportional to its determined support region a significance measure for each point is then computed dominant points are finally obtained through nonmaximum suppression unlike other dominant point detection algorithms which are sensitive to scaling and rotation of the object curve the new method will overcome this difficulty furthermore it is robust in the presence of noise the proposed new method is compared to the teh chin algorithm see c teh and r t chin ieee trans pattern anal mach intell 11 859 872 1989 in terms of the computational complexity the approximation errors and the number of detected dominant points of an object curve

an optimal control theory for discrete event systems in certain discrete event applications it may be desirable to find a particular controller within the set of acceptable controllers which optimizes some quantitative performance measure in this paper we propose a theory of optimal control to meet such design requirements for deterministic systems the discrete event system des is modeled by a formal language event and cost functions are defined which induce costs on controlled system behavior the event costs associated with the system behavior can be reduced in general only by increasing the control costs thus it is nontrivial to find the optimal amount of control to use and the formulation captures the fundamental tradeoff motivating classical optimal control results on the existence of minimally restrictive optimal solutions are presented communication protocols are analyzed to motivate the formulation and demonstrate optimal controller synthesis algorithms for the computation of optimal controllers are developed for the special case of des modeled by regular languages it is shown that this framework generalizes some of the existing literature

adaptive control of constrained markov chains adaptive control of finite state markov chains is discussed the optimal performance is characterized through the minimization of a long run average cost functional subject to constraints on several other such functionals under mild structural and feasibility conditions two explicit adaptive control policies are exhibited for the case where the transition probabilities are unknown the policies are optimal under the constrained optimization criterion they rely on a powerful estimation scheme which provides consistent estimators for the transition probabilities this scheme is of independent interest as it provides strong consistency under a large number of adaptive schemes and is independent of any identifiability conditions as an application an optimal adaptive policy is derived for a system of k competing queues with countable state space for which the constrained criteria arise naturally in the context of communication networks

an adaptive reduction procedure for the piecewise linear approximation of digitized curves a new algorithm is presented for the piecewise linear approximation of two dimensional digitized curves against a square grid the algorithm utilizes an adaptive reduction procedure in two approximation phases to select the critical points of a digitized curve such that the deviation from the digitized curve to its final approximated curve is bounded by a uniform error tolerance the time complexity of this algorithm is o m sup 2 rather than o n sup 2 on the theoretical plane in the experiments of fixing the initial and the final processing points the performance of the algorithm has been compared to those of three prominent other algorithms regarding the required number of critical points and the total execution time of the program of the four algorithms compared the present algorithm consistently has the shortest execution time of the program and it tends most to require as few critical points as the optimum algorithm that was tested

optimality of mixed policies for average continuous time markov decision processes with constraints this article concerns the average criteria for continuous time markov decision processes with n constraints we show the following a every extreme point of the space of performance vectors corresponding to the set of stable measures is generated by a deterministic stationary policy and b there exists a mixed optimal policy where the mixture is over no more than n 1 deterministic stationary policies

approximations in dynamic zero sum games ii we pursue in this paper our study of approximations of values and epsilon saddle point policies in dynamic zero sum games after extending the general theorem for approximation we study zero sum stochastic games with countable state space and unbounded immediate reward we focus on the expected average payoff criterion we use some tools developed in m m tidball and e altman siam j control optim 34 1996 pp 311 328 to obtain the convergence of the values as well as the convergence of the epsilon saddle point policies in various approximation problems we consider several schemes of truncation of the state space e g finite state approximation and approximations of games with discount factor close to one for the game with expected average cost we use the extension of the general theorem for approximation to study approximations in stochastic games with complete information finally we consider the problem of approximating the sets of policies we obtain some general results that we apply to a pursuit evasion differential game

new polygonal approximation schemes for object shape representation abstract object shape representation is an important issue in pattern recognition and computer vision applications a set of new polygonal approximation schemes namely the internal maximum area polygonal approximation imapa the external minimum area polygonal approximation emapa and the minimum area deviation polygonal approximation madpa are proposed for object shape representation the area deviation between the initial object shape and its approximation polygon is defined as the cost function instead of outputting a unique approximation polygon each proposed scheme outputs a sequence of approximation polygons with different numbers of line segments and different area deviations for various application situations for a given area deviation bound and an initial object shape each proposed scheme can give a corresponding optimal approximation polygon with a minimum number of line segments a performance comparison among the proposed schemes is included the proposed schemes are compared with two other existing schemes some experimental results show the feasibility of the proposed approaches

constrained undiscounted stochastic dynamic programming in this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function the state space and action sets are assumed to be finite constraints which are linear functions of the state action frequencies are allowed in the general multichain case an optimal policy will be a randomized nonstationary policy an algorithm to compute such an optimal policy is presented furthermore sufficient conditions for optimal policies to be stationary are derived there are many applications for constrained undiscounted stochastic dynamic programming e g in multiple objective markovian decision models

existence of optimal stationary policies in discounted markov decision processes approaches by occupation measures abstract we consider discounted markov decision processes with general state and action spaces and bounded borel measurable cost functions we extend the idea of occupation measures given in borkar 1 and constructing the occupation measure associated with a sequence of policies under a system assumption we prove the existence theorem of optimal stationary policies here the assumptions of continuity and compactness are excluded and the discussion is done in the class of borel measurable policies

finite state approximations for denumerable multidimensional state discounted markov decision processes abstract a finite state iterative scheme introduced by white in recent developments in markov decision processes r hartley l c thomas and d j white eds academic press new york 1980 to approximate the value function of denumerable state markov decision processes is extended to denumerable multidimensional state space under essentially the same assumptions given in op cit a simpler proof of the convergence theorem is obtained together with convergence rates the iterative scheme is used to determine an asymptotically discount optimal policy which in turn can be used to obtain a discount optimal stationary policy

approximating ergodic average reward continuous time controlled markov chains we study the approximation of an ergodic average reward continuous time denumerable state markov decision process mdp by means of a sequence of mdps our results include the convergence of the corresponding optimal policies and the optimal gains for a controlled upwardly skip free process we show some computational results to illustrate the convergence theorems

stochastic dominance constrained markov decision processes we are interested in risk constraints for infinite horizon discrete time markov decision processes mdps starting with average reward mdps we show that increasing concave stochastic dominance constraints on the empirical distribution of reward lead to linear constraints on occupation measures an optimal policy for the resulting class of dominance constrained mdps is obtained by solving a linear program we compute the dual of this linear program to obtain average dynamic programming optimality equations that reflect the dominance constraint in particular a new pricing term appears in the optimality equations corresponding to the dominance constraint we show that many types of stochastic orders can be used in place of the increasing concave stochastic order we also carry out a parallel development for discounted reward mdps with stochastic dominance constraints a portfolio optimization example is used to motivate the paper

on the asymptotic optimality of finite approximations to markov decision processes with borel spaces calculating optimal policies is known to be computationally difficult for markov decision processes mdps with borel state and action spaces this paper studies finite state approximations of discrete time markov decision processes with borel state and action spaces for both discounted and average costs criteria the stationary policies thus obtained are shown to approximate the optimal stationary policy with arbitrary precision under quite general conditions for discounted cost and more restrictive conditions for average cost for compact state mdps we obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases using information theoretic arguments the order optimality of the obtained convergence rates is established for a large class of problems we also show that as a pre processing step the action space can also be finitely approximated with sufficiently large number points thereby well known algorithms su

constrained discounted dynamic programming this paper deals with constrained optimization of markov decision processes with a countable state space compact action sets continuous transition probabilities and upper semicontinuous reward functions the objective is to maximize the expected total discounted reward for one reward function under several inequality constraints on similar criteria with other reward functions suppose a feasible policy exists for a problem with m constraints we prove two results on the existence and structure of optimal policies first we show that there exists a randomized stationary optimal policy which requires at most m actions more than a nonrandomized stationary one this result is known for several particular cases second we prove that there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is at most m iii the total number of actions that are added by nonstationarity is at most m we also establish pareto optimality of policies from the two classes described above for multi criteria problems we describe an algorithm to compute optimal policies with properties i iii for constrained problems the policies that satisfy properties i iii have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints in contrast a randomized stationary policy may require an infinite number of randomizations over time

a convex analytic approach to markov decision processes this paper develops a new framework for the study of markov decision processes in which the control problem is viewed as an optimization problem on the set of canonically induced measures on the trajectory space of the joint state and control process this set is shown to be compact convex one then associates with each of the usual cost criteria infinite horizon discounted cost finite horizon control up to an exit time a naturally defined occupation measure such that the cost is an integral of some function with respect to this measure these measures are shown to form a compact convex set whose extreme points are characterized classical results about existence of optimal strategies are recovered from this and several applications to multicriteria and constrained optimization problems are briefly indicated

optimal policies for controlled markov chains with a constraint abstract the time average reward for a discrete time controlled markov process subject to a time average cost constraint is maximized over the class of al causal policies each epoch a reward depending on the state and action is earned and a similarly constituted cost is assessed the time average of the former is maximized subject to a hard limit on the time average of the latter it is assumed that the state space is finite and the action space compact metric an accessibility hypothesis makes it possible to utilize a lagrange multiplier formulation involving the dynamic programming equation thus reducing the optimization problem to an unconstrained optimization parametrized by the multiplier the parametrized dynamic programming equation possesses compactness and convergence properties that lead to the following if the constraint can be satisfied by any causal policy the supremum over time average rewards respective to all causal policies is attained by either a simple or a mixed policy the latter is equivalent to choosing independently at each epoch between two specified simple policies by the throw of a biased coin

monotonicity of constrained optimal transmission policies in correlated fading channels with arq we consider transmission scheduling using an arq protocol with retransmissions given channel state information csi and a correlated fading channel the problem is formulated as a countable state infinite horizon average cost markov decision process mdp with an average delay constraint our main result is to give sufficient conditions on the channel memory and transmission cost so that the optimal transmission scheduling policy is a monotonically increasing function of the buffer occupancy in proving this result we first prove positive recurrence stability of the buffer the monotone structure proof consists of two steps first the constrained mdp cmdp is transformed into an unconstrained mdp using a lagrangian dynamic programming formulation it is proved that the unconstrained optimal policy is pure and monotonically increasing in the buffer occupancy it is then shown that the constrained optimal policy is a randomized mixture of two pure transmission policies that are monotone in the buffer state finally the monotone structure of the optimal transmission policy is exploited to derive a monotone policy q learning algorithm and a stochastic approximation based monotone policy search algorithm for estimating the optimal policy in real time

polygonal approximation using a competitive hopfield neural network abstract polygonal approximation plays an important role in pattern recognition and computer vision in this paper a parallel method using a competitive hopfield neural network chnn is proposed for polygonal approximation based on the chnn the polygonal approximation is regarded as a minimization of a criterion function which is defined as the arc to chord deviation between the curve and the polygon the chnn differs from the original hopfield network in that a competitive winner take all mechanism is imposed the winner take all mechanism adeptly precludes the necessity of determining the values for the weighting factors in the energy function in maintaining a feasible result the proposed method is compared to several existing methods by the approximation error norms l2 and l with the result that promising approximation polygons are obtained

polygonal approximation of digital planar curves through break point suppression this paper presents a new algorithm that detects a set of dominant points on the boundary of an eight connected shape to obtain a polygonal approximation of the shape itself the set of dominant points is obtained from the original break points of the initial boundary where the integral square is zero for this goal most of the original break points are deleted by suppressing those whose perpendicular distance to an approximating straight line is lower than a variable threshold value the proposed algorithm iteratively deletes redundant break points until the required approximation which relies on a decrease in the length of the contour and the highest error is achieved a comparative experiment with another commonly used algorithm showed that the proposed method produced efficient and effective polygonal approximations for digital planar curves with features of several sizes

stability and singular perturbations in constrained markov decision problems constrained markov decision problems cmdps with the average cost criterion and a single ergodic chain or the discounted cost with a general multichain structure are considered conditions for stability of the optimal value and control to changes of the parameters of the problem such as immediate costs transition probabilities and the discount factor are established singular constrained problems for which the optimal value and controls exhibit discontinuities are studied

unsupervised segmentation and approximation of digital curves with rate distortion curve modeling this paper considers the problem of unsupervised segmentation and approximation of digital curves and trajectories with a set of geometrical primitives model functions an algorithm is proposed based on a parameterized model of the rate distortion curve the multiplicative cost function is then derived from the model by analyzing the minimum of the cost function a solution is defined that produces the best possible balance between the number of segments and the approximation error the proposed algorithm was tested for polygonal approximation and multi model approximation circular arcs and line segments for digital curves and polynomials for trajectory the algorithm demonstrated its efficiency in comparisons with known methods with a heuristic cost function the proposed method can additionally be used for segmentation and approximation of signals and time series a new algorithm for unsupervised segmentation of digital curves is introduced this method gives solutions with the best balance between error and description length a multiplicative criterion for evaluation of solutions is introduced

constrained average cost markov control processes in borel spaces this paper considers constrained markov control processes in borel spaces with unbounded costs the criterion to be minimized is a long run expected average cost and the constraints can be imposed on similar average costs or on average rewards or discounted costs or rewards we give conditions under which the constrained problem cp is solvable and equivalent to an equality constrained ec linear program furthermore we show that there is no duality gap between ec and the dual program ec and that in fact the strong duality condition holds finally we introduce an explicit procedure to solve cp in some cases which is illustrated with a detailed example

constrained discounted markov decision processes with borel state spaces abstract we study discrete time discounted constrained markov decision processes cmdps with borel state and action spaces these cmdps satisfy either weak w continuity conditions that is the transition probability is weakly continuous and the reward function is upper semicontinuous in state action pairs or setwise s continuity conditions that is the transition probability is setwise continuous and the reward function is upper semicontinuous in actions our main goal is to study models with unbounded reward functions which are often encountered in applications e g in consumption investment problems we provide some general assumptions under which the optimization problems in cmdps are solvable in the class of randomized stationary policies and in the class of chattering policies introduced in this paper if the initial distribution and transition probabilities are atomless then using a general purification result of feinberg and piunovskiy we show the existence of a deterministic stationary optimal policy our main results are illustrated by examples

average sensitive and blackwell optimal policies in denumerable markov decisionchains with unbounded rewards in this paper we consider a discrete time markov decision chain with a denumerable state space and compact action sets and we assume that for all states the rewards and transition probabilities depend continuously on the actions the first objective of this paper is to develop an analysis for average optimality without assuming a special markov chain structure in doing so we present a set of conditions guaranteeing average optimality which are automatically fulfilled in the finite state and action model the second objective is to study simultaneously average and discount optimality as veinott veinott a f jr 1969 on discrete dynamic programming with sensitive discount optimality criteria ann math statist 40 1635 1660 did for the finite state and action model we investigate the concepts of n discount and blackwell optimality in the denumerable state space using a laurent series expansion for the discounted rewards under the same condition as for average optimality we establish solutions to the n discount optimality equations for every n

steering policies for controlled markov chains under a recurrence condition the authors consider the class of steering policies for controlled markov chains under a recurrence condition a steering policy is defined as one adaptively alternating between two stationary policies in order to track a sample average cost to a desired value convergence of the sample average costs is derived via direct sample path arguments and the performance of the steering policy is discussed steering policies are motivated by and particularly useful in the discussion of constrained markov chains with a single constraint

techniques for assessing polygonal approximations of curves given the enormous number of available methods for finding polygonal approximations to curves techniques are required to assess different algorithms some of the standard approaches are shown to be unsuitable if the approximations contain varying numbers of lines instead we suggest assessing an algorithm s results relative to an optimal polygon and describe a measure which combines the relative fidelity and efficiency of a curve segmentation we use this measure to compare the application of 23 algorithms to a curve first used by teh and chin 1989 their integral square errors ises are assessed relative to the optimal ise in addition using an example of pose estimation it is shown how goal directed evaluation can be used to select an appropriate assessment criterion

optimum uniform piecewise linear approximation of planar curves two dimensional digital curves are often uniformly approximated by polygons or piecewise linear curves several algorithms have been proposed in the literature to find such curves we present an algorithm that finds a piecewise linear curve with the minimal number of segments required to approximate a curve within a uniform error with fixed initial and final points we compare our optimal algorithm to several suboptimal algorithms with respect to the number of linear segments required in the approximation and the execution time of the algorithm

randomized and past dependent policies for markov decision processes with multiple constraints the markov decision problem of locating a policy to maximize the long run average reward subject to k long run average cost constraints is considered it is assumed that the state and action spaces are finite and the law of motion is unichain that is every pure policy gives rise to a markov chain with one recurrent class it is first proved that there exists an optimal stationary policy with a degree of randomization no greater than k consequently it is never necessary to randomize in more than k states a linear program produces the optimal policy with limited randomization for the special case of a single constraint we also address the problem of finding optimal nonrandomized but nonstationary policies we show that a round robin type policy is optimal and conjecture the same for a steering policy that depends on the entire past history of the process but whose implementation requires essentially no more storage than that of a pure policy

data reduction of large vector graphics fast algorithm for joint near optimal approximation of multiple polygonal curves is proposed it is based on iterative reduced search dynamic programming introduced earlier for the min eproblem of a single polygonal curve the proposed algorithm jointly optimizes the number of line segments allocated to the different individual curves and the approximation of the curves by the given number of segments trade off between time and optimality is controlled by the breadth of the search and by the numbers of iterations applied

ergodic control of markov chains with constraints the general case the problem of controlling a markov chain on a countable state space with ergodic or long run average cost is studied in the presence of additional constraints requiring finitely many say m other ergodic costs to satisfy prescribed bounds under extremely general conditions it is proved that an optimal stationary randomized strategy can be found that requires at most m randomizations this generalizes a result of ross

markov decision processes with sample path constraints the communicating case we consider time average markov decision processes mdps which accumulate a reward and cost at each decision epoch a policy meets the sample path constraint if the time average cost is below a specified value with probability one the optimization problem is to maximize the expected average reward over all policies that meet the sample path constraint the sample path constraint is compared with the more commonly studied constraint of requiring the average expected cost to be less than a specified value although the two criteria are equivalent for certain classes of mdps their feasible and optimal policies differ for many nontrivial problems in general there does not exist optimal or nearly optimal stationary policies when the expected average cost constraint is employed assuming that a policy exists that meets the sample path constraint we establish that there exist nearly optimal stationary policies for communicating mdps a parametric linear programming algorithm is given to construct nearly optimal stationary policies the discussion relies on well known results from the theory of stochastic processes and linear programming the techniques lead to simple proofs of the existence of optimal and nearly optimal stationary policies for unichain and deterministic mdps respectively

on minimum cost per unit time control of markov chains the minimum cost per unit time control problem is studied for a class of markov chains that though important in applications does not fit the conventional framework for this problem existence of optimal stationary strategies and necessary and sufficient conditions for optimality are established

optimal polygonal approximation of digitized curves using the sum of square deviations criterion abstract we address the problem of finding an optimal polygonal approximation of digitized curves several optimal algorithms have already been proposed to determine the minimum number of points that define a polygonal approximation of the curve with respect to a criterion error we present a new algorithm with reasonable complexity to determine the optimal polygonal approximation using the mean square error norm the idea is to estimate the remaining number of segments and to integrate the cost in the a algorithm the solution is optimal in the minimum number of segments

markov decision problems and state action frequencies consider a controlled markov chain with countable state and action spaces basic quantities that determine the values of average cost functionals are identified under some regularity conditions these turn out to be a collection of numbers one for each state action pair describing for each state the relative number of uses of each action these conditional frequencies which are defined pathwise are shown to determine the state action frequencies that in the finite case are known to determine the costs this is extended to the countable case allowing for unbounded costs the space of frequencies is shown to be compact and convex and the extreme points are identified with stationary deterministic policies conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given these problems include the standard markov decision process as well as constrained optimization both in terms of average cost functionals and variability sensitive optimization an application to a queueing problem is given where these results imply the existence and explicit computation of optimal policies in constrained optimization problems the pathwise definition of the conditional frequencies implies that their values can be controlled directly moreover they depend only on the limiting behavior of the control this has immediate application to adaptive control of markov chains including adaptive control under constraints

circumventing the slater conundrum in countably infinite linear programs abstract duality results on countably infinite linear programs are scarce subspaces that admit an interior point which is a sufficient condition for a zero duality gap yield a dual where the constraints cannot be expressed using the ordinary transpose of the primal constraint matrix subspaces that permit a dual with this transpose do not admit an interior point this difficulty has stumped researchers for a few decades it has recently been called the slater conundrum we find a way around this hurdle we propose a pair of primal dual spaces with three properties the series in the primal and dual objective functions converge the series defined by the rows and columns of the primal constraint matrix converge and the order of sums in a particular iterated series of a double sequence defined by the primal constraint matrix can be interchanged so that the dual is defined by the ordinary transpose weak duality and complementary slackness are then immediate instead of using interior point conditions to establish a zero duality gap we call upon the planning horizon method when the series in the primal and dual constraints are continuous we prove that strong duality holds if a sequence of optimal solutions to finite dimensional truncations of the primal and dual cilps has an accumulation point we show by counterexample that the requirement that such an accumulation point exist cannot be relaxed our results are illustrated using several examples and are applied to countable state markov decision processes and to a problem in robust optimization

extended truss theory with simplex constraints this paper traces a way of generalization of the classical truss theory in addition to the kinematic constraint expressing the distance between two nodes connected by a bar element other similar constraints involving three and four nodes are introduced derived from energy principles a general tangent stiffness formulation is given possible mechanical interpretations as well as problems of pre stressing are also discussed

time sharing policies for controlled markov chains we propose a class of nonstationary policies called policy time sharing pts which possesses several desirable properties for problems where the criteria are of the average cost type an optimal policy exists within this class the computation of optimal policies is straightforward and the implementation of this policy is easy while in the finite state case stationary policies are also known to share these properties the new policies are much more flexible in the sense that they can be applied to solve adaptive problems and they suggest new ways to incorporate the particular structure of the problem at hand into the derivation of optimal policies in addition they provide insight into the pathwise structure of controlled markov chains to use pts policies one alternates between the use of several stationary deterministic policies switching when reaching some predetermined state in some countable state cases optimal solutions of the pts type are available and easy to compute whereas optimal stationary policies are not available examples that illustrate the last point and the usefulness of the new approach are discussed involving constrained optimization problems with countable state space or compact action space

finite state approximations to discounted and average cost constrained markov decision processes in this paper we consider the finite state approximation of a discrete time constrained markov decision process mdp under the discounted and average cost criteria using the linear programming formulation of the constrained discounted cost problem we prove the asymptotic convergence of the optimal value of the finite state model to the optimal value of the original model with further continuity condition on the transition probability we also establish a method to compute approximately optimal policies for the average cost instead of using the finite state linear programming approximation method we use the original problem definition to establish the finite state asymptotic approximation of the constrained problem and compute approximately optimal policies under lipschitz type regularity conditions on the components of the mdp we also obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases

variance penalized markov decision processes we consider a markov decision process with both the expected limiting average and the discounted total return criteria appropriately modified to include a penalty for the variability in the stream of rewards in both cases we formulate appropriate nonlinear programs in the space of state action frequencies averaged or discounted whose optimal solutions are shown to be related to the optimal policies in the corresponding variance penalized mdp the analysis of one of the discounted cases is facilitated by the introduction of a cartesian product of two independent mdps

the linear programming approach to approximate dynamic programming the curse of dimensionality gives rise to prohibitive computational requirements that render infeasible the exact solution of large scale stochastic control problems we study an efficient method based on linear programming for approximating solutions to such problems the approach fits a linear combination of pre selected basis functions to the dynamic programming cost to go function we develop error bounds that offer performance guarantees and also guide the selection of both basis functions and state relevance weights that influence quality of the approximation experimental results in the domain of queueing network control provide empirical support for the methodology

sample path average optimality for markov control processes the authors consider a markov control process with borel state and actions spaces unbounded costs and under the long run sample path average cost criterion they prove that under very weak assumptions on the transition law and a moment assumption for the one step cost there exists a stationary policy with invariant probability distribution v that is sample path average cost optimal for v almost all initial states in addition every expected average cost optimal stationary policy is in fact liminf sample path average cost optimal and strongly expected average cost optimal

corner detection and curve segmentation by multiresolution chain code linking in this paper we propose a new method to characterise a curve by means of the hierarchical computation of a multiresolution structure this structure consisting of successive lower resolution versions of the same object is processed using the linked pyramid approach we adapt the multiresolution pixel linking algorithm to the processing of curve contours which are described by their chain code we also introduce a selective class selection process which allows application of the algorithm to segmentation and detection of contour features the resulting framework presents good performance for a wide range of object sizes without the need of any parameter tweaking and allows detection of shape detail at different scales

an exact iterative search algorithm for constrained markov decision processes abstract this communique provides an exact iterative search algorithm for the np hard problem of obtaining an optimal feasible stationary markovian pure policy that achieves the maximum value averaged over an initial state distribution in finite constrained markov decision processes it is based on a novel characterization of the entire feasible policy space and takes the spirit of policy iteration pi in that a sequence of monotonically improving feasible policies is generated and converges to an optimal policy in iterations of the size of the policy space at the worst case unlike pi an unconstrained mdp needs to be solved at iterations involved with feasible policies and the current best policy improves all feasible policies included in the union of the policy spaces associated with the unconstrained mdps

mimo transmission control in fading channels a constrained markov decision process formulation with monotone randomized policies this paper addresses the optimal power and rate allocation control in multiple input multiple output mimo wireless systems over markovian fading channels the problem is posed as an infinite horizon average cost constrained markov decision process cmdp with the goal of minimizing the average transmission power subject to delay constraints by using a lagrangian formulation of the cmdp we use the concepts of stochastic dominance submodularity and multimodularity to prove that the optimal randomized policies are monotone three important structural results on the nature of the optimal randomized policies are derived first we show that the action space can be exponentially reduced by decomposing the rate allocation problem into bit loading problem across individual antennas and the total rate allocation based on the current buffer occupancy and channel state second we show that the optimal rate allocation policy is a randomized mixture of two pure policies that are monotonically increasing in the buffer occupancy finally we show that the optimal power allocation is piecewise linear in the delay constraint these three structural results can be exploited to devise efficient online reinforcement learning algorithms for optimal rate allocation

multiple objective nonatomic markov decision processes with total reward criteria we consider a markov decision process with an uncountable state space and multiple rewards for each policy its performance is evaluated by a vector of total expected rewards under the standard continuity assumptions and the additional assumption that all initial and transition probabilities are nonatomic we prove that the set of performance vectors for all policies is equal to the set of performance vectors for nonrandomized markov policies this result implies the existence of optimal nonrandomized markov policies for nonatomic constrained markov decision processes with total rewards we provide two examples of applications of our results to constrained multiple objective problems in inventory control and finance

a stability result for linear markovian stochastic optimization problems in this paper we propose a semi metric for markov processes that allows to bound optimal values of linear markovian stochastic optimization problems similar to existing notions of distance for general stochastic processes our distance is based on transportation metrics as opposed to the extant literature the proposed distance is problem specific i e dependent on the data of the problem whose objective value we want to bound as a result we are able to consider problems with randomness in the constraints as well as in the objective function and therefore relax an assumption in the extant literature we derive several properties of the proposed semi metric and demonstrate its use in a stylized numerical example

exact finite approximations of average cost countable markov decision processes for a countable state markov decision process we introduce an embedding which produces a finite state markov decision process the finite state embedded process has the same optimal cost and moreover it has the same dynamics as the original process when restricting to the approximating set the embedded process can be used as an approximation which being finite is more convenient for computation and implementation

splitting randomized stationary policies in total reward markov decision processes this paper studies a discrete time total reward markov decision process mdp with a given initial state distribution a randomized stationary policy can be split on a given set of states if the occupancy measure of this policy can be expressed as a convex combination of the occupancy measures of stationary policies each selecting deterministic actions on the given set and coinciding with the original stationary policy outside of this set for a stationary policy necessary and sufficient conditions are provided for splitting it at a single state as well as sufficient conditions for splitting it on the whole state space these results are applied to constrained mdps the results are refined for absorbing including discounted mdps with finite state and actions spaces in particular this paper provides an efficient algorithm that presents the occupancy measure of a given policy as a convex combination of the occupancy measures of finitely many stationary deterministic policies this algorithm generates the splitting policies in a way that each pair of consecutive policies differs at exactly one state the results are applied to constrained problems to efficiently compute an optimal policy by computing and splitting a stationary optimal policy