structural properties of optimal transmission policies over a randomly varying channel we consider the problem of transmitting packets over a randomly varying point to point channel with the objective of minimizing the expected power consumption subject to a constraint on the average packet delay by casting it as a constrained markov decision process in discrete time with time averaged costs we prove structural results about the dependence of the optimal policy on buffer occupancy number of packet arrivals in the previous slot and the channel fading state for both i i d and markov arrivals and channel fading the techniques we use to establish such results convexity stochastic dominance decreasing differences are among the standard ones for the purpose our main contribution however is the passage to the average cost case a notoriously difficult problem for which rather limited results are available the novel proof techniques used here are likely to have utility in other stochastic control problems well beyond their immediate application considered here

long term values in markov decision processes and repeated games and a new distance for probability spaces we study long term markov decision processes mdps and gambling houses with applications to any partial observation mdps with finitely many states and zero sum repeated games with an informed controller we consider a decision maker who is maximizing the weighted sum t 1 x1d703 trt where rt is the expected reward of the t th stage we prove the existence of a very strong notion of long term value called general uniform value representing the fact that the decision maker can play well independently of the evaluations x1d703 t t 1 over stages provided the total variation or impatience t 1 x1d703 t 1 x1d703 t is small enough this result generalizes previous results of the literature that focus on arithmetic means and discounted evaluations moreover we give a variational characterization of the general uniform value via the introduction of appropriate invariant measures for the decision problems generalizing the fundamental theorem of gambling or the aumann maschler cav u formula for repeated games with incomple

duality and linear programs for stability and performance analysis of queuing networks and scheduling policies we consider the problems of performance analysis and stability instability determination of queuing networks and scheduling policies we exhibit a strong duality relationship between the performance of a system and its stability analysis via mean drift we obtain a variety of linear programs lps to conduct such stability and performance analyses a certain lp called the performance lp bounds the performance of all stationary nonidling scheduling policies if it is bounded then its dual called the drift lp has a feasible solution which is a copositive matrix the quadratic form associated with this copositive matrix has a negative drift showing that all stationary nonidling scheduling policies result in a geometrically converging exponential moment these results carry over to fluid models allowing the study of networks with nonexponential distributions if a modification of the performance lp called the monotone lp is bounded then the system is stable finally there is a another modification of the performance lp called the finite time lp it provides transient bounds on the performance of the system from any initial condition

a new weighted metric the relative metric ii in the first part of this investigation we generalized a weighted distance function of r c li s and found necessary and sufficient conditions for it being a metric in this paper some properties of this so called m relative metric are established specifically isometries and quasiconvexity results are derived we also illustrate connections between our approach and generalizations of the hyperbolic metric

on solving event based optimization with average reward over infinite stages event based optimization ebo provides a unified framework for problems in which decisions can be made only when certain events occur because the event sequence usually is not markovian the optimal policy could depend on the entire event history which is hard to implement in practice so most existing studies focus on memoryless policies which make decisions only based on the current observable events but it remains open how to find the optimal memoryless policies in general leaving alone to solve the ebo optimally in this technical note we address these two important questions for infinite stage ebos with finite state and action spaces and make the following three major contributions first we extend our previous studies on finite stage ebos and convert infinite stage ebos to partially observable markov decision processes pomdps the belief process of this pomdp is called belief event decision process bedp under certain well known conditions the optimal policies of bedps can be achieved within stationary markov deterministic policies second assuming optimal stationary policies exist the performance difference and derivative formulas are developed potentials of memoryless event based policies are shown to be piecewise linear functions and thus can be efficiently estimated through sample paths third a potential based approximate policy iteration algorithm is developed to obtain near optimal memoryless policies the convergence and performance loss bound of this algorithm are analyzed

image denoising using scale mixtures of gaussians in the wavelet domain we describe a method for removing noise from digital images based on a statistical model of the coefficients of an overcomplete multiscale oriented basis neighborhoods of coefficients at adjacent positions and scales are modeled as the product of two independent random variables a gaussian vector and a hidden positive scalar multiplier the latter modulates the local variance of the coefficients in the neighborhood and is thus able to account for the empirically observed correlation between the coefficient amplitudes under this model the bayesian least squares estimate of each coefficient reduces to a weighted average of the local linear estimates over all possible values of the hidden multiplier variable we demonstrate through simulations with images contaminated by additive white gaussian noise that the performance of this method substantially surpasses that of previously published methods both visually and in terms of mean squared error

a kernel based lagrangian method for imperfectly mixed chemical reactions abstract current lagrangian particle tracking algorithms used to simulate diffusion reaction equations must employ a certain number of particles to properly emulate the system dynamics particularly for imperfectly mixed systems the number of particles is tied to the statistics of the initial concentration fields of the system at hand systems with shorter range correlation and or smaller concentration variance require more particles potentially limiting the computational feasibility of the method for the well known problem of bimolecular reaction we show that using kernel based rather than dirac delta particles can significantly reduce the required number of particles we derive the fixed width of a gaussian kernel for a given reduced number of particles that analytically eliminates the error between kernel and dirac solutions at any specified time we also show how to solve for the fixed kernel size by minimizing the squared differences between solutions over any given time interval numerical results show that the width of the kernel should be kept below about 12 of the domain size and that the analytic equations used to derive kernel width suffer significantly from the neglect of higher order moments the simulations with a kernel width given by least squares minimization perform better than those made to match at one specific time a heuristic time variable kernel size based on the previous results performs on par with the least squares fixed kernel size

a note on the convergence rate of the value iteration scheme in controlled markov chains this work is concerned with controlled markov chains with bounded costs assuming that the transition probabilities satisfy a simultaneous doeblin condition it is shown that schweitzer s transformation on the transition law yields a strong ergodicity condition that implies that the solution to the average cost optimality equation can be approximated at a geometric rate via the value iteration scheme

on the mathematical properties of the structural similarity index since its introduction in 2004 the structural similarity ssim index has gained widespread popularity as a tool to assess the quality of images and to evaluate the performance of image processing algorithms and systems there has been also a growing interest of using ssim as an objective function in optimization problems in a variety of image processing applications one major issue that could strongly impede the progress of such efforts is the lack of understanding of the mathematical properties of the ssim measure for example some highly desirable properties such as convexity and triangular inequality that are possessed by the mean squared error may not hold in this paper we first construct a series of normalized and generalized vector valued metrics based on the important ingredients of ssim we then show that such modified measures are valid distance metrics and have many useful properties among which the most significant ones include quasi convexity a region of convexity around the minimizer and distance preservation under orthogonal or unitary transformations the groundwork laid here extends the potentials of ssim in both theoretical development and practical applications

a note on the convergence of policy iteration in markov decision processes with compact action spaces the undiscounted unichain finite state markov decision process with compact action space is studied we provide a counterexample for a result in hordijk and puterman 1987 and give an alternate proof of the convergence of policy iteration under the condition that there exists a state that is recurrent under every stationary policy the analysis essentially uses a two term matrix representation for the relative value vectors generated by policy iteration procedure

necessary and sufficient conditions for a bounded solution to the optimality equation in average reward markov decision chains abstract we consider average reward markov decision processes with discrete time parameter and denumerable state space we are concerned with the following problem find necessary and sufficient conditions so that for arbitrary bounded reward function the corresponding average reward optimality equation has a bounded solution this problem is solved for a class of systems including the case in which under the action of any stationary policy the state space is an irreducible positive recurrent class

markov decision processes with a borel measurable cost function the average case we consider a markov decision process with a borel measurable cost function we introduce a p step contraction property for the average cost case by use of this method the validity of the optimality equation and the existence of e optimal stationary policies are proved as some applications the sequential replacement model and the inventory model are considered

the vanishing discount approach in markov chains with risk sensitive criteria in this paper stochastic dynamic systems are studied modeled by a countable state space markov cost reward chain satisfying a lyapunov type stability condition for an infinite planning horizon risk sensitive exponential discounted and average cost criteria are considered the main contribution is the development of a vanishing discount approach to relate the discounted criterion problem with the average criterion one as the discount factor increases to one i e no discounting in comparison to the well established risk neutral case our results are novel and reveal several fundamental and surprising differences other contributions made include the use of convex analytic arguments to obtain appropriately convergent sequences and a verification theorem for the case of unbounded solutions to the average cost poisson equation arising in the risk sensitive case also of importance is the fact that our developments are very much self contained and employ only basic probabilistic and analysis principles

indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access in this paper we consider a class of restless multiarmed bandit processes rmabs that arises in dynamic multichannel access user server scheduling and optimal activation in multiagent systems for this class of rmabs we establish the indexability and obtain whittle index in closed form for both discounted and average reward criteria these results lead to a direct implementation of whittle index policy with remarkably low complexity when arms are stochastically identical we show that whittle index policy is optimal under certain conditions furthermore it has a semiuniversal structure that obviates the need to know the markov transition probabilities the optimality and the semiuniversal structure result from the equivalence between whittle index policy and the myopic policy established in this work for nonidentical arms we develop efficient algorithms for computing a performance upper bound given by lagrangian relaxation the tightness of the upper bound and the near optimal performance of whittle index policy are illustrated with simulation examples

randomized and past dependent policies for markov decision processes with multiple constraints the markov decision problem of locating a policy to maximize the long run average reward subject to k long run average cost constraints is considered it is assumed that the state and action spaces are finite and the law of motion is unichain that is every pure policy gives rise to a markov chain with one recurrent class it is first proved that there exists an optimal stationary policy with a degree of randomization no greater than k consequently it is never necessary to randomize in more than k states a linear program produces the optimal policy with limited randomization for the special case of a single constraint we also address the problem of finding optimal nonrandomized but nonstationary policies we show that a round robin type policy is optimal and conjecture the same for a steering policy that depends on the entire past history of the process but whose implementation requires essentially no more storage than that of a pure policy

control of markov chains with long run average cost criterion the dynamic programming equations the long run average cost control problem for discrete time markov chains on a countable state space is studied in a very general framework necessary and sufficient conditions for optimality in terms of the dynamic programming equations are given when an optimal stable stationary strategy is known to exist e g for the situations studied in stochastic differential systems stochastic control theory and applications ima vol math app 10 springer verlag new york berlin 1988 pp 57 77 a characterization of the desired solution of the dynamic programming equations is given in a special case also included is a novel convex analytic argument for deducing the existence of an optimal stable stationary strategy when that of a randomized one is known

a liapounov bound for solutions of the poisson equation in this paper we consider psi irreducible markov processes evolving in discrete or continuous time on a general state space we develop a liapounov function criterion that permits one to obtain explicit bounds on the solution to the poisson equation and in particular obtain conditions under which the solution is square integrable these results are applied to obtain sufficient conditions that guarantee the validity of a functional central limit theorem for the markov process as a second consequence of the bounds obtained a perturbation theory for markov processes is developed which gives conditions under which both the solution to the poisson equation and the invariant probability for the process are continuous functions of its transition kernel the techniques are illustrated with applications to queueing theory and autoregressive processes

a cost shaping linear program for average cost approximate dynamic programming with performance guarantees we introduce a new algorithm based on linear programming for optimization of average cost markov decision processes mdps the algorithm approximates the differential cost function of a perturbed mdp via a linear combination of basis functions we establish a bound on the performance of the resulting policy that scales gracefully with the number of states without imposing the strong lyapunov condition required by its counterpart in de farias and van roy de farias d p b van roy 2003 the linear programming approach to approximate dynamic programming oper res 51 6 850 865 we investigate implications of this result in the context of a queueing control problem

discounted approximations to the risk sensitive average cost in finite markov chains abstract this work concerns with markov chains on a finite state space which is endowed with a cost function the evolution of the chain is observed by an agent with constant risk sensitivity and assuming that the state space is a communicating class the relation between the risk sensitive discounted and average performance criteria is studied it is proved that as the discount factor increases to 1 an appropriate normalization of the discounted value function converges to the average cost also it is shown that if the classical normalization used in the risk neutral case is applied in the risk sensitive context then the normalized discounted value function converges to an arithmetic mean of the average cost

on the convergence of policy iteration in finite state undiscounted markov decision processes the unichain case we study the convergence of policy iteration for the undiscounted finite state discrete time markov decision problem with compact action space and unichain transition structure using a newton method type representation for policy iteration we establish the existence of a solution to the optimality equation we show that to find an average optimal policy it is sufficient to solve the optimality equation on the recurrent set of the maximizing policy under the additional assumption of a unique maximizing policy at each stage of the policy iteration procedure we show that the iterates are convergent and the resulting policy is blackwell optimal

approximate dynamic programming for stochastic linear control problems on compact state spaces abstract this paper addresses markov decision processes over compact state and action spaces we investigate the special case of linear dynamics and piecewise linear and convex immediate costs for the average cost criterion this model is very general and covers many interesting examples for instance in inventory management due to the curse of dimensionality the problem is intractable and optimal policies usually cannot be computed not even for instances of moderate size we show the existence of optimal policies and of convex and bounded relative value functions that solve the average cost optimality equation under reasonable and easy to check assumptions based on these insights we propose an approximate relative value iteration algorithm based on piecewise linear convex relative value function approximations besides computing good policies the algorithm also provides lower bounds to the optimal average cost which allow us to bound the optimality gap of any given policy for a given instance the algorithm is applied to the well studied multiple sourcing problem as known from inventory management multiple sourcing is known to be a hard problem and usually tackled by parametric heuristics we analyze several msp instances with two and more suppliers and compare our results to state of the art heuristics for the considered scenarios our policies are always at least as good as the best known heuristic and strictly better in most cases moreover by using the computed lower bounds we show for all instances that the optimality gap has never exceeded 5 and that it has been much smaller for most of them

partially observed semi markov zero sum games with average payoff we study a zero sum partially observed semi markov game with average payoff on a countable state space under certain ergodicity conditions we show that a saddle point equilibrium exists we achieve this by solving the corresponding average cost optimality equation using a span contraction method the average value is shown to be the unique zero of a lipschitz continuous function a value iteration scheme is developed to compute the value

sequencing and routing in multiclass queueing networks part i feedback regulation this paper establishes new criteria for stability and for instability of multiclass network models under a given stationary policy it also extends previous results on the approximation of the solution to the average cost optimality equations through an associated fluid model it is shown that an optimized network possesses a fluid limit model which is itself optimal with respect to a total cost criterion a general framework for constructing control algorithms for multiclass queueing networks is proposed based on these general results network sequencing and routing problems are considered as special cases the following aspects of the resulting feedback regulation policies are developed in the paper the policies are stabilizing and are in fact geometrically ergodic for a markovian model numerical examples are given in each case it is shown that the feedback regulation policy closely resembles the average cost optimal policy a method is proposed for reducing variance in simulation for a network controlled using a feedback regulation policy

optimality gap of constant order policies decays exponentially in the lead time for lost sales models inventory models with lost sales and large lead times have traditionally been considered intractable due to the curse of dimensionality recently goldberg and coauthors laid the foundations for a new approach to solving these models by proving that as the lead time grows large a simple constant order policy is asymptotically optimal however the bounds proven there require the lead time to be very large before the constant order policy becomes effective in contrast to the good numerical performance demonstrated by zipkin even for small lead time values in this work we prove that for the infinite horizon variant of the same lost sales problem the optimality gap of the same constant order policy actually converges exponentially fast to zero with the optimality gap decaying to zero at least as fast as the exponential rate of convergence of the expected waiting time in a related single server queue to its steady state value we also derive simple and explicit bounds for the optimality gap and demons

the policy iteration algorithm for average reward markov decision processes with general state space the average cost optimal control problem is addressed for markov decision processes with unbounded cost it is found that the policy iteration algorithm generates a sequence of policies which are c regular where c is the cost function under consideration this result only requires the existence of an initial c regular policy and an irreducibility condition on the state space furthermore under these conditions the sequence of relative value functions generated by the algorithm is bounded from below and nearly decreasing from which it follows that the algorithm is always convergent under further conditions it is shown that the algorithm does compute a solution to the optimality equations and hence an optimal average cost policy these results provide elementary criteria for the existence of optimal policies for markov decision processes with unbounded cost and recover known results for the standard linear quadratic gaussian problem in particular in the control of multiclass queueing networks it is found that there is a close connection between optimization of the network and optimal control of a far simpler fluid network model

the feedback capacity of the binary erasure channel with a no consecutive ones input constraint the input constrained erasure channel with feedback is considered where the binary input sequence contains no consecutive ones i e it satisfies the 1 infty rll constraint we derive the capacity for this setting which can be expressed as c epsilon max 0 leq p leq 0 5 frac 1 epsilon h b p 1 1 epsilon p where epsilon is the erasure probability and h b cdot is the binary entropy function moreover we prove that a priori knowledge of the erasure at the encoder does not increase the feedback capacity the feedback capacity was calculated using an equivalent dynamic programming dp formulation with an optimal average reward that is equal to the capacity furthermore we obtained an optimal encoding procedure from the solution of the dp leading to a capacity achieving zero error coding scheme for our setting dp is thus shown to be a tool not only for solving optimization problems such as capacity calculation but also for constructing optimal coding schemes the derived capacity expression also serves as the only non trivial upper bound known on the capacity of the input constrained erasure channel without feedback a problem that is still open

approximate receding horizon approach for markov decision processes average reward case abstract we consider an approximation scheme for solving markov decision processes mdps with countable state space finite action space and bounded rewards that uses an approximate solution of a fixed finite horizon sub mdp of a given infinite horizon mdp to create a stationary policy which we call approximate receding horizon control we first analyze the performance of the approximate receding horizon control for infinite horizon average reward under an ergodicity assumption which also generalizes the result obtained by white j oper res soc 33 1982 253 259 we then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub mdp the first control policy is based on a finite horizon approximation of howard s policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies along the study we also provide a simple alternative proof on the policy improvement for countable state space we finally discuss practical implementations of these schemes via simulation

from fluid relaxations to practical algorithms for high multiplicity job shop scheduling the holding cost objective we design an algorithm for the high multiplicity job shop scheduling problem with the objective of minimizing the total holding cost by appropriately rounding an optimal solution to a fluid relaxation in which we replace discrete jobs with the flow of a continuous fluid the algorithm solves the fluid relaxation optimally and then aims to keep the schedule in the discrete network close to the schedule given by the fluid relaxation if the number of jobs from each type grow linearly withn then the algorithm is within an additive factoro n from the optimal which scales aso n2 thus it is asymptotically optimal we report computational results on benchmark instances chosen from the or library comparing the performance of the proposed algorithm and several commonly used heuristic methods these results suggest that for problems of moderate to high multiplicity the proposed algorithm outperforms these methods and for very high multiplicity the overperformance is dramatic for problems of low to moderate multiplicity however the relative errors of the heuristic methods are comparable to those of the proposed algorithm and the best of these methods performs better overall than the proposed method

kernel based reinforcement learning in average cost problems reinforcement learning rl is concerned with the identification of optimal controls in markov decision processes mdps where no explicit model of the transition probabilities is available we propose a class of rl algorithms which always produces stable estimates of the value function in detail we use local averaging methods to construct an approximate dynamic programming adp algorithm nearest neighbor regression grid based approximations and trees can all be used as the basis of this approximation we provide a thorough theoretical analysis of this approach and we demonstrate that adp converges to a unique approximation in continuous state average cost mdps in addition we prove that our method is consistent in the sense that an optimal approximate strategy is identified asymptotically with regard to a practical implementation we suggest a reduction of adp to standard dynamic programming in an artificial finite state mdp

average cost markov decision processes optimality conditions we are concerned in this paper with discrete time markov decision processes mdps with borel state and action spaces x and a respectively and the long run expected average cost criterion when x is a denumerable set man y necessary and or sufficient conditions for the existence of optimal control policies are known however when x is a borel space i e a borel subset of a complete separable metric space most of the available results impose on the mdp very restrictive topological conditions e g compactness and or strong recurrence assumptions such as doeblin s condition see e g 4 9 12 and their references another related work is 7 where we have studied mdps from the viewpoint of the recurrence or ergodicity properties of the state process ln the present paper however we are concerned with the existence of average optimal policies by looking at static optimization problems see condition cs in section 4 related in sorne cases equivalent to the existence of a bounded solution to the so called optimality equation see c4 in section 4 these optimization problems are dual in the sense that under appropriate conditions the existence of an optimal solution to one of the problems implies existence of an optimal

optimal eviction policies for stochastic address traces the eviction problem for memory hierarchies is studied for the hidden markov reference model hmrm of the memory trace showing how miss minimization can be naturally formulated in the optimal control setting in addition to the traditional version assuming a buffer of fixed capacity a relaxed version is also considered in which buffer occupancy can vary and its average is constrained resorting to multiobjective optimization viewing occupancy as a cost rather than as a constraint the optimal eviction policy is obtained by composing solutions for the individual addressable items this approach is then specialized to the least recently used stack model lrusm a type of hmrm often considered for traces which includes v 1 parameters where v is the size of the virtual space a gain optimal policy for any target average occupancy is obtained which i is computable in time o v from the model parameters ii is optimal also for the fixed capacity case and iii is characterized in terms of priorities with the name of least profit rate lpr policy an o logc upper bound being c the buffer capacity is derived for the ratio between the expected miss rate of lpr and that of opt the optimal off line policy the upper bound is tightened to o 1 under reasonable constraints on the lrusm parameters using the stack distance framework an algorithm is developed to compute the number of misses incurred by lpr on a given input trace simultaneously for all buffer capacities in time o logv per access finally some results are provided for miss minimization over a finite horizon and over an infinite horizon under bias optimality a criterion more stringent than gain optimality

a mixed value and policy iteration method for stochastic control with universally measurable policies we consider stochastic optimal control models with borel spaces and universally measurable policies for such models the standard policy iteration is known to have difficult measurability issues and cannot be carried out in general we present a mixed value and policy iteration method that circumvents this difficulty the method allows the use of stationary policies in computing the optimal cost function in a manner that resembles policy iteration it can also be used to address similar difficulties of policy iteration in the context of upper and lower semicontinuous models we analyze the convergence of the method in infinite horizon total cost problems for the discounted case where the one stage costs are bounded and for the undiscounted case where the one stage costs are nonpositive or nonnegative for undiscounted total cost problems with nonnegative one stage costs we also give a new convergence theorem for value iteration that shows that value iteration converges whenever it is initialized with a f

some monotonicity results for partially observed markov decision processes this paper provides sufficient conditions for the optimal value in a discrete time finite partially observed markov decision process to be monotone on the space of state probability vectors ordered by likelihood ratios the paper also presents sufficient conditions for the optimal policy to be monotone in a simple machine replacement problem and in the general case for the optimal policy to be bounded from below by an easily calculated monotone function

average cost markov decision processes with weakly continuous transition probabilities this paper presents sufficient conditions for the existence of stationary optimal policies for average cost markov decision processes with borel state and action sets and weakly continuous transition probabilities the one step cost functions may be unbounded and the action sets may be noncompact the main contributions of this paper are i general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions ii a sufficient condition for the average cost optimality of a stationary policy in the form of optimality inequalities and iii approximations of average cost optimal actions by discount optimal actions

stability and asymptotic optimality of generalized maxweight policies it is shown that stability of the celebrated maxweight or back pressure policies is a consequence of the following interpretation either policy is myopic with respect to a surrogate value function of a very special form in which the marginal disutility at a buffer vanishes for a vanishingly small buffer population this observation motivates the h maxweight policy defined for a wide class of functions h these policies share many of the attractive properties of the maxweight policy as follows i arrival rate data is not required in the policy ii under a variety of general conditions the policy is stabilizing when h is a perturbation of a monotone linear function a monotone quadratic or a monotone lyapunov function for the fluid model iii a perturbation of the relative value function for a workload relaxation gives rise to a myopic policy that is approximately average cost optimal in heavy traffic with logarithmic regret the first results are obtained for a general markovian network model asymptotic optimality is established for a general markovian scheduling model with a single bottleneck and with homogeneous servers

the laplacian pyramid as a compact image code we describe a technique for image encoding in which local operators of many scales but identical shape serve as the basis functions the representation differs from established techniques in that the code elements are localized in spatial frequency as well as in space pixel to pixel correlations are first removed by subtracting a lowpass filtered copy of the image from the image itself the result is a net data compression since the difference or error image has low variance and entropy and the low pass filtered image may represented at reduced sample density further data compression is achieved by quantizing the difference image these steps are then repeated to compress the low pass image iteration of the process at appropriately expanded scales generates a pyramid data structure the encoding process is equivalent to sampling the image with laplacian operators of many scales thus the code tends to enhance salient image features a further advantage of the present code is that it is well suited for many image analysis tasks as well as for image compression fast algorithms are described for coding and decoding

ergodic control of partially observed markov chains the ergodic or long run average cost control problem for a partially observed finite state markov chain is studied via the associated fully observed separated control problem for the nonlinear filter dynamic programming equations for the latter are derived leading to existence and characterization of optimal stationary policies

a weighted markov decision process the two most commonly considered reward criteria for markov decision processes are the discounted reward and the long term average reward the first tends to neglect the future concentrating on the short term rewards while the second one tends to do the opposite we consider a new reward criterion consisting of the weighted combination of these two criteria thereby allowing the decision maker to place more or less emphasis on the short term versus the long term rewards by varying their weights the mathematical implications of the new criterion include the deterministic stationary policies can be outperformed by the randomized stationary policies which in turn can be outperformed by the nonstationary policies an optimal policy might not exist we present an iterative algorithm for computing an e optimal nonstationary policy with a very simple structure

potential based online policy iteration algorithms for markov decision processes performance potentials play a crucial role in performance sensitivity analysis and policy iteration of markov decision processes the potentials can be estimated on a single sample path of a markov process in this paper we propose two potential based online policy iteration algorithms for performance optimization of markov systems the algorithms are based on online estimation of potentials and stochastic approximation we prove that with these two algorithms the optimal policy can be attained after a finite number of iterations a simulation example is given to illustrate the main ideas and the convergence rates of the algorithms

on the adaptive control of a class of partially observed markov decision processes this paper is concerned with the adaptive control problem over the infinite horizon for partially observable markov decision processes whose transition functions are parameterized by an unknown vector we treat finite models and impose relatively mild assumptions on the transition function provided that a sequence of parameter estimates converging in probability to the true parameter value is available we show that the certainty equivalence adaptive policy is optimal in the long run average sense

controlled markov chains and stochastic networks controlled markov chains with average cost criterion and with special cost and transition structures are studied existence of optimal stationary strategies is established for the average cost criterion corresponding dynamic programming equations are derived a stochastic network problem that includes interconnected queues as a special case is described and studied within this framework

connecting the dots semi analytical and random walk numerical solutions of the diffusion reaction equation with stochastic initial conditions we study a system with bimolecular irreversible kinetic reaction a b a where the underlying transport of reactants is governed by diffusion and the local reaction term is given by the law of mass action we consider the case where the initial concentrations are given in terms of an average and a white noise perturbation our goal is to solve the diffusion reaction equation which governs the system and we tackle it with both analytical and numerical approaches to obtain an analytical solution we develop the equations of moments and solve them approximately to obtain a numerical solution we develop a grid less monte carlo particle tracking approach where diffusion is modeled by a random walk of the particles and reaction is modeled by annihilation of particles the probability of annihilation is derived analytically from the particles co location probability we rigorously derive the relationship between the initial number of particles in the system and the amplitude of white noise represented by that number this enables us to compare the particle simulations and the approximate analytical solution and offer an explanation of the late time discrepancies

risk sensitive optimal control of hidden markov models structural results the authors consider a risk sensitive optimal control problem for finite state and action spaces hidden markov models hmm they present results of an investigation on the nature and structure of risk sensitive controllers for hmm several general structural results are presented as well as a particular case study of a popular benchmark problem for the latter they obtain structural results for the optimal risk sensitive controller and compare it to that of the risk neutral controller furthermore they show that indeed the risk sensitive controller and its corresponding information state converge to the known solutions for the risk neutral situation as the risk factor goes to zero they also study the infinite and general risk aversion cases

completion of squares revisited and extended due to its simplicity the completion of squares technique is quite popular in linear optimal control however this simple technique is limited to linear quadratic gaussian systems in this note by interpreting the completion of squares from a new angle we extend this technique to performance optimization of general markov systems with the long run average criterion leading to a new approach to performance optimization based on direct comparisons of the performance of two policies

on the optimality equation for average cost markov control processes with feller transition probabilities we consider markov control processes with borel state space and feller transition probabilities satisfying some generalized geometric ergodicity conditions we provide a new theorem on the existence of a solution to the average cost optimality equation

a recurrence principle for stochastic difference inclusions the invariance principle is extended to a recurrence principle and is developed for stochastic difference inclusions for these systems random solutions are not unique under appropriate lyapunov like conditions it is established for every random solution that almost every complete sample path converges to the largest weakly totally recurrent set contained in a level set of the lyapunov like function such a set is not larger and is sometimes smaller than the largest weakly invariant set contained in the level set the principle is useful for establishing robust uniform asymptotic stability in probability or robust uniform strong recurrence under weak lyapunov conditions for stochastic discrete time control systems that employ discontinuous feedback laws examples demonstrate the achieved results

average cost optimal stationary policies in infinite state markov decision processes with unbounded costs we deal with infinite state markov decision processes with unbounded costs three simple conditions based on the optimal discounted value function guarantee the existence of an expected average cost optimal stationary policy sufficient conditions are the existence of a distinguished state of smallest discounted value and a single stationary policy inducing an irreducible ergodic markov chain for which the average cost of a first passage from any state to the distinguished state is finite a result to verify this is also given two examples illustrate the ease of applying the criteria

phase transitions for controlled markov chains on infinite graphs this paper is concerned with some long run average cost problems for controlled markov chains with a denumerable state space the criterion to be optimized contains both reward and penalty functions as a trade off between reward and penalty we observe certain phase transition phenomena our results also provide a stochastic optimal control interpretation for phase transitions of discrete homopolymers with finite attracting potentials

dsim a dissimilarity based image clutter metric for targeting performance previous image clutter metrics were proposed on the thought that clutter was just a perceptual effect while we identify clutter as both perceptual and cognitive effects under this identification we give a new definition of image clutter metric by analyzing the research results in the fields of visual psychology and psychophysics according to the definition we further put forward a dissimilarity dsim based image clutter metric which can also be taken as a kind of hvs based signal to clutter ratio the earlier image clutter metrics produced limited success in predicting targeting performance mainly since they did not consider brain cognitive characteristics we develop a brain cognitive dissimilarity measure bcdm as a quantitative estimate of the selection weights which are allocated by brain attentional mechanism to affect visual selection processes a human vision perceptual dissimilarity measure vpdm fully embodying vision perceptual properties is first established between the target and clutter images and then we utilize the bcdm between the two images as selection weights to pool the vpdm to be a clutter metric which can be called dsim metric the metric is tested in search 2 dataset provided by tno human factors research institute of netherlands error analysis and correlation tests demonstrate that the dsim metric makes a more significant improvement than previously proposed metrics in predicting 62 observers targeting performances including detection probability false alarm probability and search time

the n th order bias optimality for multichain markov decision processes in this paper we propose a new approach to the theory of finite multichain markov decision processes mdps with different performance optimization criteria we first propose the concept of nth order bias then using the average reward and bias difference formulas derived in this paper we develop an optimization theory for finite mdps that covers a complete spectrum from average optimality bias optimality to all high order bias optimality in a unified way the approach is simple direct natural and intuitive it depends neither on laurent series expansion nor on discounted mdps we also propose one phase policy iteration algorithms for bias and high order bias optimal policies which are more efficient than the two phase algorithms in the literature furthermore we derive high order bias optimality equations this research is a part of our effort in developing sensitivity based learning and optimization theory

markov decision processes with sample path constraints the communicating case we consider time average markov decision processes mdps which accumulate a reward and cost at each decision epoch a policy meets the sample path constraint if the time average cost is below a specified value with probability one the optimization problem is to maximize the expected average reward over all policies that meet the sample path constraint the sample path constraint is compared with the more commonly studied constraint of requiring the average expected cost to be less than a specified value although the two criteria are equivalent for certain classes of mdps their feasible and optimal policies differ for many nontrivial problems in general there does not exist optimal or nearly optimal stationary policies when the expected average cost constraint is employed assuming that a policy exists that meets the sample path constraint we establish that there exist nearly optimal stationary policies for communicating mdps a parametric linear programming algorithm is given to construct nearly optimal stationary policies the discussion relies on well known results from the theory of stochastic processes and linear programming the techniques lead to simple proofs of the existence of optimal and nearly optimal stationary policies for unichain and deterministic mdps respectively

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

workload models for stochastic networks value functions and performance evaluation this paper concerns control and performance evaluation for stochastic network models structural properties of value functions are developed for controlled brownian motion cbm and deterministic fluid workload models leading to the following conclusions outside of a null set of network parameters the following hold i the fluid value function is a smooth function of the initial state under further minor conditions the fluid value function satisfies the derivative boundary conditions that are required to ensure it is in the domain of the extended generator for the cbm model exponential ergodicity of the cbm model is demonstrated as one consequence ii the fluid value function provides a shadow function for use in simulation variance reduction for the stochastic model the resulting simulator satisfies an exact large deviation principle while a standard simulation algorithm does not satisfy any such bound iii the fluid value function provides upper and lower bounds on performance for the cbm model this follows from an extension of recent linear programming approaches to performance evaluation

continuous time markov decision processes with nth bias optimality criteria in this paper we study the nth bias optimality problem for finite continuous time markov decision processes mdps with a multichain structure we first provide nth bias difference formulas for two policies and present some interesting characterizations of an nth bias optimal policy by using these difference formulas then we prove the existence of an nth bias optimal policy by using nth bias optimal policy iteration algorithms and show that such an nth bias optimal policy can be obtained in a finite number of policy iterations

a probabilistic analysis of bias optimality in unichain markov decision processes focuses on bias optimality in unichain finite state and action space markov decision processes using relative value functions we present methods for evaluating optimal bias this leads to a probabilistic analysis which transforms the original reward problem into a minimum average cost problem the result is an explanation of how and why bias implicitly discounts future rewards

on the optimality of myopic production controls for single server continuous flow manufacturing systems in this note we consider a broad class of dynamic scheduling problems associated with a single server multiclass continuous flow manufacturing system using a general framework we provide conditions under which the solution to these problems is the implementation of a class of production controls called myopic scheduling policies the proof of optimality which is intuitively appealing applies to more general production models than existing proofs in the literature which typically either use the maximum principle or solve the hamilton jacobi bellman hjb equation we also present several counter examples that explicitly illustrate the potential limitations of myopic scheduling policies

do we really need a large number of particles to simulate bimolecular reactive transport with random walk methods a kernel density estimation approach random walk particle tracking methods are a computationally efficient family of methods to solve reactive transport problems while the number of particles in most realistic applications is in the order of 10 6 10 9 the number of reactive molecules even in diluted systems might be in the order of fractions of the avogadro number thus each particle actually represents a group of potentially reactive molecules the use of a low number of particles may result not only in loss of accuracy but also may lead to an improper reproduction of the mixing process limited by diffusion recent works have used this effect as a proxy to model incomplete mixing in porous media in this work we propose using a kernel density estimation kde of the concentrations that allows getting the expected results for a well mixed solution with a limited number of particles the idea consists of treating each particle as a sample drawn from the pool of molecules that it represents this way the actual location of a tracked particle is seen as a sample drawn from the density function of the location of molecules represented by that given particle rigorously represented by a kernel density function the probability of reaction can be obtained by combining the kernels associated to two potentially reactive particles we demonstrate that the observed deviation in the reaction vs time curves in numerical experiments reported in the literature could be attributed to the statistical method used to reconstruct concentrations fixed particle support from discrete particle distributions and not to the occurrence of true incomplete mixing we further explore the evolution of the kernel size with time linking it to the diffusion process our results show that kdes are powerful tools to improve computational efficiency and robustness in reactive transport simulations and indicates that incomplete mixing in diluted systems should be modeled based on alternative mechanistic models and not on a limited number of particles

optimal stationary policies in general state space markov decision chains with finite action sets the result of sennott 9 on the existence of optimal stationary policies in countable state markov decision chains with finite action sets is generalized to arbitrary state space markov decision chains the assumption of finite action sets occurring in a global countable action space allows a particularly simple theoretical structure for the general state space markov decision chain two examples illustrate the results example 1 is a system of parallel queues with stochastic work requirements a movable server with controllable service rate and a reject option example 2 is a system of parallel queues with stochastic controllable inputs a movable server with fixed service rates and a reject option

super overtaking optimal policies for markov control processes abstract in this paper we study the average cost optimization problem for discrete time markov control processes on borel space with possibly unbounded costs under proper ergodicity assumptions we show the existence of stationary policies for which asymptotic as n behavior of some functions of n horizon costs can be better than for overtaking optimal policies

limiting average criteria for nonstationary markov decision processes this paper deals with the so called limiting average criteria for nonstationary markov decision processes with possibly unbounded rewards and borel state space a new set of conditions is provided under which the existence of both a solution to the optimality equations and the limiting average epsilon geq 0 optimal markov policies is derived also a rolling horizon algorithm for computing limiting average epsilon 0 optimal markov policies is developed furthermore the results in this paper are illustrated by several examples such as the water regulation problem

simultaneous identification and adaptive control of unknown systems over finite parameter sets the problem considered is one of simultaneously identifying an unknown system while adequately controlling it the system can be any fairly general discrete time system and the cost criterion can be either of a discounted type or of a long term average type the chief restriction being that the unknown parameter lies in a finite parameter set for a previously introduced scheme of identification and control based on biased maximum likelihood estimates it is shown that 1 every cesaro limit point of the parameter estimates is closed loop equivalent to the unknown parameter 2 for both the discounted and long term average cost criteria the adaptive control law cesaro converges to the set of optimal control laws and 3 in the case of the long term average cost criterion the actual cost incurred by the use of the adaptive controller is optimal and cannot be bettered even if one knew the value of the unknown parameter at the start

reactive random walk particle tracking and its equivalence with the advection diffusion reaction equation we acknowledge the support of the european research council erc through the project mhetscale 617511 and the spanish ministry of economy industry and competitiveness and european regional development fund through the project mecmat cgl2016 80022 r the numerical data can be reproduced following the numerical algorithms and parameter sets given in the paper alternatively the data can be obtained upon request from the corresponding author

average cost optimal policies for markov control processes with borel state space and unbounded costs abstract we show the existence of average cost optimal stationary policies for markov control processes with borel state space and unbounded costs per stage under a set of assumptions recently introduced by l i sennott 1989 for control processes with countable state space and finite control sets

discrete time controlled markov processes with average cost criterion a survey this work is a survey of the average cost control problem for discrete time markov processes the authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades the exposition ranges from finite to borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies the authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography the authors have also identified several important questions that are still open to investigation

risk sensitive optimal control for markov decision processes with monotone cost the existence of an optimal feedback law is established for the risk sensitive optimal control problem with denumerable state space the main assumptions imposed are irreducibility and anear monotonicity condition on the one step cost function a solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law

value iteration in a class of communicating markov decision chains with the average cost criterion markov decision processes with denumerable state space and discrete time parameter are considered the performance index of a control policy is the lim sup expected average cost criterion and the the main structural restrictions on the model are the following i under the action of any stationary policy the state space is a communicating class ii the cost function has an almost monotone or penalized structure v s borkar siam j control optim 21 1983 pp 652 666 22 1984 pp 965 978 and iii some stationary policy induces an ergodic chain with finite average cost in this context it is shown that the value iteration scheme can be used to construct convergent approximations of a solution to the optimality equation as well as a sequence of stationary policies whose limit points are optimal

exponential and uniform ergodicity of markov processes general characterizations of geometric convergence for markov chains in discrete time on a general state space have been developed recently in considerable detail here we develop a similar theory for varphi irreducible continuous time processes and consider the following types of criteria for geometric convergence 1 the existence of exponentially bounded hitting times on one and then all suitably small sets 2 the existence of foster lyapunov or drift conditions for any one and then all skeleton and resolvent chains 3 the existence of drift conditions on the extended generator tilde mathscr a of the process we use the identity tilde mathscr a r beta beta r beta i connecting the extended generator and the resolvent kernels r beta to show that under a suitable aperiodicity assumption exponential convergence is completely equivalent to any of criteria 1 3 these conditions yield criteria for exponential convergence of unbounded as well as bounded functions of the chain they enable us to identify the dependence of the convergence on the initial state of the chain and also to illustrate that in general some smoothing is required to ensure convergence of unbounded functions

stochastic dominance constrained markov decision processes we are interested in risk constraints for infinite horizon discrete time markov decision processes mdps starting with average reward mdps we show that increasing concave stochastic dominance constraints on the empirical distribution of reward lead to linear constraints on occupation measures an optimal policy for the resulting class of dominance constrained mdps is obtained by solving a linear program we compute the dual of this linear program to obtain average dynamic programming optimality equations that reflect the dominance constraint in particular a new pricing term appears in the optimality equations corresponding to the dominance constraint we show that many types of stochastic orders can be used in place of the increasing concave stochastic order we also carry out a parallel development for discounted reward mdps with stochastic dominance constraints a portfolio optimization example is used to motivate the paper

optimal inventory control with jump diffusion and nonlinear dynamics in the demand in this paper we consider an inventory control problem with a nonlinear evolution and a jump diffusion demand model this work extends the earlier inventory model proposed by benkherouf and johnson math methods oper res 76 2012 pp 377 393 by including a general jump process however as those authors note their techniques are not applicable to models with demand driven by jump diffusion processes with drift therefore the combination of diffusion and general compound poisson demands is not completely solved from the dynamic programming principle we transform the problem into a set of quasi variational inequalities q v i the difficulty arises when solving the q v i because the second derivative and integration term appear in the same inequality our technique is to construct a set of coupled auxiliary functions then an analytical study of the q v i implies the existence and uniqueness of an s s policy

on optimal zero delay coding of vector markov sources optimal zero delay coding quantization of a vector valued markov source driven by a noise process is consid ered using a stochastic control problem formulation the exis tence and structure of optimal quantization policies are studied for a finite horizon problem with bounded per stage distortion measure the existence of an optimal zero delay quantization policy is shown provided that the quantizers allowed are ones with convex codecells the bounded distortion assumption is relaxed to cover cases that include the linear quadratic gaussian problem for the infinite horizon problem and a stationary markov source the optimality of deterministic markov coding policies is shown the existence of optimal stationary markov quantization policies is also shown provided randomization that is shared by the encoder and the decoder is allowed

markov decision problems and state action frequencies consider a controlled markov chain with countable state and action spaces basic quantities that determine the values of average cost functionals are identified under some regularity conditions these turn out to be a collection of numbers one for each state action pair describing for each state the relative number of uses of each action these conditional frequencies which are defined pathwise are shown to determine the state action frequencies that in the finite case are known to determine the costs this is extended to the countable case allowing for unbounded costs the space of frequencies is shown to be compact and convex and the extreme points are identified with stationary deterministic policies conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given these problems include the standard markov decision process as well as constrained optimization both in terms of average cost functionals and variability sensitive optimization an application to a queueing problem is given where these results imply the existence and explicit computation of optimal policies in constrained optimization problems the pathwise definition of the conditional frequencies implies that their values can be controlled directly moreover they depend only on the limiting behavior of the control this has immediate application to adaptive control of markov chains including adaptive control under constraints

the capacity of channels with feedback in this paper we introduce a general framework for treating channels with memory and feedback first we prove a general feedback channel coding theorem based on massey s concept of directed information second we present coding results for markov channels this requires determining appropriate sufficient statistics at the encoder and decoder we give a recursive characterization of these sufficient statistics third a dynamic programming framework for computing the capacity of markov channels is presented fourth it is shown that the average cost optimality equation acoe can be viewed as an implicit single letter characterization of the capacity fifth scenarios with simple sufficient statistics are described sixth error exponents for channels with feedback are presented

technical communique maximizing the set of recurrent states of an mdp subject to convex constraints this paper focuses on the design of time homogeneous fully observed markov decision processes mdps with finite state and action spaces the main objective is to obtain policies that generate the maximal set of recurrent states subject to convex constraints on the set of invariant probability mass functions we propose a design method that relies on a finitely parametrized convex program inspired on principles of entropy maximization a numerical example is provided to illustrate these ideas

stability and instability of fluid models for reentrant lines reentrant lines can be used to model complex manufacturing systems such as wafer fabrication facilities as the first step to the optimal or near optimal scheduling of such lines we investigate their stability in light of a recent theorem of dai dai j g 1995 on positive harris recurrence of multiclass queueing networks a unified approach via fluid models ann appl probab 5 49 77 which states that a scheduling policy is stable if the corresponding fluid model is stable we study the stability and instability of fluid models to do this we utilize piecewise linear lyapunov functions we establish stability of first buffer first served fbfs and last buffer first served lbfs disciplines in all reentrant lines and of all work conserving disciplines in any three buffer reentrant lines for the four buffer network of lu and kumar we characterize the stability region of the lu and kumar policy and show that it is also the global stability region for this network we also study stability and instability of kelly type networks in particular we show that not all work conserving policies are stable for such networks however all work conserving policies are stable in a ring network

weak conditions for average optimality in markov control processes abstract in this paper we study the existence of average cost optimal policies for markov control processes on borel spaces and with possibly unbounded costs and controls under assumptions weaker than those used in the previous literature we show that the optimality inequality holds everywhere and that there exists a stationary policy which is optimal whenever the initial state lies in a possibly proper subset of the state space this is in contrast to previous works where an implicit unichain assumption ensures optimality for all initial states

the optimal observability of partially observable markov decision processes discrete state space we consider autonomous partially observable markov decision processes where the control action influences the observation process only considering entropy as the cost incurred by the markov information state process the optimal observability problem is posed as a markov decision scheduling problem that minimizes the infinite horizon cost this scheduling problem is shown to be equivalent to minimization of an entropy measure called estimation entropy which is related to the invariant measure of the information state

average control of markov decision processes with feller transition probabilities and general action spaces abstract this paper studies the average control problem of discrete time markov decision processes mdps for short with general state space feller transition probabilities and possibly non compact control constraint sets a x two hypotheses are considered either the cost function c is strictly unbounded or the multifunctions a r x a a x c x a r are upper semicontinuous and compact valued for each real r for these two cases we provide new results for the existence of a solution to the average cost optimality equality and inequality using the vanishing discount approach we also study the convergence of the policy iteration approach under these conditions it should be pointed out that we do not make any assumptions regarding the convergence and the continuity of the limit function generated by the sequence of relative difference of the discounted value functions and the poisson equations as often encountered in the literature