transient policies in discrete dynamic programming linear programming including suboptimality tests and additional constraints this paper investigates the computation of transient optimal policies in discrete dynamic programming the model is quite general it may contain transient as well as nontransient policies and the transition matrices are not necessarily substochastic a functional equation for the so called transient value vector is derived and the concept of superharmonicity is introduced this concept provides the linear program to compute the transientvalue vector and a transient optimal policy we also discuss the elimination of suboptimal actions the solution of problems with additional constraints and the computation of an efficient policy for a multiple objective dynamic programming problem

finding the k best policies in a finite horizon markov decision process abstract directed hypergraphs represent a general modelling and algorithmic tool which have been successfully used in many different research areas such as artificial intelligence database systems fuzzy systems propositional logic and transportation networks however modelling markov decision processes using directed hypergraphs has not yet been considered in this paper we consider finite horizon markov decision processes mdps with finite state and action space and present an algorithm for finding the k best deterministic markov policies that is we are interested in ranking the first k deterministic markov policies in non decreasing order using an additive criterion of optimality the algorithm uses a directed hypergraph to model the finite horizon mdp it is shown that the problem of finding the optimal policy can be formulated as a minimum weight hyperpath problem and be solved in linear time with respect to the input data representing the mdp using different additive optimality criteria

rate of convergence of empirical measures and costs in controlled markov chains and transient optimality the purpose of this paper is two fold first bounds on the rate of convergence of empirical measures in controlled markov chains are obtained under some recurrence conditions these include bounds obtained through large deviations and central limit theorem arguments these results are then applied to optimal control problems bounds on the rate of convergence of the empirical measures that are uniform over different sets of policies are derived resulting in bounds on the rate of convergence of the costs finally new optimal control problems that involve not only average cost criteria but also measures on the transient behavior of the cost namely the rate of convergence are introduced and applied to a problem in telecommunications the solution to these problems rely on the bounds introduced in previous sections

denumerable constrained markov decision processes and finite approximations the purpose of this paper is two fold first to establish the theory of discounted constrained markov decision processes with a countable state and action spaces with general multi chain structure second to introduce finite approximation methods we define the occupation measures and obtain properties of the set of all achievable occupation measures under the different admissible policies we establish the optimality of stationary policies for the constrained control problem and obtain an lp with a countable number of decision variables through which stationary optimal policies are computed since for such an lp one cannot expect to find an optimal solution in a finite number of operations we present two schemes for finite approximations and establish the convergence of optimal values and policies for both the discounted and the expected average cost with unbounded cost sometimes it turns out to be easier to solve the problem with infinite state space than the problem with finite yet large state space based on the optimal policy for the problem with infinite state space we construct policies which are almost optimal for the problem with truncated state space this method is applied to obtain an e optimal policy for a problem of optimal priority assignment under constraints for a system of k finite queues

constrained markov decision models with weighted discounted rewards this paper deals with constrained optimization of markov decision processes both objective function and constraints are sums of standard discounted rewards but each with a different discount factor such models arise e g in production and in applications involving multiple time scales we prove that it a feasible policy exists then there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is bounded by the number of constraints optimality of such policies for multi criteria problems is also established these new policies have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints this result is new even for constrained optimization with a single discount factor where the optimality of randomized stationary policies is known however a randomized stationary policy may req

an algorithm to identify and compute average optimal policies in multichain markov decision processes this paper concerns discrete time finite state multichain mdps with compact action sets the optimality criterion is long run average cost simple examples illustrate that optimal stationary markov policies do not always exist we establish the existence of e optimal policies that are stationary markovian and develop an algorithm that computes these approximate optimal policies we establish a necessary and sufficient condition for the existence of an optimal policy that is stationary markovian and in case that such an optimal policy exists the algorithm computes it

multichain markov decision processes with a sample path constraint a decomposition approach we consider finite state finite action markov decision processes which accumulate both a reward and a cost at each decision epoch we study the problem of finding a policy that maximizes the expected long run average reward subject to the constraint that the long run average cost be no greater than a given value with probability one we establish that if there exists a policy that meets the constraint then there exists an e optimal stationary policy furthermore an algorithm is outlined to locate the e optimal stationary policy the proof of the result hinges on a decomposition of the state space into maximal recurrent classes and a set of transient states

heavy traffic limits for queues with many exponential servers two different kinds of heavy traffic limit theorems have been proved for s server queues the first kind involves a sequence of queueing systems having a fixed number of servers with an associated sequence of traffic intensities that converges to the critical value of one from below the second kind which is often not thought of as heavy traffic involves a sequence of queueing systems in which the associated sequences of arrival rates and numbers of servers go to infinity while the service time distributions and the traffic intensities remain fixed with the traffic intensities being less than the critical value of one in each case the sequence of random variables depicting the steady state number of customers waiting or being served diverges to infinity but converges to a nondegenerate limit after appropriate normalization however in an important respect neither procedure adequately represents a typical queueing system in practice because in the heavy traffic limit an arriving customer is either a

adaptive control of constrained markov chains adaptive control of finite state markov chains is discussed the optimal performance is characterized through the minimization of a long run average cost functional subject to constraints on several other such functionals under mild structural and feasibility conditions two explicit adaptive control policies are exhibited for the case where the transition probabilities are unknown the policies are optimal under the constrained optimization criterion they rely on a powerful estimation scheme which provides consistent estimators for the transition probabilities this scheme is of independent interest as it provides strong consistency under a large number of adaptive schemes and is independent of any identifiability conditions as an application an optimal adaptive policy is derived for a system of k competing queues with countable state space for which the constrained criteria arise naturally in the context of communication networks

queues with many servers the virtual waiting time process in the qed regime we consider a first come first served multiserver queue in the quality and efficiency driven qed regime in this regime which was first formalized by halfin and whitt the number of servers n is not small servers utilization is efficiency driven while waiting time is quality driven this is equivalent to having the number of servers n being approximately equal to where r is the offered load and is a positive constant for the g dk n queue in the qed regime we analyze the virtual waiting time vn t as n increases indefinitely assuming that the service time distribution has a finite support hence the dk in g dk n it is shown that in the limit the scaled virtual waiting time is representable as a supremum over a random weighted tree s denotes a service time informally it is then argued that for large n here is the averaging of over s and the process is zero mean gaussian that summarizes all relevant information about arrivals and service times arises as a limit of an infinite server g dk process appropriately scaled the results are obtained by using both combinatorial and probabilistic arguments possible applications of our approximations include fast simulation of queues and estimation prediction of customer waiting times in the qed regime

constrained undiscounted stochastic dynamic programming in this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function the state space and action sets are assumed to be finite constraints which are linear functions of the state action frequencies are allowed in the general multichain case an optimal policy will be a randomized nonstationary policy an algorithm to compute such an optimal policy is presented furthermore sufficient conditions for optimal policies to be stationary are derived there are many applications for constrained undiscounted stochastic dynamic programming e g in multiple objective markovian decision models

hazard rate scaling of the abandonment distribution for the gi m n gi queue in heavy traffic we obtain a heavy traffic limit for the gi m n gi queue which includes the entire patience time distribution our main approach is to scale the hazard rate function of the patience time distribution in such a way that our resulting diffusion approximation contains the entire hazard rate function we then show through numerical studies that for various performance measures our approximations tend to outperform those commonly used in practice the robustness of our results is also demonstrated by applying them to solving constraint satisfaction problems arising in the context of telephone call centers

achieving target state action frequencies in multichain average reward markov decision processes in this paper we address a basic problem that arises naturally in average reward markov decision processes with constraints and or nonstandard payoff criteria given a feasible state action frequency vector the target construct a policy whose state action frequencies match those of the target vector while it is well known that the solution to this problem cannot in general be found in the space of stationary randomized policies we construct a solution that has ultimately stationary structure it consists of two stationary policies where the first one is used initially and then the switch to the second one is made at a certain random switching time the computational effort required to construct this solution is minimal we also show that our problem can always be solved by a stationary policy if the original mdp is extended by adding certain states and actions the solution in the original mdp is obtained by mapping the solution in the extended mdp back to the original process

markov decision processes with sample path constraints the communicating case we consider time average markov decision processes mdps which accumulate a reward and cost at each decision epoch a policy meets the sample path constraint if the time average cost is below a specified value with probability one the optimization problem is to maximize the expected average reward over all policies that meet the sample path constraint the sample path constraint is compared with the more commonly studied constraint of requiring the average expected cost to be less than a specified value although the two criteria are equivalent for certain classes of mdps their feasible and optimal policies differ for many nontrivial problems in general there does not exist optimal or nearly optimal stationary policies when the expected average cost constraint is employed assuming that a policy exists that meets the sample path constraint we establish that there exist nearly optimal stationary policies for communicating mdps a parametric linear programming algorithm is given to construct nearly optimal stationary policies the discussion relies on well known results from the theory of stochastic processes and linear programming the techniques lead to simple proofs of the existence of optimal and nearly optimal stationary policies for unichain and deterministic mdps respectively

queues with many servers and impatient customers the asymptotic many server queue with abandonments g gi n gi is considered in the quality and efficiency driven qed regime here the number of servers and the offered load are related via the square root rule as the number of servers increases indefinitely qed performance entails short waiting times and scarce abandonments high quality jointly with high servers utilization high efficiency which is feasible when many servers cater to a single queue for the g gi n gi queue we derive diffusion approximations for both its queue length and virtual waiting time processes special cases for which closed form analysis is provided are the g m n gi and g d n gi queues thus expanding and generalizing existing results

markov decision problems and state action frequencies consider a controlled markov chain with countable state and action spaces basic quantities that determine the values of average cost functionals are identified under some regularity conditions these turn out to be a collection of numbers one for each state action pair describing for each state the relative number of uses of each action these conditional frequencies which are defined pathwise are shown to determine the state action frequencies that in the finite case are known to determine the costs this is extended to the countable case allowing for unbounded costs the space of frequencies is shown to be compact and convex and the extreme points are identified with stationary deterministic policies conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given these problems include the standard markov decision process as well as constrained optimization both in terms of average cost functionals and variability sensitive optimization an application to a queueing problem is given where these results imply the existence and explicit computation of optimal policies in constrained optimization problems the pathwise definition of the conditional frequencies implies that their values can be controlled directly moreover they depend only on the limiting behavior of the control this has immediate application to adaptive control of markov chains including adaptive control under constraints

customer abandonment in many server queues we study g g n gi queues in which customer patience times are independent identically distributed following a general distribution when a customer s waiting time in queue exceeds his patience time the customer abandons the system without service for the performance of such a system we focus on the abandonment process and the queue length process we prove that under some conditions a deterministic relationship between the two stochastic processes holds asymptotically under the diffusion scaling when the number of servers n goes to infinity these conditions include a minor assumption on the arrival processes that can be time nonhomogeneous and a key assumption that the sequence of diffusion scaled queue length processes indexed by n is stochastically bounded we also establish a comparison result that allows one to verify the stochastic boundedness by studying a corresponding sequence of systems without customer abandonment

variability sensitive markov decision processes considered are time average markov decision processes mdps with finite state and action spaces two definitions of variability are introduced namely the expected time average variability and time average expected variability the two criteria are in general different although they can both be employed to penalize for variance in the stream of rewards for communicating mdps we construct a randomized stationary policy that is e optimal for both criteria the policy is optimal and pure for a specific variability function for general multichain mdps a state space decomposition leads to a similar result for the expected time average variability we also consider the problem of the decision maker choosing the initial state along with the policy

on the invariance principle for the first passage time new results are obtained on the connection between the invariance principle for the first passage time process and that for the underlying processes the main contribution is that the passing process may have general drift an example of the application is given which shows that this is of use in deriving limit theorems for delays from limit theorems for queue lengths when studying queueing systems with queue dependent arrival or and service completion rates

open boundary control problem for navier stokes equations including a free surface adjoint sensitivity analysis this paper develops the adjoint sensitivities to the free surface barotropic navier stokes equations in order to allow for the assimilation of measurements of currents and free surface elevations into an unsteady flow solution by open boundary control to calculate a variation in a surface variable a mapping is used in the vertical to shift the problem into a fixed domain a variation is evaluated in the transformed space from the jacobian matrix of the mapping this variation is then mapped back into the original space where it completes a tangent linear model the adjoint equations are derived using the scalar product formulas redefined for a domain with variable bounds the method is demonstrated by application to an unsteady fluid flow in a one dimensional open channel in which horizontal and vertical components of velocity are represented as well as the elevation of the free surface a 2d vertical section model this requires the proper treatment of open boundaries in both the forward and adjoint models a particular application is to the construction of a fully three dimensional coastal ocean model that allows assimilation of tidal elevation and current data however the results are general and can be applied in a wider context

time sharing policies for controlled markov chains we propose a class of nonstationary policies called policy time sharing pts which possesses several desirable properties for problems where the criteria are of the average cost type an optimal policy exists within this class the computation of optimal policies is straightforward and the implementation of this policy is easy while in the finite state case stationary policies are also known to share these properties the new policies are much more flexible in the sense that they can be applied to solve adaptive problems and they suggest new ways to incorporate the particular structure of the problem at hand into the derivation of optimal policies in addition they provide insight into the pathwise structure of controlled markov chains to use pts policies one alternates between the use of several stationary deterministic policies switching when reaching some predetermined state in some countable state cases optimal solutions of the pts type are available and easy to compute whereas optimal stationary policies are not available examples that illustrate the last point and the usefulness of the new approach are discussed involving constrained optimization problems with countable state space or compact action space

variance penalized markov decision processes we consider a markov decision process with both the expected limiting average and the discounted total return criteria appropriately modified to include a penalty for the variability in the stream of rewards in both cases we formulate appropriate nonlinear programs in the space of state action frequencies averaged or discounted whose optimal solutions are shown to be related to the optimal policies in the corresponding variance penalized mdp the analysis of one of the discounted cases is facilitated by the introduction of a cartesian product of two independent mdps

hamiltonian cycles and singularly perturbed markov chains we consider the hamiltonian cycle problem embedded in a singularly perturbed markov decision process we also consider a functional on the space of deterministic policies of the process that consists of the 1 1 entry of the fundamental matrices of the markov chains induced by the same policies we show that when the perturbation parameter e is less than or equal to 1 n2 the hamiltonian cycles of the directed graph are precisely the minimizers of our functional over the space of deterministic policies in the process we derive analytical expressions for the possiblen distinct values of the functional over the typically much larger space of deterministic policies

gaussian limits for a fork join network with nonexchangeable synchronization in heavy traffic we study a fork join network of stations with multiple servers and nonexchangeable synchronization in heavy traffic under the first come first served fcfs discipline tasks are only synchronized if all the tasks associated with the same job are completed service times of parallel tasks of each job can be correlated we jointly consider the number of tasks in each waiting buffer for synchronization with the number of tasks in each parallel service station and the number of synchronized jobs we develop a new approach to show a functional central limit theorem for these processes in the quality driven regime under general assumptions on arrival and service processes specifically we represent these processes as functionals of a sequential empirical process driven by the sequence of service vectors for each job s parallel tasks all of the limiting processes are functionals of two independent processes i e the limiting arrival process and a generalized kiefer process driven by the service vector of