optimal time abstract schedulers for ctmdps and continuous time markov games we study time bounded reachability in continuous time markov decision processes ctmdps and games ctgs for time abstract scheduler classes reachability problems play a paramount role in probabilistic model checking consequently their analysis has been studied intensively and approximation techniques are well understood from a mathematical point of view however the question of approximation is secondary compared to the fundamental question whether or not optimal control exists in this article we demonstrate the existence of optimal schedulers for the time abstract scheduler classes for ctmdps for ctgs we distinguish two cases the simple case where both players face the same restriction to use time abstract strategies symmetric ctgs and the case where one player is a completely informed adversary asymmetric ctgs while for the former case optimal strategies exist we prove that for asymmetric ctgs there is not necessarily a scheduler that attains the optimum it turns out that for ctmdps and symmetric ctgs optimal time abstract schedulers have an amazingly simple structure they converge to a memoryless scheduling policy after a finite number of steps this allows us to compute time abstract strategies with finite memory

contraction conditions for average and α discount optimality in countable state markov games with unbounded rewards the goal of this paper is to provide a theory of n person markov games with unbounded cost for a countable state space and compact action spaces we investigate both the finite and infinite horizon problems for the latter we consider the discounted cost as well as the expected average cost we present conditions for the infinite horizon problems for which equilibrium policies exist for all players within the stationary policies and show that the costs in equilibrium policies exist for all players within the stationary policies and show that the costs in equilibrium satisfy the optimality equations similar results are obtained for the finite horizon costs for which equilibrium policies are shown to exist for all players within the markov policies as special case of n person games we investigate the zero sum 2 players game for which we establish the convergence of the value iteration algorithm we conclude by studying an application of a zero sum markov game in a queueing model

mean variance optimization of discrete time discounted markov decision processes abstract in this paper we study a mean variance optimization problem in an infinite horizon discrete time discounted markov decision process mdp the objective is to minimize the variance of system rewards with the constraint of mean performance different from most of works in the literature which require the mean performance already achieve optimum we can let the discounted performance equal any constant the difficulty of this problem is caused by the quadratic form of the variance function which makes the variance minimization problem not a standard mdp by proving the decomposable structure of the feasible policy space we transform this constrained variance minimization problem to an equivalent unconstrained mdp under a new discounted criterion and a new reward function the difference of the variances of markov chains under any two feasible policies is quantified by a difference formula based on the variance difference formula a policy iteration algorithm is developed to find the optimal policy we also prove the optimality of deterministic policy over the randomized policy generated in the mean constrained policy space numerical experiments demonstrate the effectiveness of our approach

n person nonzero sum games for continuous time jump processes with varying discount factors this paper is concerned with the first passage discounted payoff criterion for continuous time n person nonzero sum stochastic games in countable states and borel action spaces where the discount factor is nonconstant for the game with unbounded reward functions and transition rates the optimality is over the set of all randomized history dependent policies after showing the uniqueness of the solution to the shapley equation we give some suitable conditions imposed on the primitive data for the existence of nash equilibria then a financial system is introduced to illustrate the applications of the main result here

penalty barrier multiplier methods for convex programming problems we study a class of methods for solving convex programs which are based on nonquadratic augmented lagrangians for which the penalty parameters are functions of the multipliers this gives rise to lagrangians which are nonlinear in the multipliers each augmented lagrangian is specified by a choice of a penalty function varphi and a penalty updating function pi the requirements on varphi are mild and allow for the inclusion of most of the previously suggested augmented lagrangians more importantly a new type of penalty barrier function having a logarithmic branch glued to a quadratic branch is introduced and used to construct an efficient algorithm convergence of the algorithms is proved for the case of pi being a sublinear function of the dual multipliers the algorithms are tested on large scale quadratically constrained problems arising in structural optimization

convergence of controlled models and finite state approximation for discounted continuous time markov decision processes with constraints abstract in this paper we consider the convergence of a sequence m n of the models of discounted continuous time constrained markov decision processes mdp to the limit one denoted by m for the models with denumerable states and unbounded transition rates under reasonably mild conditions we prove that the constrained optimal policies and the optimal values of m n converge to those of m respectively using a technique of occupation measures as an application of the convergence result developed here we show that an optimal policy and the optimal value for countable state continuous time mdp can be approximated by those of finite state continuous time mdp finally we further illustrate such finite state approximation by solving numerically a controlled birth and death system and also give the corresponding error bound of the approximation

constrained discounted markov decision processes and hamiltonian cycles this paper establishes new links between stochastic and discrete optimization we consider the following three problems for discrete time markov decision processes with finite states and action sets i find an optimal deterministic policy for a discounted problem with constraints ii find an optimal stationary policy for a weighted discounted problem with constraints iii find an optimal deterministic policy for a weighted discounted problem with constraints we formulate mathematical programs for problems i iii and show that the hamiltonian cycle problem is a special case of each of these problems therefore problems i iii are np hard we also provide new mathematical programming formulations for the hamiltonian cycle and traveling salesman problems

constrained optimization for average cost continuous time markov decision processes to the best of our knowledge this note is a first study on a multiconstrained optimization problem for average cost continuous time markov decision processes in polish spaces the optimality criterion is the long run expected average cost and the constraints are imposed on similar average costs we give suitable conditions under which the existence of a constrained optimal stationary policy is shown by introducing occupation measures and using a martingale technique also we illustrate our conditions with controlled potlach processes

on reducing a constrained gradual impulsive control problem for a jump markov model to a model with gradual control only in this paper we consider a gradual impulsive control problem for continuous time markov decision processes ctmdps with total cost criteria and constraints we develop a simple and useful method

constrained markov decision models with weighted discounted rewards this paper deals with constrained optimization of markov decision processes both objective function and constraints are sums of standard discounted rewards but each with a different discount factor such models arise e g in production and in applications involving multiple time scales we prove that it a feasible policy exists then there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is bounded by the number of constraints optimality of such policies for multi criteria problems is also established these new policies have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints this result is new even for constrained optimization with a single discount factor where the optimality of randomized stationary policies is known however a randomized stationary policy may req

optimal risk probability for first passage models in semi markov decision processes this paper studies the risk minimization problem in semi markov decision processes with denumerable states the criterion to be optimized is the risk probability or risk function that a first passage time to some target set doesn t exceed a threshold value we first characterize such risk functions and the corresponding optimal value function and prove that the optimal value function satisfies the optimality equation by using a successive approximation technique then we present some properties of optimal policies and further give conditions for the existence of optimal policies in addition a value iteration algorithm and a policy improvement method for obtaining respectively the optimal value function and optimal policies are developed finally two examples are given to illustrate the value iteration procedure and essential characterization of the risk function

nonatomic total rewards markov decision processes with multiple criteria we consider a markov decision process with an uncountable state space for which the vector performance functional has the form of expected total rewards under the single condition that initial distribution and transition probabilities are nonatomic we prove that the performance space coincides with that generated by nonrandomized markov policies we also provide conditions for the existence of optimal policies when the goal is to maximize one component of the performance vector subject to inequality constraints on other components we illustrate our results with examples of production and financial problems

first passage optimality for continuous time markov decision processes with varying discount factors and history dependent policies this paper is an attempt to study the first passage optimality criterion for continuous time markov decision processes with state dependent discount factors and history dependent policies the state space is denumerable the action space is a borel space and the transition and reward rates are unbounded under suitable conditions we show the existence of a deterministic stationary optimal policy establish the bellman optimality equation to which the value function is the unique solution and give the value and policy iteration algorithms for solving at least approximating the value function and an optimal policy furthermore we give examples about reliability and controlled birth processes with killing to illustrate the potential applications of the results obtained here and also to show the difference between the main results in this paper and those in the previous literature

a comparison of global and semi local approximation in t stage stochastic optimization the paper presents a comparison between two different flavors of nonlinear models to be used for the approximate solution of t stage stochastic optimization tso problems a typical paradigm of markovian decision processes specifically the well known class of neural networks is compared with a semi local approach based on kernel functions characterized by less demanding computational requirements to this purpose two alternative methods for the numerical solution of tso are considered one corresponding to the classic approximate dynamic programming adp and the other based on a direct optimization of the optimal control functions introduced here for the first time advantages and drawbacks in the tso context of the two classes of approximators are analyzed in terms of computational burden and approximation capabilities then their performances are evaluated through simulations in two important high dimensional tso test cases namely inventory forecasting and water reservoirs management

a simple label switching algorithm for semisupervised structural svms in structured output learning obtaining labeled data for real world applications is usually costly while unlabeled examples are available in abundance semisupervised structured classification deals with a small number of labeled examples and a large number of unlabeled structured data in this work we consider semisupervised structural support vector machines with domain constraints the optimization problem which in general is not convex contains the loss terms associated with the labeled and unlabeled examples along with the domain constraints we propose a simple optimization approach that alternates between solving a supervised learning problem and a constraint matching problem solving the constraint matching problem is difficult for structured prediction and we propose an efficient and effective label switching method to solve it the alternating optimization is carried out within a deterministic annealing framework which helps in effective constraint matching and avoiding poor local minima which are not very useful the algorithm is simple and easy to implement further it is suitable for any structured output learning problem where exact inference is available experiments on benchmark sequence labeling data sets and a natural language parsing data set show that the proposed approach though simple achieves comparable generalization performance

posterior regularization for structured latent variable models we present posterior regularization a probabilistic framework for structured weakly supervised learning our framework efficiently incorporates indirect supervision via constraints on posterior distributions of probabilistic models with latent variables posterior regularization separates model complexity from the complexity of structural constraints it is desired to satisfy by directly imposing decomposable regularization on the posterior moments of latent variables during learning we retain the computational efficiency of the unconstrained model while ensuring desired constraints hold in expectation we present an efficient algorithm for learning with posterior regularization and illustrate its versatility on a diverse set of structural constraints such as bijectivity symmetry and group sparsity in several large scale experiments including multi view learning cross lingual dependency grammar induction unsupervised part of speech induction and bitext word alignment

on solutions of kolmogorovʼs equations for nonhomogeneous jump markov processes this paper studies three ways to construct a nonhomogeneous jump markov process i via a compensator of the random measure of a multivariate point process ii as a minimal solution of the backward kolmogorov equation and iii as a minimal solution of the forward kolmogorov equation the main conclusion of this paper is that for a given measurable transition intensity commonly called a q function all these constructions define the same transition function if this transition function is regular that is the probability of accumulation of jumps is zero then this transition function is the unique solution of the backward and forward kolmogorov equations for continuous q functions kolmogorov equations were studied in feller s seminal paper in particular this paper extends feller s results for continuous q functions to measurable q functions and provides additional results

discounted continuous time markov decision processes with constraints unbounded transition and loss rates this paper deals with denumerable continuous time markov decision processes mdp with constraints the optimality criterion to be minimized is expected discounted loss while several constraints of the same type are imposed the transition rates may be unbounded the loss rates are allowed to be unbounded as well from above and from below and the policies may be history dependent and randomized based on kolmogorov s forward equation and dynkin s formula we remind the reader about the bellman equation introduce and study occupation measures reformulate the optimization problem as a primary linear program provide the form of optimal policies for a constrained optimization problem here and establish the duality between the convex analytic approach and dynamic programming finally a series of examples is given to illustrate all of our results

mean semivariance optimality for continuous time markov decision processes abstract in this paper we study the mean semivariance problem for continuous time markov decision processes with borel state and action spaces and unbounded cost and transition rates the optimality criterion is to minimize the semivariance of the discounted total cost over the set of all policies satisfying the constraint that the mean of the discounted total cost is equal to a given function under reasonable conditions we show that the semivariance optimal value function is a solution to the optimality equation of the mean semivariance criterion by an iteration approach moreover we obtain the existence of mean semivariance optimal policies from the optimality equation furthermore we give a value iteration algorithm to compute approximately an optimal policy and the optimal value and analyze the convergence of the algorithm

proximal minimization methods with generalized bregman functions we consider methods for minimizing a convex function f that generate a sequence xk by taking xk 1 to be an approximate minimizer of f x dh x xk ck where ck 0 and dh is the d function of a bregman function h extensions are made to b functions that generalize bregman functions and cover more applications convergence is established under criteria amenable to implementation applications are made to nonquadratic multiplier methods for nonlinear programs

finite horizon semi markov decision processes with application to maintenance systems abstract this paper investigates finite horizon semi markov decision processes with denumerable states the optimality is over the class of all randomized history dependent policies which include states and also planning horizons and the cost rate function is assumed to be bounded below under suitable conditions we show that the value function is a minimum nonnegative solution to the optimality equation and there exists an optimal policy moreover we develop an effective algorithm for computing optimal policies derive some properties of optimal policies and in addition illustrate our main results with a maintenance system

optimal control of ergodic continuous time markov chains with average sample path rewards in this paper we study continuous time markov decision processes with the average sample path reward aspr criterion and possibly unbounded transition and reward rates we propose conditions on the system s primitive data for the existence of epsilon aspr optimal deterministic stationary policies in a class of randomized markov policies satisfying some additional continuity assumptions the proof of this fact is based on the time discretization technique the martingale stability theory and the concept of potential we also provide both policy and value iteration algorithms for computing or at least approximating the epsilon aspr optimal stationary policies we illustrate with examples our main results as well as the difference between the aspr and the average expected reward criteria

convergence of proximal like algorithms we analyze proximal methods based on entropy like distances for the minimization of convex functions subject to nonnegativity constraints we prove global convergence results for the methods with approximate minimization steps and an ergodic convergence result for the case of finding a zero of a maximal monotone operator we also consider linearly constrained convex problems and establish a quadratic convergence rate result for linear programs our analysis allows us to simplify and extend the available convergence results for these methods

optimal interventions in countable jump markov processes in this paper an intervention refers to an immediate change of the state of the system between interventions the continuous time jump markov process is uncontrollable with natural jump intensities the multicriteria control problem for such a model is considered and the constrained version is investigated with the help of the lagrange multipliers technique all of the theory is illustrated by an example of the optimal control of epidemic with carriers

optimality of mixed policies for average continuous time markov decision processes with constraints this article concerns the average criteria for continuous time markov decision processes with n constraints we show the following a every extreme point of the space of performance vectors corresponding to the set of stable measures is generated by a deterministic stationary policy and b there exists a mixed optimal policy where the mixture is over no more than n 1 deterministic stationary policies

on the first passage g mean variance optimality for discounted continuous time markov decision processes this paper considers the discounted continuous time markov decision processes mdps in borel spaces and with unbounded transition rates the discount factors are allowed to depend on states and actions main attention is concentrated on the set f g of stationary policies attaining a given mean performance g up to the first passage of the continuous time mdp to an arbitrarily fixed target set under suitable conditions we prove the existence of a g mean variance optimal policy that minimizes the first passage variance over the set f g using a transformation technique and also give the value iteration and policy iteration algorithms for computing the g variance value function and a g mean variance optimal policy respectively two examples are analytically solved to demonstrate the application of our results

brief paper towards the optimal control of markov chains with constraints an optimal control problem with constraints is considered on a finite interval for a non stationary markov chain with a finite state space the constraints are given as a set of inequalities the optimal solution existence is proved under a natural assumption that the set of admissible controls is non empty the stochastic control problem is reduced to a deterministic one and it is shown that the optimal solution satisfies the maximum principle moreover it can be chosen within a class of markov controls on the basis of this result an approach to the numerical solution is proposed and its implementation is illustrated by examples

constrained discounted dynamic programming this paper deals with constrained optimization of markov decision processes with a countable state space compact action sets continuous transition probabilities and upper semicontinuous reward functions the objective is to maximize the expected total discounted reward for one reward function under several inequality constraints on similar criteria with other reward functions suppose a feasible policy exists for a problem with m constraints we prove two results on the existence and structure of optimal policies first we show that there exists a randomized stationary optimal policy which requires at most m actions more than a nonrandomized stationary one this result is known for several particular cases second we prove that there exists an optimal policy which is i stationary nonrandomized from some step onward ii randomized markov before this step but the total number of actions which are added by randomization is at most m iii the total number of actions that are added by nonstationarity is at most m we also establish pareto optimality of policies from the two classes described above for multi criteria problems we describe an algorithm to compute optimal policies with properties i iii for constrained problems the policies that satisfy properties i iii have the pleasing aesthetic property that the amount of randomization they require over any trajectory is restricted by the number of constraints in contrast a randomized stationary policy may require an infinite number of randomizations over time

generalized potlatch and smoothing processes we consider simple generalizations of the potlatch and smoothing processes which were introduced in 8 and studied in 5 these generalizations provide relatively simple examples of infinite interacting systems which exhibit phase transition the original potlatch and smoothing processes do not exhibit phase transition our results show that for the generalized processes phase transition does not usually occur in one or two dimensions but usually does occur in higher dimensions upper and lower bounds for the relevant critical values are obtained as one application of our results we obtain the limiting behavior of the critical values for the linear contact process in d dimensions as d thus answering a question we raised in 3 this is done by comparing the contact process with an appropriate generalized smoothing process

zero sum stochastic games in borel spaces average payoff criteria this paper is concerned with two person zero sum dynamic stochastic games in borel spaces with possibly unbounded payoff function and several average or ergodic payoff criteria we give conditions under which the long run expected average payoff criterion the sample path average criterion the existence of solutions to the average payoff shapley equations and a certain martingale condition are all equivalent

inner jumping numbers of non degenerate polynomials the inner jumping numbers were introduced by budur to relate two different measures of the complexity of the singularities of an effective divisor d on a nonsingular complex variety x the jumping numbers of a pair x d and the hodge spectrum at a singular point of d we give an elementary proof for an effective and combinatorial description of inner jumping numbers 1 of non degenerate polynomials

an analysis of the em algorithm and entropy like proximal point methods the em algorithm is a popular method for maximum likelihood estimation from incomplete data this method may be viewed as a proximal point method for maximizing the log likelihood function using an integral form of the kullback leibler distance function motivated by this interpretation we consider a proximal point method using an integral form of entropy like distance function we give a convergence analysis of the resulting proximal point method in the case where the cluster points lie in the interior of the objective function domain this result is applied to a normal independent example and a gaussian mixture example to establish convergence of the em algorithm on these examples further convergence analysis of the method for maximization over an orthant is given in low dimensions sublinear convergence and schemes for accelerating convergence are also discussed

computable approximations for continuous time markov decision processes on borel spaces based on empirical measures in this paper we propose an approach for approximating the value function and an optimal policy of continuous time markov decision processes with borel state and action spaces with possibly unbounded cost and transition rates under the total expected discounted cost optimality criterion under adequate assumptions which in particular include that the transition rate has a density function with respect to a reference measure together with piecewise lipschitz continuity of the elements of the control model we approximate the original controlled process by a model with finite state and action spaces the approximation error is related to the 1 wasserstein distance between suitably defined probability measures and approximating measures with finite support we also study the case when the reference measure is approximated with empirical distributions and we show that convergence of the approximations takes place at an exponential rate in probability

discounted continuous time markov decision processes with unbounded rates the convex analytic approach this paper deals with constrained discounted continuous time markov decision processes also known as controlled jump markov processes with borel state and action spaces under some conditions imposed on the primitives allowing unbounded transition rates and unbounded from both above and below cost rates first we study the space of occupation measures then we reformulate the original problem as a linear program over the space of those measures and undertake the duality analysis finally under some compactness continuity conditions we show the existence of a stationary optimal policy out of the class of randomized history dependent policies

continuous time markov decision processes with discounted rewards the case of polish spaces this paper deals with continuous time markov decision processes in polish spaces under an expected discounted reward criterion the transition rates of underlying continuous time jump markov processes are allowed to be unbounded and the reward rates may have neither upper nor lower bounds we first give conditions on the controlled system s primitive data under these conditions we prove that the transition functions of possibly nonhomogeneous continuous time markov processes are regular by using feller s construction approach to such transition functions then under additional continuity and compactness conditions we ensure the existence of optimal stationary policies by using the technique of extended infinitesimal operators associated with the transition functions and also provide a recursive way to compute or at least to approximate the optimal reward values finally we use examples to illustrate our results and the gap between our conditions and those in the previous literature

a linear programming formulation for constrained discounted continuous control for piecewise deterministic markov processes this paper deals with the constrained discounted control of piecewise deterministic markov process pdmps in general borel spaces the control variable acts on the jump rate and transition measure and the goal is to minimize the total expected discounted cost composed of positive running and boundary costs while satisfying some constraints also in this form the basic idea is by using the special features of the pdmps to re write the problem via an embedded discrete time markov chain associated to the pdmp and re formulate the problem as an infinite dimensional linear programming lp problem via the occupation measures associated to the discrete time process it is important to stress however that our new discrete time problem is not in the same framework of a general constrained discrete time markov decision process and due to that some conditions are required to get the equivalence between the continuous time problem and the lp formulation we provide in the sequel sufficient conditions for the solvability of the associated lp problem based on a generalization of theorem 4 1 in 8 in appendix a we present the proof of this generalization which we believe is of interest on its own the paper is concluded with some examples to illustrate the obtained results

constrained average cost markov control processes in borel spaces this paper considers constrained markov control processes in borel spaces with unbounded costs the criterion to be minimized is a long run expected average cost and the constraints can be imposed on similar average costs or on average rewards or discounted costs or rewards we give conditions under which the constrained problem cp is solvable and equivalent to an equality constrained ec linear program furthermore we show that there is no duality gap between ec and the dual program ec and that in fact the strong duality condition holds finally we introduce an explicit procedure to solve cp in some cases which is illustrated with a detailed example

robustness of policies in constrained markov decision processes we consider the optimization of finite state finite action markov decision processes mdps under constraints cost and constraints are discounted we introduce a new method for investigating the continuity and a certain type of robustness of the optimal cost and the optimal policy under changes in the constraints this method is also applicable for other cost criteria such as finite horizon and infinite horizon average cost

randomized and relaxed strategies in continuous time markov decision processes one of the goals of this article is to describe a wide class of control strategies which includes the traditional relaxed strategies as well as the so called randomized strategies which appeared earlier only in the framework of semi markov decision processes if the objective is the total expected cost up to the accumulation of jumps then without loss of generality one can consider only markov relaxed strategies under a simple condition the markov randomized strategies are also sufficient an example shows that the mentioned condition is important finally without any conditions the class of so called poisson related strategies is also sufficient in the optimization problems all the results are applicable to the discounted model they may be useful also for the case of long run average cost

convergence rate analysis of nonquadratic proximal methods for convex and linear programming the divergence proximal method is an extension of the proximal minimization algorithm where the usual quadratic proximal term is substituted by a class of convex statistical distances called divergences in this paper we study the convergence rate of this nonquadratic proximal method for convex and particularly linear programming we identify a class of divergences for which superlinear convergence is attained both for optimization problems with strongly convex objectives at the optimum and linear programming problems when the regularization parameters tend to zero we prove also that with regularization parameters bounded away from zero convergence is at least linear for a wider class of divergences when the method is applied to the same kinds of problems we further analyze the associated class of augmented lagrangian methods for convex programming with nonquadratic penalty terms and prove convergence of the dual generated by these methods for linear programming problems under a weak nondegeneracy assumption

markowitz s mean variance portfolio selection with regime switching from discrete time models to their continuous time limits we study a discrete time version of markowitz s mean variance portfolio selection problem where the market parameters depend on the market mode regime that jumps among a finite number of states the random regime switching is delineated by a finite state markov chain based on which a discrete time markov modulated portfolio selection model is presented such models either arise from multiperiod portfolio selections or result from numerical solution of continuous time problems the natural connections between discrete time models and their continuous time counterpart are revealed since the markov chain frequently has a large state space to reduce the complexity an aggregated process with smaller state space is introduced and the underlying portfolio selection is formulated as a two time scale problem we prove that the process of interest yields a switching diffusion limit using weak convergence methods next based on the optimal control of the limit process obtained from our recent work we devise portfolio selection strategies for the original problem and demonstrate their asymptotic optimality

a mean variance optimization problem for discounted markov decision processes in this paper we consider a mean variance optimization problem for markov decision processes mdps over the set of deterministic stationary policies different from the usual formulation in mdps we aim to obtain the mean variance optimal policy that minimizes the variance over a set of all policies with a given expected reward for continuous time mdps with the discounted criterion and finite state and action spaces we prove that the mean variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and markov properties then we show that a mean variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations we also address related issues such as a mutual fund theorem and illustrate our results with an example

motivic milnor fiber of a quasi ordinary hypersurface let f cd 1 0 c 0 be a germ of complex analytic function such that its zero level defines an irreducible germ of quasi ordinary hypersurface s 0 subset of cd 1 0 we describe the motivic igusa zeta function the motivic milnor fibre and the hodge steenbrink spectrum of f at 0 in terms of topological invariants of s 0 subset of cd 1 0

ergodic theorems for coupled random walks and other systems with locally interacting components in 5 the second author introduced a variety of new infinite systems with locally interacting components on the basis of computations for the finite analogues of these systems he made conjectures ragarding their limiting behavior as t this paper is devoted to the construction of these processes and to the proofs of these conjectures we restrict ourselves primarily to spatially homogeneous situations interesting problems remain unsolved in inhomogeneous cases two features distinguish these processes from most other infinite particle systems which have been studied one is that the state spaces of these systems are noncompact the other that even though the invariant measures are not generally of product form one can nevertheless compute explicitly the first and second moments of the number of particles per site in equilibrium the second moment computations are of inherent interest of course and they play an important role in the proofs of the ergodic theorems as well

common information based markov perfect equilibria for stochastic games with asymmetric information finite games a model of stochastic games where multiple controllers jointly control the evolution of the state of a dynamic system but have access to different information about the state and action processes is considered the asymmetry of information among the controllers makes it difficult to compute or characterize nash equilibria using the common information among the controllers the game with asymmetric information is used to construct another game with symmetric information such that the equilibria of the new game can be transformed to equilibria of the original game further under certain conditions a markov state is identified for the new symmetric information game and its markov perfect equilibria are characterized this characterization provides a backward induction algorithm to find nash equilibria of the original game with asymmetric information in pure or behavioral strategies each step of this algorithm involves finding bayesian nash equilibria of a one stage bayesian game the class of nash equilibria of the original game that can be characterized in this backward manner are named common information based markov perfect equilibria

drift and monotonicity conditions for continuous time controlled markov chains with an average criterion we give conditions for the existence of average optimal policies for continuous time controlled markov chains with a denumerable state space and borel action sets the transition rates are allowed to be unbounded and the reward cost rates may have neither upper nor lower bounds in the spirit of the drift and monotonicity conditions for continuous time markov processes we propose a new set of conditions on the controlled process primitive data under which the existence of optimal deterministic stationary policies in the class of randomized markov policies is proved using the extended generator approach instead of kolmogorov s forward equation used in the previous literature and under which the convergence of a policy iteration method is also shown moreover we use a controlled queueing system to show that all of our conditions are satisfied whereas those in the previous literature fail to hold

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

a probabilistic analysis of bias optimality in unichain markov decision processes focuses on bias optimality in unichain finite state and action space markov decision processes using relative value functions we present methods for evaluating optimal bias this leads to a probabilistic analysis which transforms the original reward problem into a minimum average cost problem the result is an explanation of how and why bias implicitly discounts future rewards

subgame perfect equilibria for stochastic games for an n person stochastic game with borel state space s and compact metric action sets a1 a2 an sufficient conditions are given for the existence of subgame perfect equilibria one result is that such equilibria exist if the law of motion q s a is for fixed s continuous in a a1 a2 an for the total variation norm and the payoff functions f1 f2 fn are bounded borel measurable functions of the sequence of states s1 s2 sn and in addition are continuous when sn is given the product of discrete topologies on s

rescaling and stepsize selection in proximal methods using separable generalized distances this paper presents a convergence proof technique for a broad class of proximal algorithms in which the perturbation term is separable and may contain barriers enforcing interval constraints there are two key ingredients in the analysis a mild regularity condition on the differential behavior of the barrier as one approaches an interval boundary and a lower stepsize limit that takes into account the curvature of the proximal term we give two applications of our approach first we prove subsequential convergence of a very broad class of proximal minimization algorithms for convex optimization where different stepsizes can be used for each coordinate applying these methods to the dual of a convex program we obtain a wide class of multiplier methods with subsequential convergence of both primal and dual iterates and independent adjustment of the penalty parameter for each constraint the adjustment rules for the penalty parameters generalize a well established scheme for the exponential method of multipliers the results may also be viewed as a generalization of recent work by ben tal and zibulevsky siam j optim 7 1997 pp 347 366 and auslender teboulle and ben tiba comput optim appl 12 1999 pp 31 40 math oper res 24 1999 pp 645 668 on methods derived from varphi divergences the second application established full convergence under a novel stepsize condition of bregman function based proximal methods for general monotone operator problems over a box prior results in this area required strong restrictive assumptions on the monotone operator

a note on optimality conditions for continuous time markov decision processes with average cost criterion this note deals with continuous time markov decision processes with a denumerable state space and the average cost criterion the transition rates are allowed to be unbounded and the action set is a borel space we give a new set of conditions under which the existence of optimal stationary policies is ensured by using the optimality inequality our results are illustrated with a controlled queueing model moreover we use an example to show that our conditions do not imply the existence of a solution to the optimality equations in the previous literature

nonlinear rescaling and proximal like methods in convex optimization the nonlinear rescaling principle nrp consists of transforming the objective function and or the constraints of a given constrained optimization problem into another problem which is equivalent to the original one in the sense that their optimal set of solutions coincides a nonlinear transformation parameterized by a positive scalar parameter and based on a smooth sealing function is used to transform the constraints the methods based on nrp consist of sequential unconstrained minimization of the classical lagrangian for the equivalent problem followed by an explicit formula updating the lagrange multipliers we first show that the nrp leads naturally to proximal methods with an entropy like kernel which is defined by the conjugate of the scaling function and establish that the two methods are dually equivalent for convex constrained minimization problems we then study the convergence properties of the nonlinear rescaling algorithm and the corresponding entropy like proximal methods for convex constrained optimization problems special cases of the nonlinear rescaling algorithm are presented in particular a new class of exponential penalty modified barrier functions methods is introduced

continuous time discounted jump markov decision processes a discrete event approach this paper introduces and develops a new approach to the theory of continuous time jump markov decision processes ctjmdp this approach reduces discounted ctjmdps to discounted semi markov decision processes smdps and eventually to discrete time markov decision processes mdps the reduction is based on the equivalence of strategies that change actions between jumps and the randomized strategies that change actions only at jump epochs this holds both for one criterion problems and for multiple objective problems with constraints in particular this paper introduces the theory for multiple objective problems with expected total discounted rewards and constraints if a problem is feasible there exist three types of optimal policies i nonrandomized switching stationary policies ii randomized stationary policies for the ctjmdp and iii randomized stationary policies for the corresponding smdp with exponentially distributed sojourn times and these policies can be implemented as randomized strategies in the ctjmdp

constrained and unconstrained optimal discounted control of piecewise deterministic markov processes the main goal of this paper is to study the infinite horizon expected discounted continuous time optimal control problem of piecewise deterministic markov processes with the control acting continuously on the jump intensity lambda and on the transition measure q of the process but not on the deterministic flow phi the contributions of the paper are for the unconstrained as well as the constrained cases the set of admissible control strategies is assumed to be formed by policies possibly randomized and depending on the history of the process taking values in a set valued action space for the unconstrained case we provide sufficient conditions based on the three local characteristics of the process phi lambda q and the semicontinuity properties of the set valued action space to guarantee the existence and uniqueness of the integro differential optimality equation the so called bellman hamilton jacobi equation as well as the existence of an optimal and delta optimal as well

an approximation approach for the deviation matrix of continuous time markov processes with application to markov decision theory we present an update formula that allows the expression of the deviation matrix of a continuous time markov process with denumerable state space having generator matrix q through a continuous time markov process with generator matrix q we show that under suitable stability conditions the algorithm converges at a geometric rate by applying the concept to three different examples namely the m m 1 queue with vacations the m g 1 queue and a tandem network we illustrate the broad applicability of our approach for a problem in admission control we apply our approximation algorithm to markov decision theory for computing the optimal control policy numerical examples are presented to highlight the efficiency of the proposed algorithm

a statistical model for positron emission tomography abstract positron emission tomography pet still in its research stages is a technique that promises to open new medical frontiers by enabling physicians to study the metabolic activity of the body in a pictorial manner much as in x ray transmission tomography and other modes of computerized tomography the quality of the reconstructed image in pet is very sensitive to the mathematical algorithm to be used for reconstruction in this article we tailor a mathematical model to the physics of positron emissions and we use the model to describe the basic image reconstruction problem of pet as a standard problem in statistical estimation from incomplete data we describe various estimation procedures such as the maximum likelihood ml method using the em algorithm the method of moments and the least squares method a computer simulation of a pet experiment is then used to demonstrate the ml and the least squares reconstructions the main purposes of this article are to report on what we believe is an

on hodge spectrum and multiplier ideals we describe a relation between two invariants which measure the complexity of a hypersurface singularity one is the hodge spectrum which is related to the monodromy and the hodge filtration on the cohomology of the milnor fiber the other is the multiplier ideal having to do with log resolutions

approximating ergodic average reward continuous time controlled markov chains we study the approximation of an ergodic average reward continuous time denumerable state markov decision process mdp by means of a sequence of mdps our results include the convergence of the corresponding optimal policies and the optimal gains for a controlled upwardly skip free process we show some computational results to illustrate the convergence theorems

on the convergence of the exponential multiplier method for convex programming in this paper we analyze the exponential method of multipliers for convex constrained minimization problems which operates like the usual augmented lagrangian method except that it uses an exponential penalty function in place of the usual quadratic we also analyze a dual counterpart the entropy minimization algorithm which operates like the proximal minimization algorithm except that it uses a logarithmic entropy proximal term in place of a quadratic we strengthen substantially the available convergence results for these methods and we derive the convergence rate of these methods when applied to linear programs

realizable strategies in continuous time markov decision processes for the borel model of the continuous time markov decision process we introduce a wide class of control strategies in a particular case such strategies transform to the standard relaxed strategies intensively studied in the last decade in another special case if one restricts to another special subclass of the general strategies the model transforms to the semi markov decision process further we show that the relaxed strategies are not realizable for the constrained optimal control problem with total expected costs we describe the sufficient class of realizable strategies the so called poisson related strategies finally we show that for solving the formulated optimal control problems one can use all the tools developed earlier for the classical discrete time markov decision processes

entropic proximal mappings with applications to nonlinear programming we introduce a family of new transforms based on imitating the proximal mapping of moreau and the associated moreau yosida proximal approximation of a function the transforms are constructed in terms of the a divergence functional a generalization of the relative entropy and of bregman s measure of distance an analogue of moreau s theorem associated with these entropy like distances is proved we show that the resulting entropic proximal maps share properties similar to the proximal mapping and provide a fairly general framework for constructing approximation and smoothing schemes for optimization problems applications of the results to the construction of generalized augmented lagrangians for nonlinear programs and the minimax problem are presented