a general method for the existence of periodic solutions of differential systems in the plane abstract we propose a general method to prove the existence of periodic solutions for planar systems of ordinary differential equations which can be used in many different circumstances applications are given to some nonresonant cases even for systems with superlinear growth in some direction or with a singularity systems at resonance are also considered provided a landesman lazer type of condition is assumed

optimal structural policies for ambiguity and risk averse inventory and pricing models this paper discusses multiperiod stochastic joint inventory and pricing models when the decision maker is risk and ambiguity averse we study infinite horizon models with discounted and long run average optimization criteria the main result of this paper is establishing the optimality of stationary s s p policies for the infinite horizon inventory and pricing models

a generalization of the borkar meyn theorem for stochastic recursive inclusions in this paper the stability theorem of borkar and meyn is extended to include the case when the mean field is a set valued map two different sets of sufficient conditions are presented that guarantee the stability and convergence of stochastic recursive inclusions our work builds on the works of benaim hofbauer and sorin as well as borkar and meyn as a corollary to one of the main theorems a natural generalization of the borkar and meyn theorem follows in addition the original theorem of borkar and meyn is shown to hold under slightly relaxed assumptions as an application to one of the main theorems we discuss a solution to the approximate drift problem finally we analyze the stochastic gradient algorithm with constant error gradient estimators as yet another application of our main result

positive periodic solutions for planar differential systems with repulsive singularities on the axes abstract in this paper we provide an existence result of periodic solutions to planar systems having a repulsive singularity on both the axes we require a nonresonance assumption related to the spectrum of the periodic problem associated to the steen s planar system y c x 3 x x d y 3 y x 0 y 0 which is isochronous see 15 as a particular case we obtain the existence of periodic solutions to a periodically perturbed steen s planar system

a convex analytic approach to risk aware markov decision processes in classical markov decision process mdp theory we search for a policy that say minimizes the expected infinite horizon discounted cost expectation is of course a risk neutral measure which does not suffice in many applications particularly in finance we replace the expectation with a general risk functional and call such models risk aware mdp models we consider minimization of such risk functionals in two cases the expected utility framework and conditional value at risk a popular coherent risk measure later we consider risk aware mdps wherein the risk is expressed in the constraints this includes stochastic dominance constraints and the classical chance constrained optimization problems in each case we develop a convex analytic approach to solve such risk aware mdps in most cases we show that the problem can be formulated as an infinite dimensional linear program lp in occupation measures when we augment the state space we provide a discretization method and finite approximati

computational methods for risk averse undiscounted transient markov models the total cost problem for discrete time controlled transient markov models is considered the objective functional is a markov dynamic risk measure of the total cost two solution methods value and policy iteration are proposed and their convergence is analyzed in the policy iteration method we propose two algorithms for policy evaluation the nonsmooth newton method and convex programming and we prove their convergence the results are illustrated on a credit limit control problem

periodic solutions of weakly coupled superlinear systems abstract by the use of a higher dimensional version of the poincare birkhoff theorem we are able to generalize a result of jacobowitz 15 and hartman 14 thus proving the existence of infinitely many periodic solutions for a weakly coupled superlinear system

a simulation based algorithm for ergodic control of markov chains conditioned on rare events we study the problem of long run average cost control of markov chains conditioned on a rare event in a related recent work a simulation based algorithm for estimating performance measures associated with a markov chain conditioned on a rare event has been developed we extend ideas from this work and develop an adaptive algorithm for obtaining online optimal control policies conditioned on a rare event our algorithm uses three timescales or step size schedules on the slowest timescale a gradient search algorithm for policy updates that is based on one simulation simultaneous perturbation stochastic approximation spsa type estimates is used deterministic perturbation sequences obtained from appropriate normalized hadamard matrices are used here the fast timescale recursions compute the conditional transition probabilities of an associated chain by obtaining solutions to the multiplicative poisson equation for a given policy estimate further the risk parameter associated with the value function for a given policy estimate is updated on a timescale that lies in between the two scales above we briefly sketch the convergence analysis of our algorithm and present a numerical application in the setting of routing multiple flows in communication networks

online markov decision processes with kullback leibler control cost this paper considers an online real time control problem that involves an agent performing a discrete time random walk over a finite state space the agent s action at each time step is to specify the probability distribution for the next state given the current state following the setup of todorov the state action cost at each time step is a sum of a state cost and a control cost given by the kullback leibler kl divergence between the agent s next state distribution and that determined by some fixed passive dynamics the online aspect of the problem is due to the fact that the state cost functions are generated by a dynamic environment and the agent learns the current state cost only after selecting an action an explicit construction of a computationally efficient strategy with small regret i e expected difference between its actual total cost and the smallest cost attainable using noncausal knowledge of the state costs under mild regularity conditions is presented along with a demonstration of the performance of the proposed strategy on a simulated target tracking problem a number of new results on markov decision processes with kl control cost are also obtained

infinite horizon risk sensitive control of discrete time markov processes under minorization property risk sensitive control of markov processes satisfying the minorization property is studied using splitting techniques existence of solutions to the multiplicative poisson equation is shown approximation by uniformly ergodic controlled markov processes is introduced which allows us to show the existence of solutions to the infinite horizon risk sensitive bellman equation

stability of annealing schemes and related processes an approach for establishing stability of annealing schemes and related processes is described this extends the approach developed in borkar and meyn siam j control optim 38 2000 447 for stochastic approximation algorithms the proof uses a possibly degenerate stochastic differential equation obtained as a scaling limit of the interpolated algorithm

periodic solutions of second order equations via rotation numbers abstract we consider the problem of the existence and multiplicity of periodic solutions associated to a class of scalar equations of the form x f t x 0 the class considered is such that the behaviour of its solutions near zero and infinity can be compared two suitable piecewise linear systems we show how a rotation number approach together with the poincare birkhoff theorem and the phase plane analysis of the spiral properties allows to obtain multiplicity results in terms of the gap between the rotation numbers of the referred piecewise linear systems at zero and at infinity these systems may also be resonant in particular our approach can be used to deal with the problems without both global existence of the cauchy problems associated to the equation and the sign assumption on f the typical example is a partially superlinear second order equation our main result generalizes some classical results of jacobowitz and hartman among others

stochastic approximation for nonexpansive maps application to q learning algorithms we discuss synchronous and asynchronous iterations of the form xk 1 x k gamma k h x k w k where h is a suitable map and wk is a deterministic or stochastic sequence satisfying suitable conditions in particular in the stochastic case these are stochastic approximation iterations that can be analyzed using the ode approach based either on kushner and clark s lemma for the synchronous case or on borkar s theorem for the asynchronous case however the analysis requires that the iterates xk be bounded a fact which is usually hard to prove we develop a novel framework for proving boundedness in the deterministic framework which is also applicable to the stochastic case when the deterministic hypotheses can be verified in the almost sure sense this is based on scaling ideas and on the properties of lyapunov functions we then combine the boundedness property with borkar s stability analysis of odes involving nonexpansive mappings to prove convergence with probability 1 in the stochastic case we also apply our convergence analysis to q learning algorithms for stochastic shortest path problems and are able to relax some of the assumptions of the currently available results

risk sensitive control of finite state machines on an infinite horizon i in this paper we consider robust and risk sensitive control of discrete time finite state systems on an infinite horizon the solution of the state feedback robust control problem is characterized in terms of the value of an average cost dynamic game the risk sensitive stochastic optimal control problem is solved using the policy iteration algorithm and the optimal rate is expressed in terms of the value of a stochastic dynamic game with average cost per unit time criterion by taking a small noise limit a deterministic dynamic game which is closely related to the robust control problem is obtained

empirical dynamic programming we propose empirical dynamic programming algorithms for markov decision processes in these algorithms the exact expectation in the bellman operator in classical value iteration is replaced by an empirical estimate to get empirical value iteration evi policy evaluation and policy improvement in classical policy iteration are also replaced by simulation to get empirical policy iteration epi thus these empirical dynamic programming algorithms involve iteration of a random operator the empirical bellman operator we introduce notions of probabilistic fixed points for such random monotone operators we develop a stochastic dominance framework for convergence analysis of such operators we then use this to give sample complexity bounds for both evi and epi we then provide various variations and extensions to asynchronous empirical dynamic programming the minimax empirical dynamic program and show how this can also be used to solve the dynamic newsvendor problem preliminary experimental results

brief paper an adaptive optimization scheme with satisfactory transient performance adaptive optimization ao schemes based on stochastic approximation principles such as the random directions kiefer wolfowitz rdkw the simultaneous perturbation stochastic approximation spsa and the adaptive fine tuning aft algorithms possess the serious disadvantage of not guaranteeing satisfactory transient behavior due to their requirement for using random or random like perturbations of the parameter vector the use of random or random like perturbations may lead to particularly large values of the objective function which may result to severe poor performance or stability problems when these methods are applied to closed loop controller optimization applications in this paper we introduce and analyze a new algorithm for alleviating this problem mathematical analysis establishes satisfactory transient performance and convergence of the proposed scheme under a general set of assumptions application of the proposed scheme to the adaptive optimization of a large scale complex control system demonstrates the efficiency of the proposed scheme

nonzero sum risk sensitive stochastic games on a countable state space the infinite horizon risk sensitive discounted cost and ergodic cost nonzero sum stochastic games for controlled markov chains with countably many states are analyzed for the discounted cost game we prove the existence of nash equilibrium strategies in the class of markov strategies under fairly general conditions under an additional weak geometric ergodicity condition and a small cost criterion the existence of nash equilibrium strategies in the class of stationary markov strategies is proved for the ergodic cost game the key nontrivial contributions in the ergodic part are to prove the existence of a particular form of a relative value function solution to a player s bellman equation and the continuity of this solution with respect to the opponent s strategies

resonance and rotation numbers for planar hamiltonian systems multiplicity results via the poincare birkhoff theorem abstract in the general setting of a planar first order system 0 1 u g t u u r 2 with g 0 t r 2 r 2 we study the relationships between some classical nonresonance conditions including the landesman lazer one at infinity and in the unforced case i e g t 0 0 at zero and the rotation numbers of large and small solutions of 0 1 respectively such estimates are then used to establish via the poincare birkhoff fixed point theorem new multiplicity results for t periodic solutions of unforced planar hamiltonian systems j u u h t u and unforced undamped scalar second order equations x g t x 0 in particular by means of the landesman lazer condition we obtain sharp conclusions when the system is resonant at infinity

the borkar meyn theorem for asynchronous stochastic approximations in this paper we give a generalization of a result by borkar and meyn 2000 1 on the stability and convergence of synchronous update stochastic approximation algorithms to the case of asynchronous stochastic approximations with delays we then describe an interesting application of the result to asynchronous distributed temporal difference td learning with function approximation and delays c 2011 elsevier b v all rights reserved

on the peroidic bvp for the forced duffing equation with jumping nonlinearity on etudie le probleme suivant u cu f t u e t p p sur 0 t u t u 0 0 u t u 0 0 c est une fonction arbitraire dans l 1 0 t r c r et l application non lineaire f est a saut

queue aware distributive resource control for delay sensitive two hop mimo cooperative systems in this paper we consider a queue aware distributive resource control algorithm for two hop mimo cooperative systems we shall illustrate that relay buffering is an effective way to reduce the intrinsic half duplex penalty in cooperative systems the complex interactions of the queues at the source node and the relays are modeled as an average cost infinite horizon markov decision process mdp the traditional approach solving this mdp problem involves centralized control with huge complexity to obtain a distributive and low complexity solution we introduce a linear structure which approximates the value function of the associated bellman equation by the sum of per node value functions we derive a distributive two stage two winner auction based control policy which is a function of the local csi and local qsi only furthermore to estimate the best fit approximation parameter we propose a distributive online stochastic learning algorithm using stochastic approximation theory finally we establish technical conditions for almost sure convergence and show that under heavy traffic the proposed low complexity distributive control is global optimal

boundedness of iterates in q learning abstract reinforcement learning rl is a simulation based counterpart of stochastic dynamic programming in recent years it has been used in solving complex markov decision problems mdps watkins q learning is by far the most popular rl algorithm used for solving discounted reward mdps the boundedness of the iterates in q learning plays a critical role in its convergence analysis and in making the algorithm stable which makes it extremely attractive in numerical solutions previous results show boundedness asymptotically in an almost sure sense we present a new result that shows boundedness in an absolute sense under some weaker conditions for the step size also our proof is based on some simple induction arguments

risk sensitive optimal control for markov decision processes with monotone cost the existence of an optimal feedback law is established for the risk sensitive optimal control problem with denumerable state space the main assumptions imposed are irreducibility and anear monotonicity condition on the one step cost function a solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law

risk sensitive control of markov processes in countable state space in this paper we consider infinite horizon risk sensitive control of markov processes with discrete time and denumerable state space this problem is solved by proving under suitable conditions that there exists a bounded solution to the dynamic programming equation the dynamic programming equation is transformed into an isaacs equation for a stochastic game and the vanishing discount method is used to study its solution in addition we prove that the existence conditions are also necessary

stability of travelling waves for non convex scalar viscous conservation laws travelling waves of a viscous conservation law so called viscous profiles are shown to be stable in polynomialy weighted l spaces there is no assumption of convexity on the nonlinear term and thus earlier results of il in and ole nik sattinger and kawashima and matsumura are generalized the method uses the semigroup of the linearized equation with solutions of the full problem expressed by the variation of constants formula estimates are derived for the semigroup through a new technique for estimating the resolvent 1993 john wiley sons inc

simulation based optimization of markov reward processes this paper proposes a simulation based algorithm for optimizing the average reward in a finite state markov reward process that depends on a set of parameters as a special case the method applies to markov decision processes where optimization takes place within a parametrized set of policies the algorithm relies on the regenerative structure of finite state markov processes involves the simulation of a single sample path and can be implemented online a convergence result with probability 1 is provided

decay of solutions of some nonlinear wave equations abstract we study the large time behaviour of solutions to the initial value problem for the korteweg de vries equation and for the regularized long wave equation with a dissipative term appended using energy estimates a maximum principle and a transformation of cole hopf type sharp rates of temporal decay of certain norms of the solution are obtained

infinite horizon risk sensitive control of discrete time markov processes with small risk a control problem with risk sensitive ergodic performance criterion is considered for a discrete time feller process using assumptions of uniform ergodicity and small risk factor the existence and uniqueness of the solution to the bellman equation is proved uniform approximations to such solution in terms of discounted cost and discounted game problems are also shown

travelling wave solutions to the korteweg de vries burgers equation the existence and certain qualitative properties of travelling wave solutions to the korteweg de vries burgers equation are established the limiting behaviour of these waves when e tends to zero and when tends to zero is examined together with a singular limit wherein both e and tend to zero

a continuation approach to superlinear periodic boundary value problems this paper deals with the problem of the existence of t periodic solutions for the first order differential system x i t x 1 1 where f 0 t x r r is a caratheodory function in what follows we prove some results for the solvability of the periodic bvp in the case when the dimension of the space is even such a limitation is motivated by our interest in applications to the second order equation d g t u u o 1 2 which takes the form of 1 1 when it is written as the equivalent system u v 0 g t u v 1 3

linear stochastic approximation driven by slowly varying markov chains we study a linear stochastic approximation algorithm that arises in the context of reinforcement learning the algorithm employs a decreasing step size and is driven by markov noise with time varying statistics we show that under suitable conditions the algorithm can track the changes in the statistics of the markov noise as long as these changes are slower than the rate at which the step size of the algorithm goes to zero

stabilization of stochastic approximation by step size adaptation a scheme for stabilizing stochastic approximation iterates by adaptively scaling the step sizes is proposed and analyzed this scheme leads to the same limiting differential equation as the original scheme and therefore has the same limiting behavior while avoiding the difficulties associated with projection schemes the proof technique requires only that the limiting o d e descend a certain lyapunov function outside an arbitrarily large bounded set c 2012 elsevier b v all rights reserved

the value iteration algorithm in risk sensitive average markov decision chains with finite state space this work concerns discrete time markov decision chains with finite state space and bounded costs the controller has constant risk sensitivity and the performance of a control policy is measured by the corresponding risk sensitive average cost criterion assuming that the optimality equation has a solution it is shown that the value iteration scheme can be implemented to obtain in a finite number of steps 1 an approximation to the optimal sensitive average cost with an error less than a given tolerance and 2 a stationary policy whose performance index is arbitrarily close to the optimal value the argument used to establish these results is based on a modification of the original model which is an extension of a transformation introduced by schweitzer 1971 to analyze the the risk neutral case

bellman equations of risk sensitive control risk sensitive control problems are considered existence of a nonnegative solution to the bellman equation of risk sensitive control is shown the result is applied to prove that no breaking down occurs asymptotic behaviour of the nonnegative solution is studied in relation to ergodic control problems and the relationship between the asymptotics and the large deviation principle is noted

an actor critic algorithm with function approximation for discounted cost constrained markov decision processes we develop in this article the first actor critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints we consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy dependent discounted sums of certain sample path functions we apply the lagrange multiplier method to handle the inequality constraints our algorithm makes use of multi timescale stochastic approximation and incorporates a temporal difference td critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation spsa gradient estimates we prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy c 2010 elsevier b v all rights reserved

oja s algorithm for graph clustering markov spectral decomposition and risk sensitive control given a positive definite matrix m and an integer n m 1 oja s subspace algorithm will provide convergent estimates of the first n m eigenvalues of m along with the corresponding eigenvectors it is a common approach to principal component analysis this paper introduces a normalized stochastic approximation implementation of oja s subspace algorithm as well as new applications to the spectral decomposition of a reversible markov chain recall that this means that the stationary distribution satisfies the detailed balance equations meyn tweedie 2009 equivalently the statistics of the process in steady state do not change when time is reversed stability and convergence of oja s algorithm are established under conditions far milder than that assumed in previous work applications to graph clustering markov spectral decomposition and multiplicative ergodic theory are surveyed along with numerical results

the o d e method for convergence of stochastic approximation and reinforcement learning it is shown here that stability of the stochastic approximation algorithm is implied by the asymptotic stability of the origin for an associated ode this in turn implies convergence of the algorithm several specific classes of algorithms are considered as applications it is found that the results provide i a simpler derivation of known results for reinforcement learning algorithms ii a proof for the first time that a class of asynchronous stochastic approximation algorithms are convergent without using any a priori assumption of stability iii a proof for the first time that asynchronous adaptive critic and q learning algorithms are convergent for the average cost optimal control problem

a risk sensitive control dual approach to a large deviations control problem we consider a control problem of an ergodic process where the objective is to maximize over long term the probability to overperform a given level this is formulated as a large deviations control problem for which the standard dynamic programming methods may not be applied directly we solve this problem by adopting a duality approach leading to a risk sensitive ergodic control problem in a continuous time diffusion setting we state a verification theorem in terms of partial differential equations for this dual problem we then turn back to the primal problem by means of large deviations techniques we derive the optimal rate function and nearly optimal controls for the large deviations optimization problem finally explicit solutions are provided in a linear quadratic case

double resonance with landesman lazer conditions for planar systems of ordinary differential equations abstract we prove the existence of periodic solutions for first order planar systems at resonance the nonlinearity is indeed allowed to interact with two positively homogeneous hamiltonians both at resonance and some kind of landesman lazer conditions are assumed at both sides we are thus able to obtain as particular cases the existence results proposed in the pioneering papers by lazer and leach 1969 27 and by frederickson and lazer 1969 18 our theorem also applies in the case of asymptotically piecewise linear systems and in particular generalizes fabry s results in fabry 1995 10 for scalar equations with double resonance with respect to the dancer fucik spectrum

learning algorithms for markov decision processes with average cost this paper gives the first rigorous convergence analysis of analogues of watkins s q learning algorithm applied to average cost control of finite state markov chains we discuss two algorithms which may be viewed as stochastic approximation counterparts of two existing algorithms for recursively computing the value function of the average cost problem the traditional relative value iteration rvi algorithm and a recent algorithm of bertsekas based on the stochastic shortest path ssp formulation of the problem both synchronous and asynchronous implementations are considered and analyzed using the ode method this involves establishing asymptotic stability of associated ode limits the ssp algorithm also uses ideas from two time scale stochastic approximation

radially symmetric systems with a singularity and asymptotically linear growth abstract we prove the existence of infinitely many periodic solutions for radially symmetric systems with a singularity of repulsive type the nonlinearity is assumed to have a linear growth at infinity being controlled by two constants which have a precise interpretation in terms of the dancer fucik spectrum our result generalizes an existence theorem by del pino et al 1992 4 obtained in the case of a scalar second order differential equation

analyticity convergence and convergence rate of recursive maximum likelihood estimation in hidden markov models this paper considers the asymptotic properties of the recursive maximum likelihood estimator for hidden markov models the paper is focused on the analytic properties of the asymptotic log likelihood and on the point convergence and convergence rate of the recursive maximum likelihood estimator using the principle of analytic continuation the analyticity of the asymptotic log likelihood is shown for analytically parameterized hidden markov models relying on this fact and some results from differential geometry lojasiewicz inequality the almost sure point convergence of the recursive maximum likelihood algorithm is demonstrated and relatively tight bounds on the convergence rate are derived as opposed to the existing result on the asymptotic behavior of maximum likelihood estimation in hidden markov models the results of this paper are obtained without assuming that the log likelihood function has an isolated maximum at which the hessian is strictly negative definite

natural actor critic algorithms we present four new reinforcement learning algorithms based on actor critic natural gradient and function approximation ideas and we provide their convergence proofs actor critic reinforcement learning methods are online approximations to policy iteration in which the value function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent methods based on policy gradients in this way are of special interest because of their compatibility with function approximation methods which are needed to handle large or infinite state spaces the use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates the use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases our results extend prior two timescale convergence results for actor critic methods by konda and tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients our results extend prior empirical studies of natural actor critic methods by peters vijayakumar and schaal by providing the first convergence proofs and the first fully incremental algorithms

asynchronous stochastic approximations the asymptotic behavior of a distributed asynchronous stochastic approximation scheme is analyzed in terms of a limiting nonautonomous differential equation the relation between the latter and the relative values of suitably rescaled relative frequencies of updates of different components is underscored

a note on risk sensitive control of invariant models according to assaf a dynamic programming problem is called invariant if its transition mechanism depends only on the chosen action this paper studies properties of risk sensitive invariant problems with a general state space the main result establishes the optimality equation for the risk sensitive average cost criterion without any restrictions on the risk factor moreover a practical algorithm is provided for solving the optimality equation in case of a finite action space

the vanishing discount approach in markov chains with risk sensitive criteria in this paper stochastic dynamic systems are studied modeled by a countable state space markov cost reward chain satisfying a lyapunov type stability condition for an infinite planning horizon risk sensitive exponential discounted and average cost criteria are considered the main contribution is the development of a vanishing discount approach to relate the discounted criterion problem with the average criterion one as the discount factor increases to one i e no discounting in comparison to the well established risk neutral case our results are novel and reveal several fundamental and surprising differences other contributions made include the use of convex analytic arguments to obtain appropriately convergent sequences and a verification theorem for the case of unbounded solutions to the average cost poisson equation arising in the risk sensitive case also of importance is the fact that our developments are very much self contained and employ only basic probabilistic and analysis principles

the generalized korteweg de vries burgers equation in h2 r abstract the generalized kdv burgers equation u t u x x g u x u x x u f x 0 0 is considered in this paper using the parabolic regularization technique we prove local and global solvability in h 2 r of the cauchy problem for this equation several regularity properties of the approximations stay valid for solutions constructed in such a way next for when 0 we study the asymptotic behavior of the corresponding semigroup on h 2 r constructing the h 2 r h 3 r global attractor

korteweg de vries burgers system in rn abstract we study cauchy problem in r n for the korteweg de vries burgers system parabolic regularization technique is used to prove its global in time solvability the regularization effect of the laplacian term is observed for the viscous solutions constructed in the paper asymptotic behavior of weak solutions is discussed next in the language of the global attractors

adaptive importance sampling technique for markov chains using stochastic approximation for a discrete time finite state markov chain we develop an adaptive importance sampling scheme to estimate the expected total cost before hitting a set of terminal states this scheme updates the change of measure at every transition using constant or decreasing step size stochastic approximation the updates are shown to concentrate asymptotically in a neighborhood of the desired zero variance estimator through simulation experiments on simple markovian queues we observe that the proposed technique performs very well in estimating performance measures related to rare events associated with queue lengths exceeding prescribed thresholds we include performance comparisons of the proposed algorithm with existing adaptive importance sampling algorithms on some examples we also discuss the extension of the technique to estimate the infinite horizon expected discounted cost and the expected average cost

asymptotic stability of traveling wave solutions of systems for one dimensional gas motion the asymptotic stability of traveling wave solutions with shock profile is investigated for several systems in gas dynamics 1 the solution of a scalar conservation law with viscosity approaches the traveling wave solution at the ratet for some 0 ast provided that the initial disturbance is small and of integral zero and in addition decays at an algebraic rate for x 2 the traveling wave solution with nishida and smoller s condition of the system of a viscous heat conductive ideal gas is asymptotically stable provided the initial disturbance is small and of integral zero 3 the traveling wave solution with weak shock profile of the broadwell model system of the boltzmann equation is asymptotically stable provided the initial disturbance is small and its hydrodynamical moments are of integral zero each proof is given by applying an elementary energy method to the integrated system of the conservation form of the original one the property of integral zero of the initial disturbance plays a crucial role in this procedure

discounted mdp s distribution functions and exponential utility maximization the present value of the rewards associated with a discrete time markov process has a probability distribution which depends on the initial state the first part of the paper applies fixed point theory to a system of equations for the distribution functions of the present value the second part of the paper expands the model to a markov decision process mdp and considers the maximization of the expected utility of the present value when the utility function is exponential

an eigenvalue approach to the risk sensitive control problem in near monotone case abstract risk sensitive control problem under near monotonicity condition is considered we prove existence of a solution to the corresponding hamilton jacobi bellman equation by an eigenvalue approach existence of an optimal control has also been proved

a learning algorithm for risk sensitive cost a linear function approximation based reinforcement learning algorithm is proposed for markov decision processes with infinite horizon risk sensitive cost its convergence is proved using the o d e method for stochastic approximation the scheme is also extended to continuous state space processes

a mixed value and policy iteration method for stochastic control with universally measurable policies we consider stochastic optimal control models with borel spaces and universally measurable policies for such models the standard policy iteration is known to have difficult measurability issues and cannot be carried out in general we present a mixed value and policy iteration method that circumvents this difficulty the method allows the use of stationary policies in computing the optimal cost function in a manner that resembles policy iteration it can also be used to address similar difficulties of policy iteration in the context of upper and lower semicontinuous models we analyze the convergence of the method in infinite horizon total cost problems for the discounted case where the one stage costs are bounded and for the undiscounted case where the one stage costs are nonpositive or nonnegative for undiscounted total cost problems with nonnegative one stage costs we also give a new convergence theorem for value iteration that shows that value iteration converges whenever it is initialized with a f

analysis of stochastic approximation schemes with set valued maps in the absence of a stability guarantee and their stabilization in this paper we analyze the behavior of stochastic approximation schemes with set valued maps in the absence of a stability guarantee we prove that after a large number of iterations if the stochastic approximation process enters the domain of attraction of an attracting set it gets locked into the attracting set with high probability we demonstrate that the above mentioned result is an effective instrument for analyzing stochastic approximation schemes in the absence of a stability guarantee by using it to obtain an alternate criterion for convergence in the presence of a locally attracting set for the mean field and by using it to show that a feedback mechanism which involves resetting the iterates at regular time intervals stabilizes the scheme when the mean field possesses a globally attracting set thereby guaranteeing convergence the results in this paper build on the works of borkar andrieu et al and chen et al by allowing for the presence of set valued drift functions

asymptotic stability of traveling waves for scalar viscous conservation laws with non convex nonlinearity the asymptotic stability of traveling wave solutions with shock profile is considered for scalar viscous conservation lawsu t f u x u xx with the initial datau0 which tend to the constant statesu asx stability theorems are obtained in the absence of the convexity off and in the allowance ofs shock speed f u moreover the rate of asymptotics in time is investigated for the casef u s f u if the integral of the initial disturbance over x is small and decays at the algebraic rate as x then the solution approaches the traveling wave at the corresponding rate ast this rate seems to be almost optimal compared with the rate in the casef u2 2 for which an explicit form of the solution exists the rate is also obtained in the casef u s under some additional conditions proofs are given by applying an elementary weighted energy method to the integrated equation of the original one the selection of the weight plays a crucial role in those procedures

convergence rate of solutions toward traveling waves for the cauchy problem of generalized korteweg de vries burgers equations abstract in this paper we investigate the exponential time decay rate of solutions toward traveling waves for the cauchy problem of generalized korteweg de vries burgers equations e u t u x x x u x x f u x 0 t 0 x r with prescribed initial data i u x 0 u 0 x u as x here 0 and 0 are real constants u u are two given constants and the smooth nonlinear function f u is assumed to be either convex or concave an exponential time decay rate of its global solution toward traveling wave solutions is obtained by employing the space time weighted energy method which was initiated by kawashima and matsumura s kawashima a matsumura asymptotic stability of traveling wave solutions of systems for one dimensional gas motion comm math phys 101 1985 97 127 and later elaborated by matsumura nishihara a matsumura k nishihara asymptotic stability of traveling waves for scalar viscous conservation laws with non convex nonlinearity comm math phys 165 1994 83 96 and nishikawa m nishikawa convergence rates to the traveling wave for viscous conservation laws funkcial ekvac 41 1998 107 132

error bounds for constant step size q learning we provide a bound on the first moment of the error in the q function estimate resulting from fixed step size algorithms applied to finite state space discounted reward markov decision problems motivated by tsitsiklis proof for the decreasing step size case we decompose the q learning update equations into a dynamical system driven by a noise sequence and another dynamical system whose state variable is the q learning error i e the difference between the true q function and the estimate a natural persistence of excitation condition allows us to sample the system periodically and derive a simple scalar difference equation from which the convergence properties and bounds on the error of the q learning algorithm can be derived

partially observable risk sensitive markov decision processes we consider the problem of minimizing a certainty equivalent of the total or discounted cost over a finite and an infinite time horizon that is generated by a partially observable markov decision process pomdp in contrast to a risk neutral decision maker this optimization criterion takes the variability of the cost into account it contains as a special case the classical risk sensitive optimization criterion with an exponential utility we show that this optimization problem can be solved by embedding the problem into a completely observable markov decision process with extended state space and give conditions under which an optimal policy exists the state space has to be extended by the joint conditional distribution of current unobserved state and accumulated cost in case of an exponential utility the problem simplifies considerably and we rediscover what in previous literature has been named information state however since we do not use any change of measure techniques here our approach is s

process based risk measures and risk averse control of discrete time systems for controlled discrete time stochastic processes we introduce a new class of dynamic risk measures which we call process based their main feature is that they measure risk of processes that are functions of the history of a base process we introduce a new concept of conditional stochastic time consistency and we derive the structure of process based risk measures enjoying this property we show that they can be equivalently represented by a collection of static law invariant risk measures on the space of functions of the state of the base process we apply this result to controlled markov processes and we derive dynamic programming equations we also derive dynamic programming equations for multistage stochastic programming with decision dependent distributions

decay rates for the solutions of model equations for bore propagation we study the large time behaviour of solutions to the korteweg de vries burgers equation with bore like initial data this work relies on the methods of amick bona and schonbeck to obtain sharp rates of temporal decay of certain norms of the solution thus obtaining an improvement over results of naumkin and shishmarev

more risk sensitive markov decision processes we investigate the problem of minimizing a certainty equivalent of the total or discounted cost over a finite and an infinite horizon that is generated by a markov decision process mdp in contrast to a risk neutral decision maker this optimization criterion takes the variability of the cost into account it contains as a special case the classical risk sensitive optimization criterion with an exponential utility we show that this optimization problem can be solved by an ordinary mdp with extended state space and give conditions under which an optimal policy exists in the case of an infinite time horizon we show that the minimal discounted cost can be obtained by value iteration and can be characterized as the unique solution of a fixed point equation using a sandwich argument interestingly it turns out that in the case of a power utility the problem simplifies and is of similar complexity than the exponential utility case however has not been treated in the literature so far we also establish the validity and convergence of the policy improvement method a simple numerical example namely the classical repeated casino game is considered to illustrate the influence of the certainty equivalent and its parameters finally the average cost problem is also investigated surprisingly it turns out that under suitable recurrence conditions on the mdp for convex power utility the minimal average cost does not depend on the parameter of the utility function and is equal to the risk neutral average cost this is in contrast to the classical risk sensitive criterion with exponential utility

reinforcement learning for long run average cost abstract a large class of sequential decision making problems under uncertainty can be modeled as markov and semi markov decision problems smdps when their underlying probability structure has a markov chain they may be solved by using classical dynamic programming dp methods however dp methods suffer from the curse of dimensionality and break down rapidly in face of large state spaces in addition dp methods require the exact computation of the so called transition probabilities which are often hard to obtain and are hence said to suffer from the curse of modeling as well in recent years a simulation based method called reinforcement learning rl has emerged in the literature it can to a great extent alleviate stochastic dp of its curses by generating near optimal solutions to problems having large state spaces and complex transition mechanisms in this paper a simulation based algorithm that solves markov and smdps is presented along with its convergence analysis the algorithm involves a step size based transformation on two time scales its convergence analysis is based on a recent result on asynchronous convergence of iterates on two time scales we present numerical results from the new algorithm on a classical preventive maintenance case study of a reasonable size where results on the optimal policy are also available in addition we present a tutorial that explains the framework of rl in the context of long run average cost smdps

on stochastic gradient and subgradient methods with adaptive steplength sequences traditionally stochastic approximation sa schemes have been popular choices for solving stochastic optimization problems however the performance of standard sa implementations can vary significantly based on the choice of the steplength sequence and in general little guidance is provided about good choices motivated by this gap we present two adaptive steplength schemes for strongly convex differentiable stochastic optimization problems equipped with convergence theory that aim to overcome some of the reliance on user specific parameters the first scheme referred to as a recursive steplength stochastic approximation rsa scheme optimizes the error bounds to derive a rule that expresses the steplength at a given iteration as a simple function of the steplength at the previous iteration and certain problem parameters the second scheme termed as a cascading steplength stochastic approximation csa scheme maintains the steplength sequence as a piecewise constant decreasing function with the reduction in the steplength occurring when a suitable error threshold is met then we allow for nondifferentiable objectives but with bounded subgradients over a certain domain in such a regime we propose a local smoothing technique based on random local perturbations of the objective function that leads to a differentiable approximation of the function assuming a uniform distribution on the local randomness we establish a lipschitzian property for the gradient of the approximation and prove that the obtained lipschitz bound grows at a modest rate with problem size this facilitates the development of an adaptive steplength stochastic approximation framework which now requires sampling in the product space of the original measure and the artificially introduced distribution

state space formulae for all stabilizing controllers that satisfy and h norm bound and relations to risk sensitivity abstract given a linear system all stabilizing controllers such that a specified closed loop transfer function has h norm less than a given scalar are parametrized this characterization involves the solution to two algebraic riccati equations each with the same order as the system and further gives gives feasible controllers also with this order the relationship to the risk sensitive lqg stochastic control problem is established giving an equivalence between robust and stochastic control

necessary and sufficient conditions for a solution to the risk sensitive poisson equation on a finite state space a markov chain with finite state space endowed with a cost function is considered the transition mechanism is stationary the observer has a constant risk sensitivity and the overall performance of the chain is measured by the risk sensitive long run average cost in this context the existence of solutions of the corresponding poisson equation for arbitrary cost function is characterized in terms of the communication properties of the transition matrix the result in this direction establishes that the poisson equation has a solution for each cost function if and only if the transition matrix has a unique recurrent class and a strong form of the doeblin condition holds

a turnpike theorem for a risk sensitive markov decision process with stopping this paper concerns a markov decision problem with an infinite planning horizon stationary data an exponential utility function and no discounting but also with the possibility of voluntary or involuntary termination stopping this paper establishes conditions under which the expected utility of the income received by repeated use of a well chosen stationary control equals the limit as n becomes large of the optimal expected utility for the n period problem tests for these turnpike conditions are provided as is a linear program whose optimal solution determines such a control

actor critic type learning algorithms for markov decision processes algorithms for learning the optimal policy of a markov decision process mdp based on simulated transitions are formulated and analyzed these are variants of the well known actor critic or adaptive critic algorithm in the artificial intelligence literature distributed asynchronous implementations are considered the analysis involves two time scale stochastic approximations

perturbation realization potentials and sensitivity analysis of markov processes two fundamental concepts and quantities realization factors and performance potentials are introduced for markov processes the relations among these two quantities and the group inverse of the infinitesimal generator are studied it is shown that the sensitivity of the steady state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a markov process based on these results algorithms for estimating performance sensitivities on a single sample path of a markov process can be proposed the potentials in this paper are defined through realization factors and are shown to be the same as those defined by poisson equations the results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis ipa and non ipa approaches to the sensitivity analysis of steady state performance they also provide a theoretical background for the pa algorithms developed in recent years

convergence analysis of the constant modulus algorithm we study the global convergence of the stochastic gradient constant modulus algorithm cma in the absence of channel noise as well as in the presence of channel noise the case of fractionally spaced equalizer and or multiple antenna at the receiver is considered for the noiseless case we show that with proper initialization and with small step size the algorithm converges to a zero forcing filter with probability close to one in the presence of channel noise such as additive gaussian noise we prove that the algorithm diverges almost surely on the infinite time horizon however under suitable conditions the algorithm visits a small neighborhood of the wiener filters a large number of times before ultimately diverging

a stability criterion for two timescale stochastic approximation schemes we present the first sufficient conditions that guarantee stability of two timescale stochastic approximation schemes our analysis is based on the ordinary differential equation ode method and is an extension of the results in borkar and meyn 2000 for single timescale schemes as an application of our result we show the stability of iterates in a two timescale stochastic approximation scheme arising in reinforcement learning

discounted approximations to the risk sensitive average cost in finite markov chains abstract this work concerns with markov chains on a finite state space which is endowed with a cost function the evolution of the chain is observed by an agent with constant risk sensitivity and assuming that the state space is a communicating class the relation between the risk sensitive discounted and average performance criteria is studied it is proved that as the discount factor increases to 1 an appropriate normalization of the discounted value function converges to the average cost also it is shown that if the classical normalization used in the risk neutral case is applied in the risk sensitive context then the normalized discounted value function converges to an arithmetic mean of the average cost

risk sensitive control of discrete time markov processes with infinite horizon in this paper we study existence of solutions to the bellman equation corresponding to risk sensitive ergodic control of discrete time markov processes using three different approaches also for particular classes of systems asymptotics for vanishing risk factor is investigated showing that in the limit the optimal value for an average cost per unit time is obtained

zero sum risk sensitive stochastic differential games we study zero sum risk sensitive stochastic differential games on the infinite horizon with discounted and ergodic payoff criteria under certain assumptions we establish the existence of values and saddle point equilibria we obtain our results by studying the corresponding hamilton jacobi isaacs equations finally we show that the value of the ergodic payoff criterion is a constant multiple of the maximal eigenvalue of the generators of the associated nonlinear semigroups

a sensitivity formula for risk sensitive cost and the actor critic algorithm abstract we propose for risk sensitive control of finite markov chains a counterpart of the popular actor critic algorithm for classical markov decision processes the algorithm is based on a sensitivity formula for the risk sensitive cost and is shown to converge with probability one to the desired solution the proof technique is an adaptation of the ordinary differential equations approach for the analysis of two time scale stochastic approximation algorithms