stability of constrained markov modulated diffusions a family of constrained diffusions in a random environment is considered constraint set is a polyhedral cone and coefficients of the diffusion are governed by in addition to the system state a finite state markov process that is independent of the driving noise such models arise as limit objects in the heavy traffic analysis of generalized jackson networks gjn with markov modulated arrival and processing rates we give sufficient conditions which in particular includes a requirement on the regularity of the underlying skorohod map for positive recurrence and geometric ergodicity when the coefficients only depend on the modulating markov process i e they are independent of the system state a complete characterization for stability and transience is provided the case where the pathwise skorohod problem is not well posed but the underlying reflection matrix is completely s is treated as well as consequences of geometric ergodicity various results such as exponential integrability of invariant measures and central limit results clt for fluctuations of long time averages of process functionals about their stationary values are obtained conditions for stability are formulated in terms of the averaged drift where the average is taken with respect to the stationary distribution of the modulating markov process finally steady state distributions of the underlying gjn are considered and it is shown that under suitable conditions such distributions converge to the unique stationary distribution of the constrained random environment diffusion

stability of queueing networks and scheduling policies we develop a programmatic procedure for establishing the stability of queueing networks and scheduling policies the method uses linear or nonlinear programming to determine what is an appropriate quadratic functional to use as a lyapunov function if the underlying system is markovian our method establishes not only positive recurrence and the existence of a steady state probability distribution but also the geometric convergence of an exponential moment we illustrate this method on several example problems

coherent risk measures in inventory problems abstract we analyze an extension of the classical multi period single item linear cost inventory problem where the objective function is a coherent risk measure properties of coherent risk measures allow us to offer a unifying treatment of risk averse and min max type formulations for the single period newsvendor problem we show that the structure of the optimal solution of the risk averse model is similar to that of the classical expected value problem for a finite horizon dynamic inventory model we show that again the optimal policy has a similar structure as that of the expected value problem this result carries over even to the case when there is a fixed ordering cost we also analyze monotonicity properties of the optimal order quantity with respect to the degree of risk aversion for certain risk measures

the newsboy problem when customer demand is a compound renewal process the newsboy newsvendor problem is probably the simplest of all stochastic inventory problems involving a one time purchase decision and a stochastic sales outcome as an investment it can be interpreted as the simplest stochastic version of the point in point out investment problem of jevons jevons w s theory of political economy macmillan london 1871 this paper provides a compound variation of the newsboy problem instead of demand simply being known as to its distribution here demand is generated by customers arriving at different points in time requiring amounts of varying size arrivals follow a renewal process and amounts required are each taken from a second independent distribution it is shown how the optimal purchase quantity in explicit form depends on properties of the two distributions maximising the expected net present value npv of the payments involved the solution to the compound problem will be the solution to the classical problem if designing a special distribution for the demand process the developments make use of the relation between the npv and the laplace transform simultaneously using the laplace transform as a moment generating function

code shifted differential chaos shift keying with code index modulation for high data rate transmission in the general code shifted differential chaos shift keying gcs dcsk system multiple bit streams are simultaneously carried by cs dcsk signal using walsh codes however its data rate is limited by the available walsh codes in the walsh code matrix due to orthogonality constraint of gcs dcsk signal in this paper we propose a new gcs dcsk scheme with code index modulation cim cs dcsk which can achieve higher spectrum efficiency using code index modulation to carry additional bits in the proposed scheme the reference chaotic signal is the same as that of gcs dcsk and the multiple information bearing signals are simultaneously modulated by the information symbols and the selected walsh codes with specific indices which are determined by an integer sequence mapped from information symbols via the natural numbers to k combinations mapping method we derive bit error rate of the proposed scheme over the additive white gaussian noise channel the numerical results not only verify our analysis but also show the advantage of the proposed scheme over gcs dcsk through analyzing and comparing the numbers of carried bits per transmitted symbol of cim cs dcsk and gcs dcsk it is shown that the proposed scheme can obtain higher data rate than gcs dcsk with the same resource constraints

technical communique a unified approach to markov decision problems and performance sensitivity analysis we propose a simple approach that provides a unified formulation for the performance sensitivity analysis of markov chains and markov decision problems with both infinite horizon average cost and discounted performance criteria

stability convergence and performance of an adaptive control algorithm applied to a randomly varying system the stability and performance of a stochastic adaptive control algorithm applied to a randomly varying linear system are investigated the authors demonstrate that loss functions on the input output process converge to their expectation with respect to an invariant probability at a geometric rate and hence a form of stochastic exponential asymptotic stability is established and when the parameter variation and measurement noise are small it is shown that the performance is nearly optimal and if an excitation signal is added in the control law near consistency of the parameter estimates is obtained further results include central limit theorems and the law of large numbers of the input output and parameter processes

stability of stochastic approximation under verifiable conditions in this paper we address the problem of the stability and convergence of the stochastic approximation procedure theta n 1 theta n gamma n 1 h theta n xi n 1 the stability of such sequences theta n is known to heavily rely on the behavior of the mean field h at the boundary of the parameter set and the magnitude of the stepsizes used the conditions typically required to ensure convergence and in particular the boundedness or stability of theta n are either too difficult to check in practice or not satisfied at all this is the case even for very simple models the most popular technique for circumventing the stability problem consists of constraining theta n to a compact subset mathcal k in the parameter space this is obviously not a satisfactory solution as the choice of mathcal k is a delicate one in this paper we first prove a deterministic stability result which relies on simple conditions on the sequences xi n and gamma n we then propose and analyze an algorithm based on projections on adaptive truncation sets which ensures that the aforementioned conditions required for stability are satisfied we focus in particular on the case where xi n is a so called markov state dependent noise we establish both the stability and convergence with probability 1 w p 1 of the algorithm under a set of simple and verifiable assumptions we illustrate our results with an example related to adaptive markov chain monte carlo algorithms

general results on the convergence of stochastic algorithms a deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form under necessary conditions on the input noise and reasonable conditions on the nonnecessarily continuous mean field emphasis is placed on the case where more than one stationary point exists we also use this approach to prove the convergence of a stochastic algorithm with markovian dynamics

risk sensitive optimal control for markov decision processes with monotone cost the existence of an optimal feedback law is established for the risk sensitive optimal control problem with denumerable state space the main assumptions imposed are irreducibility and anear monotonicity condition on the one step cost function a solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law

on the poisson equation for piecewise deterministic markov processes in this paper we study the problem of the existence of a solution for the poisson equation pe associated to a piecewise deterministic markov process pdp it is well known that the long run average cost of a stochastic process can be obtained through a solution of the pe associated with the process our first result will show that the existence of a solution for the pe of a pdp is equivalent to the existence of a solution for the pe of an embedded discrete time markov chain associated with the pdp it is important to point out that due to the possibility of jumps from the boundary the differential formula for the pdps has a special form so that general results for the pe of continuous time stochastic processes cannot be directly applied usually discrete time conditions for the existence of a solution of a pe of a markov chain are easier to apply than the continuous time counterpart we follow this approach to derive our second result which establishes sufficient conditions for the existence of a

on the functional central limit theorem and the law of the iterated logarithm for markov processes let xt t 0 be an ergodic stationary markov process on a state space s if is its infinitesimal generator on l2 s dm where m is the invariant probability measure then it is shown that for all f in the range of hat a n 1 mathord left vphantom 1 2 right kern nulldelimiterspace 2 int limits 0 nt f x s text ds text t underline underline 0 converges in distribution to the wiener measure with zero drift and variance parameter 2 2 f g 2 g g where g is some element in the domain of such that g f theorem 2 1 positivity of 2 is proved for nonconstant f under fairly general conditions and the range of is shown to be dense in 1 a functional law of the iterated logarithm is proved when the 2 th moment of f in the range of is finite for some 0 theorem 2 7 a under the additional condition of convergence in norm of the transition probability p t x d y to m dy as t for each x the above results hold when the process starts away from equilibrium theorems 2 6 2 7 b applications to diffusions are discussed in some detail

design and performance analysis of a multiuser ofdm based differential chaos shift keying communication system in this paper a multiuser ofdm based chaos shift keying mu ofdm dcsk modulation is presented in this system the spreading operation is performed in time domain over the multicarrier frequencies to allow the multiple access scenario without using excessive bandwidth each user has n p predefined private frequencies from the n available frequencies to transmit its reference signal and share with the other users the remaining frequencies to transmit its m spread bits in this new design n p duplicated chaotic reference signals are used to transmit m bits instead of using m different chaotic reference signals as done in dcsk systems moreover given that n p the mu ofdm dcsk scheme increases spectral efficiency uses less energy and allows multiple access scenario therefore the use of ofdm technique reduces the integration complexity of the system where the parallel low pass filters are no longer needed to recover the transmitted data as in multicarrier dcsk scheme finally the bit error rate performance is investigated under multipath rayleigh fading channels in the presence of multiuser and additive white gaussian noise interferences simulation results confirm the accuracy of our analysis and show the advantages of this new hybrid design

average continuous control of piecewise deterministic markov processes this paper deals with the long run average continuous control problem of piecewise deterministic markov processes pdmps taking values in a general borel space and with compact action space depending on the state variable the control variable acts on the jump rate and transition measure of the pdmp and the running and boundary costs are assumed to be positive but not necessarily bounded our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete time optimality equation related to the embedded markov chain given by the postjump location of the pdmp our second main result guarantees the existence of a feedback measurable selector for the discrete time optimality equation by establishing a connection between this equation and an integro differential equation our final main result is to obtain some sufficient conditions for the existence of a solution for a discrete time optimality inequality and an ordinary optimal feedback control for the long run average cost using the so called vanishing discount approach two examples are presented illustrating the possible applications of the results developed in the paper

the policy iteration algorithm for average reward markov decision processes with general state space the average cost optimal control problem is addressed for markov decision processes with unbounded cost it is found that the policy iteration algorithm generates a sequence of policies which are c regular where c is the cost function under consideration this result only requires the existence of an initial c regular policy and an irreducibility condition on the state space furthermore under these conditions the sequence of relative value functions generated by the algorithm is bounded from below and nearly decreasing from which it follows that the algorithm is always convergent under further conditions it is shown that the algorithm does compute a solution to the optimality equations and hence an optimal average cost policy these results provide elementary criteria for the existence of optimal policies for markov decision processes with unbounded cost and recover known results for the standard linear quadratic gaussian problem in particular in the control of multiclass queueing networks it is found that there is a close connection between optimization of the network and optimal control of a far simpler fluid network model

stochastic approximations and adaptive control of a discrete time single server network with random routing this paper considers a discrete time system composed of k infinite capacity queues that compete for the use of a single server customers arrive in independent and identically distributed i i d batches and are served according to a server allocation policy upon completing service customers either leave the system or are routed instantaneously to another queue according to some random mechanism as an alternative to simply randomized strategies a policy based on a stochastic approximation algorithm is proposed to drive a long run average cost to a given value the underlying motivation can be traced back to implementation issues associated with constrained optimal strategies a version of the ordinary differential equation ode method as given by metivier and priouret is developed for proving almost sure convergence of this algorithm this is done by exploiting the recurrence structure of the system under nonidling policies a probabilistic representation of solutions to an associated poisson equatio

on the use of stochastic differential games against nature to ergodic control problems with unknown parameters in this paper we propose an application of the so called games against nature for giving solution to an ergodic control problem governed by a general class of markov diffusion processes whose coefficients depend on an unknown and non observable parameter to this end we assume that the values of the parameter are taken by means of actions made by some opposite player of the controller the nature then the problem reduces to finding optimality for the controller given that the nature has chosen its best strategy such a control is also known as the worst case optimal control our analysis is based on the use of the dynamic programming technique by showing among other facts the existence of classical twice differentiable solutions of the so called hamilton jacobi bellman equation we also provide an example on economic welfare to illustrate our results

theoremes de convergence presque sure pour une classe d algorithmes stochastiques a pas decroissant the paper studies the pathwise asymptotic behaviour of stochastic algorithms of the following general form theta n 1 theta n gamma n 1 f theta n y n 1 the hypotheses allowing discontinuities on the adaptation termf the process y n n 0 is a markov chain controlled by n for each fixed the markov chain y n n 0 is essentially of positive recurrent type

approximating martingales for variance reduction in markov process simulation knowledge of either analytical or numerical approximations should enable more efficient simulation estimators to be constructed this principle seems intuitively plausible and certainly attractive yet no completely satisfactory general methodology has been developed to exploit it the authors present a new approach for obtaining variance reduction in markov process simulation that is applicable to a vast array of different performance measures the approach relies on the construction of a martingale that is then used as an internal control variate

stability of piecewise deterministic markov processes in this paper we study a form of stability for a general family of nondiffusion markov processes known in the literature as piecewise deterministic markov process pdmp by stability here we mean the existence of an invariant probability measure for the pdmp it is shown that the existence of such an invariant probability measure is equivalent to the existence of a sigma finite invariant measure for a markov kernel g linked to the resolvent operator u of the pdmp satisfying a boundedness condition or equivalently a radon nikod m derivative here we generalize existing results of the literature o costa j appl prob 27 1990 pp 60 73 m davis markov models and optimization chapman and hall 1993 since we do not require any additional assumptions to establish this equivalence moreover we give sufficient conditions to ensure the existence of such a sigma finite measure satisfying the boundedness condition they are mainly based on a modified foster lyapunov criterion for the case in which the markov chain generated by g is either recurrent or weak feller to emphasize the relevance of our results we study three examples and in particular we are able to generalize the results obtained by costa and davis on the capacity expansion model

almost sure functional central limit theorem for the linear random walk on the torus let rho be a probability measure on vargamma mathrm sl d mathbb z consider the random walk defined by rho on the torus mathbb t d mathbb r d mathbb z d for any x in mathbb t d the walk starting at x is defined by begin aligned left begin array l x 0 x x n 1 g n 1 x n end array right end aligned where g n in vargamma mathbb n is chosen with the law rho otimes mathbb n bourgain furmann lindenstrauss and mozes proved that under an assumption on the group generated by the support of rho the random walk starting at any irrational point equidistributes in the torus in this article we study the functional central limit theorem and the almost sure functional central limit theorem for this walk starting at some points having good diophantine properties

on the poisson equation and diffusion approximation 3 three different results are established which turn out to be closely connected so that the first one implies the second one which in turn implies the third one the first one states the smoothness of an invariant diffusion density with respect to a parameter the second establishes a similar smoothness of the solution of the poisson equation in rd the third one states a diffusion approximation result or in other words an averaging of singularly perturbed diffusion for fully coupled sde systems or sde systems with complete dependence

a liapounov bound for solutions of the poisson equation in this paper we consider psi irreducible markov processes evolving in discrete or continuous time on a general state space we develop a liapounov function criterion that permits one to obtain explicit bounds on the solution to the poisson equation and in particular obtain conditions under which the solution is square integrable these results are applied to obtain sufficient conditions that guarantee the validity of a functional central limit theorem for the markov process as a second consequence of the bounds obtained a perturbation theory for markov processes is developed which gives conditions under which both the solution to the poisson equation and the invariant probability for the process are continuous functions of its transition kernel the techniques are illustrated with applications to queueing theory and autoregressive processes

workload models for stochastic networks value functions and performance evaluation this paper concerns control and performance evaluation for stochastic network models structural properties of value functions are developed for controlled brownian motion cbm and deterministic fluid workload models leading to the following conclusions outside of a null set of network parameters the following hold i the fluid value function is a smooth function of the initial state under further minor conditions the fluid value function satisfies the derivative boundary conditions that are required to ensure it is in the domain of the extended generator for the cbm model exponential ergodicity of the cbm model is demonstrated as one consequence ii the fluid value function provides a shadow function for use in simulation variance reduction for the stochastic model the resulting simulator satisfies an exact large deviation principle while a standard simulation algorithm does not satisfy any such bound iii the fluid value function provides upper and lower bounds on performance for the cbm model this follows from an extension of recent linear programming approaches to performance evaluation

hitting times functional inequalities lyapunov conditions and uniform ergodicity the use of lyapunov conditions for proving functional inequalities was initiated in 5 it was shown in 4 30 that there is an equivalence between a poincare inequality the existence of some lyapunov function and the exponential integrability of hitting times in the present paper we close the scheme of the interplay between lyapunov conditions and functional inequalities by showing that strong functional inequalities are equivalent to lyapunov type conditions showing that these lyapunov conditions are characterized by the finiteness of generalized exponential moments of hitting times we also give some complement concerning the link between lyapunov conditions and in tegrability property of the invariant probability measure and as such transportation inequalities and we show that some unbounded lyapunov conditions can lead to uniform ergodicity and coming down from infinity property

synchronization of coupled oscillators is a game the purpose of this paper is to understand phase transition in noncooperative dynamic games with a large number of agents applications are found in neuroscience biology and economics as well as traditional engineering applications the focus of analysis is a variation of the large population linear quadratic gaussian lqg model of huang et al 2007 comprised here of a controlled n dimensional stochastic differential equation model coupled only through a cost function the states are interpreted as phase angles for a collection of heterogeneous oscillators and in this way the model may be regarded as an extension of the classical coupled oscillator model of kuramoto a deterministic pde model is proposed which is shown to approximate the stochastic system as the population size approaches infinity key to the analysis of the pde model is the existence of a particular nash equilibrium in which the agents opt out of the game setting their controls to zero resulting in the incoherence equilibrium methods from dynamical systems theory are used in a bifurcation analysis based on a linearization of the partial differential equation pde model about the incoherence equilibrium a critical value of the control cost parameter is identified above this value the oscillators are incoherent and below this value when control is sufficiently cheap the oscillators synchronize these conclusions are illustrated with results from numerical experiments

multivariable feedback particle filter this paper presents the multivariable extension of the feedback particle filter fpf algorithm for the nonlinear filtering problem in continuous time the fpf is a control oriented approach to particle filtering the approach does not require importance sampling or resampling and offers significant variance improvements in particular the algorithm can be applied to systems that are not stable this paper describes new representations and algorithms for the fpf in the general multivariable nonlinear non gaussian setting theory surrounding the fpf is improved exactness of the fpf is established in the general setting as well as well posedness of the associated boundary value problem to obtain the filter gain a galerkin finite element algorithm is proposed for approximation of the gain its performance is illustrated in numerical experiments

consistency of the maximum likelihood estimator for general hidden markov models consider a parametrized family of general hidden markov models where both the observed and unobserved components take values in a complete separable metric space we prove that the maximum likelihood estimator mle of the parameter is strongly consistent under a rather minimal set of assumptions as special cases of our main result we obtain consistency in a large class of nonlinear state space models as well as general results on linear gaussian state space models and finite state models a novel aspect of our approach is an information theoretic technique for proving identifiability which does not require an explicit representation for the relative entropy rate our method of proof could therefore form a foundation for the investigation of mle consistency in more general dependent and non markovian time series also of independent interest is a general concentration inequality for v uniformly ergodic markov chains

perturbation realization potentials and sensitivity analysis of markov processes two fundamental concepts and quantities realization factors and performance potentials are introduced for markov processes the relations among these two quantities and the group inverse of the infinitesimal generator are studied it is shown that the sensitivity of the steady state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a markov process based on these results algorithms for estimating performance sensitivities on a single sample path of a markov process can be proposed the potentials in this paper are defined through realization factors and are shown to be the same as those defined by poisson equations the results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis ipa and non ipa approaches to the sensitivity analysis of steady state performance they also provide a theoretical background for the pa algorithms developed in recent years

estimation of continuous time markov processes sampled at random time intervals we introduce a family of generalized method of moments estimators of the parameters of a continuous time markov process observed at random time intervals the results include strong consistency asymptotic normality and a characterization of standard errors sampling is at an arrival intensity that is allowed to depend on the underlying markov process and on the parameter vector to be estimated we focus on financial applications including tick based sampling allowing for jump diffusions regime switching diffusions and reflected diffusions

variance reduction using nonreversible langevin samplers a standard approach to computing expectations with respect to a given target measure is to introduce an overdamped langevin equation which is reversible with respect to the target distribution and to approximate the expectation by a time averaging estimator as has been noted in recent papers 30 37 61 72 introducing an appropriately chosen nonreversible component to the dynamics is beneficial both in terms of reducing the asymptotic variance and of speeding up convergence to the target distribution in this paper we present a detailed study of the dependence of the asymptotic variance on the deviation from reversibility our theoretical findings are supported by numerical simulations

robust multi market newsvendor models with interval demand data we present a robust model for determining the optimal order quantity and market selection for short life cycle products in a single period newsvendor setting due to limited information about demand distribution in particular for short life cycle products stochastic modeling approaches may not be suitable we propose the minimax regret multi market newsvendor model where the demands are only known to be bounded within some given interval in the basic version of the problem a linear time solution method is developed for the capacitated case we establish some structural results to reduce the problem size and then propose an approximation solution algorithm based on integer programming finally we compare the performance of the proposed minimax regret model against the typical average case and worst case models our test results demonstrate that the proposed minimax regret model outperformed the average case and worst case models in terms of risk related criteria and mean profit respectively

orthogonal frequency division multiplexing with index modulation in this paper a novel orthogonal frequency division multiplexing ofdm scheme called ofdm with index modulation ofdm im is proposed for operation over frequency selective and rapidly time varying fading channels in this scheme the information is conveyed not only by m ary signal constellations as in classical ofdm but also by the indices of the subcarriers which are activated according to the incoming bit stream different low complexity transceiver structures based on maximum likelihood detection or log likelihood ratio calculation are proposed and a theoretical error performance analysis is provided for the new scheme operating under ideal channel conditions then the proposed scheme is adapted to realistic channel conditions such as imperfect channel state information and very high mobility cases by modifying the receiver structure the approximate pairwise error probability of ofdm im is derived under channel estimation errors for the mobility case several interference unaware aware detection methods are proposed for the new scheme it is shown via computer simulations that the proposed scheme achieves significantly better error performance than classical ofdm due to the information bits carried by the indices of ofdm subcarriers under both ideal and realistic channel conditions

semi markov decision problems and performance sensitivity analysis recent research indicates that markov decision processes mdps can be viewed from a sensitivity point of view and the perturbation analysis pa mdps and reinforcement learning rl are three closely related areas in optimization of discrete event dynamic systems that can be modeled as markov processes the goal of this paper is two fold first we develop the pa theory for semi markov processes smps and then we extend the aforementioned results about the relation among pa mdp and rl to smps in particular we show that performance sensitivity formulas and policy iteration algorithms of semi markov decision processes can be derived based on the performance potential and realization matrix both the long run average and discounted cost problems are considered this approach provides a unified framework for both problems and the long run average problem corresponds to the discounted factor being zero the results indicate that performance sensitivities and optimization depend only on first order statistics single sample path based implementations are discussed