mean variance optimization of discrete time discounted markov decision processes abstract in this paper we study a mean variance optimization problem in an infinite horizon discrete time discounted markov decision process mdp the objective is to minimize the variance of system rewards with the constraint of mean performance different from most of works in the literature which require the mean performance already achieve optimum we can let the discounted performance equal any constant the difficulty of this problem is caused by the quadratic form of the variance function which makes the variance minimization problem not a standard mdp by proving the decomposable structure of the feasible policy space we transform this constrained variance minimization problem to an equivalent unconstrained mdp under a new discounted criterion and a new reward function the difference of the variances of markov chains under any two feasible policies is quantified by a difference formula based on the variance difference formula a policy iteration algorithm is developed to find the optimal policy we also prove the optimality of deterministic policy over the randomized policy generated in the mean constrained policy space numerical experiments demonstrate the effectiveness of our approach

on actor critic algorithms in this article we propose and analyze a class of actor critic algorithms these are two time scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture and the actor is updated in an approximate gradient direction based on information provided by the critic we show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor we study actor critic algorithms for markov decision processes with polish state and action spaces we state and prove two results regarding their convergence

variability in noise driven integrator neurons neural variability in the presence of noise has been studied mainly in resonator neurons such as hodgkinhuxley or fitzhugh nagumo models here we investigate this variability for integrator neurons whose excitability is due to a saddle node bifurcation of the rest state instead of a hopf bifurcation using simple theoretical expressions for the interspike times distributions we obtain coefficients of variation in good agreement with numerical calculations in realistic neuron models the main features of this coefficient as a function of noise depend on the refractory period and on the presence of bistability the bistability is responsible for the existence of two different time scales in the spiking behavior giving an antiresonance effect

optimization of markov decision processes under the variance criterion in this paper we study a variance minimization problem in an infinite stage discrete time markov decision process mdp regardless of the mean performance for the markov chain under the variance criterion since the value of the cost function at the current stage will be affected by future actions this problem is not a standard mdp and the traditional mdp theory is not applicable in this paper we convert the variance minimization problem into a standard mdp by introducing a concept called pseudo variance then we derive a variance difference formula that quantifies the difference of variances of markov systems under any two policies with the difference formula the correlation of the variance cost function at different stages can be decoupled through a nonnegative term a necessary condition of the optimal policy is obtained it is also proved that the optimal policy with the minimal variance can be found in the deterministic policy space furthermore we propose an efficient iterative algorithm to reduce the variance of markov systems we prove that this algorithm can converge to a local optimum finally a numerical experiment is conducted to demonstrate the efficiency of our algorithm compared with the gradient based method widely adopted in the literature

on line sensitivity analysis of markov chains discrete event systems modeled as continuous time markov processes and characterized by some integer valued parameter are considered the problem addressed is that of estimating performance sensitivities with respect to this parameter by directly observing a single sample path of the system the approach is based on transforming the nominal markov chain into a reduced augmented chain the stationary state probabilities which can be easily combined to obtain stationary state probability sensitivities with respect to the given parameter under certain conditions the reduced augmented chain state transitions are observable with respect to the state transitions of the system itself and no knowledge of the nominal markov chain state of the transition rates is required applications for some queueing systems are included the approach incorporates estimation of unknown transition rates when needed and is extended to real valued parameters

limiting average criteria for nonstationary markov decision processes this paper deals with the so called limiting average criteria for nonstationary markov decision processes with possibly unbounded rewards and borel state space a new set of conditions is provided under which the existence of both a solution to the optimality equations and the limiting average epsilon geq 0 optimal markov policies is derived also a rolling horizon algorithm for computing limiting average epsilon 0 optimal markov policies is developed furthermore the results in this paper are illustrated by several examples such as the water regulation problem

modular units and cuspidal divisor classes on x0 n2m with n 24 and m squarefree abstract for a positive integer n let c n be the subgroup of j 0 n generated by the equivalence classes of cuspidal divisors of degree 0 and c n q c n j 0 n q be its q rational subgroup let also c q n be the subgroup of c n q generated by q rational cuspidal divisors we prove that when n n 2 m for some integer n dividing 24 and some squarefree integer m the two groups c n q and c q n are equal to achieve this we show that all modular units on x 0 n on such n are products of functions of the form m k h m h 2 n and k z and determine the necessary and sufficient conditions for products of such functions to be modular units on x 0 n

cuspidal class number of the tower of modular curves x1 npn we consider a cuspidal class number which is the order of a subgroup of the full cuspidal divisor class group of x1 npn with p nmid n and n 1 by studying the second generalized bernoulli numbers we obtain results similar to ones ferrero and washington in ann math 2 109 2 377 395 1979 washington in invent math 49 87 97 1978 about the relative class numbers of cyclotomic mathbb z p extension of an abelian number field

a single sample path based performance sensitivity formula for markov chains using a sample path approach we derive a new formula for performance sensitivities of discrete time markov chains a distinguished feature of this formula is that the quantities involved can be estimated by analyzing a single sample path of a markov chain thus the formula provides a new direction for sensitivity analysis and can be viewed as an extension of the perturbation realization theory to problems where infinitesimal perturbation analysis does not work well

technical communique a unified approach to markov decision problems and performance sensitivity analysis we propose a simple approach that provides a unified formulation for the performance sensitivity analysis of markov chains and markov decision problems with both infinite horizon average cost and discounted performance criteria

the control of a two level markov decision process by time aggregation the solution of markov decision processes mdps often relies on special properties of the processes for two level mdps the difference in the rates of state changes of the upper and lower levels has led to limiting or approximate solutions of such problems in this paper we solve a two level mdp without making any assumption on the rates of state changes of the two levels we first show that such a two level mdp is a non standard one where the optimal actions of different states can be related to each other then we give assumptions conditions under which such a specially constrained mdp can be solved by policy iteration we further show that the computational effort can be reduced by decomposing the mdp a two level mdp with m upper level states can be decomposed into one mdp for the upper level and m to m m 1 mdps for the lower level depending on the structure of the two level mdp the upper level mdp is solved by time aggregation a technique introduced in a recent paper cao x r ren z y bhatnagar s fu m marcus s 2002 a time aggregation approach to markov decision processes automatica 38 6 929 943 and the lower level mdps are solved by embedded markov chains

optimization of average rewards of time nonhomogeneous markov chains we study the optimization of average rewards of discrete time nonhomogeneous markov chains in which the state spaces transition probabilities and reward functions depend on time the analysis encounters a few major difficulties 1 notions crucial to homogeneous markov chains such as ergodicity stationarity periodicity and connectivity no longer apply 2 the average reward criterion is under selective i e it does not depend on the decisions in any finite period and thus dynamic programming is not amenable and 3 because of the under selectivity an optimal average reward policy may not be the best in any finite period

markov decision processes with a minimum variance criterion abstract we consider the optimization of the variance of the sum of costs as well as that of an average expected cost in markov decision processes with unbounded cost in case of general state and action space we find the stationary policy which makes the average variance as small as possible in the class of policies which are e optimal in an average expected cost

nonhomogeneous markov decision processes with borel state space the average criterion with nonuniformly bounded rewards this paper deals with nonhomogeneous markov decision processes with borel state space and nonuniformly bounded rewards under the average criterion first under the minorant type assumption we prove the existence of an appropriate solution to the optimality equations second from the optimality equations we also establish the existence of e 0 optimal markov policies under the additional conditions third some sufficient conditions for the validity of the assumptions in this paper and several examples such as inventory production systems are provided finally as an application of the optimality equations a rolling horizon algorithm is given

discrete time controlled markov processes with average cost criterion a survey this work is a survey of the average cost control problem for discrete time markov processes the authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades the exposition ranges from finite to borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies the authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography the authors have also identified several important questions that are still open to investigation

optimal control of ergodic continuous time markov chains with average sample path rewards in this paper we study continuous time markov decision processes with the average sample path reward aspr criterion and possibly unbounded transition and reward rates we propose conditions on the system s primitive data for the existence of epsilon aspr optimal deterministic stationary policies in a class of randomized markov policies satisfying some additional continuity assumptions the proof of this fact is based on the time discretization technique the martingale stability theory and the concept of potential we also provide both policy and value iteration algorithms for computing or at least approximating the epsilon aspr optimal stationary policies we illustrate with examples our main results as well as the difference between the aspr and the average expected reward criteria

quantitative non vanishing of dirichlet l values modulo p let p be an odd prime and k a non negative integer let n be a positive integer such that p not mid n and lambda a dirichlet character modulo n we obtain quantitative lower bounds for the number of dirichlet character chi modulo f with the critical dirichlet l value l k lambda chi being p indivisible here f rightarrow infty with n f 1 and p not mid f phi f we explore the indivisibility via an algebraic and a homological approach the latter leads to a bound of the form f 1 2 the p indivisibility yields results on the distribution of the associated p selmer ranks we also consider an iwasawa variant it leads to an explicit upper bound on the lowest conductor of the characters factoring through the iwasawa mathbb z ell extension of mathbb q for an odd prime ell ne p with the corresponding critical l value twists being p indivisible

general solution of the poisson equation for quasi birth and death processes we consider the poisson equation i p u g where p is the transition matrix of a quasi birth and death process with infinitely many levels g is a given infinite dimensional vector and u is the unknown our main result is to provide the general solution of this equation to this purpose we use the block tridiagonal and block toeplitz structure of the matrix p to obtain a set of matrix difference equations which are solved by constructing suitable resolvent triples

a note on the convergence rate of the value iteration scheme in controlled markov chains this work is concerned with controlled markov chains with bounded costs assuming that the transition probabilities satisfy a simultaneous doeblin condition it is shown that schweitzer s transformation on the transition law yields a strong ergodicity condition that implies that the solution to the average cost optimality equation can be approximated at a geometric rate via the value iteration scheme

modular units and cuspidal divisor class groups of x 1 n abstract in this article we consider the group f 1 n of modular units on x 1 n that have divisors supported on the cusps lying over of x 0 n called the cusps for each positive integer n we will give an explicit basis for the group f 1 n this enables us to compute the group structure of the rational torsion subgroup c 1 n of the jacobian j 1 n of x 1 n generated by the differences of the cusps in addition based on our numerical computation we make a conjecture on the structure of the p primary part of c 1 p n for a regular prime p

zero sum stochastic games in borel spaces average payoff criteria this paper is concerned with two person zero sum dynamic stochastic games in borel spaces with possibly unbounded payoff function and several average or ergodic payoff criteria we give conditions under which the long run expected average payoff criterion the sample path average criterion the existence of solutions to the average payoff shapley equations and a certain martingale condition are all equivalent

are limits of discounted optimal policies blackwell optimal a counter example abstract in markov decision chains mdc s with a finite state space and finite action sets it is a well known result that the limits of discounted optimal policies for tending to 1 are blackwell optimal it was conjectured in a recent paper by cavazos cadena and lasserre that this property of limiting policies also holds for unichain mdc s we disprove this conjecture by constructing a non blackwell limiting policy in a unichain mdc with finitely many states and compact action sets

another set of conditions for markov decision processes with average sample path costs abstract this paper deals with discrete time markov decision processes with average sample path costs aspc in borel spaces the costs may have neither upper nor lower bounds we propose new conditions for the existence of e aspc optimal deterministic stationary policies in the class of all randomized history dependent policies our conditions are weaker than those in the previous literature moreover some sufficient conditions for the existence of aspc optimal stationary policies are imposed on the primitive data of the model in particular the stochastic monotonicity condition in this paper has first been used to study the aspc criterion also the approach provided here is slightly different from the optimality equation approach widely used in the previous literature on the other hand under mild assumptions we show that average expected cost optimality and aspc optimality are equivalent finally we use a controlled queueing system to illustrate our results

optimization of queues using an infinitesimal perturbation analysis based stochastic algorithm with general update times convergence with probability one of a stochastic optimization algorithm for a single server queue is proved the parameter to be optimized is updated using an infinitesimal perturbation analysis estimate of the gradient of the performance measure and the updates are performed at general times first an algorithm in which the parameter is updated before each customer begins service is considered then it is shown how this analysis can be extended to an algorithm that updates at certain stopping times the analysis suggests that the sample path behavior of the algorithm is similar to that of an algorithm that updates at the start of every busy period

finding optimal observation based policies for constrained pomdps under the expected average reward criterion in this technical note constrained partially observable markov decision processes with discrete state and action spaces under the average reward criterion are studied from a sensitivity point of view by analyzing the derivatives of performance criteria we develop a simulation based optimization algorithm to find the optimal observation based policy on the basis of a single sample path this algorithm does not need any overly strict assumption and can be applied to the general ergodic markov systems with the imperfect state information the performance is proved to converge to the optimum with probability 1 one numerical example is provided to illustrate the applicability of the algorithm

a unified approach to markov decision problems and performance sensitivity analysis with discounted and average criteria multichain cases we propose a unified framework to markov decision problems and performance sensitivity analysis for multichain markov processes with both discounted and average cost performance criteria with the fundamental concept of performance potentials we derive both performance gradient and performance difference formulas which play the central role in performance optimization the standard policy iteration algorithms for both discounted and average reward mdps can be established using the performance difference formulas in a simple and intuitive way and the performance gradient formulas together with stochastic approximation may lead to new optimization schemes this sensitivity based point of view of performance optimization provides some insights that link perturbation analysis markov decision processes and reinforcement learning together the research is an extension of the previous work on ergodic markov chains cao automatica 36 2000 771

some new results on sample path optimality in ergodic control of diffusions we present some new results on sample path optimality for the ergodic control problem of a class of nondegenerate diffusions controlled through the drift the hypothesis most often used in the literature to ensure the existence of an almost sure sample path optimal stationary markov control requires finite second moments of the first hitting times tau of bounded domains over all admissible controls we show that this can be considerably weakened mathbb e tau 2 may be replaced with mathbb e tau ln tau thus reducing the required rate of convergence of averages from polynomial to logarithmic a foster lyapunov condition that guarantees this is also exhibited moreover we study a large class of models that are neither uniformly stable nor have a near monotone running cost and we exhibit sufficient conditions for the existence of a sample path optimal stationary markov control

necessary and sufficient conditions for a bounded solution to the optimality equation in average reward markov decision chains abstract we consider average reward markov decision processes with discrete time parameter and denumerable state space we are concerned with the following problem find necessary and sufficient conditions so that for arbitrary bounded reward function the corresponding average reward optimality equation has a bounded solution this problem is solved for a class of systems including the case in which under the action of any stationary policy the state space is an irreducible positive recurrent class

smoothed conditional perturbation analysis of discrete event dynamical systems a new sample path analysis approach based on the smoothing property of conditional expectation for estimating the performance sensitivity of discrete event dynamical systems is proposed several examples are presented to show how this approach overcomes a difficulty of the ordinary infinitesimal perturbation analysis the basic message is that one can get more knowledge about the system performance by observing and analyzing the sample path than by using the conventional simulation approach it is also pointed out that the classical queueing theory approach for getting the performance sensitivity and the sample path based infinitesimal perturbation analysis approach can be unified in the framework of the new approach the smoothed conditional perturbation analysis

bias optimality and strong n n 1 0 discount optimality for markov decision processes abstract in this paper we study both bias optimality and strong n n 1 0 discount optimality for denumerable discrete time markov decision processes the rewards may have neither upper nor lower bounds we give sufficient conditions on the system s primitive data and under which we prove 1 the existence of the bias optimality equation and bias optimal policies 2 a condition equivalent to bias optimal policies 3 average expected reward optimality and strong 1 discount optimality are equivalent 4 bias optimality and strong 0 discount optimality are equivalent 5 the existence of strong n n 1 0 discount optimal stationary policies our conditions are weaker than those in the previous literature moreover our results are illustrated by a controlled random walk

sample path optimality and variance minimization of average cost markov control processes this paper studies several average cost criteria for markov control processes on borel spaces with possibly unbounded costs under suitable hypotheses we show i the existence of a sample path average cost spac optimal stationary policy ii a stationary policy is spac optimal if and only if it is expected average cost eac optimal and iii within the class of stationary spac optimal equivalently eac optimal policies there exists one with a minimal limiting average variance

physiological gain leads to high isi variability in a simple model of a cortical regular spiking cell to understand the interspike interval isi variability displayed by visual cortical neurons softky koch 1993 it is critical to examine the dynamics of their neuronal integration as well as

recurrence conditions for average and blackwell optimality in denumerable state markov decision chains in a previous paper dekker and hordijk 1988 presented an operator theoretical approach for multichain markov decision processes with a countable state space compact action sets and unbounded rewards conditions were presented guaranteeing the existence of a laurent series expansion for the discounted rewards the existence of average and blackwell optimal policies and the existence of solutions for the average and blackwell optimality equations while these assumptions were operator oriented and formulated as conditions for the deviation matrix we will show in this paper that the same approach can also be carried out under recurrence conditions these new conditions seem easier to check in general and are especially suited for applications in queueing models

a mean variance optimization problem for discounted markov decision processes in this paper we consider a mean variance optimization problem for markov decision processes mdps over the set of deterministic stationary policies different from the usual formulation in mdps we aim to obtain the mean variance optimal policy that minimizes the variance over a set of all policies with a given expected reward for continuous time mdps with the discounted criterion and finite state and action spaces we prove that the mean variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and markov properties then we show that a mean variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations we also address related issues such as a mutual fund theorem and illustrate our results with an example

zero sum semi markov games this paper deals with borel state and action spaces zero sum semi markov games under the expected long run average payoff criterion the transition probabilities are assumed to satisfy some generalized geometric ergodicity conditions the main result states that the optimality equation has a solution which is approximated by the solutions of some varepsilon perturbed semi markov games as a corollary the existence of value and average optimal strategies for the players is established

markov decision processes with a borel measurable cost function the average case we consider a markov decision process with a borel measurable cost function we introduce a p step contraction property for the average cost case by use of this method the validity of the optimality equation and the existence of e optimal stationary policies are proved as some applications the sequential replacement model and the inventory model are considered

average sensitive and blackwell optimal policies in denumerable markov decisionchains with unbounded rewards in this paper we consider a discrete time markov decision chain with a denumerable state space and compact action sets and we assume that for all states the rewards and transition probabilities depend continuously on the actions the first objective of this paper is to develop an analysis for average optimality without assuming a special markov chain structure in doing so we present a set of conditions guaranteeing average optimality which are automatically fulfilled in the finite state and action model the second objective is to study simultaneously average and discount optimality as veinott veinott a f jr 1969 on discrete dynamic programming with sensitive discount optimality criteria ann math statist 40 1635 1660 did for the finite state and action model we investigate the concepts of n discount and blackwell optimality in the denumerable state space using a laurent series expansion for the discounted rewards under the same condition as for average optimality we establish solutions to the n discount optimality equations for every n

convergence properties of infinitesimal perturbation analysis infinitesimal perturbation analysis ipa is a method for computing a sample path derivative with respect to an input parameter in a discrete event simulation the ipa algorithm is based on the fact that for certain parameters and any realization of a simulation the change in parameter can be made small enough so that only the times of events get shifted but their order does not change this paper considers the convergence properties of the ipa sample path derivatives in particular the question of when an ipa estimate converges to the derivative of a steady state performance measure is studied necessary and sufficient conditions for this convergence are derived for a class of regenerative processes although these conditions are not guaranteed to be satisfied in general they are satisfied for the mean stationary response time in the m g 1 queue a necessary condition for multiple ipa estimates to simultaneously converge to the derivatives of steady state throughputs in a queueing network is determined the implications of this necessary condition are that except in special cases the original ipa algorithm cannot be used to consistently estimate steady state throughput derivatives in queueing networks with multiple types of customers state dependent routing or blocking numerical studies on ipa convergence properties are also presented

the potential structure of sample paths and performance sensitivities of markov systems we study the structure of sample paths of markov systems by using performance potentials as the fundamental units with a sample path based approach we show that performance sensitivity formulas performance gradients and performance differences of markov systems can be constructed intuitively by first principles with performance potentials or equivalently perturbation realization factors as building blocks in particular we derive sensitivity formulas for two markov chains with possibly different state spaces the proposed approach can be used to obtain flexibly the sensitivity formulas for a wide range of problems including those with partial information these formulas are the basis for performance optimization of discrete event dynamic systems including perturbation analysis markov decision processes and reinforcement learning the approach thus provides insight on on line learning and performance optimization and opens up new research directions sample path based algorithms can be developed

a basic formula for performance gradient estimation of semi markov decision processes abstract this paper presents a basic formula for performance gradient estimation of semi markov decision processes smdps under average reward criterion this formula directly follows from a sensitivity equation in perturbation analysis with this formula we develop three sample path based gradient estimation algorithms by using a single sample path these algorithms naturally extend many gradient estimation algorithms for discrete time markov systems to continuous time semi markov models in particular they require less storage than the algorithm in the literature

markov decision processes with sample path constraints the communicating case we consider time average markov decision processes mdps which accumulate a reward and cost at each decision epoch a policy meets the sample path constraint if the time average cost is below a specified value with probability one the optimization problem is to maximize the expected average reward over all policies that meet the sample path constraint the sample path constraint is compared with the more commonly studied constraint of requiring the average expected cost to be less than a specified value although the two criteria are equivalent for certain classes of mdps their feasible and optimal policies differ for many nontrivial problems in general there does not exist optimal or nearly optimal stationary policies when the expected average cost constraint is employed assuming that a policy exists that meets the sample path constraint we establish that there exist nearly optimal stationary policies for communicating mdps a parametric linear programming algorithm is given to construct nearly optimal stationary policies the discussion relies on well known results from the theory of stochastic processes and linear programming the techniques lead to simple proofs of the existence of optimal and nearly optimal stationary policies for unichain and deterministic mdps respectively

perturbation analysis for the design of flexible manufacturing system flow controllers dynamic allocation of stochastic capacity among competing activities in a just in time manufacturing environment is addressed by optimal flow control optimal policies are characterized by generally intractable bellman equations a near optimal controller design technique is proposed it provides an approximate numerical solution to the bellman equation a tight lower bound for the optimality gap of tractable near optimal controller designs and a building block for improved near optimal controller designs that rely on the decomposition of a multiple part type problem to smaller two or three part type problems computational experience is reported for two and three part type problems

average optimality for continuous time markov decision processes with a policy iteration approach this paper deals with the average expected reward criterion for continuous time markov decision processes in general state and action spaces the transition rates of underlying continuous time jump markov processes are allowed to be unbounded a nd the reward rates may have neither upper nor lower bounds we give conditions on the system s primitive data and under which we prove the existence of the average reward optimality equation and an average optimal stationary policy also under our conditions we ensure the existence of average optimal stationary policies moreover we study some properties of average optimal stationary policies we not only establish another average optimality equation on an average optimal stationary policy but also present an interesting martingale characterization of such a policy the approach provided in this paper is based on the policy iteration algorithm it should be noted that our way is rather different from both the usually vanishing discounting factor approach and the optimality inequality approach widely used in the previous literature

a liapounov bound for solutions of the poisson equation in this paper we consider psi irreducible markov processes evolving in discrete or continuous time on a general state space we develop a liapounov function criterion that permits one to obtain explicit bounds on the solution to the poisson equation and in particular obtain conditions under which the solution is square integrable these results are applied to obtain sufficient conditions that guarantee the validity of a functional central limit theorem for the markov process as a second consequence of the bounds obtained a perturbation theory for markov processes is developed which gives conditions under which both the solution to the poisson equation and the invariant probability for the process are continuous functions of its transition kernel the techniques are illustrated with applications to queueing theory and autoregressive processes

markov decision problems and state action frequencies consider a controlled markov chain with countable state and action spaces basic quantities that determine the values of average cost functionals are identified under some regularity conditions these turn out to be a collection of numbers one for each state action pair describing for each state the relative number of uses of each action these conditional frequencies which are defined pathwise are shown to determine the state action frequencies that in the finite case are known to determine the costs this is extended to the countable case allowing for unbounded costs the space of frequencies is shown to be compact and convex and the extreme points are identified with stationary deterministic policies conditions under which the search for optimality in several optimization problems may be restricted to stationary policies are given these problems include the standard markov decision process as well as constrained optimization both in terms of average cost functionals and variability sensitive optimization an application to a queueing problem is given where these results imply the existence and explicit computation of optimal policies in constrained optimization problems the pathwise definition of the conditional frequencies implies that their values can be controlled directly moreover they depend only on the limiting behavior of the control this has immediate application to adaptive control of markov chains including adaptive control under constraints

solutions of the average cost optimality equation for markov decision processes with weakly continuous kernel the fixed point approach revisited abstract this paper shows the existence of lower semicontinuous solutions of the average cost optimality equation for markov decision processes with borel spaces possible unbounded cost function and weakly continuous transition kernel this is done imposing a growth condition on the cost function a lyapunov stability condition on the transition kernel and a set of standard compactness continuity conditions the solution of the average cost optimality equation is obtained by means of the banach fixed point theorem

behavior of integrate and fire and hodgkin huxley models with correlated inputs we assess both numerically and theoretically how positively correlated poisson inputs affect the output of the integrate and fire and hodgkin huxley models for the integrate and fire model the variability of efferent spike trains is an increasing function of input correlation and of the ratio between inhibitory and excitatory inputs interestingly for the hodgkin huxley model the variability of efferent spike trains is a decreasing function of input correlation and for fixed input correlation it is almost independent of the ratio between inhibitory and excitatory inputs in terms of the signal to noise ratio of efferent spike trains the integrate and fire model works better in an environment of asynchronous inputs but the hodgkin huxley model has an advantage for more synchronous correlated inputs in conclusion the integrate and fire and hodgkin huxley models respond to correlated inputs in totally opposite ways

addendum to extensions and generalizations of smoothed perturbation analysis in a generalized semi markov process framework under a very general framework both in terms of finite time performance measures and system structure the authors derive smoothed perturbation analysis spa estimators and prove their unbiasedness the commuting condition which has been key in previous work is not required a priori and thus the framework includes such systems as the gi g 1 k queue and multiclass queueing networks which do not satisfy the commuting condition the generality achieved is traded off against the fact that the estimator is not always easily implementable on a single sample path the use of the commuting condition in a local sense is proposed to help simplify the estimators derived queueing and multiclass queueing networks are used as illustrative examples for a simple multiclass closed queueing network some simulation results are provided when the commuting condition is satisfied globally the framework allows the recovery of previous results on ipa and spa estimators as corollaries of the main theorems

perturbation analysis gives strongly consistent sensitivity estimates for the m g 1 queue the technique of perturbation analysis has recently been introduced as an efficient way to compute parameter sensitivities for discrete event systems thus far the statistical properties of perturbation analysis have been validated mainly through experiments this paper considers for an m g 1 queueing system the sensitivity of mean system time of a customer to a parameter of the arrival or service distribution it shows analytically that i the steady state value of the perturbation analysis estimate of this sensitivity is unbiased and ii a perturbation analysis algorithm implemented on a single sample path of the system gives asymptotically unbiased and strongly consistent estimates of this sensitivity no previous knowledge of perturbation analysis is assumed so the paper also serves to introduce this technique to the unfamiliar reader numerical extensions to gi g 1 queues and applications to optimization problems are also illustrated

modularity of residual galois extensions and the eisenstein ideal for a totally real field f a finite extension f of fp and a galois character gf f unramified away from a finite set of places p p consider the bloch kato selmer group h h1 f 1 in an earlier paper of the authors it was proved that the number d of isomorphism classes of non semisimple reducible residual representations giving rise to lines in h which are modular by some f also unramified outside satisfies d n dimfh this was proved under the assumption that the order of a congruence module is greater than or equal to that of a divisible selmer group we show here that if in addition the relevant local eisenstein ideal j is non principal then d n when f q we prove the desired bounds on the congruence module and the selmer group we also formulate a congruence condition implying the non principality of j that can be checked in practice allowing us to furnish an example where d n

structural infinitesimal perturbation analysis sipa for derivative estimation of discrete event dynamic systems in this paper we present an algorithm for derivative estimation of discrete event dynamic systems ded s this algorithm is applicable to derivative estimation with respect to most structural parameters in ded s it extends the applicability of existing perturbation analysis techniques including infinitesimal perturbation analysis on structural parameters furthermore it offers an alternative approach to derivative estimation in ded s which sheds additional light on discrete event simulation finally unifying relationships with existing techniques are discussed

perturbation realization potentials and sensitivity analysis of markov processes two fundamental concepts and quantities realization factors and performance potentials are introduced for markov processes the relations among these two quantities and the group inverse of the infinitesimal generator are studied it is shown that the sensitivity of the steady state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a markov process based on these results algorithms for estimating performance sensitivities on a single sample path of a markov process can be proposed the potentials in this paper are defined through realization factors and are shown to be the same as those defined by poisson equations the results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis ipa and non ipa approaches to the sensitivity analysis of steady state performance they also provide a theoretical background for the pa algorithms developed in recent years

extensions of infinitesimal perturbation analysis an extension of the concept of perturbation analysis pa is introduced that can be applied to cases where sample path discontinuity with respect to system parameters has prevented the application of simple pa rules both markov and generalized semi markov processes are considered the robustness of the extended pa is examined and it is found to be sufficiently robust

convergence of parameter sensitivity estimates in a stochastic experiment to reduce the error in estimating the gradient parameter sensitivity of an unknown function is of great importance in stochastic optimization problems three kinds of parameter sensitivity estimates using the monte carlo method are discussed in this paper the estimates depend on the number of replications n and the change in parameter delta d the convergence properties as n rightarrow infty and delta d rightarrow 0 for these estimates are obtained the result explains many theoretical and practical issues in the study of discrete event dynamic systems as well as continuous dynamic systems by the monte carlo method it is proved that an estimate based on averaging the gradients calculated along each sample path by a perturbation of the path is much better than the other estimates if the output functions are uniformly differentiable with probability one w p 1 it is also concluded that in computer simulations one should always choose the same seed for both d and d delta d in estimating the parameter sensitivity combining the results in this paper with existing stochastic approximation algorithms may yield algorithms with faster convergence

zero sum average semi markov games fixed point solutions of the shapley equation this paper deals with zero sum average semi markov games with borel state and action spaces unbounded payoffs and mean holding times a solution to the shapley equation is obtained via the banach fixed point theorem assuming that the model satisfies a lyapunov like condition a growth hypothesis on the payoff function and the mean holding times besides standard continuity and compactness requirements

strong 1 optimal stationary policies in denumerable markov decision processes abstract usual conditions for existence of stationary average optimal policies in denumerable mdps with general bounded rewards are shown to be also sufficient for strong 1 optimality moreover we prove that all limit points of discounted optimal stationary policies when the discount factor goes to 1 are strong 1 optimal

dynamics of membrane excitability determine interspike interval variability a link between spike generation mechanisms and cortical spike train statistics we propose a biophysical mechanism for the high interspike interval variability observed in cortical spike trains the key lies in the nonlinear dynamics of cortical spike generation which are consistent with type i membranes where saddle node dynamics underlie excitability rinzel ermentrout 1989 we present a canonical model for type i membranes the neuron the neuron is a phase model whose dynamics reflect salient features of type i membranes this model generates spike trains with coefficient of variation cv above 0 6 when brought to firing by noisy inputs this happens because the timing of spikes for a type i excitable cell is exquisitely sensitive to the amplitude of the suprathreshold stimulus pulses a noisy input current giving random amplitude kicks to the cell evokes highly irregular firing across a wide range of firing rates an intrinsically oscillating cell gives regular spike trains we corroborate the results with simulations of the morris lecar m l neural model with random synaptic inputs type i m l yields high cvs when this model is modified to have type ii dynamics periodicity arises via a hopf bifurcation however it gives regular spike trains cv below 0 3 our results suggest that the high cv values such as those observed in cortical spike trains are an intrinsic characteristic of type i membranes driven to firing by random inputs in contrast neural oscillators or neurons exhibiting type ii excitability should produce regular spike trains

sample path average optimality for markov control processes the authors consider a markov control process with borel state and actions spaces unbounded costs and under the long run sample path average cost criterion they prove that under very weak assumptions on the transition law and a moment assumption for the one step cost there exists a stationary policy with invariant probability distribution v that is sample path average cost optimal for v almost all initial states in addition every expected average cost optimal stationary policy is in fact liminf sample path average cost optimal and strongly expected average cost optimal

statistical properties of spike trains universal and stimulus dependent aspects statistical properties of spike trains measured from a sensory neuron in vivo are studied experimentally and theoretically experiments are performed on an identified neuron in the visual system of the blowfly it is shown that the spike trains exhibit universal behavior over a short time modulated by a stimulus dependent envelope over a long time a model of the neuron as a nonlinear oscillator driven by noise and by an external stimulus is suggested to account for these results in the short time universal regime the main biophysical effect is refractoriness which can be described as a repulsive 1 x interaction law among spikes a universal distribution function for intervals is found defining a point process with special symmetry properties the long time modulations in the spike train are related in a simple way to the properties of the input stimulus as seen through the neuronal nonlinearity thus our model enables a separation of the effects of internal neuronal properties from the effect of external stimulus properties explicit formulas are derived for different statistical properties which are in very good agreement with the data in both the universal and the stimulus dependent regimes

a sensitivity formula for risk sensitive cost and the actor critic algorithm abstract we propose for risk sensitive control of finite markov chains a counterpart of the popular actor critic algorithm for classical markov decision processes the algorithm is based on a sensitivity formula for the risk sensitive cost and is shown to converge with probability one to the desired solution the proof technique is an adaptation of the ordinary differential equations approach for the analysis of two time scale stochastic approximation algorithms

perturbation analysis via coupling perturbation analysis is an efficient method for performance analysis of discrete event dynamic systems it yields gradient information from only one sample path observation over the last two decades various perturbation analysis techniques have been developed to handle a large class of problems coupling is a method of generating multiple random samples it has wide range of applications in applied probability the paper is concerned with perturbation analysis via coupling this approach offers a great versatility of the form of gradient estimators which is potentially useful for variance reduction and for efficient implementation several known perturbation analysis techniques can be reviewed as special ways of coupling the coupling method is further applied to gradient estimation of markov chains the method is used not only in deriving a gradient estimator but also in its implementation it is proved that the estimator is strongly consistent finally different coupling schemes are compared using an illustrative example

semi markov decision problems and performance sensitivity analysis recent research indicates that markov decision processes mdps can be viewed from a sensitivity point of view and the perturbation analysis pa mdps and reinforcement learning rl are three closely related areas in optimization of discrete event dynamic systems that can be modeled as markov processes the goal of this paper is two fold first we develop the pa theory for semi markov processes smps and then we extend the aforementioned results about the relation among pa mdp and rl to smps in particular we show that performance sensitivity formulas and policy iteration algorithms of semi markov decision processes can be derived based on the performance potential and realization matrix both the long run average and discounted cost problems are considered this approach provides a unified framework for both problems and the long run average problem corresponds to the discounted factor being zero the results indicate that performance sensitivities and optimization depend only on first order statistics single sample path based implementations are discussed

brief paper infinitesimal and finite perturbation analysis for queueing networks the sample path perturbation analysis technique is extended to include finite and possibly large perturbations typically introduced by changes in queue sizes or other parameters it is shown that there is a natural hierarchy of perturbation analysis which takes care of increasingly large perturbations experiments with zero infinitesimal and first order finite perturbation analysis show that significant accuracy improvement can be obtained with small increase in computational effort