contraction conditions for average and α discount optimality in countable state markov games with unbounded rewards the goal of this paper is to provide a theory of n person markov games with unbounded cost for a countable state space and compact action spaces we investigate both the finite and infinite horizon problems for the latter we consider the discounted cost as well as the expected average cost we present conditions for the infinite horizon problems for which equilibrium policies exist for all players within the stationary policies and show that the costs in equilibrium policies exist for all players within the stationary policies and show that the costs in equilibrium satisfy the optimality equations similar results are obtained for the finite horizon costs for which equilibrium policies are shown to exist for all players within the markov policies as special case of n person games we investigate the zero sum 2 players game for which we establish the convergence of the value iteration algorithm we conclude by studying an application of a zero sum markov game in a queueing model

partially observed semi markov zero sum games with average payoff we study a zero sum partially observed semi markov game with average payoff on a countable state space under certain ergodicity conditions we show that a saddle point equilibrium exists we achieve this by solving the corresponding average cost optimality equation using a span contraction method the average value is shown to be the unique zero of a lipschitz continuous function a value iteration scheme is developed to compute the value

approximation schemes for infinite linear programs this paper presents approximation schemes for an infinite linear program in particular it is shown that under suitable assumptions the program s optimum value can be approximated by the values of finite dimensional linear programs and that in addition every accumulation point of a sequence of optimal solutions for the approximating programs is an optimal solution for the original problem

optimal strategies for adaptive zero sum average markov games abstract we consider a class of discrete time two person zero sum markov games with borel state and action spaces and possibly unbounded payoffs the game evolves according to the recursive equation x n 1 f x n a n b n n n 0 1 where the disturbance process n is formed by independent and identically distributed r k valued random vectors which are observable but their common density is unknown for both players combining suitable methods of statistical estimation of with optimization procedures we construct a pair of average optimal strategies

infinite dimensional linear programming approach to singularstochastic control we consider a multidimensional singular stochastic control problem with state dependent diffusion matrix and drift vector and control cost depending on the position and direction of displacement of the controlled process the objective is to minimize the total expected discounted cost we write an equivalent infinite dimensional linear programming problem on a subspace of the space conjugate to cal c bbb r n times cal c bbb r n times b where b is the unit sphere in bbb r n we write a dual linear program and prove absence of duality gap the dual program characterizes the optimal cost function as a maximal solution to the variational inequality with gradient constraints

dynamic response of non linear systems to poisson distributed random impulses abstract the dynamic response of non linear systems to external excitations in form of a poisson distributed train of random impulses is considered the state vector of the dynamical system under consideration is a poisson driven markov vector process the equations governing its joint statistical moments are obtained by making use of a generalized ito s differential rule the resulting infinite hierarchy of equations is truncated with the help of a cumulant neglect closure of different orders for a duffing oscillator the time histories of the mean value the variance and the skewness coefficient of the response are evaluated analytically and compared with those obtained from monte carlo simulations it is found that if the departure of the poisson driven excitation process from gaussianity ranges from small to moderately large i e the expected arrival rate of impulses is not very low the application of the fourth order cumulant neglect closure technique i e neglecting the cumulants above the fourth order yields very good estimates of the response mean values and variances

a proposal of testing method on the exact steady state probability density function of non linear stochastic system this paper is concerned with the exact analysis method of the response process of second order non linear stochastic systems excited by gaussian white noise a main feature is that a testing method is presented in this paper for the exact steady state probability density of two dimensions to demonstrate the effectiveness of the exact results examples are given to illustrate the applications of the analysis method

a convex analytic framework for ergodic control of semi markov processes the ergodic control problem for semi markov processes is reformulated as an optimization problem over the set of suitably defined ergodic occupation measures this set is shown to be closed and convex with its extreme points corresponding to stationary strategies this leads to the existence of optimal stationary strategies under additional hypotheses a pathwise analysis of the joint empirical occupation measures of the state and control processes shows that this optimality is in the strong i e almost sure sense

response of nonlinear systems in probability domain using stochastic averaging determination of probability density function pdf of the response for strongly nonlinear single degree of freedom system subjected to both multiplicative and additive random excitations using stochastic averaging technique is faced with few difficulties firstly the size of excitations should be small such that the response of the system converges weakly to a markov process secondly the excitations should preferably be broad banded so that the analytical results are reasonably accurate and finally the nonlinear functions should be integrable if closed formed expressions for the result are desired the above issues are examined in the paper with a view to show a limiting values of the size parameter of excitations for which stochastic averaging technique can be applied to obtain reasonable estimates of pdf and mean square value of response b the effect of the nature of the frequency contents of excitation on the response and c the use of the stochastic averaging method employing generalized harmonic functions for nonintegrable functions representing the dynamic system it is shown that the stochastic averaging procedure provides results which compare very well with those obtained from simulation analysis not only for wide band excitations but also for narrow band excitations for a wide range of the size parameter of excitation further the procedure can be easily implemented for highly irregular functions by employing a numerical scheme with fft

optimal risk probability for first passage models in semi markov decision processes this paper studies the risk minimization problem in semi markov decision processes with denumerable states the criterion to be optimized is the risk probability or risk function that a first passage time to some target set doesn t exceed a threshold value we first characterize such risk functions and the corresponding optimal value function and prove that the optimal value function satisfies the optimality equation by using a successive approximation technique then we present some properties of optimal policies and further give conditions for the existence of optimal policies in addition a value iteration algorithm and a policy improvement method for obtaining respectively the optimal value function and optimal policies are developed finally two examples are given to illustrate the value iteration procedure and essential characterization of the risk function

optimality of quasi open loop policies for discounted semi markov decision processes quasi open loop policies consist of sequences of markovian decision rules that are insensitive to one component of the state space given a semi markov decision process smdp we distinguish between exogenous and endogenous state components as follows i the decision maker s actions do not impact the evolution of an exogenous state component and ii between consecutive decision epochs the exogenous and endogenous state components are conditionally independent given the decision maker s latest action for simplicity we consider an smdp with one exogenous and one endogenous state component when transition times between epochs are conditionally independent of the exogenous state given the most recent action and the exogenous component is a multiplicative compound poisson process we provide an almost everywhere condition on the reward function sufficient for the optimality of a quasi open loop policy after adjusting the discount factor to account for the statistical properties of the exogenous state process obtaining this policy amounts to solving a reduced smdp in which the exogenous state is static depending on the relationship between the structure of the exogenous state process and the shape of the reward function we can replace the almost everywhere condition with one that applies only in expectation quasi open loop optimality holds even if the times between decision epochs depend on the poisson process underlying the exogenous state component and or the poisson process is replaced with a generic counting process

the average cost optimality equation for markov control processes on borel spaces abstract this paper deals with discrete time markov control processes with borel state space allowing unbounded costs and noncompact control sets for these models the existence of average optimal stationary policies has been recently established under very general assumptions using an optimality inequality here we give a condition which is a strengtened version of a variant of the vanishing discount factor approach for the optimality equation to hold

average cost optimal policies for markov control processes with borel state space and unbounded costs abstract we show the existence of average cost optimal stationary policies for markov control processes with borel state space and unbounded costs per stage under a set of assumptions recently introduced by l i sennott 1989 for control processes with countable state space and finite control sets

statistical linearization of the duffing oscillator under non gaussian external excitation the duffing oscillator under external non gaussian excitations is investigated by means of statistical linearization the input process is modelled as a polynomial of a gaussian process or as a renewal driven impulse process four criteria of statistical linearization are considered the interarrival times of the renewal process are distributed according to a pearson type iii law predictions of the stationary variance are compared with monte carlo simulations

updating schemes correlation structure blocking and parameterization for the gibbs sampler in this paper many convergence issues concerning the implementation of the gibbs sampler are investigated exact computable rates of convergence for gaussian target distributions are obtained different random and non random updating strategies and blocking combinations are compared using the rates the effect of dimensionality and correlation structure on the convergence rates are studied some examples are considered to demonstrate the results for a gaussian image analysis problem several updating strategies are described and compared for problems in bayesian linear models several possible parameterizations are analysed in terms of their convergence rates characterizing the optimal choice

response of stochastic dynamical systems driven by additive gaussian and poisson white noise solution of a forward generalized kolmogorov equation by a spectral finite difference method abstract a numerical method is given for the solution of the probability density function of the response process of memoryless one and two state dynamical systems having polynomial restoring forces and which are subjected to a combination of gaussian and poisson white noises the method employs the fourier transformed forward generalized kolmogorov equation to arrive at an initial boundary value problem for the characteristic function which is solved using a high order finite difference procedure the probability density function is recovered by numerical inverse fourier transformation several examples are given the results of which are compared with analytical solutions where available and with simulation otherwise

denumerable undiscounted semi markov decision processes with unbounded rewards this paper establishes the existence of a solution to the optimality equations in undis counted semi markov decision models with countable state space under conditions generalizing the hitherto obtained results in particular we merely require the existence of a finite set of states in which every pair of states can reach each other via some stationary policy instead of the traditional and restrictive assumption that every stationary policy has a single irreducible set of states a replacement model and an inventory model illustrate why this extension is essential our approach differs fundamentally from classical approaches we convert the optimality equations into a form suitable for the application of a fixed point theorem

finite horizon semi markov decision processes with application to maintenance systems abstract this paper investigates finite horizon semi markov decision processes with denumerable states the optimality is over the class of all randomized history dependent policies which include states and also planning horizons and the cost rate function is assumed to be bounded below under suitable conditions we show that the value function is a minimum nonnegative solution to the optimality equation and there exists an optimal policy moreover we develop an effective algorithm for computing optimal policies derive some properties of optimal policies and in addition illustrate our main results with a maintenance system

average optimality in markov control processes via discounted costproblems and linear programming this paper shows the existence of solutions to the average cost problem for markov control processes on borel spaces with possibly unbounded costs and noncompact control constraint sets this is done via a combination of the well known vanishing discount approach and recent results on the linear programming formulation for both discounted and average cost markov control problems

stochastic response of vibro impact duffing oscillators under external and parametric gaussian white noises abstract this study presents a solution procedure for the stationary probability density function pdf of the response of vibro impact duffing oscillators under external and parametric gaussian white noises first the zhuravlev non smooth coordinate transformation is adopted to convert a vibro impact oscillator into an oscillator without barriers the stationary pdf of the converted oscillator is governed by the fokker planck fp equation the fp equation is solved by the exponential polynomial closure epc method illustrative examples are presented with vibro impact duffing oscillators under external and parametric gaussian white noises to show the effectiveness of the solution procedure the parametric excitation is acting in displacement and the constraint is a unilateral zero offset barrier the restitution coefficient of impacts is taken as 0 90 comparison with the simulated results shows that the proposed solution procedure can provide good approximate pdfs for displacement and velocity although a little difference exists in the tail of these pdfs this difference may be due to the weak approximation on the response of the vibro impact oscillators using a continuous markov process when the restitution coefficient is not very close to unity

probabilistic solution of some multi degree of freedom nonlinear systems under external independent poisson white noises this paper studies the stationary probability density function pdf of the response of multi degree of freedom nonlinear systems under external independent poisson white noises the pdf is governed by the high dimensional generalized fokker planck kolmogorov fpk equation the state space split 3s method is adopted to reduce the high dimensional generalized fpk equation to a low dimensional equation subsequently the exponential polynomial closure epc method is further used to solve the reduced fpk equation for the pdf solution two illustrative examples are presented to examine the accuracy of the 3s epc solution procedure one example involves a two degree of freedom coupled nonlinear system the other example is concerned with a ten degree of freedom system with cubic terms in displacement a monte carlo simulation is also performed for simulating the pdf solution of the response the comparison with the simulated result shows that the 3s epc solution procedure can provide satisfactory pdf solut

an explicit link between gaussian fields and gaussian markov random fields the stochastic partial differential equation approach continuously indexed gaussian fields gfs are the most important ingredient in spatial statistical modelling and geostatistics the specification through the covariance function gives an intuitive interpretation of the field properties on the computational side gfs are hampered with the big n problem since the cost of factorizing dense matrices is cubic in the dimension although computational power today is at an all time high this fact seems still to be a computational bottleneck in many applications along with gfs there is the class of gaussian markov random fields gmrfs which are discretely indexed the markov property makes the precision matrix involved sparse which enables the use of numerical algorithms for sparse matrices that for fields in r 2 only use the square root of the time required by general algorithms the specification of a gmrf is through its full conditional distributions but its marginal properties are not transparent in such a parameterization we show that using an approximate stochastic weak solution to linear stochastic partial differential equations we can for some gfs in the matern class provide an explicit link for any triangulation of r d between gfs and gmrfs formulated as a basis function representation the consequence is that we can take the best from the two worlds and do the modelling by using gfs but do the computations by using gmrfs perhaps more importantly our approach generalizes to other covariance functions generated by spdes including oscillating and non stationary gfs as well as gfs on manifolds we illustrate our approach by analysing global temperature data with a non stationary model defined on a sphere less

exact stationary probability density for second order non linear systems under external white noise excitation abstract in this paper exact stationary solutions are constructed for non linear dynamical systems subjected to stochastic excitation the solution of the equations 6 7 in this paper is demonstrated to be unique and then the results of references 10 11 are shown to be generalized therefore new exact stationary solutions that satisfies the fokker plank kolmogorov fpk equation is obtainable

bayesian calibration of a large scale geothermal reservoir model by a new adaptive delayed acceptance metropolis hastings algorithm 1 the aim of this research is to estimate the parameters of a large scale numerical model of a geothermal reservoir using markov chain monte carlo mcmc sampling within the framework of bayesian inference all feasible parameters that are consistent with the measured data are summarized by the posterior distribution and hence parameter estimation and uncertainty quantification are both given by calculating expected values of statistics of interest over the posterior distribution it appears to be computationally infeasible to use the standard metropolis hastings algorithm mh to sample the high dimensional computationally expensive posterior distribution to improve the sampling efficiency a new adaptive delayed acceptance mh algorithm adamh is implemented to adaptively build a stochastic model of the error introduced by the use of a reduced order model this use of adaptivity differs from existing adaptive mcmc algorithms that tune proposal distributions of the metropolis hastings algorithm mh though adamh also implements that technique for the 3 d geothermal reservoir model we present here adamh shows a great improvement in the computational efficiency of the mcmc sampling and promising results for parameter estimation and uncertainty quantification are obtained this algorithm could offer significant improvement in computational efficiency when implementing sample based inference in other large scale inverse problems

measurable selection theorems for minimax stochastic optimization problems this paper resolves the measurability questions which arise in the analysis of various minimax stochastic optimization models posed in borel spaces for a large class of such models we provide a set of sufficient conditions which ensure that the questions have positive answers these conditions are essentially weaker than those described in the existing literature moreover they allow us to conclude a fundamental theorem yielding some new results on the existence of universally measurable selections of extrema for both zero sum stochastic games and minimax stochastic optimal control problems in particular a random version of the well known minimax theorem of fan is in this way established the fundamental theorem yields also a descriptive set theoretic fact concerning the projections of borel sets the paper indicates also some counterexamples to possibly more general problems

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

linear programming and average optimality of markov control processes on borel spaces unbounded costs this paper is concerned with the linear programming formulation of markov control processes with borel state and action spaces and the average cost ac criterion the one stage cost function may be unbounded a linear program ep and its dual ep are introduced their values inf ep and sup ep bound the value say inf ac of the ac problem i e sup ep leq inf ac leq inf ep conditions are provided for the existence of no duality gap viz sup ep inf ep and also for strong duality so that both ep and ep are solvable and their optimal values satisfy max ep min ep the latter implies i the existence of an ac optimal control policy and that ii the ac optimality equation holds almost everywhere these results are applied to a general vector valued additive noise system with quadratic costs

on the optimality equation for average cost markov control processes with feller transition probabilities we consider markov control processes with borel state space and feller transition probabilities satisfying some generalized geometric ergodicity conditions we provide a new theorem on the existence of a solution to the average cost optimality equation

average cost optimal stationary policies in infinite state markov decision processes with unbounded costs we deal with infinite state markov decision processes with unbounded costs three simple conditions based on the optimal discounted value function guarantee the existence of an expected average cost optimal stationary policy sufficient conditions are the existence of a distinguished state of smallest discounted value and a single stationary policy inducing an irreducible ergodic markov chain for which the average cost of a first passage from any state to the distinguished state is finite a result to verify this is also given two examples illustrate the ease of applying the criteria

probabilistic solution of nonlinear oscillators excited by combined gaussian and poisson white noises abstract the stationary probability density function pdf solution of the stochastic response of nonlinear oscillators is investigated in this paper the external excitation is assumed to be a combination of gaussian and poisson white noises the pdf solution is governed by the generalized kolmogorov equation which is solved by the exponential polynomial closure epc method in order to evaluate the effectiveness of the epc method different nonlinear oscillators are considered in numerical analysis nonlinearity exists either in displacement or in velocity for these nonlinear oscillators the impulse arrival rate mono modal pdf and bi modal pdf are also considered in this study compared to the pdf given by monte carlo simulation the epc method presents good agreement with the simulated result which can also be observed in the tail region of the pdf solution

zero sum stochastic games in borel spaces average payoff criteria this paper is concerned with two person zero sum dynamic stochastic games in borel spaces with possibly unbounded payoff function and several average or ergodic payoff criteria we give conditions under which the long run expected average payoff criterion the sample path average criterion the existence of solutions to the average payoff shapley equations and a certain martingale condition are all equivalent

response of dynamic systems to poisson white noise abstract a method is developed for finding response statistics and reliability for a non linear system subjected to poisson white noise the poisson white noise can be viewed as a sequence of independent indentically distributed pulses arriving in time according to a poisson counting process the method is based on the observation that the system experiences free vibrations between consecutive pulses of the input two algorithms are used for analysis the first uses the definition of the poisson white noise and the second is based on an approximate representation of the poisson white noise referred to as binomial white noise the binomial white noise is an independent time series with a fixed time step that can be zero with a finite probability the time step of the first and second algorithms are random and deterministic respectively the algorithms are used to calculate transition probabilities and estimate response statistics numerical examples are presented to demonstrate the proposed method of response and reliability analysis

another set of conditions for average optimality in markov control processes abstract the existence of an average cost optimal stationary policy in a countable state markov decision chain is shown under assumptions weaker than those given by sennott 1989 this treatment is a modification of that given by hu 1992 and is related to conditions of hordijk 1977 an example is given for which the new axiom set holds whereas the axiom set of sennott 1989 fails to hold

minimax control of discrete time stochastic systems this paper gives a unified self contained presentation of minimax control problems for discrete time stochastic systems on borel spaces with possibly unbounded costs the main results include conditions for the existence of minimax strategies for finite horizon problems and infinite horizon discounted and undiscounted average cost criteria the results are specialized to control systems with unknown disturbance distributions also known as games against nature two examples illustrate the theory one of them on the mold level control problem which is a key problem in the steelmaking industry

linear programming formulation for optimal stopping problems optimal stopping problems for continuous time markov processes are shown to be equivalent to infinite dimensional linear programs over a space of pairs of measures under very general conditions the measures involved represent the joint distribution of the stopping time and stopping location and the occupation measure of the process until it is stopped these measures satisfy an identity for each function in the domain of the generator which is sufficient to characterize the stochastic process finite dimensional linear programs obtained using markov chain approximations are solved in two examples to illustrate the numerical accuracy of the linear programming formulation

existence of optimal policies for semi markov decision processes using duality for infinite linear programming semi markov decision processes on borel spaces with deterministic kernels have many practical applications particularly in inventory theory most of the results from general semi markov decision processes do not carry over to a deterministic kernel since such a kernel does not provide smoothness we develop infinite dimensional linear programming theory for a general stochastic semi markov decision process we give conditions general enough to allow deterministic kernels for solvability and strong duality of the resulting linear programs by using the developed linear programming theory we give conditions for the existence of a stationary deterministic policy for deterministic kernels which is optimal among all possible policies

infinite linear programming and multichain markov control processes in uncountable spaces in this paper we use infinite linear programming to study markov control processes in borel spaces and the average cost criterion in the unichain and multichain cases under appropriate assumptions we show that in both cases the associated linear programs are solvable and that there is no duality gap moreover conditions are given for minimizing respectively maximizing sequences for the primal respectively dual programs to converge to optimal solutions

sampling from gaussian markov random fields using stationary and non stationary subgraph perturbations gaussian markov random fields gmrfs or gaussian graphical models have been widely used in many applications efficiently drawing samples from gmrfs has been an important research problem in this paper we introduce the subgraph perturbation sampling algorithm which makes use of any pre existing tractable inference algorithm for a subgraph by perturbing this algorithm so as to yield asymptotically exact samples for the intended distribution we study the stationary version where a single fixed subgraph is used in all iterations as well as the non stationary version where tractable subgraphs are adaptively selected the subgraphs used can have any structure for which efficient inference algorithms exist for example tree structured low tree width or having a small feedback vertex set we present new theoretical results that give convergence guarantees for both stationary and non stationary graphical splittings our experiments using both simulated models and large scale real models demonstrate that this subgraph perturbation algorithm efficiently yields accurate samples for many graph topologies

zero sum semi markov games this paper deals with borel state and action spaces zero sum semi markov games under the expected long run average payoff criterion the transition probabilities are assumed to satisfy some generalized geometric ergodicity conditions the main result states that the optimality equation has a solution which is approximated by the solutions of some varepsilon perturbed semi markov games as a corollary the existence of value and average optimal strategies for the players is established

sample path optimality and variance minimization of average cost markov control processes this paper studies several average cost criteria for markov control processes on borel spaces with possibly unbounded costs under suitable hypotheses we show i the existence of a sample path average cost spac optimal stationary policy ii a stationary policy is spac optimal if and only if it is expected average cost eac optimal and iii within the class of stationary spac optimal equivalently eac optimal policies there exists one with a minimal limiting average variance

exact stationary response of multi dimensional non linear hamiltonian dynamical systems under parametric and external stochastic excitations abstract a large class of multi dimensional non linear hamiltonian dissipative dynamical systems under parametric and external random excitations is studied a gaussian white noise modelling is used for the random excitations the conservative part is a general hamiltonian formulation for time invariant non linear dynamical systems the non conservative part has three terms a linear or non linear damping term an external random excitation which appears as the inhomogeneous term and a random parametric excitation term because the random parametric excitation causes some time variations of the dynamic characteristics the stability of the system and the existence of an asymptotic stationary response are studied the exact steady state probability density function is effectively constructed as the solution of the steady fokker planck equation for the radial form case of the hamiltonian function the characteristic function and the covariance matrix of the stationary response are calculated explicitly several examples are given and show clearly the effects of the non linearities and of the random parametric excitation for multi dimensional systems

dual based methods for solving infinite horizon nonstationary deterministic dynamic programs we develop novel dual ascent and primal dual methods to solve infinite horizon nonstationary deterministic dynamic programs these methods are finitely implementable and converge in value to optimality moreover the dual ascent method produces a sequence of improving dual solutions that pointwise converge to an optimal dual solution while the primal dual algorithm provides a sequence of primal basic feasible solutions with value error bounds from optimality that converge to zero our dual based methods work on a more general class of infinite network flow problems that include the shortest path formulation of dynamic programs as a special case to our knowledge these are the first dual based methods proposed in the literature to solve infinite horizon nonstationary deterministic dynamic programs

maximum entropy principle and nonlinear stochastic oscillators abstract the principle of maximum entropy introduced first in physics and successfully applied in many other fields e g statistics reliability estimation simulation has recently been developed to analyze the systems governed by stochastic differential equations in this paper we extend the maximum entropy method to the general class of stochastic nonlinear systems and apply it to stochastic nonlinear oscillators to indicate the effectiveness of the method a comparison of the results with predictions of the known procedures is shown graphically the analysis presented in this paper indicates that the maximum entropy principle can serve as a general method for determining the probability distributions of a wide class of stochastic systems when we have only a partial information in the form of moments or equations for moments

stochastic response of a ϕ6 oscillator subjected to combined harmonic and poisson white noise excitations abstract the transient and stationary probability density functions pdfs of stochastic response of the 6 duffing oscillator under combined harmonic and external and parametric poisson white noises excitations are investigated by the generalized cell mapping method in this paper based on the digraph analysis method the global qualitative properties are obtained such as attractors basins of attraction basin boundaries saddles and invariant manifolds the evolutionary process of transient and stationary pdfs are shown based on the matrix analysis method it is observed that there is a close relationship between evolutionary direction of pdf and the unstable manifold monte carlo mc simulation is used to verify the accuracy of the matrix analysis method

zero sum ergodic stochastic games with feller transition probabilities we consider zero sum stochastic games with borel state spaces satisfying a generalized geometric ergodicity condition the main objective of this paper is to establish a minimax theorem for a class of ergodic stochastic games with the feller transition probability function all previous results on ergodic stochastic games are based on a strong continuity of the transition probabilities in the actions of the players

average optimality in dynamic programming on borel spaces unbounded costs and controls abstract we show the existence of average cost optimal stationary policies in a class of discrete time markov control processes with borel state space unbounded costs and non compact control sets

a counterexample on the optimality equation in markov decision chains with the average cost criterion abstract we consider markov decision processes with denumerable state space and finite control sets the performance index of a control policy is a long run expected average cost criterion and the cost function is bounded below for these models the existence of average optimal stationary policies was recently established in 11 under very general assumptions such a result was obtained via an optimality inequality here we use a simple example to prove that the conditions in 11 do not imply the existence of a solution to the average cost optimality equation

an infinite dimensional linear programming algorithm for deterministic semi markov decision processes on borel spaces we devise an algorithm for solving the infinite dimensional linear programs that arise from general deterministic semi markov decision processes on borel spaces the algorithm constructs a sequence of approximate primal dual solutions that converge to an optimal one the innovative idea is to approximate the dual solution with continuous piecewise linear ridge functions that naturally represent functions defined on a high dimensional domain as linear combinations of functions defined on only a single dimension this approximation gives rise to a primal dual pair of semi infinite programs for which we show strong duality in addition we prove various properties of the underlying ridge functions

solutions of the average cost optimality equation for markov decision processes with weakly continuous kernel the fixed point approach revisited abstract this paper shows the existence of lower semicontinuous solutions of the average cost optimality equation for markov decision processes with borel spaces possible unbounded cost function and weakly continuous transition kernel this is done imposing a growth condition on the cost function a lyapunov stability condition on the transition kernel and a set of standard compactness continuity conditions the solution of the average cost optimality equation is obtained by means of the banach fixed point theorem

a shadow simplex method for infinite linear programs we present a simplex type algorithm that is an algorithm that moves from one extreme point of the infinite dimensional feasible region to another not necessarily adjacent extreme point for solving a class of linear programs with countably infinite variables and constraints each iteration of this method can be implemented in finite time whereas the solution values converge to the optimal value as the number of iterations increases this simplex type algorithm moves to an adjacent extreme point and hence reduces to a true infinite dimensional simplex method for the important special cases of nonstationary infinite horizon deterministic and stochastic dynamic programs

on markov games with average reward criterion and weakly continuous transition probabilities in this paper we consider two person zero sum stochastic games with the average reward criterion and borel state and action spaces a geometric drift condition is assumed we show that the optimality shapley equation has a unique solution if the transition probability function is weakly continuous the stage reward is lower semicontinuous and the set valued mappings of admissible actions satisfy some semicontinuity assumptions furthermore the minimizing player has an optimal stationary strategy and the maximizing player has an e optimal stationary strategy for every varepsilon 0

an improved path integration method for nonlinear systems under poisson white noise excitation abstract in order to overcome some unsatisfactory trends and limitations of the traditional path integration pi method for poisson white noise a novel pi method is proposed in this paper which includes two improved schemes the first one is a new transition probability density function tpdf approximation which considers the randomness of the impulse happening time during each time intervals the second one is a transformation of chapman kolmogorov ck equation by a variable substitution instead of directly using it whose numerical calculation is based on the back stepping runge kutta scheme and the triangulation based interpolation monte carlo simulations mcs are utilized to measure the accuracy of the improved algorithm with three illustrative nonlinear systems the results show that compared with the traditional pi method the improved pi method can give a more accurate description of the tpdf values and provide more precise stationary probability density function pdf results whenever the mean arrival rate is large or small the improved algorithm has a wider range of choices in time interval values to maintain the accuracy of stationary pdf results besides it is discovered that cubic interpolation deserves to be applied in the improved pi method more than linear and natural interpolations

reliability of non linear oscillators subject to poisson driven impulses abstract the reliability problem of a sdof non linear and non hysteretic oscillator subject to a stationary poisson driven train of impulses is considered in the first passage time problem of the oscillator time invariant double barriers with a deterministic start in the safe domain are assumed initially the kolmogorov feller backwards equation for the reliability function along with associated initial and boundary conditions is specified next a petrov galerkin approach with shape and weighting functions matching all differentiability and continuity requirements of the weak formulation is applied to solve the consequent partial differential equation as an example a duffing oscillator with linear damping is considered

convergence in variance of chebyshev accelerated gibbs samplers a stochastic version of a stationary linear iterative solver may be designed to converge in distribution to a probability distribution with a specified mean and covariance matrix a 1 a common example is gibbs sampling applied to a multivariate gaussian distribution which is a stochastic version of the gauss seidel linear solver the iteration operator that acts on the error in mean and covariance in the stochastic iteration is the same iteration operator that acts on the solution error in the linear solver and thus both the stationary sampler and the stationary solver have the same error polynomial and geometric convergence rate the polynomial acceleration techniques that are well known in numerical analysis for accelerating the linear solver may also be used to accelerate the stochastic iteration we derive first order and second order chebyshev polynomial acceleration for the stochastic iteration to accelerate convergence in the mean and covariance by mimicking the derivation for the linear solver in particular we show that the error polynomials are identical and hence so are the convergence rates thus optimality of the chebyshev accelerated solver implies optimality of the chebyshev accelerated sampler we give an algorithm for the stochastic version of the second order chebyshev accelerated ssor symmetric successive overrelaxation iteration and provide numerical examples of sampling from multivariate gaussian distributions to confirm that the desired convergence properties are achieved in finite precision

zero sum average semi markov games fixed point solutions of the shapley equation this paper deals with zero sum average semi markov games with borel state and action spaces unbounded payoffs and mean holding times a solution to the shapley equation is obtained via the banach fixed point theorem assuming that the model satisfies a lyapunov like condition a growth hypothesis on the payoff function and the mean holding times besides standard continuity and compactness requirements

duality and existence of optimal policies in generalized joint replenishment we establish a duality theory for a broad class of deterministic inventory control problems on continuous spaces that includes the classical joint replenishment problem and inventory routing using this theory we establish the existence of an optimal policy which has been an open question we show how a primal dual pair of infinite dimensional linear programs encode both cyclic and noncyclic schedules and provide various results regarding cyclic schedules including an example showing that they need not be optimal

stochastic averaging of quasi partially integrable hamiltonian systems under combined gaussian and poisson white noise excitations abstract a stochastic averaging method for predicting the response of quasi partially integrable and non resonant hamiltonian systems to combined gaussian and poisson white noise excitations is proposed for the case with r 1 r n independent first integrals which are in involution an r dimensional averaged generalized fokker planck kolmogorov gfpk equation for the transition probability density of r independent first integrals is derived from the stochastic integro differential equations sides of the original quasi partially integrable and non resonant hamiltonian systems by using the stochastic jump diffusion chain rule and the stochastic averaging theorem an example is given to illustrate the applications of the proposed stochastic averaging method and a combination of the finite difference method and the successive over relaxation method is used to solve the reduced gfpk equation to obtain the stationary probability density of the system the results are well verified by a monte carlo simulation

sample path average optimality for markov control processes the authors consider a markov control process with borel state and actions spaces unbounded costs and under the long run sample path average cost criterion they prove that under very weak assumptions on the transition law and a moment assumption for the one step cost there exists a stationary policy with invariant probability distribution v that is sample path average cost optimal for v almost all initial states in addition every expected average cost optimal stationary policy is in fact liminf sample path average cost optimal and strongly expected average cost optimal

the existence of minimum pair of state and policy for markov decision processes under the hypothesis of doeblin this paper studies the average cost markov decision process with compact state and action spaces and bounded lower semicontinuous cost functions following the idea of borkar s excellent papers siamj control optim 21 1983 pp 652 666 22 1984 pp 965 978 the general case where irreducibility is not assumed is considered under the hypothesis of doeblin and the existence of a minimum pair of state and policy which attains the infimum of the average expected cost over all initial states and policies is established further it is proved that under additional weak conditions there exists an optimal stationary policy in the usual sense

optimal stationary policies in general state space markov decision chains with finite action sets the result of sennott 9 on the existence of optimal stationary policies in countable state markov decision chains with finite action sets is generalized to arbitrary state space markov decision chains the assumption of finite action sets occurring in a global countable action space allows a particularly simple theoretical structure for the general state space markov decision chain two examples illustrate the results example 1 is a system of parallel queues with stochastic work requirements a movable server with controllable service rate and a reject option example 2 is a system of parallel queues with stochastic controllable inputs a movable server with fixed service rates and a reject option

weak conditions for average optimality in markov control processes abstract in this paper we study the existence of average cost optimal policies for markov control processes on borel spaces and with possibly unbounded costs and controls under assumptions weaker than those used in the previous literature we show that the optimality inequality holds everywhere and that there exists a stationary policy which is optimal whenever the initial state lies in a possibly proper subset of the state space this is in contrast to previous works where an implicit unichain assumption ensures optimality for all initial states

sampling gaussian distributions in krylov spaces with conjugate gradients this paper introduces a conjugate gradient sampler that is a simple extension of the method of conjugate gradients cg for solving linear systems the cg sampler iteratively generates samples from a gaussian probability density using either a symmetric positive definite covariance or precision matrix whichever is more convenient to model similar to how the lanczos method solves an eigenvalue problem the cg sampler approximates the covariance or precision matrix in a small dimensional krylov space as with any iterative method the cg sampler is efficient for high dimensional problems where forming the covariance or precision matrix is impractical but operating by the matrix is feasible in exact arithmetic the sampler generates gaussian samples with a realized covariance that converges to the covariance of interest in finite precision the sampler produces a gaussian sample with a realized covariance that is the best approximation to the desired covariance in the smaller dimensional krylov space

extreme value prediction for non linear stochastic oscillators via numerical solutions of the stationary fpk equation abstract the accuracy of two well established numerical methods is demonstrated and the importance of bandwidth examined for computationally efficient markov based extreme value predictions associated with finite duration stationary sample paths of a non linear oscillator driven by gaussian white noise by making the poisson assumption of independent upcrossings extreme exceedance probabilities are predicted via the mean threshold crossing rate using numerical solutions of the stationary fokker planck fpk equation with bandwidth initially ignored predicted exceedances using the weighted residual methods of bhandari and sherrer soize and kunert and the finite element method of langley are compared with nominally exact predictions for a heavily damped duffing type model obtained by using an explicit fpk solution the fe method being established as superior predictions via fe solutions are then compared with very long monte carlo simulations in which bandwidth effects are included two lightly damped non linear oscillator models are examined both with cubic stiffness but different damping mechanisms one model again being of simple duffing type with linear damping the other being appropriate to single co ordinate random vibration of a clamped clamped beam with wholly non linear damping the realistic damping parameter values assigned to the beam model are statistically equivalent to the linear damping level chosen for the simple model at just above 1 at this overall damping level results clearly demonstrate that for the probability levels and durations considered bandwidth is only important for the linearly damped model for the beam model with non linear damping bandwidth can be ignored allowing accurate extreme exceedance predictions by using only the stationary fpk equation the paper also demonstrates that the limiting decay rate of the first passage probability advocated by crandall roberts and others as a criterion for deciding the barrier height above which first passage times can be assumed independent proves grossly over conservative as a corresponding criterion for deciding the independence level for use in stationary extreme value prediction

approximate dynamic programming for stochastic linear control problems on compact state spaces abstract this paper addresses markov decision processes over compact state and action spaces we investigate the special case of linear dynamics and piecewise linear and convex immediate costs for the average cost criterion this model is very general and covers many interesting examples for instance in inventory management due to the curse of dimensionality the problem is intractable and optimal policies usually cannot be computed not even for instances of moderate size we show the existence of optimal policies and of convex and bounded relative value functions that solve the average cost optimality equation under reasonable and easy to check assumptions based on these insights we propose an approximate relative value iteration algorithm based on piecewise linear convex relative value function approximations besides computing good policies the algorithm also provides lower bounds to the optimal average cost which allow us to bound the optimality gap of any given policy for a given instance the algorithm is applied to the well studied multiple sourcing problem as known from inventory management multiple sourcing is known to be a hard problem and usually tackled by parametric heuristics we analyze several msp instances with two and more suppliers and compare our results to state of the art heuristics for the considered scenarios our policies are always at least as good as the best known heuristic and strictly better in most cases moreover by using the computed lower bounds we show for all instances that the optimality gap has never exceeded 5 and that it has been much smaller for most of them