look ahead control of conveyor serviced production station by using potential based online policy iteration we consider the look ahead control of a conveyor serviced production station csps in the context of semi markov decision process smdp model and our goal is to find an optimal control policy under either average or discounted cost criteria policy iteration pi combined with the concept of performance potential can be applied to provide a unified optimisation framework for both criteria however a major difficulty arises in the exact solution scheme that is it requires not only the full knowledge of model parameters but also a considerable amount of work to obtain and process the necessary system and performance matrices to overcome this difficulty we propose a potential based online pi algorithm in this article during implementation by analysing and utilising the historic information of all the past operation of a practical csps system the potentials and state action values are learned on line through an effective exploration scheme we finally illustrate the successful application of th

mean variance optimization of discrete time discounted markov decision processes abstract in this paper we study a mean variance optimization problem in an infinite horizon discrete time discounted markov decision process mdp the objective is to minimize the variance of system rewards with the constraint of mean performance different from most of works in the literature which require the mean performance already achieve optimum we can let the discounted performance equal any constant the difficulty of this problem is caused by the quadratic form of the variance function which makes the variance minimization problem not a standard mdp by proving the decomposable structure of the feasible policy space we transform this constrained variance minimization problem to an equivalent unconstrained mdp under a new discounted criterion and a new reward function the difference of the variances of markov chains under any two feasible policies is quantified by a difference formula based on the variance difference formula a policy iteration algorithm is developed to find the optimal policy we also prove the optimality of deterministic policy over the randomized policy generated in the mean constrained policy space numerical experiments demonstrate the effectiveness of our approach

a unified view of the ipa sf and lr gradient estimation techniques we study the links between the likelihood ratio lr gradient estimation technique sometimes called the score function sf method and infinitesimal perturbation analysis ipa we show how ipa can be viewed as a degenerate special case of the lr and sf techniques by selecting an appropriate representation of the underlying sample space for a given simulation experiment we also show how different definitions of the sample space yield different variants of the lr method some of them mixing ipa with more straightforward lr we illustrate this by many examples we also give sufficient conditions under which the gradient estimators are unbiased

large time behavior of an attraction repulsion chemotaxis system abstract in this study we consider an attraction repulsion chemotaxis system u t u u v u w x t 0 v t v u v x t 0 w t w u w x t 0 with homogeneous neumann boundary conditions where and are positive parameters under the critical condition that 0 we prove that the system possesses a unique global solution which is uniformly bounded in the physical domain r n n 2 3 in addition we assert that we can find 0 0 such that for all of the initial data u 0 that satisfy u 0 0 the solution of the system approaches the steady state u 0 u 0 u 0 exponentially as time tends to infinity where u 0 1 u 0

optimization of markov decision processes under the variance criterion in this paper we study a variance minimization problem in an infinite stage discrete time markov decision process mdp regardless of the mean performance for the markov chain under the variance criterion since the value of the cost function at the current stage will be affected by future actions this problem is not a standard mdp and the traditional mdp theory is not applicable in this paper we convert the variance minimization problem into a standard mdp by introducing a concept called pseudo variance then we derive a variance difference formula that quantifies the difference of variances of markov systems under any two policies with the difference formula the correlation of the variance cost function at different stages can be decoupled through a nonnegative term a necessary condition of the optimal policy is obtained it is also proved that the optimal policy with the minimal variance can be found in the deterministic policy space furthermore we propose an efficient iterative algorithm to reduce the variance of markov systems we prove that this algorithm can converge to a local optimum finally a numerical experiment is conducted to demonstrate the efficiency of our algorithm compared with the gradient based method widely adopted in the literature

a time aggregation approach to markov decision processes we propose a time aggregation approach for the solution of infinite horizon average cost markov decision processes via policy iteration in this approach policy update is only carried out when the process visits a subset of the state space as in state aggregation this approach leads to a reduced state space which may lead to a substantial reduction in computational and storage requirements especially for problems with certain structural properties however in contrast to state aggregation which generally results in an approximate model due to the loss of markov property time aggregation suffers no loss of accuracy because the markov property is preserved single sample path based estimation algorithms are developed that allow the time aggregation approach to be implemented on line for practical systems some numerical and simulation examples are presented to illustrate the ideas and potential computational savings

boundedness in a quasilinear fully parabolic keller segel system of higher dimension with logistic source abstract this paper deals with the higher dimension quasilinear parabolic parabolic keller segel system involving a source term of logistic type u t u u u v g u v t v v u in 0 t subject to nonnegative initial data and homogeneous neumann boundary condition where is a smooth and bounded domain in r n n 2 and g are smooth and positive functions satisfying k s p when s s 0 1 g s a s s 2 for s 0 with g 0 0 and constants a 0 0 it is known that the model without the logistic source admits both bounded and unbounded solutions identified via the critical exponent 2 n on the other hand the model is just a critical case with the balance of logistic damping and aggregation effects for which the property of solutions should be determined by the coefficients associated in the present paper it is proved that there is 0 0 such that the problem admits global bounded classical solutions whenever 0 regardless of the size of initial data and diffusion this shows the substantial effect of the logistic source has on the behavior of solutions

aggregation vs global diffusive behavior in the higher dimensional keller segel model abstract we consider the classical parabolic parabolic keller segel system u t u u v x t 0 v t v v u x t 0 under homogeneous neumann boundary conditions in a smooth bounded domain r n it is proved that in space dimension n 3 for each q n 2 and p n one can find e 0 0 such that if the initial data u 0 v 0 satisfy u 0 l q e and v 0 l p e then the solution is global in time and bounded and asymptotically behaves like the solution of a discoupled system of linear parabolic equations in particular u v approaches the steady state m m as t where m is the total mass m u 0 of the population moreover we shall show that if is a ball then for arbitrary prescribed m 0 there exist unbounded solutions emanating from initial data u 0 v 0 having total mass u 0 m

technical communique a unified approach to markov decision problems and performance sensitivity analysis we propose a simple approach that provides a unified formulation for the performance sensitivity analysis of markov chains and markov decision problems with both infinite horizon average cost and discounted performance criteria

a hierarchy of near optimal policies for multistage adaptive optimization in this paper we propose a new tractable framework for dealing with linear dynamical systems affected by uncertainty applicable to multistage robust optimization and stochastic programming we introduce a hierarchy of near optimal polynomial disturbance feedback control policies and show how these can be computed by solving a single semidefinite programming problem the approach yields a hierarchy parameterized by a single variable the degree of the polynomial policies which controls the trade off between the optimality gap and the computational requirements we evaluate our framework in the context of three classical applications two in inventory management and one in robust regulation of an active suspension system in which very strong numerical performance is exhibited at relatively modest computational expense

finding optimal memoryless policies of pomdps under the expected average reward criterion in this paper partially observable markov decision processes pomdps with discrete state and action space under the average reward criterion are considered from a recent developed sensitivity point of view by analyzing the average reward performance difference formula we propose a policy iteration algorithm with step sizes to obtain an optimal or local optimal memoryless policy this algorithm improves the policy along the same direction as the policy iteration does and suitable step sizes guarantee the convergence of the algorithm moreover the algorithm can be used in markov decision processes mdps with correlated actions two numerical examples are provided to illustrate the applicability of the algorithm

boundedness and decay behavior in a higher dimensional quasilinear chemotaxis system with nonlinear logistic source this paper is concerned with a class of quasilinear chemotaxis systems generalizing the prototype 0 1 ut um uv uur x t 0 vt vv u x t 0 u v 0 x t 0 u x 0 u0 x v x 0 v0 x x in a smooth bounded domain rn n2 with parameters m r1 and 0 the pde system in 0 1 is used in mathematical biology to model the mechanism of chemotaxis that is the movement of cells in response to the presence of a chemical signal substance which is in homogeneously distributed in space it is shown that if m 22nif11 n 22r n 2ifn 22rn 2n 1ifr n 22 and the nonnegative initial data u0 v0 c w1 0 then 0 1 possesses at least one global bounded weak solution apart from this it is proved that if 0 then both u t and v t decay to zero with respect to the norm in l as t

general results on the convergence of stochastic algorithms a deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form under necessary conditions on the input noise and reasonable conditions on the nonnecessarily continuous mean field emphasis is placed on the case where more than one stationary point exists we also use this approach to prove the convergence of a stochastic algorithm with markovian dynamics

explanation of goal softening in ordinal optimization the authors explain the role of goal softening in the convergence of alignment probability employed in ordinal optimization using the order statistics formulation they examine the exponential decrease of misalignment probability bounds when two or more designs are compared their conclusion states that by relaxing the good enough subset and selected subset criteria it is exponentially efficient in terms of matching good designs in a selected group

importance sampling for stochastic simulations importance sampling is one of the classical variance reduction techniques for increasing the efficiency of monte carlo algorithms for estimating integrals the basic idea is to replace the original random mechanism in the simulation by a new one and at the same time modify the function being integrated in this paper the idea is extended to problems arising in the simulation of stochastic systems discrete time markov chains continuous time markov chains and generalized semi markov processes are covered applications are given to a gi g 1 queueing problem and response surface estimation computation of the theoretical moments arising in importance sampling is discussed and some numerical examples given

a quasilinear fully parabolic chemotaxis system with indirect signal production and logistic source abstract in this paper we study the quasilinear fully parabolic chemotaxis system with indirect signal production and logistic source u t d u u s u v f u v t v a 1 v b 1 w w t w a 2 w b 2 u under homogeneous neumann boundary conditions in a bounded and smooth domain r n n 1 where a i b i 0 i 1 2 d s c 2 0 and f r r is a smooth function generalizing the logistic source f s b s r for all s 0 with b 0 0 and r 1 we obtain the global boundedness of solutions in four cases i the self diffusion dominates the cross diffusion ii the logistic source suppresses the cross diffusion iii the logistic dampening balances the cross diffusion with 0 suitably large iv the self diffusion and the logistic source both balance the cross diffusion to some extent with 0 arbitrary as corollaries we also consider the global boundedness of solutions for the quasilinear attraction repulsion chemotaxis model with logistic source u t d u u u z u w f u z t z z u w t w w u where 0

event based optimization of markov systems recent research indicates that markov decision processes mdps and perturbation analysis pa based optimization can be derived easily from two fundamental performance sensitivity formulas with this sensitivity point of view an event based optimization approach including event based sensitivity analysis and event based policy iteration was proposed via an example by x r cao discrete event dyn syst theory appl vol 15 pp 169 197 2005 this approach utilizes the special feature of a system and illustrates how the potentials can be aggregated using the special feature the approach applies to many practical problems that do not fit well the standard mdp formulation this note provides a mathematical formulation and proves the main results for this approach

decentralized delay optimal control for interference networks with limited renewable energy storage in this paper we consider delay minimization for interference networks with renewable energy source where the transmission power of a node comes from both the conventional utility power ac power and the renewable energy source we assume the transmission power of each node is a function of the local channel state local data queue state and local energy queue state only we consider two delay optimization formulations namely the decentralized partially observable markov decision process dec pomdp and noncooperative partially observable stochastic game posg in dec pomdp formulation we derive a decentralized online learning algorithm to determine the control actions and lagrangian multipliers lms simultaneously based on the policy gradient approach under some mild technical conditions the proposed decentralized policy gradient algorithm converges almost surely to a local optimal solution in the noncooperative posg formulation the transmitter nodes are noncooperative we extend the decentralized policy gradient solution and establish the technical proof for almost sure convergence of the learning algorithms in both cases the solutions are very robust to model variations

optimization of average rewards of time nonhomogeneous markov chains we study the optimization of average rewards of discrete time nonhomogeneous markov chains in which the state spaces transition probabilities and reward functions depend on time the analysis encounters a few major difficulties 1 notions crucial to homogeneous markov chains such as ergodicity stationarity periodicity and connectivity no longer apply 2 the average reward criterion is under selective i e it does not depend on the decisions in any finite period and thus dynamic programming is not amenable and 3 because of the under selectivity an optimal average reward policy may not be the best in any finite period

discrete time controlled markov processes with average cost criterion a survey this work is a survey of the average cost control problem for discrete time markov processes the authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades the exposition ranges from finite to borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies the authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography the authors have also identified several important questions that are still open to investigation

sample path optimization of convex stochastic performance functions in this paper we propose a method for optimizing convex performance functions in stochastic systems these functions can include expected performance in static systems and steady state performance in discrete event dynamic systems they may be nonsmooth the method is closely related to retrospective simulation optimization it appears to overcome some limitations of stochastic approximation which is often applied to such problems we explain the method and give computational results for two classes of problems tandem production lines with up to 50 machines and stochastic pert program evaluation and review technique problems with up to 70 nodes and 110 arcs

completion of squares revisited and extended due to its simplicity the completion of squares technique is quite popular in linear optimal control however this simple technique is limited to linear quadratic gaussian systems in this note by interpreting the completion of squares from a new angle we extend this technique to performance optimization of general markov systems with the long run average criterion leading to a new approach to performance optimization based on direct comparisons of the performance of two policies

simulation based optimization of markov reward processes this paper proposes a simulation based algorithm for optimizing the average reward in a finite state markov reward process that depends on a set of parameters as a special case the method applies to markov decision processes where optimization takes place within a parametrized set of policies the algorithm relies on the regenerative structure of finite state markov processes involves the simulation of a single sample path and can be implemented online a convergence result with probability 1 is provided

performance optimization algorithms based on potentials for semi markov control processes optimization algorithms are studied for a class of semi markov control processes smcps with compact action set both the long run average and discounted cost problems are considered formulas of performance potentials and optimality equations for smcps are derived a policy iteration algorithm and a value iteration algorithm are proposed which can lead to an optimal or suboptimal stationary policy in a finite number of iterations the convergence of these algorithms is established without the assumption of the corresponding iteration operator being a span contraction in addition an online policy iteration optimization algorithm based on a single sample path is provided to escape curse of dimensionality in the end a numerical example is provided to illustrate the application of the algorithms

application of an adams type inequality to a two chemical substances chemotaxis system abstract this paper deals with positive solutions of the fully parabolic system u t u u v in 0 1 v t v v w in 0 2 w t w w u in 0 under homogeneous neumann boundary conditions or mixed boundary conditions no flux and dirichlet conditions in a smooth bounded domain r n n 4 with positive parameters 1 2 0 and nonnegative smooth initial data u 0 v 0 w 0 in the lower dimensional case n 3 it is proved that for all reasonable initial data solutions of the system exist globally in time and remain bounded in the case n 4 it is shown that in the radially symmetric setting solutions to the neumann boundary value problem of the system exist globally in time and remain bounded if u 0 l 1 8 2 as to the mixed boundary value problem we will establish global existence and boundedness of solutions if u 0 l 1 8 2 without radial symmetry the key ingredients are a lyapunov functional and an adams type inequality a lyapunov functional of the above problems will be constructed and the constant 8 2 is deduced from the critical constant in the adams type inequality this result is regarded as a generalization of the well known 8 problem in the keller segel system to higher dimensions

boundedness in a quasilinear parabolic parabolic keller segel system with logistic source abstract we consider a quasilinear parabolic parabolic keller segel system involving a source term of logistic type 0 1 u t u u u v g u x t 0 t v t v v u x t 0 t with nonnegative initial data under neumann boundary condition in a smooth bounded domain r n n 1 here and are supposed to be smooth positive functions satisfying c 1 s p and c 1 s q s c 2 s q when s s 0 with some s 0 1 and we assume that g is smooth on 0 fulfilling g 0 0 and g s a s s 2 for all s 0 with constants a 0 and 0 within this framework it is proved that whenever q 1 for any sufficiently smooth initial data there exists a unique classical solution which is global in time and bounded our result is independent of p

state classification of time nonhomogeneous markov chains and average reward optimization of multi chains in a discrete time nonhomogeneous markov chain tnhmc the states spaces transition probabilities and reward functions at different times may be different in this paper with the confluencity previously introduced we show that the states of a tnhmc can be classified into the branching states and a number of classes of confluent states versus the transient and recurrent states in the time homogeneous case the optimization of average reward in tnhmc s consisting of a single confluent class uni chain have been addressed in a previous paper by the author in this paper we show that with confluencity and the state classification and under some bound conditions we can obtain the necessary and sufficient conditions for optimal policies of the average reward of tnhmcs consisting of multiple confluent classes multi chains just like in the uni chain tnhmc case the sufficient condition does not need to hold in any zero frequently visited time sequence this under selectivity makes the problem not amenable to dynamic programming a direct comparison based approach is used to prove the results the results enhance our understanding of state classification and performance optimization with the notion of confluencity

potential based online policy iteration algorithms for markov decision processes performance potentials play a crucial role in performance sensitivity analysis and policy iteration of markov decision processes the potentials can be estimated on a single sample path of a markov process in this paper we propose two potential based online policy iteration algorithms for performance optimization of markov systems the algorithms are based on online estimation of potentials and stochastic approximation we prove that with these two algorithms the optimal policy can be attained after a finite number of iterations a simulation example is given to illustrate the main ideas and the convergence rates of the algorithms

the policy iteration algorithm for average reward markov decision processes with general state space the average cost optimal control problem is addressed for markov decision processes with unbounded cost it is found that the policy iteration algorithm generates a sequence of policies which are c regular where c is the cost function under consideration this result only requires the existence of an initial c regular policy and an irreducibility condition on the state space furthermore under these conditions the sequence of relative value functions generated by the algorithm is bounded from below and nearly decreasing from which it follows that the algorithm is always convergent under further conditions it is shown that the algorithm does compute a solution to the optimality equations and hence an optimal average cost policy these results provide elementary criteria for the existence of optimal policies for markov decision processes with unbounded cost and recover known results for the standard linear quadratic gaussian problem in particular in the control of multiclass queueing networks it is found that there is a close connection between optimization of the network and optimal control of a far simpler fluid network model

finding optimal observation based policies for constrained pomdps under the expected average reward criterion in this technical note constrained partially observable markov decision processes with discrete state and action spaces under the average reward criterion are studied from a sensitivity point of view by analyzing the derivatives of performance criteria we develop a simulation based optimization algorithm to find the optimal observation based policy on the basis of a single sample path this algorithm does not need any overly strict assumption and can be applied to the general ergodic markov systems with the imperfect state information the performance is proved to converge to the optimum with probability 1 one numerical example is provided to illustrate the applicability of the algorithm

optimization of a special case of continuous time markov decision processes with compact action set performance optimization is considered for average cost multichain markov decision processes mdps with compact action set since for a general compact multichain model the optimality equation system may have no solution and also a policy iteration algorithm may yield a suboptimal policy rather than an optimal one we concentrate only on a special case of multichain models in this paper where we assume that the classifications of states are fixed identically rather than varying with policies by using the concept of performance potentials the existence of solutions to the optimality equation system is established and then a potential based policy iteration algorithm is supposed to solve this system in addition the optimality convergence for recurrent classes of the algorithm has been proved finally a numerical example is provided

stochastic optimization of regenerative systems using infinitesimal perturbation analysis we consider the problem of optimizing the steady state mean of a controlled regenerative process using a stochastic optimization algorithm driven by infinitesimal perturbation analysis ipa derivative estimates we derive ipa derivative estimates for our problem and prove almost sure convergence of the algorithm the generality of our formulation should encompass a wide variety of practical systems we illustrate our framework and results via several examples

asymptotic theory for estimators under random censorship the product limit estimator hat f of an unknown distributionf is represented as au statistic plus an error of the ordero 1 n using this the maximum likelihood estimator of the specific risk rate in the time interval 0 m is shown to admit a two term edgeworth expansion this risk rate for a specific cause of death is defined as the ratio of the probability of death due to that particular cause in the time interval 0 m to the mean life time of an individual up to that time pointm similar expansions for the bootstrapped statistics are used to show that the bootstrap distribution of the studentized estimator of the risk rate approximates the sampling distribution better than the corresponding normal distribution

a unified approach to markov decision problems and performance sensitivity analysis with discounted and average criteria multichain cases we propose a unified framework to markov decision problems and performance sensitivity analysis for multichain markov processes with both discounted and average cost performance criteria with the fundamental concept of performance potentials we derive both performance gradient and performance difference formulas which play the central role in performance optimization the standard policy iteration algorithms for both discounted and average reward mdps can be established using the performance difference formulas in a simple and intuitive way and the performance gradient formulas together with stochastic approximation may lead to new optimization schemes this sensitivity based point of view of performance optimization provides some insights that link perturbation analysis markov decision processes and reinforcement learning together the research is an extension of the previous work on ergodic markov chains cao automatica 36 2000 771

approximate receding horizon approach for markov decision processes average reward case abstract we consider an approximation scheme for solving markov decision processes mdps with countable state space finite action space and bounded rewards that uses an approximate solution of a fixed finite horizon sub mdp of a given infinite horizon mdp to create a stationary policy which we call approximate receding horizon control we first analyze the performance of the approximate receding horizon control for infinite horizon average reward under an ergodicity assumption which also generalizes the result obtained by white j oper res soc 33 1982 253 259 we then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub mdp the first control policy is based on a finite horizon approximation of howard s policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies along the study we also provide a simple alternative proof on the policy improvement for countable state space we finally discuss practical implementations of these schemes via simulation

the potential structure of sample paths and performance sensitivities of markov systems we study the structure of sample paths of markov systems by using performance potentials as the fundamental units with a sample path based approach we show that performance sensitivity formulas performance gradients and performance differences of markov systems can be constructed intuitively by first principles with performance potentials or equivalently perturbation realization factors as building blocks in particular we derive sensitivity formulas for two markov chains with possibly different state spaces the proposed approach can be used to obtain flexibly the sensitivity formulas for a wide range of problems including those with partial information these formulas are the basis for performance optimization of discrete event dynamic systems including perturbation analysis markov decision processes and reinforcement learning the approach thus provides insight on on line learning and performance optimization and opens up new research directions sample path based algorithms can be developed

boundedness in a quasilinear parabolic parabolic keller segel system with subcritical sensitivity abstract we consider the quasilinear parabolic parabolic keller segel system u t d u u s u v x t 0 v t v v u x t 0 under homogeneous neumann boundary conditions in a convex smooth bounded domain r n with n 1 it is proved that if s u d u c u with 2 n and some constant c 0 for all u 1 then the classical solutions to the above system are uniformly in time bounded provided that d u satisfies some technical conditions such as algebraic upper and lower growth resp decay estimates as u this boundedness result is optimal according to a recent result by the second author winkler 2010 27 which says that if s u d u c u for u 1 with c 0 and some 2 n n 2 then for each mass m 0 there exist blow up solutions with mass u 0 m in addition this paper also proves a general boundedness result for quasilinear non uniformly parabolic equations by modifying the iterative technique of moser alikakos alikakos 1979 1

the n th order bias optimality for multichain markov decision processes in this paper we propose a new approach to the theory of finite multichain markov decision processes mdps with different performance optimization criteria we first propose the concept of nth order bias then using the average reward and bias difference formulas derived in this paper we develop an optimization theory for finite mdps that covers a complete spectrum from average optimality bias optimality to all high order bias optimality in a unified way the approach is simple direct natural and intuitive it depends neither on laurent series expansion nor on discounted mdps we also propose one phase policy iteration algorithms for bias and high order bias optimal policies which are more efficient than the two phase algorithms in the literature furthermore we derive high order bias optimality equations this research is a part of our effort in developing sensitivity based learning and optimization theory

brief paper average cost temporal difference learning we propose a variant of temporal difference learning that approximates average and differential costs of an irreducible aperiodic markov chain approximations are comprised of linear combinations of fixed basis functions whose weights are incrementally updated during a single endless trajectory of the markov chain we present a proof of convergence with probability 1 and a characterization of the limit of convergence we also provide a bound on the resulting approximation error that exhibits an interesting dependence on the mixing time of the markov chain the results parallel previous work by the authors involving approximations of discounted cost to go

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

nonparametric estimation of specific occurrence exposure rate in risk and survival analysis abstract a cohort of individuals exposed to some risk is followed up to a point of time m and observations on two random variables y are recorded for each individual the variable refers to one of the four possible events that can occur for an individual in the period 0 m i dies of a specific disease say cancer ii dies of a natural cause iii withdraws from the study and iv is alive and still under study at time m the variable y refers to the time at which an event occurs based on such data for n individuals we consider the problem of estimation of a specific occurrence exposure rate soer which is a risk ratio defined as the ratio of probability of death due to cancer in the interval 0 m to the mean lifetime of all individuals up to the time point m the asymptotic distribution of a nonparametric estimator of soer is shown to be normal and the asymptotic variance involves unknown parameters various ways of bootstrapping are discussed for construction of confidence interva

continuous time markov decision processes with nth bias optimality criteria in this paper we study the nth bias optimality problem for finite continuous time markov decision processes mdps with a multichain structure we first provide nth bias difference formulas for two policies and present some interesting characterizations of an nth bias optimal policy by using these difference formulas then we prove the existence of an nth bias optimal policy by using nth bias optimal policy iteration algorithms and show that such an nth bias optimal policy can be obtained in a finite number of policy iterations

a probabilistic analysis of bias optimality in unichain markov decision processes focuses on bias optimality in unichain finite state and action space markov decision processes using relative value functions we present methods for evaluating optimal bias this leads to a probabilistic analysis which transforms the original reward problem into a minimum average cost problem the result is an explanation of how and why bias implicitly discounts future rewards

actor critic type learning algorithms for markov decision processes algorithms for learning the optimal policy of a markov decision process mdp based on simulated transitions are formulated and analyzed these are variants of the well known actor critic or adaptive critic algorithm in the artificial intelligence literature distributed asynchronous implementations are considered the analysis involves two time scale stochastic approximations

positive effects of repulsion on boundedness in a fully parabolic attraction repulsion chemotaxis system with logistic source abstract in this paper we study the global boundedness of solutions to the fully parabolic attraction repulsion chemotaxis system with logistic source u t u u v u w f u v t v v u w t w w u subject to homogeneous neumann boundary conditions in a bounded and smooth domain r n n 1 where and are positive constants and f r r is a smooth function generalizing the logistic source f s a b s for all s 0 with a 0 b 0 and 1 it is shown that when the repulsion cancels the attraction i e the solution is globally bounded if n 3 or n min n 2 4 n n 2 6 n 17 n 2 3 n 4 4 with n 2 therefore due to the inhibition of repulsion to the attraction in any spatial dimension the exponent is allowed to take values less than 2 such that the solution is uniformly bounded in time

perturbation realization potentials and sensitivity analysis of markov processes two fundamental concepts and quantities realization factors and performance potentials are introduced for markov processes the relations among these two quantities and the group inverse of the infinitesimal generator are studied it is shown that the sensitivity of the steady state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a markov process based on these results algorithms for estimating performance sensitivities on a single sample path of a markov process can be proposed the potentials in this paper are defined through realization factors and are shown to be the same as those defined by poisson equations the results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis ipa and non ipa approaches to the sensitivity analysis of steady state performance they also provide a theoretical background for the pa algorithms developed in recent years

confidence interval estimation for the ratio of simple and standardized rates in cohort studies summary computer simulation has been used to compare four methods for calculating confidence intervals for simple rate ratios estimated from cohort studies the method proposed by cornfield 1956 in proceedings of the third berkeley symposium on mathematical statistics and probability vol iv 135148 for interval estimation for the odds ratio in a 2 x 2 table adapted to the ratio of two poisson variables gives reasonable and conservative results and is recommended in view of its relative computational simplicity three methods are compared for interval estimation for the ratio of standardized rates such as the standardized morbidity or mortality ratio when the data are stratified by some confounding factor the adapted cornfield method again is the recommended procedure an examination of the use of a summary chi square test with and without continuity correction indicates that the test size closest to the nominal is given by excluding such a correction when the test statistic is computed cohort studies of the possible association between an exposure factor and the risk of a disease are conducted by following groups of individuals with different exposures for a period of time the morbidity or mortality experience of the cohort is monitored and the rates of disease occurrence among the different groups are compared a parameter of particular interest in comparing the disease experience of two groups of individuals with different exposures is the simple rate ratio rr estimated by rr x2 m2 1 rr xi ml where xi and x2 are the numbers of events cases or deaths from disease occurring in the two groups and ml and m2 are the respective denominators the denominator usually used is person years of observation in order to allow for individuals being under observation for different lengths of time when the relative rate rr in the population of which the cohort is a sample remains constant during the time period of the study rr is a consistent estimator of rr if rr is not constant then t1rr will be a consistent estimator only when both exposed and nonexposed subjects are followed during the same time interval and when the sampling fraction of both groups remains constant i e when there is no loss to follow up in order to allow for possible confounding factors such as age with distributions that may differ between the two groups a parameter of more general interest is the standardized rate ratio srr estimated by

boundedness in quasilinear keller segel systems of parabolic parabolic type on non convex bounded domains abstract this paper deals with the quasilinear fully parabolic keller segel system u t d u u s u v x t 0 v t v v u x t 0 under homogeneous neumann boundary conditions in a bounded domain r n with smooth boundary n n the diffusivity d u is assumed to satisfy some further technical conditions such as algebraic growth and d 0 0 which says that the diffusion is allowed to be not only non degenerate but also degenerate the global in time existence and uniform in time boundedness of solutions are established under the subcritical condition that s u d u k u e for u 0 with 2 n k 0 and e 0 when d 0 0 this paper represents an improvement of tao and winkler 17 because the domain does not necessarily need to be convex in this paper in the case r n and d 0 0 uniform in time boundedness is an open problem left in a previous paper 7 this paper also gives an answer to it in bounded domains

centralized optimization for dec pomdps under the expected average reward criterion in this paper the decentralized partially observable markov decision process dec pomdp systems with discrete state and action spaces are studied from a gradient point of view dec pomdps have recently emerged as a promising approach to optimizing multiagent decision making in the partially observable stochastic environment however the decentralized nature of the dec pomdp framework results in a lack of shared belief state which makes the decision maker impossible to estimate the system state based on local information in contrast to the belief based policy this paper focuses on optimizing the decentralized observation based policy which is easily to be applied and does not have the sharing problem by analyzing the gradient of the objective function we have developed a centralized stochastic gradient policy iteration algorithm to find the optimal policy on the basis of gradient estimates from a single sample path this algorithm does not need any specific assumption and can be applied to most practical dec pomdp problems one numerical example is provided to demonstrate the effectiveness of the algorithm

boundedness of the attraction repulsion keller segel system a b s t r a c t this paper considers the initial boundary value problem of the attraction repulsion keller segel model describing aggregation of microglia in the central nervous system in alzheimer s disease due to the interaction of chemoattractant and chemorepellent if repulsion dominates over attraction we show the global existence of classical solution in two dimensions and weak solution in three dimensions with large initial data u x 0 u0 x v x 0 v 0 x w x 0 w 0 x x

boundedness of solutions to a quasilinear parabolic parabolic keller segel system with a logistic source abstract we study global solutions of a class of chemotaxis systems generalizing the prototype u t u u u v a u b u r x t 0 v t v v u x t 0 in a bounded domain r n n 1 with smooth boundary u u 1 u u u 1 1 and the parameters r 1 a 0 b 0 and r it is proved that if 0 max r 1 2 n or b is big enough if r 1 then the classical solutions to the above system are uniformly in time bounded our results improve the results of wang et al 2014 28 and cao 2014 3 and also enlarge the range of the results of tao and winkler 2012 25 and ishida et al 2014 14

semi markov decision problems and performance sensitivity analysis recent research indicates that markov decision processes mdps can be viewed from a sensitivity point of view and the perturbation analysis pa mdps and reinforcement learning rl are three closely related areas in optimization of discrete event dynamic systems that can be modeled as markov processes the goal of this paper is two fold first we develop the pa theory for semi markov processes smps and then we extend the aforementioned results about the relation among pa mdp and rl to smps in particular we show that performance sensitivity formulas and policy iteration algorithms of semi markov decision processes can be derived based on the performance potential and realization matrix both the long run average and discounted cost problems are considered this approach provides a unified framework for both problems and the long run average problem corresponds to the discounted factor being zero the results indicate that performance sensitivities and optimization depend only on first order statistics single sample path based implementations are discussed

sensitivity analysis for simulations via likelihood ratios we present a simple method of estimating the sensitivity of quantities obtained from simulation with respect to a class of parameters here sensitivity means the derivative of an expectation with respect to a parameter the class of parameters includes for example poisson rates discrete probabilities and the mean and variance of a normal distribution the method is extremely well suited to regenerative simulation and can be implemented on extant simulations with little effort increase in running time or memory requirements it is based on some change of measure formulas derived from likelihood ratios in addition to the theorems that underly the technique we present some numerical examples