a new stochastic derivative estimator for discontinuous payoff functions with application to financial derivatives motivated by infinitesimal perturbation analysis ipa and the likelihood ratio lr method we derive a new unbiased stochastic derivative estimator for a class of discontinuous payoff functions that arise in many options pricing settings from finance our method includes ipa and the lr method as special cases and can be applied to functions of more general forms containing indicator functions this new estimator can be computed from a single sample path or simulation whereas existing estimators generally require additional simulations for the class of discontinuous payoff functions considered here we apply this method to sensitivity analysis for european call options and american style call options and numerical experiments indicate that the estimator is computationally more efficient than other estimators

on the interchange of derivative and expectation for likelihood ratio derivative estimators sufficient conditions for the validity of interchange between derivative and expectation in the context of likelihood ratio gradient estimation were given in l ecuyer 1990 the aim of this paper is to shed additional light on these conditions and introduce specific variants of them which are often easier to check sufficient conditions for the derivative estimator to have finite moments up to a given order are also given and illustrated by examples in particular we give an example of an unbiased derivative estimator which satisfies the interchange conditions but which has infinite variance

single run gradient estimation via measure valued differentiation we show how single run based measure valued differentiation gradient estimators can be obtained the key idea is to apply a change of measure a posterior to the mathematical analysis of the derivative from the point of view of the likelihood ratio method we show that likelihood ratio type gradient estimators can be applied in situations where the mathematical conditions needed for applying a likelihood ratio analysis are not met

discontinuous perturbation analysis of discrete event dynamic systems the discontinuous perturbation analysis dpa method is presented to deal with discontinuous sample performance functions this method is applicable to derivative estimation with respect to most threshold type problems in discrete event dynamic systems deds through modeling of discontinuities by step function we are able to provide a unified framework for constructing the derivative estimation of a deds as a result we offer an alternate approach to derive derivative estimation in deds s finally some relationships with other pa techniques are discussed and a numerical example is presented

a unified view of the ipa sf and lr gradient estimation techniques we study the links between the likelihood ratio lr gradient estimation technique sometimes called the score function sf method and infinitesimal perturbation analysis ipa we show how ipa can be viewed as a degenerate special case of the lr and sf techniques by selecting an appropriate representation of the underlying sample space for a given simulation experiment we also show how different definitions of the sample space yield different variants of the lr method some of them mixing ipa with more straightforward lr we illustrate this by many examples we also give sufficient conditions under which the gradient estimators are unbiased

a measure valued differentiation approach to sensitivities of quantiles quantiles play an important role in modelling quality of service in the service industry and in modelling risk in the financial industry the recent discovery that efficient simulation based estimators can be obtained for quantile sensitivities has led to an intensive search for sample path differentiation based estimators for quantile sensitivities in this paper we present a novel approach to quantile sensitivity estimation our approach elaborates on the concept of measure valued differentiation thereby we overcome the main obstacle of the sample path approach which is the requirement that the sample cost have to be lipschitz continuous with respect to the parameter of interest specifically we perform a sensitivity analysis of the value at risk in financial models in addition we discuss an application of our sensitivity estimator to queueing networks

on sampling controlled stochastic approximation the authors examine a novel class of stochastic approximation procedures which are based on carefully controlling the number of observations or measurements taken before each computational iteration this method referred to as sampling controlled stochastic approximation has advantages over standard stochastic approximation such as requiring less computation and the ability to handle bias in estimation the authors address the growth rate required of the number of samples and prove a general convergence theorem for the proposed stochastic approximation method in addition they present applications to optimize and also derive a sampling controlled version of the classic robbins munro algorithm

simterpolation a simulation based interpolation approximation for queueing systems in this paper we approximate moments of the sojourn time distribution in open networks of priority queues via an interpolation approximation the interpolation is constructed from five random variables and their covariance matrix that are simultaneously estimated from a single regenerative simulation of the system at any arrival rate the random variables are consistent estimates of the zeroth and first order light traffic limits the heavy traffic limit and the value of the function and its derivative at the arrival rate of the simulation the data yield a least squares estimate of a normalized version of the original function which in turn yields an estimate of the function of interest this procedure is useful for complex systems because the light and heavy traffic limits needed for an interpolation are not easy to calculate analytically in those cases

asymptotics of likelihood ratio derivative estimators in simulations of highly reliable markovian systems we discuss the estimation of derivatives of a performance measure using the likelihood ratio method in simulations of highly reliable markovian systems we compare the difficulties of estimating the performance measure and of estimating its partial derivatives with respect to component failure rates as the component failure rates tend to 0 and the component repair rates remain fixed we first consider the case when the quantities are estimated using naive simulation i e when no variance reduction technique is used in particular we prove that in the limit some of the partial derivatives can be estimated as accurately as the performance measure itself this result is of particular interest in light of the somewhat pessimistic empirical results others have obtained when applying the likelihood ratio method to other types of systems however the result only holds for certain partial derivatives of the performance measure when using naive simulation more specifically we can estimate a certain partial

stochastic optimization by simulation convergence proofs for the gi g 1 queue in steady state approaches like finite differences with common random numbers infinitesimal perturbation analysis and the likelihood ratio method have drawn a great deal of attention recently as ways of estimating the gradient of a performance measure with respect to continuous parameters in a dynamic stochastic system in this paper we study the use of such estimators in stochastic approximation algorithms to perform so called single run optimizations of steady state systems under mild conditions for an objective function that involves the mean system time in a gi g 1 queue we prove that many variant of these algorithms converge to the minimizer in most cases however the simulation length must be increased from iteration to iteration otherwise the algorithm may converge to the wrong value one exception is a particular implementation of infinitesimal perturbation analysis for which the single run optimization converges to the optimum even with a fixed and small number of ends of service per iteration as a by product of our convergence proofs we obtain some properties of the derivative estimators that could be of independent interest our analysis exploits the regenerative structure of the system but our derivative estimation and optimization algorithms do not always take advantage of that regenerative structure in a companion paper we report numerical experiments with an m m 1 queue which illustrate the basis convergence properties and possible pitfalls of the various techniques

sensitivity estimation for gaussian systems in this paper we address the construction of efficient algorithms for the estimation of gradients of general performance measures of gaussian systems exploiting a clever coupling between the normal and the maxwell distribution we present a new gradient estimator and we show that it outperforms both the single run based infinitesimal perturbation analysis ipa estimator and the score function sf estimator in the one dimensional case for a dense class of test functions next we present an example of the multi dimensional case with a system from the area of stochastic activity networks our numerical experiments show that this new estimator also has significantly smaller sample variance than ipa and sf to increase efficiency in addition to variance reduction we present an optimized method for generating the maxwell distribution which minimizes the cpu time

residence time and collision statistics for exponential flights the rod problem revisited many random transport phenomena such as radiation propagation chemical biological species migration or electron motion can be described in terms of particles performing exponential flights for such processes we sketch a general approach based on the feynman kac formalism that is amenable to explicit expressions for the moments of the number of collisions and the residence time that the walker spends in a given volume as a function of the particle equilibrium distribution we then illustrate the proposed method in the case of the so called rod problem a one dimensional system and discuss the relevance of the obtained results in the context of monte carlo estimators

sensitivity analysis from sample paths using likelihoods we modify the likelihood based method for obtaining derivatives with respect to the rate of a poisson process to that it is not necessary to know the exact value of that rate this type of modification is necessary if the method is to be used on a sample path from a real system the method is also applicable to simulation studies of certain real time control policies and may be useful in trace driven simulations the modification to the likelihood estimator is simply to use the value of the poisson rate estimated during the sample interval for regenerative systems this produces a strongly consistent asymptotically normal and asymptotically unbiased estimate of the derivative the strong law and central limit theorem are generalized to the case of estimating a derivative with respect to an unknown parameter from the exponential class of probability density functions numerical results for the m m 1 queue illustrate little difference between the estimates for the derivative of the expected delay with respect to arrival rate obtained when the arrival rate is known and unknown however both estimates are highly biased for small sample sizes this bias can be reduced by jackknifing

bayesian policy gradient and actor critic algorithms policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate many conventional policy gradient methods use monte carlo techniques to estimate this gradient the policy is improved by adjusting the parameters in the direction of the gradient estimate since monte carlo methods tend to have high variance a large number of samples is required to attain accurate estimates resulting in slow convergence in this paper we first propose a bayesian framework for policy gradient based on modeling the policy gradient as a gaussian process this reduces the number of samples needed to obtain accurate gradient estimates moreover estimates of the natural gradient as well as a measure of the uncertainty in the gradient estimates namely the gradient covariance are provided at little extra cost since the proposed bayesian framework considers system trajectories as its basic observable unit it does not require the dynamics within trajectories to be of any particular form and thus can be easily extended to partially observable problems on the downside it cannot take advantage of the markov property when the system is markovian to address this issue we proceed to supplement our bayesian policy gradient framework with a new actor critic learning model in which a bayesian class of non parametric critics based on gaussian process temporal difference learning is used such critics model the action value function as a gaussian process allowing bayes rule to be used in computing the posterior distribution over action value functions conditioned on the observed data appropriate choices of the policy parameterization and of the prior covariance kernel between action values allow us to obtain closed form expressions for the posterior distribution of the gradient of the expected return with respect to the policy parameters we perform detailed experimental comparisons of the proposed bayesian policy gradient and actor critic algorithms with classic monte carlo based policy gradient methods as well as with each other on a number of reinforcement learning problems

residence times and other functionals of reflected brownian motion we study the residence and local times for a brownian particle confined by reflecting boundaries and propose a general solution to the problem of finding the related probability distribution its fourier transform characteristic function and laplace transform survival probability are obtained in a compact matrix form involving the laplace operator eigenbasis explicit combinatorial relations are derived for the moments and the probability distribution is shown to be nearly gaussian when the exploration time is long enough when the eigenbasis or a part of it is known the numerical computation of the residence time distributions is straightforward and accurate the present approach can also be applied to investigate other functionals of reflected brownian motion describing in particular restricted diffusion in an external field or potential e g nuclei diffusing in an inhomogeneous magnetic field theoretical results for the local times are confronted with monte carlo simulations on the unit interval disk and sphere

a basic formula for online policy gradient algorithms this note presents a new basic formula for sample path based estimates for performance gradients for markov systems called policy gradients in reinforcement learning literature with this basic formula many policy gradient algorithms including those that have previously appeared in the literature can be easily developed the formula follows naturally from a sensitivity equation in perturbation analysis new research direction is discussed

likelihood ratio derivative estimation for finite time performance measures in generalized semi markov processes this paper investigates the likelihood ratio method for estimating derivatives of finite time performance measures in generalized semi markov processes gsmps we develop readily verifiable conditions for the applicability of this method our conditions mainly place restrictions on the basic building blocks i e the transition probabilities the distribution and density functions of the event lifetimes and the initial distribution of the gsmp which is in contrast to the structural conditions needed for infinitesimal perturbation analysis we explicitly show that our conditions hold in many practical settings and in particular for large classes of queueing and reliability models one intermediate result we obtain in this study which is of independent value is to formally show that the random variable representing the number of occurring events in a gsmp in a finite time horizon has finite exponential moments in a neighborhood of zero

ring theoretical properties of affine cellular algebras abstract as a generalisation of graham and lehrer s cellular algebras affine cellular algebras have been introduced in 12 in order to treat affine hecke algebras of type a and affine versions of diagram algebras like affine temperley lieb algebras in a unifying fashion affine cellular algebras include kleshchev s graded quasihereditary algebras khovanov lauda rouquier algebras and various other classes of algebras in this paper we will study ring theoretical properties of affine cellular algebras we show that any affine cellular algebra a satisfies a polynomial identity furthermore we show that a can be embedded into its asymptotic algebra if the occurring commutative affine k algebras b j are reduced and the determinants of the swich matrices are non zero divisors as a consequence we show that the gelfand kirillov dimension of a is less than or equal to the largest krull dimension of the algebras b j and that equality holds in case all affine cell ideals are idempotent or if the krull dimension of the algebras b j is less than or equal to 1 special emphasis is given to the question when an affine cell ideal is idempotent generated by an idempotent or finitely generated

first and second derivative estimators for cyclic closed queueing networks we consider a cyclic closed queueing network with arbitrary service time distributions and derive first and second derivative estimators of some finite horizon performance metrics with respect to a parameter of any one of the service distributions our approach is based on observing a single sample path of this system and evaluating first and second order effects on departure times as a result of the parameter perturbation we then define an estimator as a conditional expectation over appropriate observable quantities using smoothed perturbation analysis techniques this process recovers the first derivative estimator along the way and gives new insights into event order change phenomena which are of higher order despite the complexity of the analysis the final algorithms we obtain are relatively simple further we show that our estimators are unbiased and include some numerical examples we also show the use of our estimators in obtaining approximations of the entire system response surface as a function of system parameters

some applications of persistent random walks and the telegrapher s equation abstract a persistent random walk can be regarded as a multidimensional markov process the bias free telegraphers equation is 2 p t 2 1 t p t v 2 2 p it can be regarded as interpolating between the wave equation t and the diffusion equation t 0 previously it has found application in thermodynamics cf the review in rev mod phys 61 1989 41 62 1990 375 more recent applications are reviewed in the present article

collision statistics for random flights with anisotropic scattering and absorption for a broad class of random walks with anisotropic scattering kernels and absorption we derive explicit formulas that allow expressing the moments of the collision number n v performed in a volume v as a function of the particle equilibrium distribution our results apply to arbitrary domains v and boundary conditions and allow assessing the hitting statistics for systems where the typical displacements are comparable to the domain size so that the diffusion limit is possibly not attained an example is discussed for one dimensional random flights with exponential displacements where analytical calculations can be carried out

affine cellularity of affine yokonuma hecke algebras abstract we establish an explicit algebra isomorphism between the affine yokonuma hecke algebra y r n q and a direct sum of matrix algebras with coefficients in tensor products of affine hecke algebras of type a as an application of this result we show that y r n q is affine cellular in the sense of koenig and xi and further prove that it has finite global dimension when the parameter q is not a root of the poincare polynomial as another application we also recover the modular representation theory of y r n q previously obtained in 7

derivative estimates from simulation of continuous time markov chains countable state continuous time markov chains are often analyzed through simulation when simple analytical expressions are unavailable simulation is typically used to estimate costs or performance measures associated with the chain and also characteristics like state probabilities and mean passage times here we consider the problem of estimating derivatives of these types of quantities with respect to a parameter of the process in particular we consider the case where some or all transition rates depend on a parameter we derive derivative estimates of the infinitesimal perturbation analysis type for markov chains satisfying a simple condition and argue that the condition has significant scope the unbiasedness of these estimates may be surprising a naive estimator would fail in our setting what makes our estimates work is a special construction of specially structured parameteric families of markov chains in addition to proving unbiasedness we consider a variance reduction technique and make comparisions with derivative estimates based on likelihood ratios

variance reduction techniques for gradient estimates in reinforcement learning policy gradient methods for reinforcement learning avoid some of the undesirable properties of the value function approaches such as policy degradation baxter and bartlett 2001 however the variance of the performance gradient estimates obtained from the simulation is sometimes excessive in this paper we consider variance reduction methods that were developed for monte carlo estimates of integrals we study two commonly used policy gradient techniques the baseline and actor critic methods from this perspective both can be interpreted as additive control variate variance reduction methods we consider the expected average reward performance measure and we focus on the gpomdp algorithm for estimating performance gradients in partially observable markov decision processes controlled by stochastic reactive policies we give bounds for the estimation error of the gradient estimates for both baseline and actor critic algorithms in terms of the sample size and mixing properties of the controlled system for the baseline technique we compute the optimal baseline and show that the popular approach of using the average reward to define the baseline can be suboptimal for actor critic algorithms we show that using the true value function as the critic can be suboptimal we also discuss algorithms for estimating the optimal baseline and approximate value function

affine cellularity of bln algebras abstract we show that the bln algebra which was introduced by mcgerty is affine cellular in the sense of koenig and xi in fact we establish a stronger property namely that the affine cell ideals are generated by idempotents this particularly implies that the global dimension of the bln algebra is finite for affine type a we obtain that the affine q schur algebra u d n n when d n is affine cellular and has finite global dimension

convergence rates of finite difference sensitivity estimates for stochastic systems a mean square error analysis of finite difference sensitivity estimators for stochastic systems is presented and an expression for the optimal size of the increment is derived the asymptotic behavior of the optimal increments and the behavior of the corresponding optimal finite difference fd estimators are investigated for finite horizon experiments steady state estimation is also considered for regenerative systems and in this context a convergence analysis of ratio estimators is presented the use of variance reduction techniques for these fd estimates such as common random numbers in simulation experiments is not considered here in the case here direct gradient estimation techniques such as perturbation analysis and likelihood ratio methods whenever applicable are shown to converge asymptotically faster than the optimal fd estimators

permuted derivative and importance sampling estimators for regenerative simulations abstract in a previous paper we introduced a new variance reduction technique for regenerative simulations based on permuting regeneration cycles in this paper we apply this idea to new classes of estimators in particular we derive permuted versions of likelihood ratio derivative estimators for steady state performance measures importance sampling estimators of the mean cumulative reward until hitting a set of states and tin estimators for steady state ratio formulas empirical results are presented showing that modest to significant variance reductions can be obtained

light traffic derivatives via likelihood ratios the steady state behavior of open queuing systems with poisson arrival processes in light traffic that is as the arrival rate tends to zero is considered expressions are provided for the derivatives with respect to the arrival rate of various quantities of interest such as moments of steady state sojourn times and queue lengths evaluated at an arrival rate of zero these expressions are obtained using the regenerative structure of the queuing system and a change of measure formula based on likelihood ratios the derivatives which can be used in interpolation approximations can be evaluated analytically in simple cases and by simulation in general

studies of bacterial branching growth using reaction diffusion models for colonial development abstract various bacterial strains exhibit colonial branching patterns during growth on poor substrates these patterns reflect bacterial cooperative self organization and cybernetic processes of communication regulation and control employed during colonial development one method of modeling is the continuous or coupled reaction diffusion approach in which continuous time evolution equations describe the bacterial density and the concentration of the relevant chemical fields in the context of branching growth this idea has been pursued by a number of groups we present an additional model which includes a lubrication fluid excreted by the bacteria we also add fields of chemotactic agents to the other models we then present a critique of this whole enterprise with focus on the models potential for revealing new biological features

affine cellular algebras abstract graham and lehrer have defined cellular algebras and developed a theory that allows in particular to classify simple representations of finite dimensional cellular algebras many classes of finite dimensional algebras including various hecke algebras and diagram algebras have been shown to be cellular and the theory due to graham and lehrer successfully has been applied to these algebras we will extend the framework of cellular algebras to algebras that need not be finite dimensional over a field affine hecke algebras of type a and infinite dimensional diagram algebras like the affine temperley lieb algebras are shown to be examples of our definition the isomorphism classes of simple representations of affine cellular algebras are shown to be parameterised by the complement of finitely many subvarieties in a finite disjoint union of affine varieties in this way representation theory of non commutative algebras is linked with commutative algebra moreover conditions on the cell chain are identified that force the algebra to have finite global cohomological dimension and its derived category to admit a stratification these conditions are shown to be satisfied for the affine hecke algebra of type a if the quantum parameter is not a root of the poincare polynomial

residence time statistics for normal and fractional diffusion in a force field we investigate statistics of occupation times for an over damped brownian particle in an external force field using a backward fokker planck equation introduced by majumdar and comtet for an arbitrary potential field the distribution of occupation times is expressed in terms of solutions of the corresponding first passage time problem this general relationship between occupation times and first passage times is valid for normal markovian diffusion and for non markovian sub diffusion the latter modeled using the fractional fokker planck equation for binding potential fields we find in the long time limit ergodic behavior for normal diffusion while for the fractional framework weak ergodicity breaking is found in agreement with previous results of bel and barkai on the continuous time random walk on a lattice for non binding cases rich physical behaviors are obtained and classification of occupation time statistics is made possible according to whether or not the underlying random walk is recurrent and the averaged first return time to the origin is finite our work establishes a link between fractional calculus and ergodicity breaking

counting statistics a feynman kac perspective by building upon a feynman kac formalism we assess the distribution of the number of collisions in a given region for a broad class of discrete time random walks in absorbing and nonabsorbing media we derive the evolution equation for the generating function of the number of collisions and we complete our analysis by examining the moments of the distribution and their relation to the walker equilibrium density some significant applications are discussed in detail in particular we revisit the gambler s ruin problem and generalize to random walks with absorption the arcsine law for the number of collisions on the half line

weak differentiability of product measures in this paper we study cost functions over a finite collection of random variables for these types of models a calculus of differentiation is developed that allows us to obtain a closed form expression for derivatives where differentiation has to be understood in the weak sense the technique for proving the results is new and establishes an interesting link between functional analysis and gradient estimation the key contribution of this paper is a product rule of weak differentiation in addition a product rule of weak analyticity is presented that allows for taylor series approximations of finite products measures in particular from characteristics of the individual probability measures a lower bound i e domain of convergence can be established for the set of parameter values for which the taylor series converges to the true value applications of our theory to the ruin problem from insurance mathematics and to stochastic activity networks arising in project evaluation review techniques are provided

likelihood ratio sensitivity analysis for markovian models of highly dependable systems this paper discusses the application of the likelihood ratio gradient estimator to simulations of large markovian models of highly dependable systems extensive empirical work as well as some mathematical analysis of small dependability models suggests that in this model setting the gradient estimators are not significantly more noisy than the estimates of the performance measures themselves the paper also discusses implementation issues associated with likelihood ratio gradient estimation as well as some theoretical complements associated with application of the technique to continuous time markov chains

structural conditions for perturbation analysis derivative estimation finite time performance indices in recent years there has been a surge of research into methods for estimating derivatives of performance measures from sample paths of stochastic systems in the case of queueing systems typical performance measures are mean queue lengths throughputs etc and the derivatives estimated are with respect to system parameters such as parameters of service and interarrival time distributions derivative estimates potentially offer a general means of optimizing performance and are useful in sensitivity analysis this paper concerns one approach to derivative estimation known as infinitesimal perturbation analysis we first develop a general framework for these types of estimates then give simple sufficient conditions for them to be unbiased the key to our results is identifying conditions under which certain finite horizon performance measures are almost surely continuous functions of the parameter of differentiation throughout an interval the sufficient conditions we introduce are formulated in the setting of generalized semi markov processes but translate into readily verifiable conditions for queueing systems these results substantially extend the domain of problems in which infinitesimal perturbation analysis is provably applicable

smoothed conditional perturbation analysis of discrete event dynamical systems a new sample path analysis approach based on the smoothing property of conditional expectation for estimating the performance sensitivity of discrete event dynamical systems is proposed several examples are presented to show how this approach overcomes a difficulty of the ordinary infinitesimal perturbation analysis the basic message is that one can get more knowledge about the system performance by observing and analyzing the sample path than by using the conventional simulation approach it is also pointed out that the classical queueing theory approach for getting the performance sensitivity and the sample path based infinitesimal perturbation analysis approach can be unified in the framework of the new approach the smoothed conditional perturbation analysis

note pathwise convexity and its relation to convergence of time average derivatives in this note we further develop the pathwise convexity approach introduced by hu 1991 to prove consistency of infinitesimal perturbation analysis for the derivative of the steady state waiting time of the g g 1 queue in addition to generalizing the argument we illustrate the technique with applications to stochastic storage theory and networks of queues

average sensitive and blackwell optimal policies in denumerable markov decisionchains with unbounded rewards in this paper we consider a discrete time markov decision chain with a denumerable state space and compact action sets and we assume that for all states the rewards and transition probabilities depend continuously on the actions the first objective of this paper is to develop an analysis for average optimality without assuming a special markov chain structure in doing so we present a set of conditions guaranteeing average optimality which are automatically fulfilled in the finite state and action model the second objective is to study simultaneously average and discount optimality as veinott veinott a f jr 1969 on discrete dynamic programming with sensitive discount optimality criteria ann math statist 40 1635 1660 did for the finite state and action model we investigate the concepts of n discount and blackwell optimality in the denumerable state space using a laurent series expansion for the discounted rewards under the same condition as for average optimality we establish solutions to the n discount optimality equations for every n

fractional feynman kac equation for weak ergodicity breaking the continuous time random walk ctrw is a model of anomalous subdiffusion in which particles are immobilized for random times between successive jumps a power law distribution of the waiting times 1 leads to subdiffusion x 2 t for 0 1 in closed systems the long stagnation periods cause time averages to divert from the corresponding ensemble averages which is a manifestation of weak ergodicity breaking the time average of a general observable u t 1 t 0 t u x d is a functional of the path and is described by the well known feynman kac equation if the motion is brownian here we derive forward and backward fractional feynman kac equations for functionals of ctrw in a binding potential we use our equations to study two specific time averages the fraction of time spent by a particle in half box and the time average of the particle s position in a harmonic field in both cases we obtain the probability density function of the time averages for t and the first two moments our results show that both the occupation fraction and the time averaged position are random variables even for long times except for 1 when they are identical to their ensemble averages using our fractional feynman kac equation we also study the dynamics leading to weak ergodicity breaking namely the convergence of the fluctuations to their asymptotic values

convergence properties of infinitesimal perturbation analysis infinitesimal perturbation analysis ipa is a method for computing a sample path derivative with respect to an input parameter in a discrete event simulation the ipa algorithm is based on the fact that for certain parameters and any realization of a simulation the change in parameter can be made small enough so that only the times of events get shifted but their order does not change this paper considers the convergence properties of the ipa sample path derivatives in particular the question of when an ipa estimate converges to the derivative of a steady state performance measure is studied necessary and sufficient conditions for this convergence are derived for a class of regenerative processes although these conditions are not guaranteed to be satisfied in general they are satisfied for the mean stationary response time in the m g 1 queue a necessary condition for multiple ipa estimates to simultaneously converge to the derivatives of steady state throughputs in a queueing network is determined the implications of this necessary condition are that except in special cases the original ipa algorithm cannot be used to consistently estimate steady state throughput derivatives in queueing networks with multiple types of customers state dependent routing or blocking numerical studies on ipa convergence properties are also presented

the diffusion limit of transport equations derived from velocity jump processes in this paper we study the diffusion approximation to a transport equation that describes the motion of individuals whose velocity changes are governed by a poisson process we show that under an appropriate scaling of space and time the asymptotic behavior of solutions of such equations can be approximated by the solution of a diffusion equation obtained via a regular perturbation expansion in general the resulting diffusion tensor is anisotropic and we give necessary and sufficient conditions under which it is isotropic we also give a method to construct approximations of arbitrary high order for large times in a second paper part ii we use this approach to systematically derive the limiting equation under a variety of external biases imposed on the motion depending on the strength of the bias it may lead to an anisotropic diffusion equation to a drift term in the flux or to both our analysis generalizes and simplifies previous derivations that lead to the classical patlak keller segel alt

affine cellularity of affine hecke algebras of rank two we show that affine hecke algebras of rank two with generic parameters are affine cellular in the sense of koenig xi

first passage times for random walks in bounded domains we present a novel computational method of first passage times between a starting site and a target site of regular bounded lattices we derive accurate expressions for all the moments of this first passage time validated by numerical simulations their range of validity is discussed we also consider the case of a starting site and two targets in addition we present the extension to continuous brownian motion these results are of great relevance to any system involving diffusion in confined media

collision densities and mean residence times for d dimensional exponential flights in this paper we analyze some aspects of exponential flights a stochastic process that governs the evolution of many random transport phenomena such as neutron propagation chemical or biological species migration and electron motion we introduce a general framework for d dimensional setups and emphasize that exponential flights represent a deceivingly simple system where in most cases closed form formulas can hardly be obtained we derive a number of exact where possible or asymptotic results among which are the stationary probability density for two dimensional systems a long standing issue in physics and the mean residence time in a given volume bounded or unbounded domains as well as scattering or absorbing domains are examined and monte carlo simulations are performed so as to support our findings

estimating quantile sensitivities quantiles of a random performance serve as important alternatives to the usual expected value they are used in the financial industry as measures of risk and in the service industry as measures of service quality to manage the quantile of a performance we need to know how changes in the input parameters affect the output quantiles which are called quantile sensitivities in this paper we show that the quantile sensitivities can be written in the form of conditional expectations based on the conditional expectation form we first propose an infinitesimal perturbation analysis ipa estimator the ipa estimator is asymptotically unbiased but it is not consistent we then obtain a consistent estimator by dividing data into batches and averaging the ipa estimates of all batches the estimator satisfies a central limit theorem for the i i d data and the rate of convergence is strictly slower than n 1 3 the numerical results show that the estimator works well for practical problems

stochastic approximation methods for systems over an infinitehorizon the paper develops efficient and general stochastic approximation sa methods for improving the operation of parametrized systems of either the continuous or discrete event dynamical systems types and which are of interest over a long time period for example one might wish to optimize or improve the stationary or average cost per unit time performance by adjusting the systems parameters the number of applications and the associated literature are increasing at a rapid rate this is partly due to the increasing activity in computing pathwise derivatives and adapting them to the average cost problem although the original motivation and the examples come from an interest in the infinite horizon problem the techniques and results are of general applicability in sa we present an updating and review of powerful ordinary differential equation type methods in a fairly general context and based on weak convergence ideas the results and proof techniques are applicable to a wide variety of applications exploiting the full potential of these ideas can greatly simplify and extend much current work their breadth as well as the relative ease of using the basic ideas are illustrated in detail via typical examples drawn from discrete event dynamical systems piecewise deterministic dynamical systems and a stochastic differential equations model in these particular illustrations we use either infinitesimal perturbation analysis type estimators mean square derivative type estimators or finite difference type estimators markov and non markov models are discussed the algorithms for distributed asynchronous updating as well as the fully synchronous schemes are developed

an interpolation approximation for queueing systems with poisson input for many queueing systems of practical interest it is difficult to obtain exact analytical results it is however often possible to obtain asymptotic results for light and heavy traffic heavy traffic limit theorems yield expressions for normalized quantities of interest in light traffic we can obtain in addition to limits more sensitive information by calculating what are effectively derivatives of the quantity of interest with respect to the arrival rate we can then combine the light and heavy traffic results to yield a polynomial in the arrival rate as an approximation to the normalized quantity of interest for instance by utilizing the heavy traffic limit the light traffic limit and the first derivative we can obtain a quadratic approximation then by reversing the normalization process we can obtain an approximation for the original quantity of interest in this paper we present the details of the above approximation focusing via several examples on applications of the method we then compare the results of the approximation for the examples considered with exact and simulation results in addition we compare our results to some related approximations for the examples considered the approximation works extremely well

branching exponential flights travelled lengths and collision statistics the evolution of several physical and biological systems ranging from neutron transport in multiplying media to epidemics or population dynamics can be described in terms of branching exponential flights a stochastic process which couples a galton watson birth death mechanism with random spatial displacements within this context one is often called to assess the length lv that the process travels in a given region v of the phase space or the number of visits nv to this same region in this paper we address this issue by resorting to the feynman kac formalism which allows characterizing the full distribution of lv and nv and in particular deriving explicit moment formulas some other significant physical observables associated to lv and nv such as the survival probability are discussed as well and results are illustrated by revisiting the classical example of the rod model in nuclear reactor physics

paper a new approach to the analysis of discrete event dynamic systems we present a new time domain approach to the study of discrete event dynamical systems deds typified by queueing networks and production systems a general state space representation is developed and perturbation analysis is carried out observation of a single sample realization of such a system can be used to predict behavior over other sample realizations when some parameter is perturbed without having to make additional observations conditions under which this is always possible are investigated and explicit results for some special cases are included

measure valued differentiation for stationary markov chains we study general state space markov chains that depend on a parameter say sufficient conditions are established for the stationary performance of such a markov chain to be differentiable with respect to specifically we study the case of unbounded performance functions and thereby extend the result on weak differentiability of stationary distributions of markov chains to unbounded mappings first a closed form formula for the derivative of the stationary performance of a general state space markov chain is given using an operator theoretic approach in a second step we translate the derivative formula into unbiased gradient estimators specifically we establish phantom type estimators and score function estimators we illustrate our results with examples from queueing theory

perturbation analysis gives strongly consistent sensitivity estimates for the m g 1 queue the technique of perturbation analysis has recently been introduced as an efficient way to compute parameter sensitivities for discrete event systems thus far the statistical properties of perturbation analysis have been validated mainly through experiments this paper considers for an m g 1 queueing system the sensitivity of mean system time of a customer to a parameter of the arrival or service distribution it shows analytically that i the steady state value of the perturbation analysis estimate of this sensitivity is unbiased and ii a perturbation analysis algorithm implemented on a single sample path of the system gives asymptotically unbiased and strongly consistent estimates of this sensitivity no previous knowledge of perturbation analysis is assumed so the paper also serves to introduce this technique to the unfamiliar reader numerical extensions to gi g 1 queues and applications to optimization problems are also illustrated

structural infinitesimal perturbation analysis sipa for derivative estimation of discrete event dynamic systems in this paper we present an algorithm for derivative estimation of discrete event dynamic systems ded s this algorithm is applicable to derivative estimation with respect to most structural parameters in ded s it extends the applicability of existing perturbation analysis techniques including infinitesimal perturbation analysis on structural parameters furthermore it offers an alternative approach to derivative estimation in ded s which sheds additional light on discrete event simulation finally unifying relationships with existing techniques are discussed

observable augmented systems for sensitivity analysis of markov and semi markov processes previous work by the authors 1989 on sensitivity analysis of discrete event systems dess where an augmented chain approach for estimating performance sensitivities from a single nominal sample path was developed is extended in contrast to existing methods this approach can be applied to cases involving discrete e g integer valued parameters such as queue capacities or routing thresholds however the basic method is limited by some rather restrictive observability conditions these conditions are relaxed through an observability transformation which generalizes the method s applicability direct extensions to more general non markovian des are also presented via a simplified generalized semi markov process reformulation of the approach experimental verification of the techniques is included

performance gradient estimation for the very large finite markov chains using an embedded markov chain the steady state performance gradient estimation for very large markov chains is decomposed into two smaller problems one solved by the stochastic recursive sr method and the other by the likelihood ratio lr method the combination named sr lr can have several magnitudes of lower space requirement than the sr and several magnitudes of lower time consumptions than the lr both the lr and the sr are the extreme cases of the combined algorithm the authors demonstrate the sr lr methods on markov chains with hundreds of millions of states which are created using queuing networks

strong points of weak convergence a study using rpa gradient estimation for automatic learning this paper analyzes the behavior of adaptive control schemes for automatic learning estimates of the sensitivities are used in a gradient based stochastic approximation procedure in order to drive the process along the steepest descent trajectory in search for the optimum the learning rates are kept constant for adaptability for such procedures convergence can be established in a weak sense a model problem of a flexible machine is presented for which the control parameter is a probability vector we propose a new sensitivity estimator generalizing the phantom rare perturbation analysis rpa estimator to multi valued decisions from the basic properties of the estimators we build several updating rules based on the weak convergence theory to ensure asymptotic optimality we illustrate the predicted theoretical behavior with computer simulations finally we present the comparison between the behavior of our proposed scheme with a regenerative one for which we can establish strong convergence our results show that weak convergence yields a dramatic improvement in the rate of convergence in addition to the capability of adaptation or tracking

on distributions of functionals of anomalous diffusion paths functionals of brownian motion have diverse applications in physics mathematics and other fields the probability density function pdf of brownian functionals satisfies the feynman kac formula which is a schrodinger equation in imaginary time in recent years there is a growing interest in particular functionals of non brownian motion or anomalous diffusion but no equation existed for their pdf here we derive a fractional generalization of the feynman kac equation for functionals of anomalous paths based on sub diffusive continuous time random walk we also derive a backward equation and a generalization to levy flights solutions are presented for a wide number of applications including the occupation time in half space and in an interval the first passage time the maximal displacement and the hitting probability we briefly discuss other fractional schrodinger equations that recently appeared in the literature

cauchy s formulas for random walks in bounded domains cauchy s formula was originally established for random straight paths crossing a body b rn and basically relates the average chord length through b to the ratio between the volume and the surface of the body itself the original statement was later extended in the context of transport theory so as to cover the stochastic paths of pearson random walks with exponentially distributed flight lengths traversing a bounded domain some heuristic arguments suggest that cauchy s formula may also hold true for pearson random walks with arbitrarily distributed flight lengths for such a broad class of stochastic processes we rigorously derive a generalized cauchy s formula for the average length travelled by the walkers in the body and show that this quantity depends indeed only on the ratio between the volume and the surface provided that some constraints are imposed on the entrance step of the walker in b similar results are also obtained for the average number of collisions performed by the walker in b

potential based online policy iteration algorithms for markov decision processes performance potentials play a crucial role in performance sensitivity analysis and policy iteration of markov decision processes the potentials can be estimated on a single sample path of a markov process in this paper we propose two potential based online policy iteration algorithms for performance optimization of markov systems the algorithms are based on online estimation of potentials and stochastic approximation we prove that with these two algorithms the optimal policy can be attained after a finite number of iterations a simulation example is given to illustrate the main ideas and the convergence rates of the algorithms

sensitivity analysis for simulations via likelihood ratios we present a simple method of estimating the sensitivity of quantities obtained from simulation with respect to a class of parameters here sensitivity means the derivative of an expectation with respect to a parameter the class of parameters includes for example poisson rates discrete probabilities and the mean and variance of a normal distribution the method is extremely well suited to regenerative simulation and can be implemented on extant simulations with little effort increase in running time or memory requirements it is based on some change of measure formulas derived from likelihood ratios in addition to the theorems that underly the technique we present some numerical examples