stochastic approximation methods for systems over an infinitehorizon the paper develops efficient and general stochastic approximation sa methods for improving the operation of parametrized systems of either the continuous or discrete event dynamical systems types and which are of interest over a long time period for example one might wish to optimize or improve the stationary or average cost per unit time performance by adjusting the systems parameters the number of applications and the associated literature are increasing at a rapid rate this is partly due to the increasing activity in computing pathwise derivatives and adapting them to the average cost problem although the original motivation and the examples come from an interest in the infinite horizon problem the techniques and results are of general applicability in sa we present an updating and review of powerful ordinary differential equation type methods in a fairly general context and based on weak convergence ideas the results and proof techniques are applicable to a wide variety of applications exploiting the full potential of these ideas can greatly simplify and extend much current work their breadth as well as the relative ease of using the basic ideas are illustrated in detail via typical examples drawn from discrete event dynamical systems piecewise deterministic dynamical systems and a stochastic differential equations model in these particular illustrations we use either infinitesimal perturbation analysis type estimators mean square derivative type estimators or finite difference type estimators markov and non markov models are discussed the algorithms for distributed asynchronous updating as well as the fully synchronous schemes are developed

gradient estimation for sensitivity analysis and adaptive multiuser interference rejection in code division multiple access systems in this paper we consider a direct sequence code division multiple access ds cdma system in the framework of a discrete event dynamic system deds in order to optimize the system performance based on this formulation we develop infinitesimal perturbation analysis ipa for estimating the sensitivity of the average probability of bit error to factors ranging from near far effects to imperfections in power control the above estimates are shown to be unbiased and this technique is then further incorporated into a stochastic gradient algorithm for achieving adaptive multiuser interference rejection for such systems which is also subject to frequency nonselective slow fading we use an ipa based stochastic training algorithm for developing an adaptive linear detector with the average probability of error being the minimization criterion we also develop a practical implementation of such an adaptive detector where we use a joint estimation detection algorithm for minimizing the average probability of bit error a sequential implementation that does not require a stochastic training sequence or a preamble is also developed

bayesian electromagnetic spatio temporal imaging of extended sources with markov random field and temporal basis expansion estimating the locations and spatial extents of brain sources poses a long standing challenge for electroencephalography and magnetoencephalography e meg source imaging in the present work a novel source imaging method bayesian electromagnetic spatio temporal imaging of extended sources besties which is built upon a bayesian framework that determines the spatio temporal smoothness of source activities in a fully data driven fashion is proposed to address this challenge in particular a markov random field mrf which can precisely capture local cortical interactions is employed to characterize the spatial smoothness of source activities the temporal dynamics of which are modeled by a set of temporal basis functions tbfs crucially all of the unknowns in the mrf and tbf models are learned from the data to accomplish model inference efficiently on high resolution source spaces a scalable algorithm is developed to approximate the posterior distribution of the source activities which is based on the variational bayesian inference and convex analysis the performance of besties is assessed using both simulated and actual human e meg data compared with l2 norm constrained methods besties is superior in reconstructing extended sources with less spatial diffusion and less localization error by virtue of the mrf besties also overcomes the drawback of over focal estimates in sparse constrained methods

machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

asymptotic efficiency of perturbation analysis based stochastic approximation with averaging central limit theorems are obtained for the perturbation analysis robbins monro single run parmsr algorithm updated either after every regenerative cycle or after every fixed length observation period and with averaging of the iterates for one dependent regenerative processes when the convergence to the optimizer is expressed in terms of the total observation time of the system or the total computing budget in the case of a simulation the convergence rate and the limit covariance matrix turn out to be the same for all updating schemes and are optimal within the class of stochastic approximation algorithms under certain assumptions a bound on the strong convergence rate of the usual parmsr algorithm updated after every fixed length observation period is established using a limit theorem on double array martingales this is the key step for obtaining central limit theorems for the algorithms with averaging and has interest in its own right

stochastic optimization of regenerative systems using infinitesimal perturbation analysis we consider the problem of optimizing the steady state mean of a controlled regenerative process using a stochastic optimization algorithm driven by infinitesimal perturbation analysis ipa derivative estimates we derive ipa derivative estimates for our problem and prove almost sure convergence of the algorithm the generality of our formulation should encompass a wide variety of practical systems we illustrate our framework and results via several examples

perturbation realization potentials and sensitivity analysis of markov processes two fundamental concepts and quantities realization factors and performance potentials are introduced for markov processes the relations among these two quantities and the group inverse of the infinitesimal generator are studied it is shown that the sensitivity of the steady state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a markov process based on these results algorithms for estimating performance sensitivities on a single sample path of a markov process can be proposed the potentials in this paper are defined through realization factors and are shown to be the same as those defined by poisson equations the results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis ipa and non ipa approaches to the sensitivity analysis of steady state performance they also provide a theoretical background for the pa algorithms developed in recent years

finely harmonic functions with finite dirichlet integral with respect to the green measure we consider finely harmonic functions h on a fine greenian domain v c rd with finite dirichlet integral wrt gm i e f ivh y 2 g x y dm y oo for x c v where m denotes the lebesgue measure g x y the green function we use brownian motion and stochastic calculus to prove that such functions h always have boundary values h along a a brownian paths this partially extends results by doob brelot and godefroid who considered ordinary harmonic functions with finite dirichlet integral wrt m and green lines instead of brownian paths as a consequence of theorem 1 we obtain several properties equivalent to one of these being that h is the harmonic extension to v of a random boundary function h of a certain type i e h x ex h for all x c v another application is that the polar sets are removable singularity sets for finely harmonic functions satisfying this is in contrast with the situation for finely harmonic functions with finite dirichlet integral wrt m

event based optimization of markov systems recent research indicates that markov decision processes mdps and perturbation analysis pa based optimization can be derived easily from two fundamental performance sensitivity formulas with this sensitivity point of view an event based optimization approach including event based sensitivity analysis and event based policy iteration was proposed via an example by x r cao discrete event dyn syst theory appl vol 15 pp 169 197 2005 this approach utilizes the special feature of a system and illustrates how the potentials can be aggregated using the special feature the approach applies to many practical problems that do not fit well the standard mdp formulation this note provides a mathematical formulation and proves the main results for this approach

strong points of weak convergence a study using rpa gradient estimation for automatic learning this paper analyzes the behavior of adaptive control schemes for automatic learning estimates of the sensitivities are used in a gradient based stochastic approximation procedure in order to drive the process along the steepest descent trajectory in search for the optimum the learning rates are kept constant for adaptability for such procedures convergence can be established in a weak sense a model problem of a flexible machine is presented for which the control parameter is a probability vector we propose a new sensitivity estimator generalizing the phantom rare perturbation analysis rpa estimator to multi valued decisions from the basic properties of the estimators we build several updating rules based on the weak convergence theory to ensure asymptotic optimality we illustrate the predicted theoretical behavior with computer simulations finally we present the comparison between the behavior of our proposed scheme with a regenerative one for which we can establish strong convergence our results show that weak convergence yields a dramatic improvement in the rate of convergence in addition to the capability of adaptation or tracking

automated model selection in covariance estimation and spatial whitening of meg and eeg signals abstract magnetoencephalography and electroencephalography m eeg measure non invasively the weak electromagnetic fields induced by post synaptic neural currents the estimation of the spatial covariance of the signals recorded on m eeg sensors is a building block of modern data analysis pipelines such covariance estimates are used in brain computer interfaces bci systems in nearly all source localization methods for spatial whitening as well as for data covariance estimation in beamformers the rationale for such models is that the signals can be modeled by a zero mean gaussian distribution while maximizing the gaussian likelihood seems natural it leads to a covariance estimate known as empirical covariance ec it turns out that the ec is a poor estimate of the true covariance when the number of samples is small to address this issue the estimation needs to be regularized the most common approach downweights off diagonal coefficients while more advanced regularization methods are based on shrinkage techniques or generative models with low rank assumptions probabilistic pca ppca and factor analysis fa using cross validation all of these models can be tuned and compared based on gaussian likelihood computed on unseen data we investigated these models on simulations one electroencephalography eeg dataset as well as magnetoencephalography meg datasets from the most common meg systems first our results demonstrate that different models can be the best depending on the number of samples heterogeneity of sensor types and noise properties second we show that the models tuned by cross validation are superior to models with hand selected regularization hence we propose an automated solution to the often overlooked problem of covariance estimation of m eeg signals the relevance of the procedure is demonstrated here for spatial whitening and source localization of meg signals

trace operator and the dirichlet problem for elliptic equations on arbitrary bounded open sets abstract we consider the dirichlet problem and the weak dirichlet problem on a general possibly nonregular bounded domain for elliptic linear equation with uniformly elliptic divergence form operator we investigate carefully the relationship between weak integral probabilistic soft and the perron wiener brelot solutions of the problem we also study the relationship between solutions of the dirichlet problem and weak dirichlet problem to this end we extend the usual notion of the trace operator and consider the dirichlet problem with martin boundary

stochastic optimization by simulation convergence proofs for the gi g 1 queue in steady state approaches like finite differences with common random numbers infinitesimal perturbation analysis and the likelihood ratio method have drawn a great deal of attention recently as ways of estimating the gradient of a performance measure with respect to continuous parameters in a dynamic stochastic system in this paper we study the use of such estimators in stochastic approximation algorithms to perform so called single run optimizations of steady state systems under mild conditions for an objective function that involves the mean system time in a gi g 1 queue we prove that many variant of these algorithms converge to the minimizer in most cases however the simulation length must be increased from iteration to iteration otherwise the algorithm may converge to the wrong value one exception is a particular implementation of infinitesimal perturbation analysis for which the single run optimization converges to the optimum even with a fixed and small number of ends of service per iteration as a by product of our convergence proofs we obtain some properties of the derivative estimators that could be of independent interest our analysis exploits the regenerative structure of the system but our derivative estimation and optimization algorithms do not always take advantage of that regenerative structure in a companion paper we report numerical experiments with an m m 1 queue which illustrate the basis convergence properties and possible pitfalls of the various techniques

continuous optimization via simulation using golden region search simulation optimization so is a class of mathematical optimization techniques in which the objective function can only be numerically evaluated through simulation in this paper a new so approach called golden region gr search is developed for continuous problems gr divides the feasible region into a number of sub regions and selects one region in each iteration for further search based on the quality and distribution of simulated points in the feasible region and the result of scanning the response surface through a metamodel monte carlo experiments show that the gr method is efficient compared to three well established approaches in the literature we also prove the asymptotic convergence in probability to a global optimum for a large class of random search methods in general and gr in particular

perturbation analysis of discrete event systems concepts algorithms and applications abstract perturbation analysis of discrete event systems is a technique that provides estimates of performance sensitivities with respect to system parameters by analyzing a single sample path of a discrete event system this technique saves a great amount of computation in simulation and can be applied to on line performance optimization of real world systems where changing the values of parameters is infeasible this paper gives an overview of the main concepts general methodologies applications and further directions of the area

stochastic optimization by simulation numerical experiments with the m m 1 queue in steady state this paper gives numerical illustrations of the behavior of stochastic approximation combined with different derivative estimation techniques to optimize a steady state system it is a companion paper to l ecuyer and glynn 1993 which gives convergence proofs for most of the variants experimented here the numerical experiments are made with a simple m m 1 queue which while simple serves to illustrate the basic convergence properties and possible pitfalls of the various techniques

convergence rates of perturbation analysis robbins monro single run algorithms for single server queues in this paper the perturbation analysis robbins monro single run algorithm is applied to estimating the optimal parameter of a performance measure for the gi g 1 queueing systems where the algorithm is updated after every fixed length observation period our aim is to analyze the limiting behavior of the algorithm the almost sure convergence rate of the algorithm is established it is shown that the convergence rate depends on the second derivative of the performance measure at the optimal point

optimization of queues using an infinitesimal perturbation analysis based stochastic algorithm with general update times convergence with probability one of a stochastic optimization algorithm for a single server queue is proved the parameter to be optimized is updated using an infinitesimal perturbation analysis estimate of the gradient of the performance measure and the updates are performed at general times first an algorithm in which the parameter is updated before each customer begins service is considered then it is shown how this analysis can be extended to an algorithm that updates at certain stopping times the analysis suggests that the sample path behavior of the algorithm is similar to that of an algorithm that updates at the start of every busy period