machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

adaptive treatment allocation and the multi armed bandit problem a class of simple adaptive allocation rules is proposed for the problem often called the multi armed bandit problem of sampling x 1 cdots x n sequentially from k populations with densities belonging to an exponential family in order to maximize the expected value of the sum s n x 1 cdots x n these allocation rules are based on certain upper confidence bounds which are developed from boundary crossing theory for the k population parameters the rules are shown to be asymptotically optimal as n rightarrow infty from both bayesian and frequentist points of view monte carlo studies show that they also perform very well for moderate values of the horizon n

a survey of some results in stochastic adaptive control some results in discrete time stochastic adaptive control are surveyed the survey divides itself into two parts bayesian and non bayesian adaptive control in the former area the problems of converting an incompletely observed system into a completely observed one multi armed bandit processes bayesian adaptive control of markov chains and bayesian adaptive control of linear systems are exposed and surveyed in the latter area non bayesian adaptive control of markov chains and the self tuning regulator are dealt with proofs are given where appropriate to illustrate the methods involved

transverse vibration of simply supported elliptical and circular plates using boundary characteristic orthogonal polynomials in two variables abstract the first four frequencies and mode shapes have been computed for elliptical and circular plates of uniform thickness with simply supported edges by employing orthogonal polynomials in the rayleigh ritz method in general this is found to give better results than traditional methods successive approximations have been worked out to ensure convergence comparisons have been made with existing results in some cases calculations have been done for several values of b a i e the ratio of the semi major and semi minor axes mode shapes have been used to obtain three dimensional plots of plates in displaced configurations

a characterization of the sporadic group of lyons abstract we show that lyons sporadic simple group is the unique finite simple group g that is simultaneously of weak characteristic r type and characteristic p type for odd primes r and p with r p and m r p g 2

adaptive controller design for tracking and disturbance attenuation in parametric strict feedback nonlinear systems the authors develop a systematic procedure for obtaining robust adaptive controllers that achieve asymptotic tracking and disturbance attenuation for a class of nonlinear systems which are described in the parametric strict feedback form and are subject to additional exogenous disturbance inputs their approach to adaptive control is performance based where the objective for the controller design is not only to find an adaptive controller but also to construct an appropriate cost functional compatible with desired asymptotic tracking and disturbance attenuation specifications with respect to which the adaptive controller is worst case optimal three main issues of the paper are the backstepping methodology worst case identification schemes and singular perturbations analysis closed form expressions have been obtained for an adaptive controller and the corresponding value function a numerical example involving a third order system is given

bayesian prediction of deterministic functions with applications to the design and analysis of computer experiments abstract this article is concerned with prediction of a function y t over a multidimensional domain t given the function values at a set of sites t 1 t 2 t n in t and with the design that is with the selection of those sites the motivating application is the design and analysis of computer experiments where t determines the input to a computer model of a physical or behavioral system and y t is a response that is part of the output or is calculated from it following a bayesian formulation prior uncertainty about the function y is expressed by means of a random function y which is taken here to be a gaussian stochastic process the mean of the posterior process can be used as the prediction function t and the variance can be used as a measure of uncertainty this kind of approach has been used previously in bayesian interpolation and is strongly related to the kriging methods used in geostatistics here emphasis is placed on product linear and product cubic correlation func

the uniqueness of groups of lyons type we give the first computer free proof of the uniqueness of groups of type ly we also show that certain simpicial complexes associated to the lyons group ly have the same homotopy type as the quillen complex for ly at the prime 3 we show this complex is simply connected and we calculate its homology finally we supply simplified proofs of some properties of groups of type ly such as the group order a group of type ly is a finite group g possessing an involution t such that h cg t is the covering group of the alternating group a1l and t is not weakly closed in h with respect to g we prove

vibration of tapered thickness annular sector plates by spline element method abstract this paper deals with the determination of the vibrations of thin annular sector plates of tapered thickness by using the spline element method to demonstrate the accuracy of the present method several examples are solved and results are compared with those obtained by analytical methods and other numerical methods good accuracy is obtained frequencies of tapered annular sector plates with several boundary conditions are analyzed for the different sector angles and thickness parameters

certainty equivalence control with forcing revisited abstract certainty equivalence control with forcing has been shown to be optimal for several stochastic adaptive control problems with the average cost per unit time criterion recently researchers have started looking at stochastic adaptive control problems with a view to minimizing the rate of increase of the learning loss this criterion is stronger than the average cost per unit time criterion certainty equivalence control with forcing does not usually suffice for the learning loss criterion and one has to develop fairly complicated schemes in order to achieve optimality the objective of this paper is to see how well one might be able to do with a certainty equivalence control with forcing type of scheme in particular we construct a class of such schemes whose learning loss is o log n 1 for 0 whereas optimal schemes typically have a o log n learning loss

adaptive control of constrained markov chains adaptive control of finite state markov chains is discussed the optimal performance is characterized through the minimization of a long run average cost functional subject to constraints on several other such functionals under mild structural and feasibility conditions two explicit adaptive control policies are exhibited for the case where the transition probabilities are unknown the policies are optimal under the constrained optimization criterion they rely on a powerful estimation scheme which provides consistent estimators for the transition probabilities this scheme is of independent interest as it provides strong consistency under a large number of adaptive schemes and is independent of any identifiability conditions as an application an optimal adaptive policy is derived for a system of k competing queues with countable state space for which the constrained criteria arise naturally in the context of communication networks

transverse vibrations of a trapezoidal cantilever plate of variable thickness n atural frequencies of a trapezoidal cantilever plate of variable thickness which model an aircraft wing structure approximately are determined using energy techniques characteristic orthogonal polynomials in two variables are constructed to describe the structural deflections which are used in the rayleigh ritz method to obtain the natural frequencies the fundamental natural frequencies for different parameter values are also obtained using a deflection function containing an optimized exponent the first technique can be used to obtain the natural frequencies and corresponding mode shapes for the preliminary design of aircraft wing structures

adaptive control of markov chains with local updates abstract an iterative algorithm is proposed for adaptive expected average cost control of finite state markov chains the algorithm estimates the unknown parameter using a strongly consistent parameter estimator and uses this estimate to update the control policy each update may apply to as few as one or as many as all states as long as all states are updated infinitely often as the number of iterations goes to infinity in this sense the algorithm does local updates no prior knowledge of the optimal control policy is assumed the algorithm requires only a small amount of computation at each step is easily implementable and converges to the optimal policy in finite time its complexity for queueing systems governed by birth death processes is comparable to that of a learning automaton but is based on the optimality equation of dynamic programming

tuning of gaussian stochastic control systems a closed loop system consisting of a control system and an adaptive controller is called tuning for a specified control objective if the real system and the ideal system defined below achieve the same value for the control objective the real system is the system consisting of the unknown control system in closed loop with the adaptive controller in which the parameters of the adaptive controller have been determined by identification under feedback or in closed loop the ideal system is the system consisting of the unknown control system in closed loop with a controller in which the controller has been synthesized with knowledge of the unknown control system and such that the closed loop system satisfies the control objective both the gaussian stochastic control system with full observations and with partial observations are considered the approach to the problem is based on stochastic realization theory for gaussian systems the control objectives of minimum variance control and pole placement are also given necessary conditions for tuning are discussed

asymptotic behavior of stochastic systems possessing markovian realizations markovian stochastic systems of the form phi k 1 f phi k w k 1 are considered where bf w is an independent and identically distributed process and bf phi is a markov chain both evolving in euclidean space a condition called weak stochastic controllability w s c is introduced which is a generalization of the concept of controllability in linear system theory two different formulations of stability boundedness in probability and boundedness in probability on average are presented and it is shown that these conditions are equivalent under the w s c condition these results are related to the ergodic behavior of bf phi by a set of results that includes the following if bf phi is bounded in probability and w s c then i sample path averages of functions of the state process converge for every initial condition and ii if the stability is uniform then state process probabilities converge to a periodic orbit of probabilities finally if the origin o

computationally efficient algorithms for on line optimization of markov decision processes abstract iterative algorithms are proposed for on line optimization of finite state markov decision processes the optimization criterion is the minimization of long run average cost the transition probabilities of the underlying markov chain are assumed to depend on certain unknown parameters the algorithms estimate the unknown para meters using a strongly consistent estimator and use this estimate in place of the true parameter to generate a sequence of control policies which converge to an optimal policy the main feature of the algorithms is that the updating step does not have to be carried out in all states at each iteration at each iteration the updating step may be performed in any number of states as few as one as long as all states are updated often enough as the number of iterations increases in this sense the algorithms are asynchronous the asynchronous nature of our algorithms allows us to update the costly states more often as will be illustrated through simulation results the asynchronism of our algorithms makes them feasible for on line optimization

theory for automatic learning under partially observed markov dependent noise a vigorous branch of automatic learning is directed at the task of locating a global minimum of an unknown multimodal function f theta on the basis of noisy observations l theta i f theta i w theta i taken at sequentially chosen control points theta i in all preceding convergence deviations known to the authors the noise is postulated to depend on the past only through control selection here they allow the observation noise sequence to be stochastically dependent in particular a function of an unknown underlying markov decision process the observations being the stagewise losses in a sense in order to be made precise the algorithm offered is shown to attain asymptotically optimal performance and rates are assured a motivating example from queueing theory is offered and connections with classical problems of markov control theory and other disciplines are mentioned

parallel recursive algorithms in asymptotically efficient adaptive control of linear stochastic systems first a review of some recent developments in stochastic adaptive control of linear stochastic systems is given by integrating and refining several basic ideas in these developments a relatively complete asymptotic solution to the adaptive control problem is then provided for such systems the solution involves parallel implementation of a few basic recursive identification algorithms that serve to complement each other it also involves occasional use of a dither signal to probe the system for information

numerical experiments on the determination of natural frequencies of transverse vibrations of rectangular plates of non uniform thickness abstract a comparison of natural frequencies of transverse vibration of thin rectangular plates of non uniform thickness with different combinations of boundary conditions is presented in this paper as obtained by using the following methodologies 1 the rayleigh ritz method with characteristic orthogonal polynomial shape functions 2 the rayleigh ritz method with a shape function which includes two exponents that are to be determined by minimizing the fundamental frequency coefficient 3 the optimized kantorovich method which has been proposed rather recently 4 the finite element method the results are presented in tabular form and discussed

stochastic dynamic programming caution and probing the purpose of this paper is to unify the concepts of caution and probing put forth by feldbaum 14 with the mathematical technique of stochastic dynamic programming originated by bellman 5 the decomposition of the expected cost in a stochastic control problem recently developed in 8 is used to assess quantitatively the caution and probing effects of the system uncertainties on the control it is shown how in some problems because of the uncertainties the control becomes cautious less aggressive while in other problems it will probe by becoming more aggressive in order to enhance the estimation identification while controlling the system following this a classification of stochastic control problems according to the dominant effect is discussed this is then used to point out which are the stochastic control problems where substantial improvements can be expected from using a sophisticated algorithm versus a simple one

a globally convergent stochastic approximation by combining a constrained kiefer wolfowitz search with an automatic learning algorithm it is shown that asymptotically normal convergence of an estimator to a global optimum under reasonably lenient assumptions can be attained it is enough that the objective function be smooth and locally strictly convex at its minima the central conclusion is that if theta n is the estimate produced by the method shown at the nth decision epoch then for some global minimizer theta n 1 3 theta n theta is asymptotically normally distributed this coincides with the conventional kiefer wolfowitz convergence rate to a local optimum whereas this study was motivated by needs of machine learning the basic plan would seem applicable to root finding tasks and to other types of stochastic approximation algorithms

flexural vibration of polygonal plates using characteristic orthogonal polynomials in two variables a set of characteristic orthogonal polynomials in two variables is used as deflection functions to obtain the natural frequencies and mode shapes of polygonal plates by the rayleigh ritz method the set of orthogonal polynomials is constructed by employing the gram schmidt orthogonalization procedure the first six natural frequencies are numerically evaluated for triangular plates of different configurations by using these orthogonal functions and the corresponding mode shapes are plotted results are compared with those obtained previously by other methods