thompson sampling for stochastic control the continuous parameter case recently thompson sampling has been shown to achieve good theoretical performance guarantees for stochastic control problems with parameter uncertainty when the state control and parameter spaces are all finite much less is known however about the performance of thompson sampling when applied to continuous or more general spaces which constitutes an important class of problems in practice in this paper we study thompson sampling when applied to a broad class of average cost stochastic control problems where the state control and parameter spaces are all general measurable spaces the main contributions of our paper are establishing theoretical performance guarantees for thompson sampling as measured by first expected posterior sampling error and second average per period regret

the multi armed bandit problem with covariates we consider a multi armed bandit problem in a setting where each arm produces a noisy reward realization which depends on an observable random covariate as opposed to the traditional static multi armed bandit problem this setting allows for dynamically changing rewards that better describe applications where side information is available we adopt a nonparametric model where the expected rewards are smooth functions of the covariate and where the hardness of the problem is captured by a margin parameter to maximize the expected cumulative reward we introduce a policy called adaptively binned successive elimination abse that adaptively decomposes the global problem into suitably localized static bandit problems this policy constructs an adaptive partition using a variant of the successive elimination se policy our results include sharper regret bounds for the se policy in a static bandit problem and minimax optimal regret bounds for the abse policy in the dynamic problem

the knowledge gradient algorithm for a general class of online learning problems we derive a one period look ahead policy for finite and infinite horizon online optimal learning problems with gaussian rewards our approach is able to handle the case where our prior beliefs about the rewards are correlated which is not handled by traditional multiarmed bandit methods experiments show that our kg policy performs competitively against the best known approximation to the optimal policy in the classic bandit problem and it outperforms many learning policies in the correlated case

rough path limits of the wong zakai type with a modified drift term abstract the wong zakai theorem asserts that odes driven by reasonable e g piecewise linear approximations of brownian motion converge to the corresponding stratonovich stochastic differential equation with the aid of rough path analysis we study non reasonable approximations and go beyond a well known criterion of ikeda watanabe north holland 1989 in the sense that our result applies to perturbations on all levels exhibiting additional drift terms involving any iterated lie brackets of the driving vector fields in particular this applies to the approximations by mcshane 72 and sussmann 91 our approach is not restricted to brownian driving signals at last these ideas can be used to prove optimality of certain rough path estimates

the average distances in random graphs with given expected degrees abstract random graph theory is used to examine the small world phenomenon any two strangers are connected through a short chain of mutual acquaintances we will show that for certain families of random graphs with given expected degrees the average distance is almost surely of order log n log d where d is the weighted average of the sum of squares of the expected degrees of particular interest are power law random graphs in which the number of vertices of degree k is proportional to 1 k for some fixed exponent for the case of 3 we prove that the average distance of the power law graphs is almost surely of order log n log d however many internet social and citation networks are power law graphs with exponents in the range 2 3 for which the power law random graphs have average distance almost surely of order log log n but have diameter of order log n provided having some mild constraints for the average distance and maximum degree in particular these graphs contain a dense subgraph which we call the core having nc log log n vertices almost all vertices are within distance log log n of the core although there are vertices at distance log n from the core

machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

adaptive treatment allocation and the multi armed bandit problem a class of simple adaptive allocation rules is proposed for the problem often called the multi armed bandit problem of sampling x 1 cdots x n sequentially from k populations with densities belonging to an exponential family in order to maximize the expected value of the sum s n x 1 cdots x n these allocation rules are based on certain upper confidence bounds which are developed from boundary crossing theory for the k population parameters the rules are shown to be asymptotically optimal as n rightarrow infty from both bayesian and frequentist points of view monte carlo studies show that they also perform very well for moderate values of the horizon n

a survey of some results in stochastic adaptive control some results in discrete time stochastic adaptive control are surveyed the survey divides itself into two parts bayesian and non bayesian adaptive control in the former area the problems of converting an incompletely observed system into a completely observed one multi armed bandit processes bayesian adaptive control of markov chains and bayesian adaptive control of linear systems are exposed and surveyed in the latter area non bayesian adaptive control of markov chains and the self tuning regulator are dealt with proofs are given where appropriate to illustrate the methods involved

one armed bandit problems with covariates as does woodroofe we consider a bayesian sequential allocation between two treatments that incorporates a covariate the goal is to maximize the total discounted expected reward from an infinite population of patients although our model is more general than woodroofe s we are able to duplicate his main result the myopic rule is asymptotically optimal

better algorithms for benign bandits the online multi armed bandit problem and its generalizations are repeated decision making problems where the goal is to select one of several possible decisions in every round and incur a cost associated with the decision in such a way that the total cost incurred over all iterations is close to the cost of the best fixed decision in hindsight the difference in these costs is known as the regret of the algorithm the term bandit refers to the setting where one only obtains the cost of the decision used in a given iteration and no other information a very general form of this problem is the non stochastic bandit linear optimization problem where the set of decisions is a convex set in some euclidean space and the cost functions are linear only recently an efficient algorithm attaining o t regret was discovered in this setting in this paper we propose a new algorithm for the bandit linear optimization problem which obtains a tighter regret bound of o q where q is the total variation in the cost functions this regret bound previously conjectured to hold in the full information case shows that it is possible to incur much less regret in a slowly changing environment even in the bandit setting our algorithm is efficient and applies several new ideas to bandit optimization such as reservoir sampling

decomposable markov decision processes a fluid optimization approach decomposable markov decision processes mdps are problems where the stochastic system can be decomposed into multiple individual components although such mdps arise naturally in many practical applications they are often difficult to solve exactly due to the enormous size of the state space of the complete system which grows exponentially with the number of components in this paper we propose an approximate solution approach to decomposable mdps that is based on re solving a fluid linear optimization formulation of the problem at each decision epoch this formulation tractably approximates the problem by modeling transition behavior at the level of the individual components rather than the complete system we prove that our fluid formulation provides a tighter bound on the optimal value function than three state of the art formulations the approximate linear optimization formulation the classical lagrangian relaxation formulation and a novel alternate lagrangian relaxation that is based on relax

learning to optimize via information directed sampling we propose information directed sampling a new approach to online optimization problems in which a decision maker must balance between exploration and exploitation while learning from partial feedback each action is sampled in a manner that minimizes the ratio between squared expected single period regret and a measure of information gain the mutual information between the optimal action and the next observation we establish an expected regret bound for information directed sampling that applies across a very general class of models and scales with the entropy of the optimal action distribution we illustrate through simple analytic examples how information directed sampling accounts for kinds of information that alternative approaches do not adequately address and that this can lead to dramatic performance gains for the widely studied bernoulli gaussian and linear bandit problems we demonstrate state of the art simulation performance the electronic companion is available at https doi org 10 128

weighted distances in scale free configuration models in this paper we study first passage percolation in the configuration model with empirical degree distribution that follows a power law with exponent tau in 2 3 we assign independent and identically distributed i i d weights to the edges of the graph we investigate the weighted distance the length of the shortest weighted path between two uniformly chosen vertices called typical distances when the underlying age dependent branching process approximating the local neighborhoods of vertices is found to produce infinitely many individuals in finite time called explosive branching process baroni hofstad and the second author showed in baroni et al j appl probab 54 1 146 164 2017 that typical distances converge in distribution to a bounded random variable the order of magnitude of typical distances remained open for the tau in 2 3 case when the underlying branching process is not explosive we close this gap by determining the first order of magnitude of typical distances in this regime for arbitrary not necessary continuous edge weight distributions that produce a non explosive age dependent branching process with infinite mean power law offspring distributions this sequence tends to infinity with the amount of vertices and by choosing an appropriate weight distribution can be tuned to be any growing function that is o log log n where n is the number of vertices in the graph we show that the result remains valid for the the erased configuration model as well where we delete loops and any second and further edges between two vertices

response surface bandits in this paper we define a response surface bandit as the sequential design problem that maximizes an expected bandit utility but where the outcomes y n are continuous and can be related through a response surface to a set of controllable variables x n x 1n x 2n x kn we link this problem to other traditional optimization problems from industrial engineering and to the traditional bandit problem we consider two approaches to the problem the first is based on a myopic sequential design the second approach uses the best design out of a family of designs related to upper bounds for the predicted surface the family includes myopic and sequential versions of d optimal designs these approaches can be generalized to more broadly defined sequential problems

generating simple random graphs with prescribed degree distribution let f be a probability distribution with support on the non negative integers four methods for generating a simple undirected graph with approximate degree distribution f are described and compared two methods are based on the so called configuration model with modifications ensuring a simple graph one method is an extension of the classical erdos renyi graph where the edge probabilities are random variables and the last method starts with a directed random graph which is then modified to a simple undirected graph all methods are shown to give the correct distribution in the limit of large graph size but under different assumptions on the degree distribution f and also using different order of operations

batched bandit problems motivated by practical applications chiefly clinical trials we study the regret achievable for stochastic bandits under the constraint that the employed policy must split trials into a small number of batches we propose a simple policy and show that a very small number of batches gives close to minimax optimal regret bounds as a byproduct we derive optimal policies with low switching cost for stochastic bandits

first passage percolation on the newman watts small world model the newman watts model is given by taking a cycle graph of n vertices and then adding each possible edge i j i j ne 1 mod n with probability rho n for some rho 0 constant in this paper we add i i d exponential edge weights to this graph and investigate typical distances in the corresponding random metric space given by the least weight paths between vertices we show that typical distances grow as frac 1 lambda log n for a lambda 0 and determine the distribution of smaller order terms in terms of limits of branching process random variables we prove that the number of edges along the shortest weight path follows a central limit theorem and show that in a corresponding epidemic spread model the fraction of infected vertices follows a deterministic curve with a random shift

non asymptotic analysis of a new bandit algorithm for semi bounded rewards in this paper we consider a stochastic multiarmed bandit problem it is known in this problem that deterministic minimum empirical divergence dmed policy achieves the asymptotic theoretical bound for the model where each reward distribution is supported in a known bounded interval say 0 1 however the regret bound of dmed is described in an asymptotic form and the performance in finite time has been unknown we modify this policy and derive a finite time regret bound for the new policy indexed minimum empirical divergence imed by refining large deviation probabilities to a simple nonasymptotic form further the refined analysis reveals that the finite time regret bound is valid even in the case that the reward is not bounded from below therefore our finite time result applies to the case that the minimum reward that is the maximum loss is unknown or unbounded we also present some simulation results which shows that imed much improves dmed and performs competitively to other state of the art policies

boundary crossing problems for sample means on etudie une classe de problemes de passages a la limite lies a plusieurs problemes classiques de la methodologie statistique sequentielle

efficient reinforcement learning in deterministic systems with value function generalization we consider the problem of reinforcement learning over episodes of a finite horizon deterministic system and as a solution propose optimistic constraint propagation ocp an algorithm designed to synthesize efficient exploration and value function generalization we establish that when the true value function lies within a given hypothesis class ocp selects optimal actions over all but at most d episodes where d is the eluder dimension of the given hypothesis class we establish further efficiency and asymptotic performance guarantees that apply even if the true value function does not lie in the given hypothesis class for the special case where the hypothesis class is the span of prespecified indicator functions over disjoint sets we also discuss the computational complexity of ocp and present computational results involving two illustrative examples

optimal strategies for a class of sequential control problems with precedence relations consider the following multi phase project management problem each project is divided into several phases all projects enter the next phase at the same point chosen by the decision maker based on observations up to that point within each phase one can pursue the projects in any order when pursuing the project with one unit of resource the project state changes according to a markov chain the probability distribution of the markov chain is known up to an unknown parameter when pursued the project generates a random reward depending on the phase and the state of the project and the unknown parameter the decision maker faces two problems a how to allocate resources to projects within each phase and b when to enter the next phase so that the total expected reward is as large as possible in this paper we formulate the preceding problem as a stochastic scheduling problem and propose asymptotic optimal strategies which minimize the shortfall from perfect information payoff concrete examples are given to illustrate our method

hierarchical knowledge gradient for sequential sampling we propose a sequential sampling policy for noisy discrete global optimization and ranking and selection in which we aim to efficiently explore a finite set of alternatives before selecting an alternative as best when exploration stops each alternative may be characterized by a multi dimensional vector of categorical and numerical attributes and has independent normal rewards we use a bayesian probability model for the unknown reward of each alternative and follow a fully sequential sampling policy called the knowledge gradient policy this policy myopically optimizes the expected increment in the value of sampling information in each time period we propose a hierarchical aggregation technique that uses the common features shared by alternatives to learn about many alternatives from even a single measurement this approach greatly reduces the measurement effort required but it requires some prior knowledge on the smoothness of the function in the form of an aggregation function and computational issues limit the number of alternatives that can be easily considered to the thousands we prove that our policy is consistent finding a globally optimal alternative when given enough measurements and show through simulations that it performs competitively with or significantly better than other policies

learning to optimize via posterior sampling this paper considers the use of a simple posterior sampling algorithm to balance between exploration and exploitation when learning to optimize actions such as in multiarmed bandit problems the algorithm also known as thompson sampling and as probability matching offers significant advantages over the popular upper confidence bound ucb approach and can be applied to problems with finite or infinite action spaces and complicated relationships among action rewards we make two theoretical contributions the first establishes a connection between posterior sampling and ucb algorithms this result lets us convert regret bounds developed for ucb algorithms into bayesian regret bounds for posterior sampling our second theoretical contribution is a bayesian regret bound for posterior sampling that applies broadly and can be specialized to many model classes this bound depends on a new notion we refer to as the eluder dimension which measures the degree of dependence among action rewards compared to ucb

truthful mechanisms with implicit payment computation it is widely believed that computing payments needed to induce truthful bidding is somehow harder than simply computing the allocation we show that the opposite is true creating a randomized truthful mechanism is essentially as easy as a single call to a monotone allocation rule our main result is a general procedure to take a monotone allocation rule for a single parameter domain and transform it via a black box reduction into a randomized mechanism that is truthful in expectation and individually rational for every realization the mechanism implements the same outcome as the original allocation rule with probability arbitrarily close to 1 and requires evaluating that allocation rule only once we also provide an extension of this result to multiparameter domains and cycle monotone allocation rules under mild star convexity and nonnegativity hypotheses on the type space and allocation rule respectively because our reduction is simple versatile and general it has many applications to mechanism design problems in which re evaluating the allocation rule is either burdensome or informationally impossible applying our result to the multiarmed bandit problem we obtain truthful randomized mechanisms whose regret matches the information theoretic lower bound up to logarithmic factors even though prior work showed this is impossible for truthful deterministic mechanisms we also present applications to offline mechanism design showing that randomization can circumvent a communication complexity lower bound for deterministic payments computation and that it can also be used to create truthful shortest path auctions that approximate the welfare of the vcg allocation arbitrarily well while having the same running time complexity as dijkstra s algorithm

exploration exploitation tradeoff using variance estimates in multi armed bandits algorithms based on upper confidence bounds for balancing exploration and exploitation are gaining popularity since they are easy to implement efficient and effective this paper considers a variant of the basic algorithm for the stochastic multi armed bandit problem that takes into account the empirical variance of the different arms in earlier experimental works such algorithms were found to outperform the competing algorithms we provide the first analysis of the expected regret for such algorithms as expected our results show that the algorithm that uses the variance estimates has a major advantage over its alternatives that do not use such estimates provided that the variances of the payoffs of the suboptimal arms are low we also prove that the regret concentrates only at a polynomial rate this holds for all the upper confidence bound based algorithms and for all bandit problems except those special ones where with probability one the payoff obtained by pulling the optimal arm is larger than the expected payoff for the second best arm hence although upper confidence bound bandit algorithms achieve logarithmic expected regret rates they might not be suitable for a risk averse decision maker we illustrate some of the results by computer simulations

local neighbourhoods for first passage percolation on the configuration model we consider first passage percolation on the configuration model once the network has been generated each edge is assigned an i i d weight modeling the passage time of a message along this edge then independently two vertices are chosen uniformly at random a sender and a recipient and all edges along the geodesic connecting the two vertices are coloured in red in the case that both vertices are in the same component in this article we prove local limit theorems for the coloured graph around the recipient in the spirit of benjamini and schramm we consider the explosive regime in which case the random distances are of finite order and the malthusian regime in which case the random distances are of logarithmic order

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

on bayesian index policies for sequential resource allocation this paper is about index policies for minimizing frequentist regret in a stochastic multi armed bandit model that are inspired by a bayesian view on the problem our main contribution is to prove the asymptotic optimality of bayes ucb an algorithm based on quantiles of posterior distributions when the rewards distributions belong to a one dimensional exponential family for a large class of prior distributions we also show that the bayesian literature gives new insight on what kind of exploration rates could be used in frequentist ucb type algorithms indeed approximations of the bayesian optimal solution or the finite horizon gittins indices suggest the introduction of two algorithms kl ucb and kl ucb h whose asymptotic optimality is also established

the nonstochastic multiarmed bandit problem in the multiarmed bandit problem a gambler must decide which arm of k nonidentical slot machines to play in a sequence of trials so as to maximize his reward this classical problem has received much attention because of the simple model it provides of the trade off between exploration trying out each arm to find the best one and exploitation playing the arm believed to give the best payoff past solutions for the bandit problem have almost always relied on assumptions about the statistics of the slot machines in this work we make no statistical assumptions whatsoever about the nature of the process generating the payoffs of the slot machines we give a solution to the bandit problem in which an adversary rather than a well behaved stochastic process has complete control over the payoffs in a sequence of t plays we prove that the per round payoff of our algorithm approaches that of the best arm at the rate o t 1 2 we show by a matching lower bound that this is the best possible we also prove that our algorithm approaches the per round payoff of any set of strategies at a similar rate if the best strategy is chosen from a pool of n strategies then our algorithm approaches the per round payoff of the strategy at the rate o log n1 2 t 1 2 finally we apply our results to the problem of playing an unknown repeated matrix game we show that our algorithm approaches the minimax payoff of the unknown game at the rate o t 1 2

bayesian heuristic for multiperiod control in multiperiod control problems one needs to sequentially choose the value of a set of multiple input variables xn to keep the value of a single output variable yn close to a target t at any stage n the selection of xn poses a difficult dilemma on the one hand xn has to be such that yn xn t 2 is small in expectation on the other hand xn has to be such that it maximizes the amount of information in yn xn about the model relating yn and xn to help keep the value of future y s as close to t as possible the main approach to this problem is through certainty equivalence designs which select xn such that yn xn is estimated to be equal to t instead myopic designs select the xn that would be optimal if yn xn was the last observation to be made we propose designs that improve on certainty equivalence and on myopic designs by searching for the best design in a family that contains both in that way the proposed design reaches a good compromise between the certainty equivalence choice know

posterior sampling based reinforcement learning for control of unknown linear systems we propose a posterior sampling based learning algorithm for the linear quadratic lq control problem with unknown system parameters the algorithm is called posterior sampling based reinforcement learning for lq regulator psrl lq where two stopping criteria determine the lengths of the dynamic episodes in posterior sampling the first stopping criterion controls the growth rate of episode length the second stopping criterion is triggered when the determinant of the sample covariance matrix is less than half of the previous value we show under some conditions on the prior distribution that the expected bayesian regret of psrl lq accumulated up to time t is bounded by tilde o sqrt t here tilde o cdot hides constants and logarithmic factors numerical simulations are provided to illustrate the performance of psrl lq

universality for first passage percolation on sparse random graphs we consider rst passage percolation on sparse random graphs with prescribed degree distributions and general independent and identically distributed edge weights assumed to have a density assuming that the degree distribution satises a uniform x 2 logx condition we analyze the asymptotic distribution for the minimal weight path between a pair of typical vertices as well the number of edges on this path or hopcount the hopcount satises a central limit theorem where the norming constants are expressible in terms of the parameters of an associated continuous time branching process centered by a multiple of logn where the constant is the inverse of the malthusian rate of growth of the associated branching process the minimal weight converges in distribution the limiting random variable equals the sum of the logarithms of the martingale limits of the branching processes that measure the relative growth of neighborhoods about the two vertices and a gumbel random variable and thus shows a remarkably universal behavior the proofs rely on a rened coupling between the shortest path problems on these graphs and continuous time branching processes and on a poisson point process limit for the potential closing edges of shortest weight paths between the source and destination the results extend to a host of related random graph models ranging from random rregular graphs inhomogeneous random graphs and uniform random graphs with a prescribed degree sequence

a learning approach for interactive marketing to a customer segment when a marketer in an interactive environment decides which messages to send to her customers she may send messages currently thought to be most promising exploitation or use poorly understood messages for the purpose of information gathering exploration we assume that customers are already clustered into homogeneous segments and we consider the adaptive learning of message effectiveness within a customer segment we present a bayesian formulation of the problem in which decisions are made for batches of customers simultaneously although decisions may vary within a batch this extends the classical multiarmed bandit problem for sampling one by one from a set of reward populations our solution methods include a lagrangian decomposition based approximate dynamic programming approach and a heuristic based on a known asymptotic approximation to the multiarmed bandit solution computational results show that our methods clearly outperform approaches that ignore the effects of information gain

kullback leibler upper confidence bounds for optimal sequential allocation we consider optimal sequential allocation in the context of the so called stochastic multi armed bandit model we describe a generic index policy in the sense of gittins 1979 based on upper confidence bounds of the arm payoffs computed using the kullback leibler divergence we consider two classes of distributions for which instances of this general idea are analyzed the kl ucb algorithm is designed for one parameter exponential families and the empirical kl ucb algorithm for bounded and finitely supported distributions our main contribution is a unified finite time analysis of the regret of these algorithms that asymptotically matches the lower bounds of lai and robbins 1985 and burnetas and katehakis 1996 respectively we also investigate the behavior of these algorithms when used with general bounded rewards showing in particular that they provide significant improvements over the state of the art

adaptive strategy for stratified monte carlo sampling we consider the problem of stratified sampling for monte carlo integration of a random variable we model this problem in a k armed bandit where the arms represent the k strata the goal is to estimate the integral mean that is a weighted average of the mean values of the arms the learner is allowed to sample the variable n times but it can decide on line which stratum to sample next we propose an ucb type strategy that samples the arms according to an upper bound on their estimated standard deviations we compare its performance to an ideal sample allocation that knows the standard deviations of the arms for sub gaussian arm distributions we provide bounds on the total regret a distribution dependent bound of order poly min 1 o n 3 2 1 that depends on a measure of the disparity min of the per stratum variances and a distribution free bound poly k o n 7 6 that does not we give similar but somewhat sharper bounds on a proxy of the regret the problem independent bound for this proxy matches its recent minimax lower bound in terms of n up to a log n factor

on the splitting up method for rough partial differential equations abstract this article introduces the splitting method to systems driven by rough paths the focus is on nonlinear partial differential equations with rough noise but we also cover rough differential equations applications to stochastic partial differential equations arising in control theory and nonlinear filtering are given

lower bounds and selectivity of weak consistent policies in stochastic multi armed bandit problem this paper is devoted to regret lower bounds in the classical model of stochastic multi armed bandit a well known result of lai and robbins which has then been extended by burnetas and katehakis has established the presence of a logarithmic bound for all consistent policies we relax the notion of consistency and exhibit a generalisation of the bound we also study the existence of logarithmic bounds in general and in the case of hannan consistency moreover we prove that it is impossible to design an adaptive policy that would select the best of two algorithms by taking advantage of the properties of the environment to get these results we study variants of popular upper confidence bounds ucb policies

a note on performance limitations in bandit problems with side information we consider a sequential adaptive allocation problem which is formulated as a traditional two armed bandit problem but with one important modification at each time step t before selecting which arm to pull the decision maker has access to a random variable xt which provides information on the reward in each arm performance is measured as the fraction of time an inferior arm generating lower mean reward is pulled we derive a minimax lower bound that proves that in the absence of sufficient statistical diversity in the distribution of the covariate x a property that we shall refer to as lack of persistent excitation no policy can improve on the best achievable performance in the traditional bandit problem without side information

pac bayesian inequalities for martingales we present a set of high probability inequalities that control the concentration of weighted averages of multiple possibly uncountably many simultaneously evolving and interdependent martingales our results extend the pac bayesian probably approximately correct analysis in learning theory from the i i d setting to martingales opening the way for its application to importance weighted sampling reinforcement learning and other interactive learning domains as well as many other domains in probability theory and statistics where martingales are encountered we also present a comparison inequality that bounds the expectation of a convex function of a martingale difference sequence shifted to the 0 1 interval by the expectation of the same function of independent bernoulli random variables this inequality is applied to derive a tighter analog of hoeffding azuma s inequality

information theoretic regret bounds for gaussian process optimization in the bandit setting many applications require optimizing an unknown noisy function that is expensive to evaluate we formalize this task as a multiarmed bandit problem where the payoff function is either sampled from a gaussian process gp or has low norm in a reproducing kernel hilbert space we resolve the important open problem of deriving regret bounds for this setting which imply novel convergence rates for gp optimization we analyze an intuitive gaussian process upper confidence bound gp ucb algorithm and bound its cumulative regret in terms of maximal in formation gain establishing a novel connection between gp optimization and experimental design moreover by bounding the latter in terms of operator spectra we obtain explicit sublinear regret bounds for many commonly used covariance functions in some important cases our bounds have surprisingly weak dependence on the dimensionality in our experiments on real sensor data gp ucb compares favorably with other heuristical gp optimization approaches

normal bandits of unknown means and variances consider the problem of sampling sequentially from a finite number of n 2 populations specified by random variables xki i 1 n and k 1 2 where xki denotes the outcome from population i the kth time it is sampled it is assumed that for each fixed i xki k 1 is a sequence of i i d normal random variables with unknown mean i and unknown variance i2 the objective is to have a policy for deciding from which of the n populations to sample from at any time t 1 2 so as to maximize the expected sum of outcomes of n total samples or equivalently to minimize the regret due to lack on information of the parameters i and i2 in this paper we present a simple inflated sample mean ism index policy that is asymptotically optimal in the sense of theorem 4 below this resolves a standing open problem from burnetas and katehakis 1996b additionally finite horizon regret bounds are given

x armed bandits we consider a generalization of stochastic bandits where the set of arms x is allowed to be a generic measurable space and the mean payoff function is locally lipschitz with respect to a dissimilarity function that is known to the decision maker under this condition we construct an arm selection policy called hoo hierarchical optimistic optimization with improved regret bounds compared to previous results for a large class of problems in particular our results imply that if x is the unit hypercube in a euclidean space and the mean payoff function has a finite number of global maxima around which the behavior of the function is locally continuous with a known smoothness degree then the expected regret of hoo is bounded up to a logarithmic factor by n that is the rate of growth of the regret is independent of the dimension of the space we also prove the minimax optimality of our algorithm when the dissimilarity is a metric our basic strategy has quadratic computational complexity as a function of the number of time steps and does not rely on the doubling trick we also introduce a modified strategy which relies on the doubling trick but runs in linearithmic time both results are improvements with respect to previous approaches

first passage percolation on locally treelike networks i dense random graphs we study various properties of least cost paths under independent and identically distributed iid disorder for the complete graph and dense erd s renyi random graphs in the connected phase with iid exponential and uniform weights on edges using a simple heuristic we compute explicitly limiting distributions for properly recentered lengths of shortest paths between typical nodes as well as multiple source destination pairs we also derive asymptotics for the number of edges on the shortest path namely the hopcount and find that the addition of edge weights converts these graphs from ultrasmall world networks to small world networks finally we study the vickrey clarke groves measure of overpayment for the complete graph with exponential edge weights and show that the complete graph is far from monopolistic for large n

pure exploration in finitely armed and continuous armed bandits we consider the framework of stochastic multi armed bandit problems and study the possibilities and limitations of forecasters that perform an on line exploration of the arms these forecasters are assessed in terms of their simple regret a regret notion that captures the fact that exploration is only constrained by the number of available rounds not necessarily known in advance in contrast to the case when the cumulative regret is considered and when exploitation needs to be performed at the same time we believe that this performance criterion is suited to situations when the cost of pulling an arm is expressed in terms of resources rather than rewards we discuss the links between the simple and the cumulative regret one of the main results in the case of a finite number of arms is a general lower bound on the simple regret of a forecaster in terms of its cumulative regret the smaller the latter the larger the former keeping this result in mind we then exhibit upper bounds on the simple regret of some forecasters the paper ends with a study devoted to continuous armed bandit problems we show that the simple regret can be minimized with respect to a family of probability distributions if and only if the cumulative regret can be minimized for it based on this equivalence we are able to prove that the separable metric spaces are exactly the metric spaces on which these regrets can be minimized with respect to the family of all probability distributions with continuous mean payoff functions

long paths in first passage percolation on the complete graph ii global branching dynamics we study the random geometry of first passage percolation on the complete graph equipped with independent and identically distributed positive edge weights we consider the case where the lower extreme values of the edge weights are highly separated this model exhibits strong disorder and a crossover between local and global scales local neighborhoods are related to invasion percolation that display self organised criticality globally the edges with relevant edge weights form a barely supercritical erd s renyi random graph that can be described by branching processes this near critical behaviour gives rise to optimal paths that are considerably longer than logarithmic in the number of vertices interpolating between random graph and minimal spanning tree path lengths crucial to our approach is the quantification of the extreme value behavior of small edge weights in terms of a sequence of parameters s n n ge 1 that characterises the different universality classes for first passage percolation on the complete graph we investigate the case where s n rightarrow infty with s n o n 1 3 which corresponds to the barely supercritical setting we identify the scaling limit of the weight of the optimal path between two vertices and we prove that the number of edges in this path obeys a central limit theorem with mean approximately s n log n s n 3 and variance s n 2 log n s n 3 remarkably our proof also applies to n dependent edge weights of the form e s n where e is an exponential random variable with mean 1 thus settling a conjecture of bhamidi et al weak disorder asymptotics in the stochastic meanfield model of distance ann appl probab 22 1 29 69 2012 the proof relies on a decomposition of the smallest weight tree into an initial part following invasion percolation dynamics and a main part following branching process dynamics the initial part has been studied in eckhoff et al long paths in first passage percolation on the complete graph i local pwit dynamics electron j probab 25 1 45 2020 https doi org 10 1214 20 ejp484 the current paper focuses on the global branching dynamics

on wong zakai approximations with δ martingales we present a class of families of absolutely continuous stochastic processes approximating the wiener process we estimate the rate of convergence of the solutions of the stochastic differential equations driven by these families

regret bounds and minimax policies under partial monitoring this work deals with four classical prediction settings namely full information bandit label efficient and bandit label efficient as well as four different notions of regret pseudo regret expected regret high probability regret and tracking the best expert regret we introduce a new forecaster inf implicitly normalized forecaster based on an arbitrary function for which we propose a unified analysis of its pseudo regret in the four games we consider in particular for x exp x k inf reduces to the classical exponentially weighted average forecaster and our analysis of the pseudo regret recovers known results while for the expected regret we slightly tighten the bounds on the other hand with x x q k which defines a new forecaster we are able to remove the extraneous logarithmic factor in the pseudo regret bounds for bandits games and thus fill in a long open gap in the characterization of the minimax rate for the pseudo regret in the bandit game we also provide high probability bounds depending on the cumulative reward of the optimal action finally we consider the stochastic bandit game and prove that an appropriate modification of the upper confidence bound policy ucb1 auer et al 2002a achieves the distribution free optimal rate while still having a distribution dependent rate logarithmic in the number of plays

on optimal allocation of indivisibles under uncertainty the optimal allocation of indivisible resources is formalized as a stochastic optimization problem involving discrete decision variables a general stochastic search procedure is proposed which develops the concept of the branch and bound method the main idea is to process large collections of possible solutions and to devote more attention to the most promising groups by gathering more information to reduce the uncertainty and by narrowing the search area the optimal solution can be found with probability one special techniques for calculating stochastic lower and upper bounds are discussed the results are illustrated by a computational experiment

when is a scale free graph ultra small in this paper we study typical distances in the configuration model when the degrees have asymptotically infinite variance we assume that the empirical degree distribution follows a power law with exponent tau in 2 3 up to value n beta n for some beta n gg log n gamma and gamma in 0 1 this assumption is satisfied for power law i i d degrees and also includes truncated power law empirical degree distributions where the possibly exponential truncation happens at n beta n these examples are commonly observed in many real life networks we show that the graph distance between two uniformly chosen vertices centers around 2 log log n beta n log tau 2 1 beta n 3 tau with tight fluctuations thus the graph is an ultrasmall world whenever 1 beta n o log log n we determine the distribution of the fluctuations around this value in particular we prove these form a sequence of tight random variables with distributions that show log log periodicity and as a result it is non converging we describe the topology and number of shortest paths we show that the number of shortest paths is of order n f n beta n where f n in 0 1 is a random variable that oscillates with n we decompose shortest paths into three segments two end segments starting at each of the two uniformly chosen vertices and a middle segment the two end segments of any shortest path have length log log n beta n log tau 2 tight and the total degree is increasing towards the middle of the path on these segments the connecting middle segment has length 1 beta n 3 tau tight and it contains only vertices with degree at least of order n 1 f n beta n thus all the degrees on this segment are comparable to the maximal degree our theorems also apply when instead of truncating the degrees we start with a configuration model and we remove every vertex with degree at least n beta n and the edges attached to these vertices this sheds light on the attack vulnerability of the configuration model with infinite variance degrees

linearly parameterized bandits we consider bandit problems involving a large possibly infinite collection of arms in which the expected reward of each arm is a linear function of an r dimensional random vector z rr where r 2 the objective is to minimize the cumulative regret and bayes risk when the set of arms corresponds to the unit sphere we prove that the regret and bayes risk is of order r t by establishing a lower bound for an arbitrary policy and showing that a matching upper bound is obtained through a policy that alternates between exploration and exploitation phases the phase based policy is also shown to be effective if the set of arms satisfies a strong convexity condition for the case of a general set of arms we describe a near optimal policy whose regret and bayes risk admit upper bounds of the form o r t log3 2t

using confidence bounds for exploitation exploration trade offs we show how a standard tool from statistics namely confidence bounds can be used to elegantly deal with situations which exhibit an exploitation exploration trade off our technique for designing and analyzing algorithms for such situations is general and can be applied when an algorithm has to make exploitation versus exploration decisions based on uncertain information provided by a random process we apply our technique to two models with such an exploitation exploration trade off for the adversarial bandit problem with shifting our new algorithm suffers only o st 1 2 regret with high probability over t trials with s shifts such a regret bound was previously known only in expectation the second model we consider is associative reinforcement learning with linear value functions for this model our technique improves the regret from o t3 4 to o t1 2

efficient strong integrators for linear stochastic systems we present numerical schemes for the strong solution of linear stochastic differential equations driven by an arbitrary number of wiener processes these schemes are based on the neumann stochastic taylor and magnus expansions first we consider the case when the governing linear diffusion vector fields commute with each other but not with the linear drift vector field we prove that numerical methods based on the magnus expansion are more accurate in the mean square sense than corresponding stochastic taylor integration schemes second we derive the maximal rate of convergence for arbitrary multidimensional stochastic integrals approximated by their conditional expectations consequently for general nonlinear stochastic differential equations with noncommuting vector fields we deduce explicit formulae for the relation between error and computational costs for methods of arbitrary order third we consider the consequences in two numerical studies one of which is an application arising in stochastic linear quadratic optimal control

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

online pca with optimal regret we investigate the online version of principle component analysis pca where in each trial t the learning algorithm chooses a k dimensional subspace and upon receiving the next instance vector xt suffers the compression loss which is the squared euclidean distance between this instance and its projection into the chosen subspace when viewed in the right parameterization this compression loss is linear i e it can be rewritten as tr wtxtxt where wt is the parameter of the algorithm and the outer product xtxt with xt 1 is the instance matrix in this paper generalize pca to arbitrary positive definite instance matrices xt with the linear loss tr wtxt we evaluate online algorithms in terms of their worst case regret which is a bound on the additional total loss of the online algorithm on all instances matrices over the compression loss of the best k dimensional subspace chosen in hindsight we focus on two popular online algorithms for generalized pca the gradient descent gd and matrix exponentiated gradient meg algorithms we show that if the regret is expressed as a function of the number of trials then both algorithms are optimal to within a constant factor on worst case sequences of positive definite instances matrices with trace norm at most one which subsumes the original pca problem with outer products this is surprising because meg is believed be suboptimal in this case we also show that when considering regret bounds as a function of a loss budget then meg remains optimal and strictly outperforms gd when the instance matrices are trace norm bounded next we consider online pca when the adversary is allowed to present the algorithm with positive semidefinite instance matrices whose largest eigenvalue is bounded rather than their trace which is the sum of their eigenvalues again we can show that meg is optimal and strictly better than gd in this setting

dynamic assortment optimization with a multinomial logit choice model and capacity constraint we consider an assortment optimization problem where a retailer chooses an assortment of products that maximizes the profit subject to a capacity constraint the demand is represented by a multinomial logit choice model we consider both the static and dynamic optimization problems in the static problem we assume that the parameters of the logit model are known in advance we then develop a simple algorithm for computing a profit maximizing assortment based on the geometry of lines in the plane and derive structural properties of the optimal assortment for the dynamic problem the parameters of the logit model are unknown and must be estimated from data by exploiting the structural properties found for the static problem we develop an adaptive policy that learns the unknown parameters from past data and at the same time optimizes the profit numerical experiments based on sales data from an online retailer indicate that our policy performs well

regret in online combinatorial optimization we address online linear optimization problems when the possible actions of the decision maker are represented by binary vectors the regret of the decision maker is the difference between her realized loss and the minimal loss she would have achieved by picking in hindsight the best possible action our goal is to understand the magnitude of the best possible minimax regret we study the problem under three different assumptions for the feedback the decision maker receives full information and the partial information models of the so called semi bandit and bandit problems in the full information case we show that the standard exponentially weighted average forecaster is a provably suboptimal strategy for the semi bandit model by combining the mirror descent algorithm and the inf implicitely normalized forecaster strategy we are able to prove the first optimal bounds finally in the bandit case we discuss existing results in light of a new lower bound and suggest a conjecture on the optimal regret in that case

conceptual proofs of l log l criteria for mean behavior of branching processes the kesten stigum theorem is a fundamental criterion for the rate of growth of a supercritical branching process showing that an l log l condition is decisive in critical and subcritical cases results of kolmogorov and later authors give the rate of decay of the probability that the process survives at least n generations we give conceptual proofs of these theorems based on comparisons of galton watson measure to another measure on the space of trees this approach also explains yaglom s exponential limit law for conditioned critical branching processes via a simple characterization of the exponential distribution

universality for the distance in finite variance random graphs we generalize the asymptotic behavior of the graph distance between two uniformly chosen nodes in the configuration model to a wide class of random graphs among others this class contains the poissonian random graph the expected degree random graph and the generalized random graph including the classical erdos renyi graph in the paper we assign to each node a deterministic capacity and the probability that there exists an edge between a pair of nodes is equal to a function of the product of the capacities of the pair divided by the total capacity of all the nodes we consider capacities which are such that the degrees of a node have uniformly bounded moments of order strictly larger than two so that in particular the degrees have finite variance we prove that the graph distance grows like log nu n where the nu depends on the capacities and n denotes the size of the graph in addition the random fluctuations around this asymptotic mean log nu n are shown to be tight we also consider the case where the capacities are independent copies of a positive random lambda with p lambda x 3 againg resulting in graphs where the degrees have finite variance the method of proof of these results is to couple each member of the class to the poissonian random graph for which we then give the complete proof by adapting the arguments of van der hofstad et al

on the convergence rates of expected improvement methods we consider a ranking and selection problem with independent normal observations and we analyze the asymptotic sampling rates of expected improvement ei methods in this setting such methods often perform well in practice but a tractable analysis of their convergence rates is difficult because of the nonlinearity and nonconvexity of the ei calculations we present new results indicating that for known sampling noise variants of ei produce asymptotic simulation allocations that are essentially identical to those chosen by the optimal computing budget allocation ocba methodology which is known to yield near optimal asymptotic performance in ranking and selection this is the first general equivalence result between ei and ocba and it provides insight into the good practical performance of ei we also derive the limiting allocation for ei under unknown sampling variance

bayesian adaptive stochastic process termination this paper considers the problem of optimally terminating a number of stochastic processes when the time varying random rewards have distributions belonging to an exponential family with an unknown parameter the problem is formulated as a bayesian adaptive control model with the objective of minimizing the difference between the expected reward and the optimal reward when the parameter is known the paper establishes an asymptotic lower bound on this difference and constructs policies based on a kullback leibler index that obtain this lower bound the results are applied to models of tree harvesting and destructive testing a simulation study shows that these policies are efficient when the number of processes is large