A Closer Look at the Worst-case Behavior of Multi-armed Bandit Algorithms

p n log n minimax regret when the gap can be arbitrarily “ small. ” This paper provides new results on the arm-sampling behavior of UCB , leading to several important insights . Among these , it is shown that arm-sampling rates under UCB are asymptotically deterministic , regardless of the problem complexity . This discovery facilitates new sharp asymptotics and a novel alternative proof for the O p n log n minimax regret of UCB . Furthermore , the paper also provides the first complete process-level characterization of the MAB problem under UCB in the conventional diffusion scaling . Among other things , the “ small ” gap worst-case lens adopted in this paper also reveals profound distinctions between the behavior of UCB and Thompson Sampling , such as an incomplete learning phenomenon characteristic of the latter . 1 Introduction . Background and motivation . The MAB paradigm provides a succinct abstraction of the quintessential exploration vs. exploitation trade-offs inherent in many sequential decision making problems . This has origns in clinical trial studies dating back to 1933 [ 30 ] which gave rise to the earliest known MAB heuristic , Thompson Sampling [ 1 ] . Today , the MAB problem manifests itself in various forms with applications ranging from dynamic pricing and online auctions to packet routing , scheduling , e-commerce and matching markets among others ( see [ 9 ] for a comprehensive survey of different formulations ) . In the canonical stochastic MAB problem , a decision maker ( DM ) pulls one of K arms sequentially at each time t 2 { 1 , 2 , ... } , and receives a random payoff drawn according to an arm-dependent distribution . The DM , oblivious to the statistical properties of the arms , must balance exploring new arms and exploiting the best arm played thus far in order to maximize her cumulative payoff over the horizon of play . This objective is equivalent to minimizing the regret relative to an oracle with perfect ex ante knowledge of the optimal arm ( the one with the highest mean reward ) . The classical stochastic MAB problem is fully specified by the tuple ⇣ ( Pi ) 16i6K , n ⌘ , where Pi denotes the distribution of rewards associated with the ith arm , and n the horizon of play . The statistical complexity of regret minimization in the stochastic MAB problem is governed by a key primitive called the gap , denoted by , which accounts for the difference between the top two arm mean rewards in the problem . For a “ well-separated ” or “ large gap ” instance , i.e. , a fixed bounded away from 0 , the seminal paper [ 22 ] showed that the order of the smallest achievable regret 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is logarithmic in the horizon . There has been a plethora of subsequent work involving algorithms which can be fine-tuned to achieve a regret arbitrarily close to the optimal rate discovered in [ 22 ] ( see [ 6 , 14 , 13 , 2 , 4 ] , etc. , for a few notable examples ) . On the other hand , no algorithm can achieve an expected regret smaller than C p n for a fixed n ( the constant hides dependence on the number of arms ) uniformly over all problem instances ( also called minimax regret ) ; see , e.g. , [ 23 ] , Chapter 15 . The saddle-point in this minimax formulation occurs at a gap that satisfies ⇣ 1/ p n. This has a natural interpretation : approximately 1/ 2 samples are required to distinguish between two distributions with means separated by ; at the 1/ p n-scale , it becomes statistically impossible to distinguish between samples from the top two arms within n rounds of play . If the gap is smaller , despite the increased difficulty in the hypothesis test , the problem becomes “ easier ” from a regret perspective . Thus , ⇣ 1/ p n is the statistically “ hardest ” scale for regret minimization . A number of popular algorithms achieve the p n minimax-optimal rate ( modulo constants ) , see , e.g. , [ 4 , 2 ] , and many more do this within poly-logarithmic factors in n. Many of these are variations of the celebrated upper confidence bound algorithms , e.g. , UCB1 [ 7 ] , that achieve a minimax regret of O p n log n , and at the same time also deliver the logarithmic regret achievable in the instance-dependent setting of [ 22 ] . A major driver of the regret performance of an algorithm is its arm-sampling characteristics . For example , in the instance-dependent ( large gap ) setting , optimal regret guarantees imply that the fraction of time the optimal arm ( s ) are played approaches 1 in probability , as n grows large . However , this fails to provide any meaningful insights as to the distribution of arm-pulls for smaller gaps , e.g. , the ⇣ 1/ p n “ small gap ” that governs the “ worst-case ” instance-independent setting . An illustrative numerical example involving “ small gap. ” Consider an A/B testing problem ( e.g. , a vaccine clinical trial ) where the experimenter is faced with two competing objectives : first , to estimate the efficacy of each alternative with the best possible precision given a budget of samples , and second , keeping the overall cost of the experiment low . This is a fundamentally hard task and algorithms incurring a low cumulative cost typically spend little time exploring sub-optimal alternatives , resulting in a degraded estimation precision ( see , e.g. , [ 5 ] ) . In other words , algorithms tailored for ( cumulative ) regret minimization may lack statistical power [ 31 ] . While this trade-off is unavoidable in “ well-separated ” instances , numerical evidence suggests a plausible resolution in instances with “ small ” gaps as illustrated below . For example , such a situation might arise in trials conducted using two similarly efficacious vaccines ( abstracted away as ⇡ 0 ) . To illustrate the point more vividly , consider the case where is exactly 0 ( of course , this information is not known to the experimenter ) . This setting is numerically illustrated in Figure 1 , which shows the empirical distribution of N1 ( n ) /n ( the fraction of time arm 1 is played until time n ) in a two-armed bandit with = 0 , under two different algorithms ( UCB and Thompson Sampling ) , and reward configurations . A desirable property of the outcome in this setting is to have a linear allocation of the sampling budget per arm on almost every sample-path of the algorithm , as this leads to “ complete learning : ” an algorithm ’ s ability to discern statistical indistinguishability of the arm-means , and induce a “ balanced ” allocation in that event . However , despite the simplicity of the zero-gap scenario , it is far from obvious whether the aforementioned property may be satisfied for standard bandit algorithms such as UCB and Thompson Sampling . Indeed , Figure 1 exhibits a striking difference between the two . The concentration around 1/2 observable in Figure 1 ( a ) indicates that UCB results in an approximately “ balanced ” sample-split , i.e. , the allocation is roughly n/2 per arm for large n ( and this is observed for “ most ” sample-paths ) . In fact , we will later see that the “ bell curve ” in Figure 1 ( a ) eventually collapses into the Dirac measure at 1/2 ( Theorem 1 ) . On the other hand , under Thompson Sampling , the allocation of samples across arms may be arbitrarily “ imbalanced ” despite the arms being statistically identical , as seen in Figure 1 ( b ) ( see , for contrast , Figure 1 ( c ) , where the allocation is perfectly “ balanced ” ) . Namely , the distribution of the posterior may be such that arm 1 is allocated anywhere from almost no sampling effort all the way to receiving almost the entire sampling budget , as Figure 1 ( b ) suggests . Non-degeneracy of arm-sampling rates is observable also under the more widely used version of the algorithm that is based on Gaussian priors and Gaussian likelihoods ( Algorithm 2 in [ 2 ] ) ; see Figure 2 ( a ) . Such behavior can be detrimental for ex post causal inference in the general A/B testing context , and the vaccine testing problem referenced earlier . This is demonstrated via an instructional example of a two-armed bandit with one deterministic reference arm ( aka the “ one-armed ” bandit paradigm ) , illustrated in Figure 2 , and discussed below . A numerical example illustrating inference implications . Consider a model where arm 1 returns a constant reward of 0.5 , while arm 2 yields rewards distributed as Bernoulli ( 0.5 ) . In this setup , the estimate of the gap ( average treatment effect in causal inference parlance ) after n rounds of play is given by ̂ = X̄2 ( n ) 0.5 , where X̄2 ( n ) denotes the empirical mean reward of arm 2 at time n. The Z statistic associated with this gap estimator is given by Z = 2 p N2 ( n ) ̂ , where N2 ( n ) is the visitation count of arm 2 at time n. In the absence of any sample-adaptivity in the arm 2 data , results from classical statistics such as the Central Limit Theorem ( CLT ) would posit an asymptotically Normal distribution for Z . However , since the algorithms that play the arms are adaptive in nature , e.g. , UCB and Thompson Sampling , asymptotic-normality may no longer be guaranteed . Indeed , the numerical evidence in Figure 2 ( b ) strongly points to a significant departure from asymptotic-normality of the Z statistic associated with the gap estimator under Thompson Sampling ( TS ) . Non-normality of the Z statistic can be problematic for inferential tasks , e.g. , it can lead to statistically unsupported inferences in the binary hypothesis test H0 : = 0 vs. H1 : 6= 0 performed using confidence intervals constructed as per the conventional CLT approximation . In sharp contrast , our work shows that UCB satisfies a certain “ balanced ” sampling property ( such as that in Figure 1 ( a ) ) in instances with “ small ” gaps , formally stated as Theorem 1 , that drives the Z statistic towards asymptoticnormality in the aforementioned binary hypothesis testing example ( asymptotic-normality being a consequence of Theorem 4 ) . Furthermore , since the p n-normalized “ stochastic ” regret ( defined in ( 1 ) in §2 ) equals ⇣p N2 ( n ) / ( 4n ) ⌘ Z , it follows that this too , satisfies asymptotic-normality under UCB ( due to Theorem 4 , in conjunction with Theorem 1 ) . These properties are evident in Figure 2 ( c ) below , and signal reliability of ex post causal inference ( under classical assumptions like validity of CLT ) from “ small gap ” data collected by UCB vis-à-vis Thompson Sampling ( TS ) . The reliability of inference under TS may be doubtful even in the limit of infinite data , as Figure 2 ( b ) suggests . While traditional literature on stochastic bandits is dedicated primarily to the regret minimization problem , there has been significant recent interest also in finer-grain properties of popular “ adaptive ” MAB algorithms such as UCB and Thompson Sampling . For example , a recent line of work ( [ 27 , 28 , 29 ] ) investigates the “ bias ” of optimistic algorithms like UCB . The focus of our work is on understanding the distribution of arm-pulls , which as discussed earlier , has significant bearings on ex post causal inference from data collected adaptively by bandit algorithms ( see , e.g. , [ 33 , 16 , 32 ] , etc. , and references therein for recent developments ) , algorithmic fairness in the broader context of fairness in machine learning ( see [ 25 ] for a survey ) , as well as on novel formulations of the MAB problem such as [ 20 ] . Below , we discuss extant literature relevant to our line of work . Previous work . The study of “ well-separated ” instances , or the large gap regime , is supported by rich literature . For example , [ 6 ] provides high-probability bounds on arm-sampling rates under a parametric family of UCB algorithms . However , as the gap diminishes , leading to the so called small gap regime , the aforementioned bounds become vacuous . The understanding of armsampling behavior remains relatively under-studied here even for popular algorithms such as UCB and Thompson Sampling . This regime is of special interest in that it also covers the classical diffusion scaling1 , where ⇣ 1/ p n , which as discussed earlier , corresponds to instances that statistically constitute the “ worst-case ” for hypothesis testing and regret minimization . Recently , a partial diffusion-limit characterization of the arm-sampling distribution under a version of Thompson Sampling with horizon-dependent prior variances2 , was provided in [ 32 ] as a solution to a certain stochastic differential equation ( SDE ) . The numerical solution to said SDE was observed to have a non-degenerate distribution on [ 0 , 1 ] . Similar numerical observations on non-degeneracy of the arm-sampling distribution also under standard versions of Thompson Sampling were reported in [ 10 , 20 ] , among others , albeit limited only to the special case of = 0 , and absent a theoretical explanation for the aforementioned observations . Thus , outside of the so called “ easy ” problems , where is bounded away from 0 by an absolute constant , theoretical understanding of the sampling behavior of bandit algorithms remains an open area of research . Contributions . In this paper , we provide the first complete asymptotic characterization of armsampling distributions under canonical UCB ( Algorithm 1 ) as a function of the gap ( Theorem 1 ) . This gives rise to a fundamental insight : arm-sampling rates are asymptotically deterministic under UCB regardless of the hardness of the instance . We also provide the first theoretical explanation for an “ incomplete learning ” phenomenon under Thompson Sampling ( Algorithm 2 ) alluded to in Figure 1 , as well as a sharp dichotomy between Thompson Sampling and UCB evident therein ( Theorem 2 ) . This result earmarks an “ instability ” of Thompson Sampling in terms of the limiting arm-sampling distribution . As a sequel to Theorem 1 , we provide the first complete characterization of the worst-case performance of canonical UCB ( Theorem 3 ) . One consequence is that the O p n log n minimax regret of UCB is strictly unimprovable in a precise sense . Moreover , our work also leads to the first process-level characterization of the two-armed bandit problem under canonical UCB in the classical diffusion limit , according to which a suitably normalized cumulative reward process converges in law to a Brownian motion with fully characterized drift and infinitesimal variance ( Theorem 4 ) . To the best of our knowledge , this is the first such characterization of UCB-type algorithms . Theorem 4 facilitates a complete distribution-level characterization of UCB ’ s diffusion-limit regret , thereby providing sharp insights as to the problem ’ s minimax complexity . Such distribution-level information may also be useful for a variety of inferential tasks , e.g. , construction of confidence intervals ( see the binary hypothesis testing example referenced in Figure 2 ( c ) ) , among others . We believe our results may also present new design considerations , in particular , how to achieve , loosely speaking , the “ best of both worlds ” for Thompson Sampling , by addressing its “ small gap ” instability . Lastly , we note that our proof techniques are markedly different from the conventional methodology adopted in MAB literature ( [ 6 , 9 , 2 ] ) , and may be of independent interest in the study of related learning algorithms . Organization of the paper . A formal description of the model and the canonical UCB algorithm is provided in §2 . All theoretical propositions are stated in §3 , along with a high-level overview of their scope and proof sketch ; detailed proofs and ancillary results are relegated to the supplementary material . Finally , concluding remarks and open problems are presented in §4 .

Summary: Classical bandit algorithms such as UCB and Thompson Sampling are well-understood in terms of their performance—for example, an upper bound on their regret—but, until recently, little attention has been given to their actual behavior—for example, the rates at which specific arms are sampled. This work seeks to address these questions by analyzing the arm sampling distribution achieved by both UCB and Thompson Sampling. They provide results characterizing these distributions asymptotically and, in addition, characterize the distribution of the rewards obtained by running UCB.