A Closer Look at the Worst-case Behavior of Multi-armed Bandit Algorithms

p n log n minimax regret when the gap can be arbitrarily “ small. ” This paper provides new results on the arm-sampling behavior of UCB , leading to several important insights . Among these , it is shown that arm-sampling rates under UCB are asymptotically deterministic , regardless of the problem complexity . This discovery facilitates new sharp asymptotics and a novel alternative proof for the O p n log n minimax regret of UCB . Furthermore , the paper also provides the first complete process-level characterization of the MAB problem under UCB in the conventional diffusion scaling . Among other things , the “ small ” gap worst-case lens adopted in this paper also reveals profound distinctions between the behavior of UCB and Thompson Sampling , such as an incomplete learning phenomenon characteristic of the latter . 1 Introduction . Background and motivation . The MAB paradigm provides a succinct abstraction of the quintessential exploration vs. exploitation trade-offs inherent in many sequential decision making problems . This has origns in clinical trial studies dating back to 1933 [ 30 ] which gave rise to the earliest known MAB heuristic , Thompson Sampling [ 1 ] . Today , the MAB problem manifests itself in various forms with applications ranging from dynamic pricing and online auctions to packet routing , scheduling , e-commerce and matching markets among others ( see [ 9 ] for a comprehensive survey of different formulations ) . In the canonical stochastic MAB problem , a decision maker ( DM ) pulls one of K arms sequentially at each time t 2 { 1 , 2 , ... } , and receives a random payoff drawn according to an arm-dependent distribution . The DM , oblivious to the statistical properties of the arms , must balance exploring new arms and exploiting the best arm played thus far in order to maximize her cumulative payoff over the horizon of play . This objective is equivalent to minimizing the regret relative to an oracle with perfect ex ante knowledge of the optimal arm ( the one with the highest mean reward ) . The classical stochastic MAB problem is fully specified by the tuple ⇣ ( Pi ) 16i6K , n ⌘ , where Pi denotes the distribution of rewards associated with the ith arm , and n the horizon of play . The statistical complexity of regret minimization in the stochastic MAB problem is governed by a key primitive called the gap , denoted by , which accounts for the difference between the top two arm mean rewards in the problem . For a “ well-separated ” or “ large gap ” instance , i.e. , a fixed bounded away from 0 , the seminal paper [ 22 ] showed that the order of the smallest achievable regret 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is logarithmic in the horizon . There has been a plethora of subsequent work involving algorithms which can be fine-tuned to achieve a regret arbitrarily close to the optimal rate discovered in [ 22 ] ( see [ 6 , 14 , 13 , 2 , 4 ] , etc. , for a few notable examples ) . On the other hand , no algorithm can achieve an expected regret smaller than C p n for a fixed n ( the constant hides dependence on the number of arms ) uniformly over all problem instances ( also called minimax regret ) ; see , e.g. , [ 23 ] , Chapter 15 . The saddle-point in this minimax formulation occurs at a gap that satisfies ⇣ 1/ p n. This has a natural interpretation : approximately 1/ 2 samples are required to distinguish between two distributions with means separated by ; at the 1/ p n-scale , it becomes statistically impossible to distinguish between samples from the top two arms within n rounds of play . If the gap is smaller , despite the increased difficulty in the hypothesis test , the problem becomes “ easier ” from a regret perspective . Thus , ⇣ 1/ p n is the statistically “ hardest ” scale for regret minimization . A number of popular algorithms achieve the p n minimax-optimal rate ( modulo constants ) , see , e.g. , [ 4 , 2 ] , and many more do this within poly-logarithmic factors in n. Many of these are variations of the celebrated upper confidence bound algorithms , e.g. , UCB1 [ 7 ] , that achieve a minimax regret of O p n log n , and at the same time also deliver the logarithmic regret achievable in the instance-dependent setting of [ 22 ] . A major driver of the regret performance of an algorithm is its arm-sampling characteristics . For example , in the instance-dependent ( large gap ) setting , optimal regret guarantees imply that the fraction of time the optimal arm ( s ) are played approaches 1 in probability , as n grows large . However , this fails to provide any meaningful insights as to the distribution of arm-pulls for smaller gaps , e.g. , the ⇣ 1/ p n “ small gap ” that governs the “ worst-case ” instance-independent setting . An illustrative numerical example involving “ small gap. ” Consider an A/B testing problem ( e.g. , a vaccine clinical trial ) where the experimenter is faced with two competing objectives : first , to estimate the efficacy of each alternative with the best possible precision given a budget of samples , and second , keeping the overall cost of the experiment low . This is a fundamentally hard task and algorithms incurring a low cumulative cost typically spend little time exploring sub-optimal alternatives , resulting in a degraded estimation precision ( see , e.g. , [ 5 ] ) . In other words , algorithms tailored for ( cumulative ) regret minimization may lack statistical power [ 31 ] . While this trade-off is unavoidable in “ well-separated ” instances , numerical evidence suggests a plausible resolution in instances with “ small ” gaps as illustrated below . For example , such a situation might arise in trials conducted using two similarly efficacious vaccines ( abstracted away as ⇡ 0 ) . To illustrate the point more vividly , consider the case where is exactly 0 ( of course , this information is not known to the experimenter ) . This setting is numerically illustrated in Figure 1 , which shows the empirical distribution of N1 ( n ) /n ( the fraction of time arm 1 is played until time n ) in a two-armed bandit with = 0 , under two different algorithms ( UCB and Thompson Sampling ) , and reward configurations . A desirable property of the outcome in this setting is to have a linear allocation of the sampling budget per arm on almost every sample-path of the algorithm , as this leads to “ complete learning : ” an algorithm ’ s ability to discern statistical indistinguishability of the arm-means , and induce a “ balanced ” allocation in that event . However , despite the simplicity of the zero-gap scenario , it is far from obvious whether the aforementioned property may be satisfied for standard bandit algorithms such as UCB and Thompson Sampling . Indeed , Figure 1 exhibits a striking difference between the two . The concentration around 1/2 observable in Figure 1 ( a ) indicates that UCB results in an approximately “ balanced ” sample-split , i.e. , the allocation is roughly n/2 per arm for large n ( and this is observed for “ most ” sample-paths ) . In fact , we will later see that the “ bell curve ” in Figure 1 ( a ) eventually collapses into the Dirac measure at 1/2 ( Theorem 1 ) . On the other hand , under Thompson Sampling , the allocation of samples across arms may be arbitrarily “ imbalanced ” despite the arms being statistically identical , as seen in Figure 1 ( b ) ( see , for contrast , Figure 1 ( c ) , where the allocation is perfectly “ balanced ” ) . Namely , the distribution of the posterior may be such that arm 1 is allocated anywhere from almost no sampling effort all the way to receiving almost the entire sampling budget , as Figure 1 ( b ) suggests . Non-degeneracy of arm-sampling rates is observable also under the more widely used version of the algorithm that is based on Gaussian priors and Gaussian likelihoods ( Algorithm 2 in [ 2 ] ) ; see Figure 2 ( a ) . Such behavior can be detrimental for ex post causal inference in the general A/B testing context , and the vaccine testing problem referenced earlier . This is demonstrated via an instructional example of a two-armed bandit with one deterministic reference arm ( aka the “ one-armed ” bandit paradigm ) , illustrated in Figure 2 , and discussed below . A numerical example illustrating inference implications . Consider a model where arm 1 returns a constant reward of 0.5 , while arm 2 yields rewards distributed as Bernoulli ( 0.5 ) . In this setup , the estimate of the gap ( average treatment effect in causal inference parlance ) after n rounds of play is given by ̂ = X̄2 ( n ) 0.5 , where X̄2 ( n ) denotes the empirical mean reward of arm 2 at time n. The Z statistic associated with this gap estimator is given by Z = 2 p N2 ( n ) ̂ , where N2 ( n ) is the visitation count of arm 2 at time n. In the absence of any sample-adaptivity in the arm 2 data , results from classical statistics such as the Central Limit Theorem ( CLT ) would posit an asymptotically Normal distribution for Z . However , since the algorithms that play the arms are adaptive in nature , e.g. , UCB and Thompson Sampling , asymptotic-normality may no longer be guaranteed . Indeed , the numerical evidence in Figure 2 ( b ) strongly points to a significant departure from asymptotic-normality of the Z statistic associated with the gap estimator under Thompson Sampling ( TS ) . Non-normality of the Z statistic can be problematic for inferential tasks , e.g. , it can lead to statistically unsupported inferences in the binary hypothesis test H0 : = 0 vs. H1 : 6= 0 performed using confidence intervals constructed as per the conventional CLT approximation . In sharp contrast , our work shows that UCB satisfies a certain “ balanced ” sampling property ( such as that in Figure 1 ( a ) ) in instances with “ small ” gaps , formally stated as Theorem 1 , that drives the Z statistic towards asymptoticnormality in the aforementioned binary hypothesis testing example ( asymptotic-normality being a consequence of Theorem 4 ) . Furthermore , since the p n-normalized “ stochastic ” regret ( defined in ( 1 ) in §2 ) equals ⇣p N2 ( n ) / ( 4n ) ⌘ Z , it follows that this too , satisfies asymptotic-normality under UCB ( due to Theorem 4 , in conjunction with Theorem 1 ) . These properties are evident in Figure 2 ( c ) below , and signal reliability of ex post causal inference ( under classical assumptions like validity of CLT ) from “ small gap ” data collected by UCB vis-à-vis Thompson Sampling ( TS ) . The reliability of inference under TS may be doubtful even in the limit of infinite data , as Figure 2 ( b ) suggests . While traditional literature on stochastic bandits is dedicated primarily to the regret minimization problem , there has been significant recent interest also in finer-grain properties of popular “ adaptive ” MAB algorithms such as UCB and Thompson Sampling . For example , a recent line of work ( [ 27 , 28 , 29 ] ) investigates the “ bias ” of optimistic algorithms like UCB . The focus of our work is on understanding the distribution of arm-pulls , which as discussed earlier , has significant bearings on ex post causal inference from data collected adaptively by bandit algorithms ( see , e.g. , [ 33 , 16 , 32 ] , etc. , and references therein for recent developments ) , algorithmic fairness in the broader context of fairness in machine learning ( see [ 25 ] for a survey ) , as well as on novel formulations of the MAB problem such as [ 20 ] . Below , we discuss extant literature relevant to our line of work . Previous work . The study of “ well-separated ” instances , or the large gap regime , is supported by rich literature . For example , [ 6 ] provides high-probability bounds on arm-sampling rates under a parametric family of UCB algorithms . However , as the gap diminishes , leading to the so called small gap regime , the aforementioned bounds become vacuous . The understanding of armsampling behavior remains relatively under-studied here even for popular algorithms such as UCB and Thompson Sampling . This regime is of special interest in that it also covers the classical diffusion scaling1 , where ⇣ 1/ p n , which as discussed earlier , corresponds to instances that statistically constitute the “ worst-case ” for hypothesis testing and regret minimization . Recently , a partial diffusion-limit characterization of the arm-sampling distribution under a version of Thompson Sampling with horizon-dependent prior variances2 , was provided in [ 32 ] as a solution to a certain stochastic differential equation ( SDE ) . The numerical solution to said SDE was observed to have a non-degenerate distribution on [ 0 , 1 ] . Similar numerical observations on non-degeneracy of the arm-sampling distribution also under standard versions of Thompson Sampling were reported in [ 10 , 20 ] , among others , albeit limited only to the special case of = 0 , and absent a theoretical explanation for the aforementioned observations . Thus , outside of the so called “ easy ” problems , where is bounded away from 0 by an absolute constant , theoretical understanding of the sampling behavior of bandit algorithms remains an open area of research . Contributions . In this paper , we provide the first complete asymptotic characterization of armsampling distributions under canonical UCB ( Algorithm 1 ) as a function of the gap ( Theorem 1 ) . This gives rise to a fundamental insight : arm-sampling rates are asymptotically deterministic under UCB regardless of the hardness of the instance . We also provide the first theoretical explanation for an “ incomplete learning ” phenomenon under Thompson Sampling ( Algorithm 2 ) alluded to in Figure 1 , as well as a sharp dichotomy between Thompson Sampling and UCB evident therein ( Theorem 2 ) . This result earmarks an “ instability ” of Thompson Sampling in terms of the limiting arm-sampling distribution . As a sequel to Theorem 1 , we provide the first complete characterization of the worst-case performance of canonical UCB ( Theorem 3 ) . One consequence is that the O p n log n minimax regret of UCB is strictly unimprovable in a precise sense . Moreover , our work also leads to the first process-level characterization of the two-armed bandit problem under canonical UCB in the classical diffusion limit , according to which a suitably normalized cumulative reward process converges in law to a Brownian motion with fully characterized drift and infinitesimal variance ( Theorem 4 ) . To the best of our knowledge , this is the first such characterization of UCB-type algorithms . Theorem 4 facilitates a complete distribution-level characterization of UCB ’ s diffusion-limit regret , thereby providing sharp insights as to the problem ’ s minimax complexity . Such distribution-level information may also be useful for a variety of inferential tasks , e.g. , construction of confidence intervals ( see the binary hypothesis testing example referenced in Figure 2 ( c ) ) , among others . We believe our results may also present new design considerations , in particular , how to achieve , loosely speaking , the “ best of both worlds ” for Thompson Sampling , by addressing its “ small gap ” instability . Lastly , we note that our proof techniques are markedly different from the conventional methodology adopted in MAB literature ( [ 6 , 9 , 2 ] ) , and may be of independent interest in the study of related learning algorithms . Organization of the paper . A formal description of the model and the canonical UCB algorithm is provided in §2 . All theoretical propositions are stated in §3 , along with a high-level overview of their scope and proof sketch ; detailed proofs and ancillary results are relegated to the supplementary material . Finally , concluding remarks and open problems are presented in §4 .

Summary: The paper studies the asymptotical behaviors (with respect to the suboptimality gap $\Delta$) of key statistics in the standard upper confidence bound (UCB) algorithm in standard multi-armed bandits (MAB). The contributions are as follows.  1. The paper proves that the **asymptotical arm-sampling rate** converges to a constant, and gives its **analytical form** (Eq. 2). In particular, the paper discovers a non-trivial "**moderate gap**" regime for UCB.  2. The paper proves an asymptotic regret lower bound for UCB **up to $(1 + o(1))$**  in two-armed bandits.  3. The paper proves that the **asymptotical empirical sum** converges to a Brownian motion for UCB in two-armed bandits. 4. The paper proves the asymptotical arm-sampling rate for Bernoulli Thompson sampling in two-armed *deterministic* bandits.  5. The authors extend their results to MAB.