From Optimality to Robustness: Adaptive Re-Sampling Strategies in Stochastic Bandits

The stochastic multi-arm bandit problem has been extensively studied under standard assumptions on the arm ’ s distribution ( e.g bounded with known support , exponential family , etc ) . These assumptions are suitable for many real-world problems but sometimes they require knowledge ( on tails for instance ) that may not be precisely accessible to the practitioner , raising the question of the robustness of bandit algorithms to model misspecification . In this paper we study a generic Dirichlet Sampling ( DS ) algorithm , based on pairwise comparisons of empirical indices computed with re-sampling of the arms ’ observations and a data-dependent exploration bonus . We show that different variants of this strategy achieve provably optimal regret guarantees when the distributions are bounded and logarithmic regret for semi-bounded distributions with a mild quantile condition . We also show that a simple tuning achieve robustness with respect to a large class of unbounded distributions , at the cost of slightly worse than logarithmic asymptotic regret . We finally provide numerical experiments showing the merits of DS in a decision-making problem on synthetic agriculture data . 1 Introduction . The K-armed stochastic bandit model is a decision-making problem in which a learner sequentially picks an action among K alternatives , called arms , and collects a random reward . In this setting , all rewards drawn from an arm are independent and identically distributed . Hence , we can formally associate each arm k ∈ { 1 , . . . , K } with its reward distribution νk , with mean µk . The objective of the learner is to adapt her strategy ( At ) t∈ [ T ] in order to maximize the expected sum of rewards obtained after T selections ( where T is the horizon , unknown to the learner ) . This is equivalent to minimizing the regret , defined as the difference between the expected total reward of an oracle strategy always selecting an arm with largest mean and that of the algorithm , which is equal to RT = E [ T∑ t=1 µ ? − µAt ] = K∑ k=1 ∆kE [ Nk ( T ) ] . ( 1 ) Here , Nk ( T ) = ∑T t=1 1 ( At = k ) denotes the number of selections of arm k after T time steps , µ ? = maxj∈ { 1 , ... , K } µj and ∆k = µ ? −µk is called the gap between arm k and the largest mean . To assess the performance of a bandit algorithm , one naturally studies the best guarantees achievable by a uniformly efficient algorithm , i.e with sub-linear regret on any instance of a given class of problems . This guarantee was first provided by Lai and Robbins ( 1985 ) for 1-dimensional parametric families of distributions , and then extended by Burnetas and Katehakis ( 1996 ) for more general families . It 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . states that any algorithm that is uniformly efficient1 on a family of distributions F must satisfy lim inf T→∞ RT log ( T ) ≥ ∑ k : ∆k > 0 ∆k KFinf ( νk , µ ? ) , KFinf ( νk , µ ? ) = inf G∈F { KL ( νk , G ) : EG ( X ) > µ ? } . ( 2 ) A bandit algorithm is then called asymptotically optimal for a family of distributions F when its regret matches this lower bound . When F is a Single-Parameter Exponential Family ( SPEF ) , KFinf is simply the Kullback-Leibler divergence between the distribution of mean µk and that of mean µ ? in F , making for a theoretically appealing setting . The quantity KBinf , corresponding to the family F [ −∞ , B ] of distributions supported in ( −∞ , B ] is also often considered in the literature , see e.g ( Honda and Takemura , 2010 , 2015 ; Cappé et al. , 2013 ) . Overview of existing strategies An efficient strategy faces the classical exploration/exploitation dilemma : it needs to obtain enough information from arms that have not been sampled a lot ( exploration ) , but also to sample arms that are well-performing sufficiently often ( exploitation ) . Many algorithms have been proposed for the multi-armed bandits problem ( see Lattimore and Szepesvári ( 2020 ) for a survey ) , and we propose in the following a non-exhaustive list of such methods . A first category contains the deterministic index policies , built on the concept of Optimism in Face of Uncertainty , the most celebrated of which being the Upper Confidence Bound ( UCB ) algorithms ( Agrawal , 1995 ; Auer et al. , 2002 ) . These algorithms can obtain a logarithmic regret under classical hypothesis on the distributions ( e.g bounded , sub-gaussian , sub-exponential , . . . ) , and the strongest guarantees have been achieved by kl-UCB Cappé et al . ( 2013 ) , DMED ( Honda and Takemura , 2010 ) , and IMED ( Honda and Takemura , 2015 ) , which share a common pattern of solving a convex optimization problem at each round . To be asymptotically optimal , these algorithms require either 1 ) the knowledge of a specific SPEF for each arm , or 2 ) a known upper bound on the support of each arm . A second general category is that of randomized bandit algorithms , which has been formulated for instance in ( Kveton et al. , 2019b ) as General Randomized Exploration ( GRE ) . The common feature of these methods is that , at each time step and for each arm , the algorithm draws an index from a distribution that depends on 1 ) the rewards observed from the arm , and 2 ) some knowledge on the arms distributions and chooses the arm with the largest index . Thompson Sampling ( TS ) ( Thompson , 1933 ; Agrawal and Goyal , 2012 ) belongs to this category , and a proper choice of Bayesian prior/posterior ensures optimality of TS in SPEF ( Korda et al. , 2013 ) . Different algorithms using Bootstrapping schemes have also been proposed ( Osband and Roy , 2015 ; Kveton et al. , 2019a , b ; Wang et al. , 2020 ; Riou and Honda , 2020 ) : they share the idea of computing a noisy mean for empirical samples , enhanced by some exploration aid appropriately tuned to the family of distributions they consider . A last category contains the methods based on sub-sampling Baransi et al . ( 2014 ) ; Chan ( 2020 ) ; Baudry et al . ( 2020 , 2021b ) , that achieve asymptotic optimality in SPEF without knowing which family , when all arms share the same . However the proofs heavily rely on properties of the tails of SPEF so the results seem difficult to generalize outside these families . Motivations While many algorithms achieve optimal regret for bounded distributions with the sole knowledge of the upper bound , the assumptions needed for algorithms working with unbounded distributions ( e.g SPEF , sub-Gaussian , sub-exponential ) generally assume a known parametric model for the tails . While such assumption entails convenient properties on the theoretical side , the practitioner may have some difficulty to determine which setting/parameters correspond to her problem . Furthermore , this uncertainty raises the question of robustness with respect to these hypotheses . Several works have considered this question : Hadiji and Stoltz ( 2020 ) shows that adapting to an unknown bounded range requires a tradeoff between instance-dependent and worstcase regret , and recently ( Agrawal et al. , 2020 ; Ashutosh et al. , 2021 ) proved the impossibility of an instance-dependent logarithmic regret for light-tailed distributions without explicit control on the tail parameters . The root cause for this is the lack of compactness of such families F , which allows mass to `` leak '' at infinity so that maximally confusing distributions with mean µ∗ exist arbitrarily close to νk , meaning KFinf ( νk , µ∗ ) = 0 . The latter work also introduces a robust variant of UCB , that trades off logarithmic regret for O ( f ( T ) log ( T ) ) , where f essentially tracks the possible mass leakage at infinity . These results puts into question the usual hypotheses under which bandit algorithms are designed : considering a parametric control of the tails is indeed sensitive to model mis-specification , but on the other hand the examples chosen to prove infeasability results seem a bit extreme for the practitioner . In this paper , we propose simple alternative setups allowing unspecified tail shapes 1That is , for each bandit on F , for each arm k with ∆k > 0 , then E [ Nk ( T ) ] =o ( Tα ) for all α∈ ( 0 , 1 ] . but avoiding `` mass leakage '' to infinity , for instance with mild conditions linking the quantiles and the means of the distributions . We consider in this paper light-tailed distributions ( see definition in Appendix A.1 ) . This problem is already non-trivial , so we let possible extensions for heavy-tail distributions for future work ( e.g with tools like median-of-means , see ( Bubeck et al. , 2013 ) ) . Outline In the novel settings we consider , we want algorithms that require the smallest level of knowledge on the tails of distributions . To this extent , the Non-Parametric Thompson Sampling ( NPTS , Riou and Honda ( 2020 ) ) algorithm is a good candidate , considering how little knowledge it requires to reach asymptotic optimality for bounded distributions with known bounds . Furthermore , the flexibility of this algorithm has been recently demonstrated with its adaptation in a risk-aware setting ( Baudry et al. , 2021a ) . We provide a generalization of NPTS that we call Dirichlet Sampling ( DS ) : we combine the core elements of NPTS and a duel-based framework inspired by ( Chan , 2020 ) , introducing data-dependent exploration bonuses . We present the resulting algorithm and detail the technical motivations of this approach in Section 2 . We then introduce in Section 3 a first regret decomposition of DS algorithms under general assumptions , and the technical results that allow to fine-tune the algorithm for different families ( see Section 3.1 ) . We provide three instances of DS algorithms and their regret guarantees in Section 3.2 : Bounded Dirichlet Sampling ( BDS ) tackles bounded distributions with possibly unknown upper bounds , Quantile Dirichlet Sampling proposes a first generalization to the unbounded case using truncated distributions . Last , Robust Dirichlet Sampling ( RDS ) has a slightly larger than logarithmic regret for any unspecified light-tailed unbounded distributions , making it a competitor to the Robust-UCB algorithm of Ashutosh et al . ( 2021 ) . Finally , we study in Section 4 a use-case in agriculture using the DSSAT simulator ( see Hoogenboom et al . ( 2019 ) ) , which naturally faces all the questions ( robustness , model specification ) that motivate this work and shows the merit of DS over state-of-the-art methods for this problem . 2 Dirichlet Sampling Algorithms . In this section we introduce Dirichlet Sampling , a strategy that aims at generalizing the NonParametric Thompson Sampling algorithm of Riou and Honda ( 2020 ) outside the scope of bounded distributions with a known support upper bound . For this purpose , we build an adaptive strategy in a duel-based framework , already used in sub-sampling based algorithms like SSMC ( Chan , 2020 ) . Background Non-Parametric Thompson Sampling is an index strategy where the index of each arm is a random re-weighting of their observations , augmented by an exploration bonus . The weights are drawn from the Dirichlet distribution Dn = Dir ( ( 1 , . . . , 1 ) ) for n data , which is the uniform distribution on the simplex Pn = { w ∈ [ 0 , 1 ] n : wt1 = 1 } and matches the Bayesian posterior ( i.e Thompson Sampling ) for multinomial arms . The exploration bonus is simply the known upper bound of the support , and avoids under-exploration of potentially `` unlucky '' good arm . We provide further explanations on the Dirichlet distribution and NPTS respectively in Appendix C.1 and A.3 . The simplicity of NPTS and its strong theoretical guarantees are appealing for further generalization . As we fully depart from the Bayesian approach , considering other exploration bonuses , we derive a new family of algorithms under the name of Dirichlet Sampling . We keep the two principles of re-weighting the observations using a Dirichlet distribution and the exploration aid , and explore how to apply them to more general ( e.g unbounded ) distributions . In particular , we allow in DS some pre-processing of the observations before re-weighting ( see section 3.1 and 3.2 ) and motivate in Section 3.1 the use of a data-dependent bonus , that use information from several arms . The complexity introduced by such bonus in the analysis requires a change of algorithm structure , dropping the index policy for a leader vs challenger approach ( Chan , 2020 ) . Round-based algorithm We define a round as a step of the algorithm at the end of which a set of ( possibly several ) arms are selected to be pulled . Let Ar ⊂ { 1 , . . . , K } be the subset of the arms pulled at the beginning of a round r , we call T -round regret the quantity RT = E [ T∑ r=1 K∑ k=1 ∆k1 ( k ∈ Ar ) ] = K∑ k=1 ∆kE [ Nk ( T ) ] , ( 3 ) where we slightly change the definition of Nk ( compared with 1 ) to Nk ( T ) = ∑T r=1 1 ( k ∈ Ar ) . We consider the T -round regret for simplicity , as it is a simple upper bound of the regret after T pulls . At the beginning of each round we define a reference arm ( leader ) , and then organize pairwise comparisons called duels between this arm and the other arms ( challengers ) . The leader is chosen as the arm with largest sample size , ` r ∈ argmax k∈ { 1 , ... , K } Nk ( r ) , where ties are broken first in favor of the best empirical arm , then with a random choice . A major motivation for this choice is that the leader will have a sample size that is linear in the number of rounds , as at least one arm is chosen at each round . This ensures strong statistical properties that we will exploit to design the exploration bonus of DS strategies . Randomizing the index of the leader is also unnecessary : it competes against each challenger with its empirical mean . We also dismiss all the arms k that satisfy Nk ( r ) = N ` ( r ) with the same argument . These choices have a practical interest as they avoid the computation time of drawing the largest weight vectors . We believe this can be an alternative of independent interest to computationally intensive index policies . Challenger ’ s index We fix an index that is not dependent on the round , but only on the history of the challenger and the leader available at this round , that we denote respectively by X = ( X1 , . . . , Xn ) , Y = ( Y1 , . . . , YN ) for simplicity of notations . We denote by µ : RN → R the function that computes the average of a set of observations . The duel can includes two steps , and the challenger wins if 1. µ ( X ) ≥ µ ( Y ) ( first compare the empirical means ) , or 2. µ̃ ( X , Y ) ≥ µ ( Y ) , where µ̃ : RN × RN → R denotes the chosen DS index . We summarize in Algorithm 1 the steps of Dirichlet Sampling , that we completely detail in Appendix A.2 . We write it for a generic `` Dirichlet Sampling index '' µ̃ that must be computed by a re-weighting of the observations augmented by an exploration bonus . As in NPTS , the weights are drawn with a Dirichlet distribution . For instance , a canonical example of Dirichlet Sampling index with a data-depend ( instead of fixed ) bonus B ( X , Y ) is µ̃ ( X , Y ) = n∑ i=1 wiXi + wn+1B ( X , Y ) , w = ( w1 , . . . , wn+1 ) ∼ Dn+1 . However , the algorithm structure in Algorithm 1 could be combined with any randomized index , which is of independent interest as we will see in Section 3 . In the next section we study the theoretical properties of Dirichlet Sampling , and discuss the choice of the index µ̃ for different families of distributions . Algorithm 1 Generic Dirichlet Sampling Input : K arms , horizon T , Dirichlet Sampling index µ̃ Init . : t = 1 , r = 1 , ∀k ∈ { 1 , ... , K } : Xk = { Xk1 } , Nk = 1 ; . Draw each arm once while t < T do A = { } ; . Arm ( s ) to pull at the end of the round ` = Leader ( ( X1 , N1 ) , . . . , ( Xk , Nk ) ) ; . Choose a Leader for k ∈ { 1 , . . . , K } : Nk < N ` do if max ( µ ( Xk ) , µ̃ ( Xk , X ` ) ) ≥ µ ( X ` ) then A = A ∪ { k } ; . Play the duels Draw arms from |A| if A is non-empty , else draw arm ` . Update t , r , ( Nk ) k∈ { 1 , ... , K } , ( Xk ) k∈ { 1 , ... , K } . ; . Collect Reward ( s ) and update data

Summary: This paper considers robustness of MAB algorithm to model misspecification when knowledge on the distribution (e.g., tails) is not directly accessible. They study a generic Dirichlet Sampling algorithm and provide a generic theorem for regret decomposition. They applied the algorithm in three examples: 1) bounded distributions where optimal regret can be achieved, 2) light-tailed unbounded distributions where we can achieve slightly-worse-than-log regret through simple tuning strategies, and 3) unbounded distributions with some quantile condition where log regret is achievable.