From Optimality to Robustness: Adaptive Re-Sampling Strategies in Stochastic Bandits

The stochastic multi-arm bandit problem has been extensively studied under standard assumptions on the arm ’ s distribution ( e.g bounded with known support , exponential family , etc ) . These assumptions are suitable for many real-world problems but sometimes they require knowledge ( on tails for instance ) that may not be precisely accessible to the practitioner , raising the question of the robustness of bandit algorithms to model misspecification . In this paper we study a generic Dirichlet Sampling ( DS ) algorithm , based on pairwise comparisons of empirical indices computed with re-sampling of the arms ’ observations and a data-dependent exploration bonus . We show that different variants of this strategy achieve provably optimal regret guarantees when the distributions are bounded and logarithmic regret for semi-bounded distributions with a mild quantile condition . We also show that a simple tuning achieve robustness with respect to a large class of unbounded distributions , at the cost of slightly worse than logarithmic asymptotic regret . We finally provide numerical experiments showing the merits of DS in a decision-making problem on synthetic agriculture data . 1 Introduction . The K-armed stochastic bandit model is a decision-making problem in which a learner sequentially picks an action among K alternatives , called arms , and collects a random reward . In this setting , all rewards drawn from an arm are independent and identically distributed . Hence , we can formally associate each arm k ∈ { 1 , . . . , K } with its reward distribution νk , with mean µk . The objective of the learner is to adapt her strategy ( At ) t∈ [ T ] in order to maximize the expected sum of rewards obtained after T selections ( where T is the horizon , unknown to the learner ) . This is equivalent to minimizing the regret , defined as the difference between the expected total reward of an oracle strategy always selecting an arm with largest mean and that of the algorithm , which is equal to RT = E [ T∑ t=1 µ ? − µAt ] = K∑ k=1 ∆kE [ Nk ( T ) ] . ( 1 ) Here , Nk ( T ) = ∑T t=1 1 ( At = k ) denotes the number of selections of arm k after T time steps , µ ? = maxj∈ { 1 , ... , K } µj and ∆k = µ ? −µk is called the gap between arm k and the largest mean . To assess the performance of a bandit algorithm , one naturally studies the best guarantees achievable by a uniformly efficient algorithm , i.e with sub-linear regret on any instance of a given class of problems . This guarantee was first provided by Lai and Robbins ( 1985 ) for 1-dimensional parametric families of distributions , and then extended by Burnetas and Katehakis ( 1996 ) for more general families . It 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . states that any algorithm that is uniformly efficient1 on a family of distributions F must satisfy lim inf T→∞ RT log ( T ) ≥ ∑ k : ∆k > 0 ∆k KFinf ( νk , µ ? ) , KFinf ( νk , µ ? ) = inf G∈F { KL ( νk , G ) : EG ( X ) > µ ? } . ( 2 ) A bandit algorithm is then called asymptotically optimal for a family of distributions F when its regret matches this lower bound . When F is a Single-Parameter Exponential Family ( SPEF ) , KFinf is simply the Kullback-Leibler divergence between the distribution of mean µk and that of mean µ ? in F , making for a theoretically appealing setting . The quantity KBinf , corresponding to the family F [ −∞ , B ] of distributions supported in ( −∞ , B ] is also often considered in the literature , see e.g ( Honda and Takemura , 2010 , 2015 ; Cappé et al. , 2013 ) . Overview of existing strategies An efficient strategy faces the classical exploration/exploitation dilemma : it needs to obtain enough information from arms that have not been sampled a lot ( exploration ) , but also to sample arms that are well-performing sufficiently often ( exploitation ) . Many algorithms have been proposed for the multi-armed bandits problem ( see Lattimore and Szepesvári ( 2020 ) for a survey ) , and we propose in the following a non-exhaustive list of such methods . A first category contains the deterministic index policies , built on the concept of Optimism in Face of Uncertainty , the most celebrated of which being the Upper Confidence Bound ( UCB ) algorithms ( Agrawal , 1995 ; Auer et al. , 2002 ) . These algorithms can obtain a logarithmic regret under classical hypothesis on the distributions ( e.g bounded , sub-gaussian , sub-exponential , . . . ) , and the strongest guarantees have been achieved by kl-UCB Cappé et al . ( 2013 ) , DMED ( Honda and Takemura , 2010 ) , and IMED ( Honda and Takemura , 2015 ) , which share a common pattern of solving a convex optimization problem at each round . To be asymptotically optimal , these algorithms require either 1 ) the knowledge of a specific SPEF for each arm , or 2 ) a known upper bound on the support of each arm . A second general category is that of randomized bandit algorithms , which has been formulated for instance in ( Kveton et al. , 2019b ) as General Randomized Exploration ( GRE ) . The common feature of these methods is that , at each time step and for each arm , the algorithm draws an index from a distribution that depends on 1 ) the rewards observed from the arm , and 2 ) some knowledge on the arms distributions and chooses the arm with the largest index . Thompson Sampling ( TS ) ( Thompson , 1933 ; Agrawal and Goyal , 2012 ) belongs to this category , and a proper choice of Bayesian prior/posterior ensures optimality of TS in SPEF ( Korda et al. , 2013 ) . Different algorithms using Bootstrapping schemes have also been proposed ( Osband and Roy , 2015 ; Kveton et al. , 2019a , b ; Wang et al. , 2020 ; Riou and Honda , 2020 ) : they share the idea of computing a noisy mean for empirical samples , enhanced by some exploration aid appropriately tuned to the family of distributions they consider . A last category contains the methods based on sub-sampling Baransi et al . ( 2014 ) ; Chan ( 2020 ) ; Baudry et al . ( 2020 , 2021b ) , that achieve asymptotic optimality in SPEF without knowing which family , when all arms share the same . However the proofs heavily rely on properties of the tails of SPEF so the results seem difficult to generalize outside these families . Motivations While many algorithms achieve optimal regret for bounded distributions with the sole knowledge of the upper bound , the assumptions needed for algorithms working with unbounded distributions ( e.g SPEF , sub-Gaussian , sub-exponential ) generally assume a known parametric model for the tails . While such assumption entails convenient properties on the theoretical side , the practitioner may have some difficulty to determine which setting/parameters correspond to her problem . Furthermore , this uncertainty raises the question of robustness with respect to these hypotheses . Several works have considered this question : Hadiji and Stoltz ( 2020 ) shows that adapting to an unknown bounded range requires a tradeoff between instance-dependent and worstcase regret , and recently ( Agrawal et al. , 2020 ; Ashutosh et al. , 2021 ) proved the impossibility of an instance-dependent logarithmic regret for light-tailed distributions without explicit control on the tail parameters . The root cause for this is the lack of compactness of such families F , which allows mass to `` leak '' at infinity so that maximally confusing distributions with mean µ∗ exist arbitrarily close to νk , meaning KFinf ( νk , µ∗ ) = 0 . The latter work also introduces a robust variant of UCB , that trades off logarithmic regret for O ( f ( T ) log ( T ) ) , where f essentially tracks the possible mass leakage at infinity . These results puts into question the usual hypotheses under which bandit algorithms are designed : considering a parametric control of the tails is indeed sensitive to model mis-specification , but on the other hand the examples chosen to prove infeasability results seem a bit extreme for the practitioner . In this paper , we propose simple alternative setups allowing unspecified tail shapes 1That is , for each bandit on F , for each arm k with ∆k > 0 , then E [ Nk ( T ) ] =o ( Tα ) for all α∈ ( 0 , 1 ] . but avoiding `` mass leakage '' to infinity , for instance with mild conditions linking the quantiles and the means of the distributions . We consider in this paper light-tailed distributions ( see definition in Appendix A.1 ) . This problem is already non-trivial , so we let possible extensions for heavy-tail distributions for future work ( e.g with tools like median-of-means , see ( Bubeck et al. , 2013 ) ) . Outline In the novel settings we consider , we want algorithms that require the smallest level of knowledge on the tails of distributions . To this extent , the Non-Parametric Thompson Sampling ( NPTS , Riou and Honda ( 2020 ) ) algorithm is a good candidate , considering how little knowledge it requires to reach asymptotic optimality for bounded distributions with known bounds . Furthermore , the flexibility of this algorithm has been recently demonstrated with its adaptation in a risk-aware setting ( Baudry et al. , 2021a ) . We provide a generalization of NPTS that we call Dirichlet Sampling ( DS ) : we combine the core elements of NPTS and a duel-based framework inspired by ( Chan , 2020 ) , introducing data-dependent exploration bonuses . We present the resulting algorithm and detail the technical motivations of this approach in Section 2 . We then introduce in Section 3 a first regret decomposition of DS algorithms under general assumptions , and the technical results that allow to fine-tune the algorithm for different families ( see Section 3.1 ) . We provide three instances of DS algorithms and their regret guarantees in Section 3.2 : Bounded Dirichlet Sampling ( BDS ) tackles bounded distributions with possibly unknown upper bounds , Quantile Dirichlet Sampling proposes a first generalization to the unbounded case using truncated distributions . Last , Robust Dirichlet Sampling ( RDS ) has a slightly larger than logarithmic regret for any unspecified light-tailed unbounded distributions , making it a competitor to the Robust-UCB algorithm of Ashutosh et al . ( 2021 ) . Finally , we study in Section 4 a use-case in agriculture using the DSSAT simulator ( see Hoogenboom et al . ( 2019 ) ) , which naturally faces all the questions ( robustness , model specification ) that motivate this work and shows the merit of DS over state-of-the-art methods for this problem . 2 Dirichlet Sampling Algorithms . In this section we introduce Dirichlet Sampling , a strategy that aims at generalizing the NonParametric Thompson Sampling algorithm of Riou and Honda ( 2020 ) outside the scope of bounded distributions with a known support upper bound . For this purpose , we build an adaptive strategy in a duel-based framework , already used in sub-sampling based algorithms like SSMC ( Chan , 2020 ) . Background Non-Parametric Thompson Sampling is an index strategy where the index of each arm is a random re-weighting of their observations , augmented by an exploration bonus . The weights are drawn from the Dirichlet distribution Dn = Dir ( ( 1 , . . . , 1 ) ) for n data , which is the uniform distribution on the simplex Pn = { w ∈ [ 0 , 1 ] n : wt1 = 1 } and matches the Bayesian posterior ( i.e Thompson Sampling ) for multinomial arms . The exploration bonus is simply the known upper bound of the support , and avoids under-exploration of potentially `` unlucky '' good arm . We provide further explanations on the Dirichlet distribution and NPTS respectively in Appendix C.1 and A.3 . The simplicity of NPTS and its strong theoretical guarantees are appealing for further generalization . As we fully depart from the Bayesian approach , considering other exploration bonuses , we derive a new family of algorithms under the name of Dirichlet Sampling . We keep the two principles of re-weighting the observations using a Dirichlet distribution and the exploration aid , and explore how to apply them to more general ( e.g unbounded ) distributions . In particular , we allow in DS some pre-processing of the observations before re-weighting ( see section 3.1 and 3.2 ) and motivate in Section 3.1 the use of a data-dependent bonus , that use information from several arms . The complexity introduced by such bonus in the analysis requires a change of algorithm structure , dropping the index policy for a leader vs challenger approach ( Chan , 2020 ) . Round-based algorithm We define a round as a step of the algorithm at the end of which a set of ( possibly several ) arms are selected to be pulled . Let Ar ⊂ { 1 , . . . , K } be the subset of the arms pulled at the beginning of a round r , we call T -round regret the quantity RT = E [ T∑ r=1 K∑ k=1 ∆k1 ( k ∈ Ar ) ] = K∑ k=1 ∆kE [ Nk ( T ) ] , ( 3 ) where we slightly change the definition of Nk ( compared with 1 ) to Nk ( T ) = ∑T r=1 1 ( k ∈ Ar ) . We consider the T -round regret for simplicity , as it is a simple upper bound of the regret after T pulls . At the beginning of each round we define a reference arm ( leader ) , and then organize pairwise comparisons called duels between this arm and the other arms ( challengers ) . The leader is chosen as the arm with largest sample size , ` r ∈ argmax k∈ { 1 , ... , K } Nk ( r ) , where ties are broken first in favor of the best empirical arm , then with a random choice . A major motivation for this choice is that the leader will have a sample size that is linear in the number of rounds , as at least one arm is chosen at each round . This ensures strong statistical properties that we will exploit to design the exploration bonus of DS strategies . Randomizing the index of the leader is also unnecessary : it competes against each challenger with its empirical mean . We also dismiss all the arms k that satisfy Nk ( r ) = N ` ( r ) with the same argument . These choices have a practical interest as they avoid the computation time of drawing the largest weight vectors . We believe this can be an alternative of independent interest to computationally intensive index policies . Challenger ’ s index We fix an index that is not dependent on the round , but only on the history of the challenger and the leader available at this round , that we denote respectively by X = ( X1 , . . . , Xn ) , Y = ( Y1 , . . . , YN ) for simplicity of notations . We denote by µ : RN → R the function that computes the average of a set of observations . The duel can includes two steps , and the challenger wins if 1. µ ( X ) ≥ µ ( Y ) ( first compare the empirical means ) , or 2. µ̃ ( X , Y ) ≥ µ ( Y ) , where µ̃ : RN × RN → R denotes the chosen DS index . We summarize in Algorithm 1 the steps of Dirichlet Sampling , that we completely detail in Appendix A.2 . We write it for a generic `` Dirichlet Sampling index '' µ̃ that must be computed by a re-weighting of the observations augmented by an exploration bonus . As in NPTS , the weights are drawn with a Dirichlet distribution . For instance , a canonical example of Dirichlet Sampling index with a data-depend ( instead of fixed ) bonus B ( X , Y ) is µ̃ ( X , Y ) = n∑ i=1 wiXi + wn+1B ( X , Y ) , w = ( w1 , . . . , wn+1 ) ∼ Dn+1 . However , the algorithm structure in Algorithm 1 could be combined with any randomized index , which is of independent interest as we will see in Section 3 . In the next section we study the theoretical properties of Dirichlet Sampling , and discuss the choice of the index µ̃ for different families of distributions . Algorithm 1 Generic Dirichlet Sampling Input : K arms , horizon T , Dirichlet Sampling index µ̃ Init . : t = 1 , r = 1 , ∀k ∈ { 1 , ... , K } : Xk = { Xk1 } , Nk = 1 ; . Draw each arm once while t < T do A = { } ; . Arm ( s ) to pull at the end of the round ` = Leader ( ( X1 , N1 ) , . . . , ( Xk , Nk ) ) ; . Choose a Leader for k ∈ { 1 , . . . , K } : Nk < N ` do if max ( µ ( Xk ) , µ̃ ( Xk , X ` ) ) ≥ µ ( X ` ) then A = A ∪ { k } ; . Play the duels Draw arms from |A| if A is non-empty , else draw arm ` . Update t , r , ( Nk ) k∈ { 1 , ... , K } , ( Xk ) k∈ { 1 , ... , K } . ; . Collect Reward ( s ) and update data

Summary: The goal of this paper is to construct a sequential decision-making algorithm under non-standard distributional assumptions. The authors propose a Dirichlet Sampling algorithm, employing pairwise comparisons of indices computed with the arms’ observations and a data-dependent exploration bonus.  They suggest three specific variants of this strategy under different distributional assumptions having distinct tail behaviors.  When it is applied to the bounded distributions, the algorithm is shown to be optimal. Numerical studies display some advantages of the proposed algorithms.