Stochastic bandits with groups of similar arms.

1 Introduction . The finite stochastic multi-armed bandit problem is a popular framework for studying sequential decision making problems in which a learner sequentially samples from a finite set of distributions called arms . It was first introduced in the context of medical trials [ Thompson , 1933b , 1935 ] and later formalized under this name by Robbins in Robbins [ 1952 ] . We refer the interested reader to Lattimore and Szepesvári [ 2020 ] for a recent survey . This is one of the simplest theoretical framework in which one can study the notion of exploration-exploitation tradeoff . This tension between exploration and exploitation arises from the sequential optimization problem a learner is trying to perform while being uncertain about the very problem it is optimizing . Formally , a multi-armed bandit configuration is specified by a set of unknown real-valued probability distributions ν= ( νa ) a∈A with means ( µa ) a∈A , indexed by a set of arms A . We hereafter consider a finite A , and that all νa , a∈A belong to the same family of distributions F ( e.g . Bernoulli , Gaussian , etc . ) , that is ν ∈ FA . The bandit game proceeds as follows . At each time t∈N , the learner chooses an arm at ∈A based on the past observations and decisions , then receives and observes a sample Xt ( called the reward ) , conditionally independent , sampled from νat . Her goal is to maximize the cumulative reward received over time . The mean of each arm is unknown , which makes the problem non-trivial , hence the learner should adjust her sampling strategy based on past information obtained from drawing different arms in order to maximize the expected sum of rewards . The maximal expected value of a finite bandit configuration is denoted by µ ? , defined as µ ? = max a∈A µa . The performance of the strategy used by the agent is measured by the ( pseudo ) regret , that compares the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . expected sum of rewards obtained by an oracle that would constantly pull an optimal arm and the ones obtained by the learner , up to some time horizon T ( that we assume is unknown to the learner ) . Definition 1 ( Regret ) . The regret incurred by a sampling strategy after T time steps on a bandit configuration ν is given by : R ( ν , T ) = Eν ( T∑ t=1 ( µ∗ − µat ) ) = ∑ a∈A ( µ∗ − µa ) Eν ( Na ( T ) ) , where Na ( T ) = T∑ t=1 I { at = a } denotes the number of selection of arm a after T time steps . Group of similar arms Motivated by various practical reasons , one may want to restrict to a subset B ⊂ FA of allowed bandit configurations instead of the full set FA . In this paper , we study a variant of the multi-armed bandit problem in which the reward function , µ : a ∈ A → µa , is assumed to satisfy a cluster-like structural property . A bandit configuration ν is said to satisfy the q-equivalence property if for every arm a ∈ A , there are at least q−1 distinct arms having the same expected value : ∀a ∈ A , | { a′ ∈ A|µa′ = µa } | > q . Assuming the set of arms A and base distributions D are known to the learner , we denote by Bq the set of bandit configurations having the q-equivalence property . We also denote by Bq ( µ ) the set of all expected values in Bq . Formally , Bq ( µ ) is the image of Bq under the µ mapping . Definition 2 ( Arm equivalence and equivalence class ) . Given a bandit configuration ν , two arms a , a′ ∈ A are said to be equivalent if their associated distributions have the same expected values : a ∼ a′ ⇔ µa = µa′ . An equivalence class c in ν is a maximal subset of arms in A having the same mean , i.e. , for all arms a , a′ in c , µa = µa′ and for all arm a ∈ c and a′ ∈ A \ c , µa 6= µa′ . This situation typically appears in practical situations when each arm can be described with a list of categorical attributes , and the ( unknown ) mean reward function only depends on a subset of them , the others being redundant . In this case , q is naturally linked to the number of attributes considered redundant ( or useless descriptors ) , and the number of categories of each attribute . Precisely , q = ∏ i∈R ci whereR is the set of redundant attributes and ci the number of categories for attribute i . The learner may know that there exists such a structure while not knowing a closed form formula mapping the list of categorical attributes to the significant subset . In this case , q might be a lower bound on the sizes of the class since the setR might not be the largest possible one or because the number of redundant attributes depends on the number of relevant attributes . In all cases , the smallest possible number of redundant attributes can be naturally linked to q . We hereafter consider the learner only knows q but would like to exploit the prior knowledge of this structure in a bandit problem . Regret lower bounds overview In order to assess the performance of a bandit algorithm on a set of configurations B , one naturally studies the best guarantee achievable by a uniformly efficient algorithm on B , i.e with sub-linear regret on any instance ν ∈ B of the bandit problem . When B = FA , such a guarantee was first provided by Lai and Robbins [ 1985 ] for parametric families F , and then extended by Burnetas and Katehakis [ 1996 ] for more general families . It states that any algorithm that is uniformly efficient1 on a family of distributions F must satisfy lim inf T→∞ R ( ν , T ) log ( T ) > ∑ a : µ ? > µa0 µ ? − µa KF ( νa‖µ ? ) , KF ( νa‖µ ? ) = inf G∈F { KL ( νa‖G ) : EG ( X ) > µ ? } . ( 1 ) This popular result entails that any strategy having the desirable property to have sub-linear regret on any instance in F must incur a non-trivial minimal regret . When B is a strict subset of FA , the bandit problem is called structured , as in this case pulling an arm may reveal information that makes it possible to refine estimation of other arms ( e.g . think of the set of bandit configurations having Lipschitz mean function with respect to A⊂Rd ) . The presence of structure may considerably modify the achievable lowest regret , as shown in Burnetas and Katehakis [ 1996 ] and Graves and Lai 1Formally , for each bandit on F , for each arm k with ∆k > 0 , then E [ Nk ( T ) ] =o ( Tα ) for all α∈ ( 0 , 1 ] . [ 1997 ] , who extended the ( unstructured ) lower bounds to arbitrarily structured bandit problems ( and beyond ) . These lower bound take the generic form lim infT→∞ R ( ν , T ) log ( T ) > CB ( ν ) , where CB ( ν ) is a constant term solution of a constrained linear-optimization problem . A bandit algorithm is then called asymptotically optimal for a set B when its regret asymptotically matches this lower bound . Existing strategies In order to minimize the regret , a learner faces the classical exploration/exploitation dilemma : it needs to balance exploration , that is gaining information about the expected values of the arms by sampling them , and exploitation , that is playing the most promising arm sufficiently often . Many algorithms have been proposed to solve the multi-armed bandits problem ( see Lattimore and Szepesvári [ 2020 ] for a survey ) . The study of the lower bounds had a crucial impact on the development of provably asymptotically optimal strategies . In the case of unstructured bandit B = FA , this includes strategies that build on the concept of Optimism in Face of Uncertainty ( the most celebrated of which being the Upper Confidence Bound ( UCB ) algorithms Agrawal [ 1995 ] , Auer et al . [ 2002 ] ) , such as KLUCB [ Lai , 1987 , Cappé et al. , 2013 , Maillard , 2018 ] , DMED and IMED Honda and Takemura [ 2011 , 2015 ] , that are proven asymptotically optimal for various families F ( e.g . one-dimensional exponential families ) , and directly exploit the lower bound in their structure . Alternative asymptotically optimal strategies include the Thompson Sampling ( TS ) Thompson [ 1933a ] , Agrawal and Goyal [ 2012 ] , which uses a Bayesian posterior distribution given a specific prior , whose optimality was shown in Korda et al . [ 2013 ] . See also Kveton et al . [ 2019 ] for other randomized algorithms and Kveton et al . [ 2020 ] , Chan [ 2020 ] , Baudry et al . [ 2020 ] for recent non-parametric extensions using re-sampling methods . Further , some authors also allow many optimal arms , see de Heide et al . [ 2021 ] , or even countably many arms , see Kalvit and Zeevi [ 2020 ] . However , these works do not consider nor exploit a constraint on the level-sets of the mean function and follow an optimistic paradigm while we follow an information minimization targeting optimality . On the other hand , several instances of structured bandits received considerable attention in the last few years . This is the case for instance of linear bandits , see [ Abbasi-Yadkori et al. , 2011 , Srinivas et al. , 2010 , Durand et al. , 2017 , Kveton et al. , 2020 ] and Lattimore and Szepesvari [ 2017 ] , Lipschitz bandits Magureanu et al . [ 2014 ] , Wang et al . [ 2020 ] , Lu et al . [ 2019 ] , unimodal bandits Yu and Mannor [ 2011 ] , Combes and Proutiere [ 2014 ] , Saber et al . [ 2020 ] , or combinatorial bandits Kveton et al . [ 2015 ] , Magureanu [ 2018 ] , and more recently Cuvelier et al . [ 2021b ] . A generic asymptotically optimal algorithm , called OSSB ( Optimal Structured Stochastic Bandit ) , has been introduced in the work of Combes et al . [ 2017 ] , and proven to be asymptotically optimal for all structures satisfying some weak properties that include all the aforementioned structures . Although being asymptotically optimal this algorithm often suffers from a long burn-in phase that may hinder its finite practical performance . It further comes with high computational price as it requires to solve an empirical version of the optimization problem CB ( ν ) at each step . This motivates the quest for alternative strategies , perhaps less generic but better suited to a specific structure . Inspired by combinatorial structures for which computing CD ( ν ) is simply not feasible , a relaxation of the generic constrained optimization problem was recently proposed in Cuvelier et al . [ 2021a ] . The authors show that this comes at the price of trading-off regret optimality for computational efficiency . Indeed in some structure , combinatorial properties are at stake and asymptotically optimal algorithms may require solving combinatorial optimization problems ( see Cuvelier et al . [ 2021a ] ) related to CB ( ν ) . In order to exploit the combinatorial structures in a numerically efficient way , research has been made in how to relax these combinatorial optimization problems while preserving theoretical properties on the regret of the relaxed algorithms ( see Cuvelier et al . [ 2021b , a ] ) . Our work consider similar computational issues , with a different perspective . Goal For the structure Bq , as we show in Theorem 1 below , the term CBq ( ν ) unfortunately makes appear in general a combinatorial optimization problem . This makes resorting to OSSB or any strategy targeting exact asymptotic optimality a daunting task for the practitioner . In this paper , our goal is to provide a computationally efficient strategy adapted to the structure Bq , that is able to reach optimality up to controlled error term . Outline and contributions The rest of this paper is organized as follows . In section 2 , we derive a lower bound on the regret for the structured set of bandit configurations Bq . This bound makes appear two components , one that we call non-combinatorial as optimizing it can be done efficiently , and a second term that we term combinatorial as it involves solving a combinatorial problem . Interestingly , using in Lemma 1 and Theorem 3 that the contribution of the combinatorial part of the lower bound can be controlled . Owing to this key insight , we introduce in section 3 , IMED-EC , an adaptation of the IMED strategy from Honda and Takemura [ 2015 ] to the structured set Bq . One advantage of IMED over a KL-UCB alternative is its reduced complexity , which translates to the equivalence class setup . At each time step , the complexity of computing the next arm to be pulled by IMED-EC is no more than the one of sorting a list of |A| elements once the IMED indexes have been computed , which is only log |A| times larger than looking for the minimal IMED index . In Section 4 , we prove that IMED-EC achieves a controlled asymptotic regret that matches the non-combinatorial part of the lower bound and is at most ( less than ) a factor of 2 times the optimal regret bound . Last , we illustrate the benefit of the IMED-EC over its unstructured version in section 5 , where it shows a substantial improvement . Our experiments also highlights the robustness of the algorithm to a misspecified parameter q , which is a desirable feature for the practitioner .

Summary: The paper studies a version of the stochastic multi-armed bandit problem where arm-means have a multiplicity of at least $q$, meaning, for any given arm "a," there exist at least $q-1$ other (distinct) arms with a mean reward same as that of "a." The parameter $q$ is assumed known. The reward distributions are assumed to belong to a one-dimensional exponential family. Under this premise, the authors show refined lower bounds for the expected sampling times of inferior arms (also regret) that exhibit a $q^{-1}$-scaling. Additionally, leveraging the underlying problem structure, a refinement of the KL-divergence-based terms appearing in the lower bound is also shown. The authors propose a modification of the IMED algorithm that suffers a regret within a multiplicative "control" factor of the lower bound. All in all, the takeaway is that the achievable regret is reduced by a factor of $q$. The reported numerical experiments indicate robustness to misspecification of $q$.