Indexed Minimum Empirical Divergence for Unimodal Bandits

1 Introduction . The multi-armed bandit problem is a popular framework to formalize sequential decision making problems . It was first introduced in the context of medical trials [ Thompson , 1933 , 1935 ] and later formalized by Robbins [ 1952 ] : A bandit is specified by a configuration , that is a set of unknown probability distributions , ν= ( νa ) a∈A with means ( µa ) a∈A . At each time t∈N , the learner chooses an arm at∈A , based only on the past , the learner then receives and observes a reward Xt , conditionally independent , sampled according to νat . The goal of the learner is to maximize the expected sum of rewards received over time ( up to some unknown horizon T ) , or equivalently minimize the regret with respect to the algorithm constantly receiving the highest mean reward R ( ν , T ) = Eν [ T∑ t=1 µ ? −Xt ] where µ ? = max a∈A µa . Both means and distributions are unknown , which makes the problem non trivial , and the learner only knows that ν ∈ D where D is a given set of bandit configurations . This problem received increased attention in the middle of the 20th century , and the seminal paper Lai and Robbins [ 1985 ] established the first lower bound on the cumulative regret , showing that designing an algorithm that is optimal uniformly over a given set of configurations D comes with a price . The study of the lower performance bounds in multi-armed bandits successfully lead to the development of asymptotically optimal algorithms for specific configuration sets , such as the KLUCB algorithm [ Lai , 1987 , Cappé et al. , 2013 , Maillard , 2018 ] for exponential families , or alternatively the DMED and IMED algorithms from Honda and Takemura [ 2011 , 2015 ] . The lower bounds from Lai and Robbins [ 1985 ] , later extended by Burnetas and Katehakis [ 1997 ] did not cover all possible configurations , and in particular structured configuration sets were not handled until Agrawal et al . [ 1989 ] and then Graves and Lai [ 1997 ] established generic lower bounds . Here , structure refers to the fact that pulling an arm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . may reveals information that enables to refine estimation of other arms . Unfortunately , designing numerical efficient algorithms that are provably optimal remains a challenge for many structures . Structured configurations . Motivated by the growing popularity of bandits in a number of industrial and societal application domains , the study of structured configuration sets has received increasing attention over the last few years : The linear bandit problem is one typical illustration Abbasi-Yadkori et al . [ 2011 ] , Srinivas et al . [ 2010 ] , Durand et al . [ 2017 ] , for which the linear structure considerably modifies the achievable lower bound , see Lattimore and Szepesvari [ 2017 ] . The study of a unimodal structure naturally appears in many contexts , e.g . single-peak preference economics , voting theory or wireless communications , and has been first considered in Yu and Mannor [ 2011 ] from a bandit perspective , then in Combes and Proutiere [ 2014 ] and Trinh et al . [ 2020 ] providing an explicit lower bound together with an algorithm exploiting this specific structure . Other structures include Lipschitz bandits Magureanu et al . [ 2014 ] , and we refer to the manuscript Magureanu [ 2018 ] for other examples , such as cascading bandits that are useful in the context of recommender systems . In Combes et al . [ 2017 ] , a generic algorithm is introduced called OSSB ( Optimal Structured Stochastic Bandit ) , stepping the path towards generic multi-armed bandit algorithms that are adaptive to a given structure . More recently in Degenne et al . [ 2020 ] , the authors introduce an adaptation of the KLUCB strategy to handle structured multi-armed bandit problems . Unimodal-structure . We assume a unimodal structure similar to that considered in Yu and Mannor [ 2011 ] and Combes and Proutiere [ 2014 ] . That is , there exists an undirected graph G = ( A , E ) whose vertices are arms A , and whose edges E characterize a partial order among means ( µa ) a∈A . This partial order is assumed unknown to the learner . We assume that there exists a unique optimal arm a ? = argmaxa∈A µa and that for all sub-optimal arm a 6= a ? , there exists a path Pa = ( a1 = a , . . . , a ` a = a ? ) ∈ A ` a of length ` a > 2 such that for all i ∈ [ 1 , ` a − 1 ] , ( ai , ai+1 ) ∈ E and µai < µai+1 . Lastly , we assume that ν⊂P : = { p ( µ ) , µ∈Θ } , where p ( µ ) is an exponential-family distribution probability with density f ( · , µ ) with respect to some positive measure λ on R and mean µ∈Θ⊂R . P is assumed to be known to the learner . Thus , for all a∈A we have νa=p ( µa ) . We denote by D ( P , G ) or simply D the structured set of such unimodal-bandit distributions characterized by ( P , G ) . In the following , we assume that P is a set of one-dimensional exponential family distributions . Contributions . In this paper , we provide novel regret minimization results related to the unimodal structure . We first revisit the Indexed Minimum Empirical Divergence ( IMED ) algorithm from Honda and Takemura [ 2015 ] introduced for unstructured multi-armed bandits , and adapt it to the unimodalstructured setting . We introduce in Section 3 the IMED-UB algorithm that is limited to the pulling of the current best arm or their no more than d nearest arms at each time step , with d the maximum degree of nodes in G. Being constructed from IMED , IMED-UB does not require any optimization procedure and does not separate exploration from exploitation rounds . IMED-UB appears to be a local algorithm . We prove in Theorem 6 that IMED-UB is asymptotically optimal . Furthermore , this novel algorithm competes with the state-of-the-art algorithms in practice . This is confirmed by numerical illustrations on synthetic data . We believe that the construction of this algorithm together with the proof techniques developed in this paper are of independent interest for the bandit community . Notations . Let ν∈D . Let µ ? =maxa∈A µa be the optimal mean and a ? =argmaxa∈A µa be the optimal arm of ν . We define for an arm a∈A its sub-optimality gap ∆a=µ ? −µa . Considering an horizon T > 1 , thanks to the chain rule we can rewrite the regret as follows : R ( ν , T ) = ∑ a∈A ∆a Eν [ Na ( T ) ] , ( 1 ) where Na ( t ) = ∑t s=1 I { as=a } is the number of pulls of arm a at time t . 2 Regret lower bound . In this subsection , we recall for completeness the known lower bound on the regret when we assume a unimodal structure . In order to obtain non trivial lower bound we consider algorithms that are consistent ( aka uniformly-good ) . Definition 1 ( Consistent algorithm ) An algorithm is consistent on D if for all configuration ν ∈ D , for all sub-optimal arm a , for all α > 0 , lim T→∞ Eν [ Na ( T ) Tα ] = 0 . We can derive from the notion of consistency an asymptotic lower bound on the regret , see Combes and Proutiere [ 2014 ] . Proposition 2 ( Lower bounds on the regret ) Let us consider a consistent algorithm . Then , for all configuration ν∈D , it must be that lim inf T→∞ R ( ν , T ) log ( T ) > c ( ν ) : = ∑ a∈Va ? ∆a KL ( µa|µ ? ) , where KL ( µ|µ′ ) = ∫ Rlog ( f ( x , µ ) /f ( x , µ ′ ) ) f ( x , µ ) λ ( dx ) denotes the Kullback-Leibler divergence between ν=p ( µ ) and ν′=p ( µ′ ) , for µ , µ′∈Θ . Remark 3 The quantity c ( ν ) is a fully explicit function of ν ( it does not require solving any optimization problem ) for some set of distributions ν ( see Remark 4 ) . This useful property no longer holds in general for arbitrary structures . Also , it is noticeable that c ( ν ) does not involve all the suboptimal arms but only the ones in Va ? . This indicates that sub-optimal arms outside Va ? are sampled o ( log ( T ) ) , which contrasts with the unstructured stochastic multi-armed bandits . See Combes and Proutiere [ 2014 ] for further insights . Remark 4 For Bernoulli distributions , a possible setting is to assume λ = δ0 + δ1 ( with δ0 , δ1 Dirac measures ) , Θ = ( 0 , 1 ) and for µ ∈Θ , f ( · , µ ) = : x ∈ { 0 , 1 } 7→ µx ( 1 − µ ) 1−x . Then for all µ , µ′ ∈ ( 0 , 1 ) , KL ( µ|µ′ ) =µ log ( µ/µ′ ) + ( 1−µ ) log ( ( 1−µ ) / ( 1−µ′ ) ) . For Gaussian distributions ( variance σ2 =1 ) , we assume λ to be the Lebesgue measure , Θ=R , and for µ∈R , f ( · , µ ) = : x∈R 7→ ( √ 2π ) −1e− ( x−µ ) 2/2 . Then for all µ , µ′∈R , KL ( µ|µ′ ) = ( µ′−µ ) 2/2 . For Exponential distributions , we assume λ to be the Lebesgue measure , Θ= ] 0 ; +∞ [ , and for µ > 0 , f ( · , µ ) = : x > 0 7→ e−x/µ/µ . Then for all µ , µ′ > 0 , KL ( µ|µ′ ) =log ( µ′/µ ) +µ/µ′−1 . 3 Optimal algorithm for unimodal-structured bandits . We present in this section a novel algorithm that matches the asymptotic lower bound of Proposition 2 . This algorithm is inspired by the Indexed Minimum Empirical Divergence ( IMED ) proposed by Honda and Takemura [ 2011 ] . The general idea behind this algorithm is , following the intuition given by the lower bound , to narrow on the current best arm and its neighbourhood for pulling an arm at a given time step . Notations . The empirical mean of the rewards from the arm a is denoted by µ̂a ( t ) =∑t s=1 I { as=a } Xs/Na ( t ) if Na ( t ) > 0 , 0 otherwise . We also denote by µ̂ ? ( t ) = maxa∈A µ̂a ( t ) and Â ? ( t ) =argmax a∈A µ̂a ( t ) respectively the current best mean and the current set of optimal arms . 3.1 The IMED-UB algorithm . We first pull each arm once . For all arm a∈A and time step t > 1 we introduce the IMED index Ia ( t ) = Na ( t ) KL ( µ̂a ( t ) |µ̂ ? ( t ) ) + log ( Na ( t ) ) , with the convention 0×∞= 0 . This index can be seen as a transportation cost for moving a suboptimal arm to an optimal one plus an exploration term : the logarithm of the number of pulls . When an optimal arm is considered , the transportation cost is null and there is only the exploration part . Note that , as stated in Honda and Takemura [ 2011 ] , Ia ( t ) is an index in the weaker sense since it can not be determined only by samples from the arm a but also uses the empirical mean of the current optimal arm . We define IMED-UB ( Indexed Minimum Empirical Divergence for Unimodal Bandits ) , described in Algorithm 1 , to be the algorithm consisting of pulling an arm at ∈ { â ? t } ∪Vâ ? t with minimum index at each time step t , where is â ? t ∈argminâ ? ∈Â ? ( t ) Nâ ? ( t ) is a current best arm . This is a natural algorithm since the lower bound on the regret given in Proposition 2 involves only the arms in Va ? , the neighbourhood of the arm a ? of maximal mean . Algorithm 1 IMED-UB Pull each arm once for t = |A| . . . T − 1 do Choose â ? t ∈ argmin â ? ∈Â ? ( t ) Nâ ? ( t ) ( chosen arbitrarily ) Pull at+1 ∈ argmin a∈ { â ? t } ∪Vâ ? t Ia ( t ) ( chosen arbitrarily ) end for 3.2 Asymptotic optimality of IMED-UB In this section , we state the main theoretical result of this paper . Theorem 5 ( Upper bounds ) Let us consider a set of distributions ν∈D and let a ? its optimal arm . Let Va ? be the sub-optimal arms in the neighbourhood of a ? . Then under the IMED-UB algorithm for all 0 < ε < εν , for all horizon time T > 1 , for all a∈Va ? , Eν [ Na ( T ) ] 6 1 + αν ( ε ) KL ( µa|µa ? ) log ( T ) + 2dCε √ log ( cεT ) + d ( 1 + c−1εν ) + d ( 2d+ 3 ) 2σ2ενe ε2ν/2σ 2 ε ε2 + 1 and , for all a /∈ { a ? } ∪Va ? , Eν [ Na ( T ) ] 6 2dCε √ log ( cεT ) + d ( 1 + c−1εν ) + d ( 2d+ 3 ) 2σ2ενe ε2ν/2σ 2 ε ε2 + 1 , where d is the maximum degree of nodes in G , εν =mina 6=a′ |µa−µa′ |/2 , σ2ε =max a∈A { V X∼p ( µ′ ) ( X ) : µ ′∈ [ µa−ε , µa ] } and cε , Cε > 0 are the constants involved in Theorem 15. αν ( · ) is a non-negative function depending only on ν such that lim ε→0 αν ( ε ) =0 ( see Section 4.1 for more details ) . In particular one can note that the arms in the neighbourhood of the optimal one are pulledO ( log ( T ) ) times while the other sub-optimal arms are pulled O ( √ log ( T ) ) of times under IMED-UB . This is coherent with the lower bound that only involves the neighbourhood of the best arm . More precisely , combining Theorem 5 and the chain rule ( 1 ) gives the asymptotic optimality of IMED-UB with respect to the lower bound of Proposition 2 . Corollary 6 ( Asymptotic optimality ) With the same notations as in Theorem 5 , then under the IMED-UB algorithm lim sup T→∞ R ( ν , T ) log ( T ) 6 c ( ν ) = ∑ a∈Va ? ∆a KL ( µa|µa ? ) . A finite time analysis of IMED-UB is provided in following Section 4 . 4 IMED-UB finite time analysis At a high level , the key interesting step of the proof is to realize that the considered algorithm implies empirical lower and empirical upper bounds on the numbers of pulls ( see Lemma 7 , Lemma 8 ) . Then , based on concentration lemmas ( see Section B ) , the algorithm-based empirical lower bounds ensure the reliability of the estimators of interest ( Lemma 12 ) . Interestingly , this makes use of arguments based on recent concentration of measure that enable to control the concentration without adding some log log bonus ( such a bonus was required for example in the initial analysis of the KL-UCB strategy from Cappé et al . [ 2013 ] ) . Then , combining the reliability of these estimators with the obtained algorithm-base empirical upper bounds , we obtain upper bounds on the average numbers of pulls ( Theorem 5 ) . The proof is concise to fit mostly in the next few pages .

Summary: The paper tackles the problem of unimodal bandits where it looks at an algorithm based on the IMED index which follows the general idea of the OSUB algorithm. The main novelty of the approach is the analysis which does not rely on distinctions between exploration and exploitation rounds and the added benefit of the IMED indexes not requiring solving an optimisation problem. The paper is mostly dedicated to the regret analysis but also provides a set of experiments showcasing the numerical performance of the algorithm.