Indexed Minimum Empirical Divergence for Unimodal Bandits

1 Introduction . The multi-armed bandit problem is a popular framework to formalize sequential decision making problems . It was first introduced in the context of medical trials [ Thompson , 1933 , 1935 ] and later formalized by Robbins [ 1952 ] : A bandit is specified by a configuration , that is a set of unknown probability distributions , ν= ( νa ) a∈A with means ( µa ) a∈A . At each time t∈N , the learner chooses an arm at∈A , based only on the past , the learner then receives and observes a reward Xt , conditionally independent , sampled according to νat . The goal of the learner is to maximize the expected sum of rewards received over time ( up to some unknown horizon T ) , or equivalently minimize the regret with respect to the algorithm constantly receiving the highest mean reward R ( ν , T ) = Eν [ T∑ t=1 µ ? −Xt ] where µ ? = max a∈A µa . Both means and distributions are unknown , which makes the problem non trivial , and the learner only knows that ν ∈ D where D is a given set of bandit configurations . This problem received increased attention in the middle of the 20th century , and the seminal paper Lai and Robbins [ 1985 ] established the first lower bound on the cumulative regret , showing that designing an algorithm that is optimal uniformly over a given set of configurations D comes with a price . The study of the lower performance bounds in multi-armed bandits successfully lead to the development of asymptotically optimal algorithms for specific configuration sets , such as the KLUCB algorithm [ Lai , 1987 , Cappé et al. , 2013 , Maillard , 2018 ] for exponential families , or alternatively the DMED and IMED algorithms from Honda and Takemura [ 2011 , 2015 ] . The lower bounds from Lai and Robbins [ 1985 ] , later extended by Burnetas and Katehakis [ 1997 ] did not cover all possible configurations , and in particular structured configuration sets were not handled until Agrawal et al . [ 1989 ] and then Graves and Lai [ 1997 ] established generic lower bounds . Here , structure refers to the fact that pulling an arm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . may reveals information that enables to refine estimation of other arms . Unfortunately , designing numerical efficient algorithms that are provably optimal remains a challenge for many structures . Structured configurations . Motivated by the growing popularity of bandits in a number of industrial and societal application domains , the study of structured configuration sets has received increasing attention over the last few years : The linear bandit problem is one typical illustration Abbasi-Yadkori et al . [ 2011 ] , Srinivas et al . [ 2010 ] , Durand et al . [ 2017 ] , for which the linear structure considerably modifies the achievable lower bound , see Lattimore and Szepesvari [ 2017 ] . The study of a unimodal structure naturally appears in many contexts , e.g . single-peak preference economics , voting theory or wireless communications , and has been first considered in Yu and Mannor [ 2011 ] from a bandit perspective , then in Combes and Proutiere [ 2014 ] and Trinh et al . [ 2020 ] providing an explicit lower bound together with an algorithm exploiting this specific structure . Other structures include Lipschitz bandits Magureanu et al . [ 2014 ] , and we refer to the manuscript Magureanu [ 2018 ] for other examples , such as cascading bandits that are useful in the context of recommender systems . In Combes et al . [ 2017 ] , a generic algorithm is introduced called OSSB ( Optimal Structured Stochastic Bandit ) , stepping the path towards generic multi-armed bandit algorithms that are adaptive to a given structure . More recently in Degenne et al . [ 2020 ] , the authors introduce an adaptation of the KLUCB strategy to handle structured multi-armed bandit problems . Unimodal-structure . We assume a unimodal structure similar to that considered in Yu and Mannor [ 2011 ] and Combes and Proutiere [ 2014 ] . That is , there exists an undirected graph G = ( A , E ) whose vertices are arms A , and whose edges E characterize a partial order among means ( µa ) a∈A . This partial order is assumed unknown to the learner . We assume that there exists a unique optimal arm a ? = argmaxa∈A µa and that for all sub-optimal arm a 6= a ? , there exists a path Pa = ( a1 = a , . . . , a ` a = a ? ) ∈ A ` a of length ` a > 2 such that for all i ∈ [ 1 , ` a − 1 ] , ( ai , ai+1 ) ∈ E and µai < µai+1 . Lastly , we assume that ν⊂P : = { p ( µ ) , µ∈Θ } , where p ( µ ) is an exponential-family distribution probability with density f ( · , µ ) with respect to some positive measure λ on R and mean µ∈Θ⊂R . P is assumed to be known to the learner . Thus , for all a∈A we have νa=p ( µa ) . We denote by D ( P , G ) or simply D the structured set of such unimodal-bandit distributions characterized by ( P , G ) . In the following , we assume that P is a set of one-dimensional exponential family distributions . Contributions . In this paper , we provide novel regret minimization results related to the unimodal structure . We first revisit the Indexed Minimum Empirical Divergence ( IMED ) algorithm from Honda and Takemura [ 2015 ] introduced for unstructured multi-armed bandits , and adapt it to the unimodalstructured setting . We introduce in Section 3 the IMED-UB algorithm that is limited to the pulling of the current best arm or their no more than d nearest arms at each time step , with d the maximum degree of nodes in G. Being constructed from IMED , IMED-UB does not require any optimization procedure and does not separate exploration from exploitation rounds . IMED-UB appears to be a local algorithm . We prove in Theorem 6 that IMED-UB is asymptotically optimal . Furthermore , this novel algorithm competes with the state-of-the-art algorithms in practice . This is confirmed by numerical illustrations on synthetic data . We believe that the construction of this algorithm together with the proof techniques developed in this paper are of independent interest for the bandit community . Notations . Let ν∈D . Let µ ? =maxa∈A µa be the optimal mean and a ? =argmaxa∈A µa be the optimal arm of ν . We define for an arm a∈A its sub-optimality gap ∆a=µ ? −µa . Considering an horizon T > 1 , thanks to the chain rule we can rewrite the regret as follows : R ( ν , T ) = ∑ a∈A ∆a Eν [ Na ( T ) ] , ( 1 ) where Na ( t ) = ∑t s=1 I { as=a } is the number of pulls of arm a at time t . 2 Regret lower bound . In this subsection , we recall for completeness the known lower bound on the regret when we assume a unimodal structure . In order to obtain non trivial lower bound we consider algorithms that are consistent ( aka uniformly-good ) . Definition 1 ( Consistent algorithm ) An algorithm is consistent on D if for all configuration ν ∈ D , for all sub-optimal arm a , for all α > 0 , lim T→∞ Eν [ Na ( T ) Tα ] = 0 . We can derive from the notion of consistency an asymptotic lower bound on the regret , see Combes and Proutiere [ 2014 ] . Proposition 2 ( Lower bounds on the regret ) Let us consider a consistent algorithm . Then , for all configuration ν∈D , it must be that lim inf T→∞ R ( ν , T ) log ( T ) > c ( ν ) : = ∑ a∈Va ? ∆a KL ( µa|µ ? ) , where KL ( µ|µ′ ) = ∫ Rlog ( f ( x , µ ) /f ( x , µ ′ ) ) f ( x , µ ) λ ( dx ) denotes the Kullback-Leibler divergence between ν=p ( µ ) and ν′=p ( µ′ ) , for µ , µ′∈Θ . Remark 3 The quantity c ( ν ) is a fully explicit function of ν ( it does not require solving any optimization problem ) for some set of distributions ν ( see Remark 4 ) . This useful property no longer holds in general for arbitrary structures . Also , it is noticeable that c ( ν ) does not involve all the suboptimal arms but only the ones in Va ? . This indicates that sub-optimal arms outside Va ? are sampled o ( log ( T ) ) , which contrasts with the unstructured stochastic multi-armed bandits . See Combes and Proutiere [ 2014 ] for further insights . Remark 4 For Bernoulli distributions , a possible setting is to assume λ = δ0 + δ1 ( with δ0 , δ1 Dirac measures ) , Θ = ( 0 , 1 ) and for µ ∈Θ , f ( · , µ ) = : x ∈ { 0 , 1 } 7→ µx ( 1 − µ ) 1−x . Then for all µ , µ′ ∈ ( 0 , 1 ) , KL ( µ|µ′ ) =µ log ( µ/µ′ ) + ( 1−µ ) log ( ( 1−µ ) / ( 1−µ′ ) ) . For Gaussian distributions ( variance σ2 =1 ) , we assume λ to be the Lebesgue measure , Θ=R , and for µ∈R , f ( · , µ ) = : x∈R 7→ ( √ 2π ) −1e− ( x−µ ) 2/2 . Then for all µ , µ′∈R , KL ( µ|µ′ ) = ( µ′−µ ) 2/2 . For Exponential distributions , we assume λ to be the Lebesgue measure , Θ= ] 0 ; +∞ [ , and for µ > 0 , f ( · , µ ) = : x > 0 7→ e−x/µ/µ . Then for all µ , µ′ > 0 , KL ( µ|µ′ ) =log ( µ′/µ ) +µ/µ′−1 . 3 Optimal algorithm for unimodal-structured bandits . We present in this section a novel algorithm that matches the asymptotic lower bound of Proposition 2 . This algorithm is inspired by the Indexed Minimum Empirical Divergence ( IMED ) proposed by Honda and Takemura [ 2011 ] . The general idea behind this algorithm is , following the intuition given by the lower bound , to narrow on the current best arm and its neighbourhood for pulling an arm at a given time step . Notations . The empirical mean of the rewards from the arm a is denoted by µ̂a ( t ) =∑t s=1 I { as=a } Xs/Na ( t ) if Na ( t ) > 0 , 0 otherwise . We also denote by µ̂ ? ( t ) = maxa∈A µ̂a ( t ) and Â ? ( t ) =argmax a∈A µ̂a ( t ) respectively the current best mean and the current set of optimal arms . 3.1 The IMED-UB algorithm . We first pull each arm once . For all arm a∈A and time step t > 1 we introduce the IMED index Ia ( t ) = Na ( t ) KL ( µ̂a ( t ) |µ̂ ? ( t ) ) + log ( Na ( t ) ) , with the convention 0×∞= 0 . This index can be seen as a transportation cost for moving a suboptimal arm to an optimal one plus an exploration term : the logarithm of the number of pulls . When an optimal arm is considered , the transportation cost is null and there is only the exploration part . Note that , as stated in Honda and Takemura [ 2011 ] , Ia ( t ) is an index in the weaker sense since it can not be determined only by samples from the arm a but also uses the empirical mean of the current optimal arm . We define IMED-UB ( Indexed Minimum Empirical Divergence for Unimodal Bandits ) , described in Algorithm 1 , to be the algorithm consisting of pulling an arm at ∈ { â ? t } ∪Vâ ? t with minimum index at each time step t , where is â ? t ∈argminâ ? ∈Â ? ( t ) Nâ ? ( t ) is a current best arm . This is a natural algorithm since the lower bound on the regret given in Proposition 2 involves only the arms in Va ? , the neighbourhood of the arm a ? of maximal mean . Algorithm 1 IMED-UB Pull each arm once for t = |A| . . . T − 1 do Choose â ? t ∈ argmin â ? ∈Â ? ( t ) Nâ ? ( t ) ( chosen arbitrarily ) Pull at+1 ∈ argmin a∈ { â ? t } ∪Vâ ? t Ia ( t ) ( chosen arbitrarily ) end for 3.2 Asymptotic optimality of IMED-UB In this section , we state the main theoretical result of this paper . Theorem 5 ( Upper bounds ) Let us consider a set of distributions ν∈D and let a ? its optimal arm . Let Va ? be the sub-optimal arms in the neighbourhood of a ? . Then under the IMED-UB algorithm for all 0 < ε < εν , for all horizon time T > 1 , for all a∈Va ? , Eν [ Na ( T ) ] 6 1 + αν ( ε ) KL ( µa|µa ? ) log ( T ) + 2dCε √ log ( cεT ) + d ( 1 + c−1εν ) + d ( 2d+ 3 ) 2σ2ενe ε2ν/2σ 2 ε ε2 + 1 and , for all a /∈ { a ? } ∪Va ? , Eν [ Na ( T ) ] 6 2dCε √ log ( cεT ) + d ( 1 + c−1εν ) + d ( 2d+ 3 ) 2σ2ενe ε2ν/2σ 2 ε ε2 + 1 , where d is the maximum degree of nodes in G , εν =mina 6=a′ |µa−µa′ |/2 , σ2ε =max a∈A { V X∼p ( µ′ ) ( X ) : µ ′∈ [ µa−ε , µa ] } and cε , Cε > 0 are the constants involved in Theorem 15. αν ( · ) is a non-negative function depending only on ν such that lim ε→0 αν ( ε ) =0 ( see Section 4.1 for more details ) . In particular one can note that the arms in the neighbourhood of the optimal one are pulledO ( log ( T ) ) times while the other sub-optimal arms are pulled O ( √ log ( T ) ) of times under IMED-UB . This is coherent with the lower bound that only involves the neighbourhood of the best arm . More precisely , combining Theorem 5 and the chain rule ( 1 ) gives the asymptotic optimality of IMED-UB with respect to the lower bound of Proposition 2 . Corollary 6 ( Asymptotic optimality ) With the same notations as in Theorem 5 , then under the IMED-UB algorithm lim sup T→∞ R ( ν , T ) log ( T ) 6 c ( ν ) = ∑ a∈Va ? ∆a KL ( µa|µa ? ) . A finite time analysis of IMED-UB is provided in following Section 4 . 4 IMED-UB finite time analysis At a high level , the key interesting step of the proof is to realize that the considered algorithm implies empirical lower and empirical upper bounds on the numbers of pulls ( see Lemma 7 , Lemma 8 ) . Then , based on concentration lemmas ( see Section B ) , the algorithm-based empirical lower bounds ensure the reliability of the estimators of interest ( Lemma 12 ) . Interestingly , this makes use of arguments based on recent concentration of measure that enable to control the concentration without adding some log log bonus ( such a bonus was required for example in the initial analysis of the KL-UCB strategy from Cappé et al . [ 2013 ] ) . Then , combining the reliability of these estimators with the obtained algorithm-base empirical upper bounds , we obtain upper bounds on the average numbers of pulls ( Theorem 5 ) . The proof is concise to fit mostly in the next few pages .

Summary: The paper handles the unimodal bandit setting with reward distributions in an exponential family. The proposed algorithm, IMED-UB, is based on IMED's index and a logarithmic finite-time upper-bound of its regret is proven. The second term in IMED-UB's bound is in $\sqrt{\log T}$, which is worse than the second term in OSUB's and UTS's bounds. However, empirically the proposed algorithm has the same regret or a smaller regret than OSUB and UTS on a linear graph with 9 nodes.  The algorithm and the proof of its regret-bound are interesting. However, some points would benefit from a more thorough investigation.