Dueling Bandits with Adversarial Sleeping

∑K−1 i=1 ∑K j=i+1 log T ∆ ( i , j ) ) , where K is the number of items and ∆ ( i , j ) is the gap between items i and j . This indicates that the sleeping problem with preference feedback is inherently more difficult than that for classical multi-armed bandits ( MAB ) . We then propose two algorithms , with near optimal regret guarantees . Our results are corroborated empirically . 1 Introduction . The problem of Dueling-Bandits has gained much attention in the machine learning community [ 34 , 38 , 36 ] , which is an online learning framework that generalizes the standard multiarmed bandit ( MAB ) [ 6 ] setting for identifying a set of ‘ good ’ arms from a fixed decision-space ( set of arms/items ) by querying preference feedback of actively chosen item-pairs . More formally , in dueling bandits , the learning proceeds in rounds : At each round , the learner selects a pair of arms and observes stochastic preference feedback of the winner of the comparison ( duel ) between the selected arms ; the objective of the learner is to minimize the regret with respect to a ( or set of ) ‘ best ’ arm ( s ) in hindsight . Towards this several algorithms have been proposed [ 2 , 37 , 21 , 14 ] . Due to the inherent exploration-vs-exploitation tradeoff of the learning framework and several advantages of preference feedback [ 9 , 35 ] , many real-world applications can be modeled as dueling bandits , including movie recommendations , retail management , search engine optimization , job scheduling , etc . However , in reality , the decision spaces might often change over time due to the non-availability of some items , which are considered to be ‘ sleeping ’ . This ‘ sleeping-aspect ’ of online decision making problems has been widely studied in the standard multiarmed bandit ( MAB ) literature [ 17 , 24 , 15 , 19 , 18 , 11 ] . There the goal is to learn a ‘ no-regret ’ policy that maps to the ‘ best awake item ’ of any available ( non-sleeping ) subset of items , and the learner ’ s performance is measured with respect to the optimal policy in hindsight . This setting is famously known as Sleeping Bandits in MAB [ 17 , 24 , 15 , 11 ] . More discussions are given in Related Works . Surprisingly , however , the ‘ sleeping problem ’ is completely unaddressed in the preference bandits literature , even for the special case of pairwise preference feedback , which is famously studied ∗Microsoft Research , New York , US ; aasa @ microsoft.com . †Univ . Grenoble Alpes , Inria , CNRS , Grenoble INP , LJK , 38000 Grenoble , France . pierre.gaillard @ inria.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as Dueling Bandits [ 37 , 34 ] , even though the setup of changing decision spaces are quite relevant in almost every practical applications : Be that in retail stores where some items might go out of production over time , for search engine optimization some websites could be down on certain days , in recommender systems some restaurants might be closed or movies could be outdated , in clinical trials certain drugs could be out of stock , and many more . This work is the first to consider the problem of Sleeping Dueling Bandits , where we formulated the stochastic K-armed dueling bandit problem with adversarial item availabilities . Here at each round t ∈ { 1 , 2 , . . . , T } the item preferences are considered to be generated from a fixed underlying ( and of course unknown ) preference matrix P ∈ [ 0 , 1 ] K×K , however , the set of available actions St ⊆ { 1 , 2 , . . . , K } is assumed to be adversarially chosen by the environment . We call the problem as Sleeping-Dueling Bandit with Stochastic Preferences and Adversarial Availabilities or in brief DB-SPAA ( P , ST ) , where ST = { S1 , S2 , . . . ST } denotes the sequence of available subsets over T rounds . We also assume the preference P follows a ‘ total-ordering assumption to ensure the existence of a best-item per available subset St. We describe the setting in Fig . 1 with a formal description in Sec . 2 . Our specific contributions are as follows : 1 . We first analyze the fundamental performance limit for the DB-SPAA ( P , ST ) problem in Sec . 3 : Thm . 1 gives an instance-specific regret lower bound of Ω ( ∑K−1 i=1 ∑K j=i+1 log T ∆ ( i , j ) ) , with ∆ ( i , j ) being the ‘ preference gap ’ of item i-vs-j ( see Eqn . ( 2 ) ) . Our lower bound , which can be of order Ω ( K2 log T/∆ ) , ∆ : = mini , j ∆ ( i , j ) being the worst case gap , indicates that the problem of sleeping dueling bandits is inherently more difficult that standard sleeping bandits ( MAB ) , unlike the ‘ non-sleeping ’ case where both dueling bandits ( with ‘ total-ordering ’ assumption on P ) and MAB are known to have the same fundamental performance limit of Ω ( K log T/∆ ) ( Rem . 1 ) . 2 . We next design a ‘ fixed confidence regret ’ algorithm SlDB-UCB ( Alg . 1 ) , inspired from the pairwise upper confidence bound ( UCB ) based algorithm [ 37 ] . However due to the fixed confidence and ‘ adversarial-sleeping ’ nature of the problem , we need to differently maintain pairwise confidence bounds per item ( based on the availability sequence { St } ) , which makes the resulting algorithm and its subsequent analysis significantly different than standard UCB based dueling bandit algorithms : Precisely given any δ > 0 , SlDB-UCB achieves a regret of O ( K3 log ( 1/δ ) ∆2 ) with probability at least 1− δ over any problem instance of DB-SPAA ( P , ST ) ( Sec . 4 ) . 3 . In Sec . 5 , we design another computationally efficient algorithm , SlDB-ED ( Alg . 2 ) , for ‘ expected regret ’ guarantee . Unlike the previous algorithm ( SlDB-UCB ) , SlDB-ED uses empirical divergence ( ED ) based measures to filter out the ‘ good ’ set of arms , inspired from the idea of RMED algorithm of [ 21 ] for standard dueling bandits ; however , due to sleeping nature of the items , it requires a different maintenance of ‘ good ’ arms and the regret analysis of the algorithm requires derivation of new results ( as described in Sec . 5 ) . The algorithm is shown to perform near optimally with an expected non-asymptotic regret upper-bound of ( Thm . 6 ) . Note that for any problem instance with constant suboptimality gaps ∆ ( i , j ) = ∆ for all i < j , regret bound of SlDB-ED is tight and matches the lower-bound ensuring the near optimality of SlDB-ED in the worst case . Furthermore , a novelty of our finite time regret analysis lies in showing a cleaner tradeoff between regret vs. availability sequence ST which automatically adapts to the inherent ‘ hardness ’ of the sequence of available subsets ST , compared to existing sleeping bandits work for adversarial availabilities in the MAB setting [ 19 ] which only gives a worst-case regret bound over all possible availability sequences ( Rem . 3 ) . 4 . Finally we corroborate our theoretical results with extensive empirical evaluations . ( Sec . 6 ) . Related Works . The problem of regret minimization for stochastic multiarmed bandits ( MAB ) is extremely well studied in the online learning literature [ 6 , 1 , 22 , 5 , 16 ] , where the learner gets to see a noisy draw of absolute reward feedback of an arm upon playing a single arm per round . A well motivated generalization of MAB framework is Sleeping Bandits [ 17 , 24 , 18 , 15 ] , much studied in the online learning community , where at any round the set of available actions could vary stochastically based on some unknown distributions over the decision space of K items [ 24 , 11 ] or adversarially [ 15 , 19 , 18 ] . Besides the reward model , the set of available actions could also vary stochastically or adversarially [ 17 , 24 ] . The problem is NP-hard when both rewards and availabilities are adversarial [ 19 , 18 , 15 ] . In case of stochastic reward and adversarial availabilities [ 19 ] proposed an UCB based no-regret algorithm , which was also shown to be provably optimal . The case of adversarial reward and stochastic availabilities has also been studied where the achievable regret lower bound is known be Ω ( √ KT ) by the inefficient EXP4 algorithm [ 19 , 15 ] . On the other hand over the last decade , the relative feedback variants of stochastic MAB problem has seen a widespread resurgence in the form of the Dueling Bandit problem , where , instead of getting noisy feedback of the reward of the chosen arm , the learner only gets to see a noisy feedback on the pairwise preference of two arms selected by the learner . The objective of the learner is to minimize the regret with respect to ‘ best arm in the stochastic model . Several algorithms have been proposed to address this dueling bandits problem , for different notions of ‘ best arms ’ or preference models [ 10 , 31 , 38 , 37 , 36 , 21 , 33 , 13 ] , or even extending the pairwise preference to subsetwise preferences [ 29 , 8 , 26 , 27 , 25 ] . However , surprisingly , unlike the ‘ sleeping bandits generalization ’ of MAB , no parallel has been drawn for dueling bandits , which remains our main focus . 2 Problem Formulation . Notations . Decision space ( or item/arm set ) [ K ] : = { 1 , 2 , . . . , K } . The available set of items at round t is denoted by St ⊆ [ K ] . For any matrix M ∈ RK×K , we definemij : = M ( i , j ) , ∀i , j ∈ [ K ] . We write S\i = S \ { i } , for any S ⊆ [ K ] and i ∈ S. 1 ( · ) denotes the indicator random variable which takes value 1 if the predicate is true and 0 otherwise and . a rough inequality which holds up to universal constants . For any two items x , y ∈ [ K ] , we use the symbol x y to denote x is preferred over y. ΣK denotes the set of all permutations of the items in set [ K ] . The KL-divergence of two Bernoullis with biases p and q respectively is written kl ( p , q ) : = p log ( p/q ) + ( 1− p ) log ( ( 1−p ) / ( 1−q ) ) . We assume 00 : = 0.5 ( in Alg . 1 and 2 ) . Setup . We consider the problem of stochastic K-armed dueling bandits with adversarial availabilities : At every iteration t = 1 , . . . , T , a set of available items ( actions ) St ⊆ [ K ] is revealed , and the learner is asked to choose two items xt , yt ∈ St. Then , the learner receives a preference feedback ot = 1 ( xt yt ) ∼ Ber ( P ( xt , yt ) ) , where P ∈ [ 0 , 1 ] K×K is an underlying pairwise preference matrix , unknown to the learner . The setting is described in Figure 1 . We assume that P respects a ‘ total ordering ’ , say σ∗ ∈ ΣK . Without loss of generality , we set σ∗ = ( 1 , 2 , . . . , K ) thoughout the paper . This implies P ( i , j ) ≥ 0.5 for i ≤ j . One possible pairwise probability model which respects ‘ total ordering ’ is Plackett-Luce [ 7 ] , where it is assumed that the K items are associated to positive score parameters θ1 , . . . , θK , and P ( i , j ) = θi/ ( θi + θj ) for all i , j ∈ [ K ] . In fact any well random utility ( RUM ) based preference model would have the above property , like [ 7 , 28 ] . Note also that our assumption corresponds to assuming the existence of a Condorcet winner for every subset St ⊆ [ K ] . Objective . The objective of the learner is to minimize his regret over T rounds with respect to the best policy in the policy class Π = { π : 2K 7→ [ K ] | ∀t ∈ [ T ] , π ( St ) ∈ St } , i.e . any π ∈ Π is such that for any t ∈ [ T ] , π ( St ) ∈ St. More formally we define the regret as follows : RT = max π∈Π T∑ t=1 P ( π ( St ) , xt ) + P ( π ( St ) , yt ) − 1 2 . ( 1 ) We analyze both fixed-confidence and expected regret guarantees in this paper respectively in Sec . 4 ( see Thm . 3 ) and Sec . 5 ( see Thm . 6 ) . It is easy to note that under our preference modelling assumptions , the best policy , say π∗ , turns out to be π∗ ( S ) = min { S } for any S ⊆ [ K ] . We henceforth denote by i∗t = π ∗ ( St ) . We define the above problem to be Sleeping-Dueling Bandit with Stochastic Preferences and Adversarial Availabilities over the stochastic preference matrix P ∈ [ 0 , 1 ] K×K and the sequence of available subsets ST = { S1 , . . . , ST } , or in short DB-SPAA ( P , ST ) . For ease of notation we respectively define the gaps and the non-zeros gaps as ∆ ( i , j ) : = P ( i , j ) −1/2 , and ∆ ( i , j ) + : = { ∆ ( i , j ) if ∆ ( i , j ) 6= 0 +∞ if ∆ ( i , j ) = 0 ( 2 ) The regret thus can be rewritten as RT : = ∑T t=1 rt , where rt : = ( ∆ ( i ∗ t , xt ) + ∆ ( i ∗ t , yt ) ) /2 denotes the instantaneous regret . We also denote by nij ( t ) : = ∑t τ=1 1 ( { xt , yt } = { i , j } ) the number of times the pair ( i , j ) is played until time t and by wij ( t ) the number of times i beats j in t rounds .

Summary: This paper studies the classic dueling bandits problem in the sleeping bandits setting, i.e. where each round an adversarially selected subset of actions is available. As in dueling bandits, each round the learner is allowed to choose two of the available actions, and learns a noisy signal about which action is better. We also assume there is a consistent total ordering of the items. The learner would like to minimize their total regret over T rounds compared to an algorithm which knows this ordering in advance.  This paper proves the following results: - They show an Omega(K^2 log T) (really Omega(log T * sum 1/gap_{i, j})) lower bound on the regret of any learning algorithm. - They then give an algorithm (S1DB-UCB) that matches this lower bound. The idea, roughly, is to maintain a confidence interval for each pair of items representing the probability that one beats the other. Each round, the algorithm picks one random element from the set of “potential winners” (roughly, items which have a chance of being the best item), and then a second item being the one with the highest UCB probability of beating the first item. - They give a slightly more computationally efficient algorithm (taking ~O(K) time per step instead of O(K^2) time per step) with a slightly worse regret bound. - Finally, they run some empirical simulations comparing the two above algorithms on synthetic data.