Dueling Bandits with Adversarial Sleeping

∑K−1 i=1 ∑K j=i+1 log T ∆ ( i , j ) ) , where K is the number of items and ∆ ( i , j ) is the gap between items i and j . This indicates that the sleeping problem with preference feedback is inherently more difficult than that for classical multi-armed bandits ( MAB ) . We then propose two algorithms , with near optimal regret guarantees . Our results are corroborated empirically . 1 Introduction . The problem of Dueling-Bandits has gained much attention in the machine learning community [ 34 , 38 , 36 ] , which is an online learning framework that generalizes the standard multiarmed bandit ( MAB ) [ 6 ] setting for identifying a set of ‘ good ’ arms from a fixed decision-space ( set of arms/items ) by querying preference feedback of actively chosen item-pairs . More formally , in dueling bandits , the learning proceeds in rounds : At each round , the learner selects a pair of arms and observes stochastic preference feedback of the winner of the comparison ( duel ) between the selected arms ; the objective of the learner is to minimize the regret with respect to a ( or set of ) ‘ best ’ arm ( s ) in hindsight . Towards this several algorithms have been proposed [ 2 , 37 , 21 , 14 ] . Due to the inherent exploration-vs-exploitation tradeoff of the learning framework and several advantages of preference feedback [ 9 , 35 ] , many real-world applications can be modeled as dueling bandits , including movie recommendations , retail management , search engine optimization , job scheduling , etc . However , in reality , the decision spaces might often change over time due to the non-availability of some items , which are considered to be ‘ sleeping ’ . This ‘ sleeping-aspect ’ of online decision making problems has been widely studied in the standard multiarmed bandit ( MAB ) literature [ 17 , 24 , 15 , 19 , 18 , 11 ] . There the goal is to learn a ‘ no-regret ’ policy that maps to the ‘ best awake item ’ of any available ( non-sleeping ) subset of items , and the learner ’ s performance is measured with respect to the optimal policy in hindsight . This setting is famously known as Sleeping Bandits in MAB [ 17 , 24 , 15 , 11 ] . More discussions are given in Related Works . Surprisingly , however , the ‘ sleeping problem ’ is completely unaddressed in the preference bandits literature , even for the special case of pairwise preference feedback , which is famously studied ∗Microsoft Research , New York , US ; aasa @ microsoft.com . †Univ . Grenoble Alpes , Inria , CNRS , Grenoble INP , LJK , 38000 Grenoble , France . pierre.gaillard @ inria.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as Dueling Bandits [ 37 , 34 ] , even though the setup of changing decision spaces are quite relevant in almost every practical applications : Be that in retail stores where some items might go out of production over time , for search engine optimization some websites could be down on certain days , in recommender systems some restaurants might be closed or movies could be outdated , in clinical trials certain drugs could be out of stock , and many more . This work is the first to consider the problem of Sleeping Dueling Bandits , where we formulated the stochastic K-armed dueling bandit problem with adversarial item availabilities . Here at each round t ∈ { 1 , 2 , . . . , T } the item preferences are considered to be generated from a fixed underlying ( and of course unknown ) preference matrix P ∈ [ 0 , 1 ] K×K , however , the set of available actions St ⊆ { 1 , 2 , . . . , K } is assumed to be adversarially chosen by the environment . We call the problem as Sleeping-Dueling Bandit with Stochastic Preferences and Adversarial Availabilities or in brief DB-SPAA ( P , ST ) , where ST = { S1 , S2 , . . . ST } denotes the sequence of available subsets over T rounds . We also assume the preference P follows a ‘ total-ordering assumption to ensure the existence of a best-item per available subset St. We describe the setting in Fig . 1 with a formal description in Sec . 2 . Our specific contributions are as follows : 1 . We first analyze the fundamental performance limit for the DB-SPAA ( P , ST ) problem in Sec . 3 : Thm . 1 gives an instance-specific regret lower bound of Ω ( ∑K−1 i=1 ∑K j=i+1 log T ∆ ( i , j ) ) , with ∆ ( i , j ) being the ‘ preference gap ’ of item i-vs-j ( see Eqn . ( 2 ) ) . Our lower bound , which can be of order Ω ( K2 log T/∆ ) , ∆ : = mini , j ∆ ( i , j ) being the worst case gap , indicates that the problem of sleeping dueling bandits is inherently more difficult that standard sleeping bandits ( MAB ) , unlike the ‘ non-sleeping ’ case where both dueling bandits ( with ‘ total-ordering ’ assumption on P ) and MAB are known to have the same fundamental performance limit of Ω ( K log T/∆ ) ( Rem . 1 ) . 2 . We next design a ‘ fixed confidence regret ’ algorithm SlDB-UCB ( Alg . 1 ) , inspired from the pairwise upper confidence bound ( UCB ) based algorithm [ 37 ] . However due to the fixed confidence and ‘ adversarial-sleeping ’ nature of the problem , we need to differently maintain pairwise confidence bounds per item ( based on the availability sequence { St } ) , which makes the resulting algorithm and its subsequent analysis significantly different than standard UCB based dueling bandit algorithms : Precisely given any δ > 0 , SlDB-UCB achieves a regret of O ( K3 log ( 1/δ ) ∆2 ) with probability at least 1− δ over any problem instance of DB-SPAA ( P , ST ) ( Sec . 4 ) . 3 . In Sec . 5 , we design another computationally efficient algorithm , SlDB-ED ( Alg . 2 ) , for ‘ expected regret ’ guarantee . Unlike the previous algorithm ( SlDB-UCB ) , SlDB-ED uses empirical divergence ( ED ) based measures to filter out the ‘ good ’ set of arms , inspired from the idea of RMED algorithm of [ 21 ] for standard dueling bandits ; however , due to sleeping nature of the items , it requires a different maintenance of ‘ good ’ arms and the regret analysis of the algorithm requires derivation of new results ( as described in Sec . 5 ) . The algorithm is shown to perform near optimally with an expected non-asymptotic regret upper-bound of ( Thm . 6 ) . Note that for any problem instance with constant suboptimality gaps ∆ ( i , j ) = ∆ for all i < j , regret bound of SlDB-ED is tight and matches the lower-bound ensuring the near optimality of SlDB-ED in the worst case . Furthermore , a novelty of our finite time regret analysis lies in showing a cleaner tradeoff between regret vs. availability sequence ST which automatically adapts to the inherent ‘ hardness ’ of the sequence of available subsets ST , compared to existing sleeping bandits work for adversarial availabilities in the MAB setting [ 19 ] which only gives a worst-case regret bound over all possible availability sequences ( Rem . 3 ) . 4 . Finally we corroborate our theoretical results with extensive empirical evaluations . ( Sec . 6 ) . Related Works . The problem of regret minimization for stochastic multiarmed bandits ( MAB ) is extremely well studied in the online learning literature [ 6 , 1 , 22 , 5 , 16 ] , where the learner gets to see a noisy draw of absolute reward feedback of an arm upon playing a single arm per round . A well motivated generalization of MAB framework is Sleeping Bandits [ 17 , 24 , 18 , 15 ] , much studied in the online learning community , where at any round the set of available actions could vary stochastically based on some unknown distributions over the decision space of K items [ 24 , 11 ] or adversarially [ 15 , 19 , 18 ] . Besides the reward model , the set of available actions could also vary stochastically or adversarially [ 17 , 24 ] . The problem is NP-hard when both rewards and availabilities are adversarial [ 19 , 18 , 15 ] . In case of stochastic reward and adversarial availabilities [ 19 ] proposed an UCB based no-regret algorithm , which was also shown to be provably optimal . The case of adversarial reward and stochastic availabilities has also been studied where the achievable regret lower bound is known be Ω ( √ KT ) by the inefficient EXP4 algorithm [ 19 , 15 ] . On the other hand over the last decade , the relative feedback variants of stochastic MAB problem has seen a widespread resurgence in the form of the Dueling Bandit problem , where , instead of getting noisy feedback of the reward of the chosen arm , the learner only gets to see a noisy feedback on the pairwise preference of two arms selected by the learner . The objective of the learner is to minimize the regret with respect to ‘ best arm in the stochastic model . Several algorithms have been proposed to address this dueling bandits problem , for different notions of ‘ best arms ’ or preference models [ 10 , 31 , 38 , 37 , 36 , 21 , 33 , 13 ] , or even extending the pairwise preference to subsetwise preferences [ 29 , 8 , 26 , 27 , 25 ] . However , surprisingly , unlike the ‘ sleeping bandits generalization ’ of MAB , no parallel has been drawn for dueling bandits , which remains our main focus . 2 Problem Formulation . Notations . Decision space ( or item/arm set ) [ K ] : = { 1 , 2 , . . . , K } . The available set of items at round t is denoted by St ⊆ [ K ] . For any matrix M ∈ RK×K , we definemij : = M ( i , j ) , ∀i , j ∈ [ K ] . We write S\i = S \ { i } , for any S ⊆ [ K ] and i ∈ S. 1 ( · ) denotes the indicator random variable which takes value 1 if the predicate is true and 0 otherwise and . a rough inequality which holds up to universal constants . For any two items x , y ∈ [ K ] , we use the symbol x y to denote x is preferred over y. ΣK denotes the set of all permutations of the items in set [ K ] . The KL-divergence of two Bernoullis with biases p and q respectively is written kl ( p , q ) : = p log ( p/q ) + ( 1− p ) log ( ( 1−p ) / ( 1−q ) ) . We assume 00 : = 0.5 ( in Alg . 1 and 2 ) . Setup . We consider the problem of stochastic K-armed dueling bandits with adversarial availabilities : At every iteration t = 1 , . . . , T , a set of available items ( actions ) St ⊆ [ K ] is revealed , and the learner is asked to choose two items xt , yt ∈ St. Then , the learner receives a preference feedback ot = 1 ( xt yt ) ∼ Ber ( P ( xt , yt ) ) , where P ∈ [ 0 , 1 ] K×K is an underlying pairwise preference matrix , unknown to the learner . The setting is described in Figure 1 . We assume that P respects a ‘ total ordering ’ , say σ∗ ∈ ΣK . Without loss of generality , we set σ∗ = ( 1 , 2 , . . . , K ) thoughout the paper . This implies P ( i , j ) ≥ 0.5 for i ≤ j . One possible pairwise probability model which respects ‘ total ordering ’ is Plackett-Luce [ 7 ] , where it is assumed that the K items are associated to positive score parameters θ1 , . . . , θK , and P ( i , j ) = θi/ ( θi + θj ) for all i , j ∈ [ K ] . In fact any well random utility ( RUM ) based preference model would have the above property , like [ 7 , 28 ] . Note also that our assumption corresponds to assuming the existence of a Condorcet winner for every subset St ⊆ [ K ] . Objective . The objective of the learner is to minimize his regret over T rounds with respect to the best policy in the policy class Π = { π : 2K 7→ [ K ] | ∀t ∈ [ T ] , π ( St ) ∈ St } , i.e . any π ∈ Π is such that for any t ∈ [ T ] , π ( St ) ∈ St. More formally we define the regret as follows : RT = max π∈Π T∑ t=1 P ( π ( St ) , xt ) + P ( π ( St ) , yt ) − 1 2 . ( 1 ) We analyze both fixed-confidence and expected regret guarantees in this paper respectively in Sec . 4 ( see Thm . 3 ) and Sec . 5 ( see Thm . 6 ) . It is easy to note that under our preference modelling assumptions , the best policy , say π∗ , turns out to be π∗ ( S ) = min { S } for any S ⊆ [ K ] . We henceforth denote by i∗t = π ∗ ( St ) . We define the above problem to be Sleeping-Dueling Bandit with Stochastic Preferences and Adversarial Availabilities over the stochastic preference matrix P ∈ [ 0 , 1 ] K×K and the sequence of available subsets ST = { S1 , . . . , ST } , or in short DB-SPAA ( P , ST ) . For ease of notation we respectively define the gaps and the non-zeros gaps as ∆ ( i , j ) : = P ( i , j ) −1/2 , and ∆ ( i , j ) + : = { ∆ ( i , j ) if ∆ ( i , j ) 6= 0 +∞ if ∆ ( i , j ) = 0 ( 2 ) The regret thus can be rewritten as RT : = ∑T t=1 rt , where rt : = ( ∆ ( i ∗ t , xt ) + ∆ ( i ∗ t , yt ) ) /2 denotes the instantaneous regret . We also denote by nij ( t ) : = ∑t τ=1 1 ( { xt , yt } = { i , j } ) the number of times the pair ( i , j ) is played until time t and by wij ( t ) the number of times i beats j in t rounds .

Summary: This paper studies the sleeping dueling bandits with stochastic preferences and adversarial availabilities (DB-SPAA). The paper provides a lower bound for this problem, and proposes two algorithms with near optimal regret guarantees. Experimental results are shown to demonstrate the effectiveness of the proposed algorithms.