Optimal Algorithms for Stochastic Contextual Preference Bandits 

√ T ) regret guarantee , while the second algorithm is shown to achieve the optimal Õ ( √ dT ) regret , as follows from our Ω ( √ dT ) matching lower bound analysis . We then proceed to analyze the problem for any general q-subsetwise preferences ( q ≥ 2 ) , where surprisingly , our lower bound proves the fundamental performance limit to be Ω ( √ dT ) yet again , independent of the subsetsize q . Following this , we propose a matching upper bound algorithm justifying the tightness of our results . This implies having access to subsetwise preferences does not help in faster information aggregation for our feedback model . All the results are corroborated empirically against existing baselines . 1 Introduction . Sequential decision-making problems with side information ( in the form of features or attributes ) , have been popular in machine learning as contextual bandits [ 16 , 13 , 26 ] . A contextual bandit learner , at each round , observes a context before taking action based on it . The resulting payoff is typically assumed to depend on the context and the action taken according to an unknown map . The learner aims to play the best possible action for the current context at each time and thus minimize its regret with respect to an oracle that knows the payoff function . In many learning settings , however , it is more common to be able to only relatively compare actions , in a decision step , instead of being able to gauge their absolute utilities . E.g. , information retrieval , search engine optimization , recommender systems , crowdsourcing , drug testing , tournament ranking , social surveys etc . [ 18 , 21 ] . A specific application example is : Consider the problem of recommending items from a catalog to users on a shopping website . Each time , the context is determined by the visiting user ’ s features together with all items ’ features . When a subset of items is presented , the user clicks on one of them according to a relative preference model ; only the items presented matter – this plausibly models comparative cognitive choices being made by humans . The aim is to converge to identify the overall best item in the catalog . Additionally , since our model is designed to leverage ∗Microsoft Research , New York , US ; aasa @ microsoft.com . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the structured preference feedback , we can handle very large decision spaces , unlike state of the art dueling bandits algorithms that only deals with pairwise preferences [ 47 ] , [ 23 ] [ 43 ] , [ 41 ] etc. , or even some of the recent works on general subsetwise preference bandits ( e.g . [ 30 , 33 , 38 ] ) . All of them need to maintain a K2 matrix explicitly , and their regret scales as O ( K ) . This becomes impractical for large ( or infinite ) action spaces of size K. In this work , we consider a natural structured contextual preference bandit setting , comprised of items with intrinsic ( absolute ) scores depending on their features in an unknown way , e.g. , linear with unknown weights . In the most general setup , the learner plays ( compares ) a subset 2 ≤ q ≤ K items and gets to see a ( noisy ) ` -length rank-ordered feedback ( where 1 ≤ ` ≤ q ) of the top-ranked items of the selected subset with a probability distribution governed by the items ’ scores . We are primarily interested in developing adaptive subset-selection algorithms for which guarantees can be given for a suitably defined measure of regret . To the best of our knowledge , we are the first to give theoretical guarantees for the above problem of regret minimization in contextual preference bandits for potentially infinitely large decision spaces and design provably optimal algorithms for the same . Some recent works [ 17 , 38 ] have considered this problem in the subset selection setup but their algorithms do not guarantee any finite time regret bounds , neither validate their performance optimality theoretically . In [ 15 ] , authors considered a version of adversarial contextual dueling bandit problem , which only takes into account the special case of pairwise feedback ( q = 2 ) . Though similar in names , their problem framework and consequently the analysis are very different from ours . In their setup , at each time the preference matrix is determined by a randomly chosen known context variable , and the algorithm is assumed to have access to a pool of finite set of policies . The goal of the learner is to compete with the ‘ optimal policy in the pool ’ for which they proposed an sparring-EXP4 like algorithm—their regret bound though scales as O ( √ T ) in T , it depends linearly on the size of arm space ( K in their notation ) , and thus become vacuous for infinitely large decision space . Moreover , their computational complexity also scales linearly with the size of the policy class , which tends to be prohibitively large in practice . 1 . We propose the problem of structured contextual preference bandits , where at each iteration , the learner is presented with a set of K-arms St ( each represented by d-dimensional feature vectors ) , and the task of the learner is to select a subset of at most q-items ( 1 ≤ q ≤ K ) , with the objective being to identify the ‘ best arm ’ of St in every round . The novelty of the framework lies in the relative preference based feedback model , which only allows the learner to see a noisy draw of the top-ranked items of the selected subset , unlike the absolute reward feedback used for standard bandits setup . 2 . We first address the problem for the special case of dueling bandits , where q = 2 , i.e . the learner only have access to pairwise preferences of the selected item pairs at each round . We propose two algorithms for the basic dueling bandit setting : Our first algorithm , Maximum-Informative-Pair ( Alg 1 ) , is based on the idea of selecting the most uncertain pair ( ‘ max-variance ’ ) from the set of ‘ promising candidates ’ , and we prove an O ( d √ T ) regret for the same ( Thm . 3 ) . 3 . Our second algorithm Stagewise-Adaptive-Duel ( Alg . 3 ) , is developed on the idea of tracking , in a phased fashion , the best arm of the context set , which ensures a sharper concentration rate of the pairwise scores . This results in Õ ( √ dT ) 2 regret guarantee ( Thm . 5 ) , improving upon the regret bound of our previous algorithm by a √ d factor . 4 . We also show that fundamental regret lower bound of Ω ( √ dT ) for the contextual dueling bandit problem addressed here ( Thm . 3 ) . Thus theoretically our second algorithm ( Alg . 3 ) is provably optimal , however our Alg . 1 often works better in practice as we show in the experiments ( Sec . 6 ) . 5 . We then analyze the problem for more general subsetwise preference feedback , where at each round , the learner is allowed to play a subset of q-items ( q ≥ 2 ) , upon which the winner feedback of the played subset is revealed [ 10 , 30 , 33 ] . We first prove an Ω ( √ dT ) regret lower bound ( Thm . 11 ) for the problem , which establishes that , interestingly , having access to subsetwise preferences does not really help in faster information aggregation ( for the specific preference model considered here ) . Subsequently , we also discuss an algorithm with near-optimal regret guarantee , whose regret bound is also independent of the subsetsize q up to logarithmic factors ( Thm . 12 ) . 2The notation Õ ( · ) hides logarithmic dependencies . Finally , we corroborate our theoretical findings with empirical evaluations . Detailed Related works , Experiments and most of the technical proofs are moved to the appendix . Related Works . Surprisingly , following the same spirit of extending standard multiarmed bandits ( MAB ) to continuous decision spaces ( as in linear or GP-bandits ) , there has been really very little work on the continuous extension of the Dueling Bandit problem [ 24 ] . The works in [ 38 , 17 ] did attempt a similar objective , however , without any satisfactory theoretical performance guarantees . [ 24 ] considers the problem of dueling bandits on continuous arm set but under rather restrictive sets of assumptions : Twice continuously differentiable , lipschitz , strongly convex and smooth score/reward function , which are often impractical for modeling any real-world preference feedback . Recently , [ 28 ] consider the problem of k-way assortment selection , where the problem is to minimize regret against the set of highest revenue . However , their objective is much different from ours . We focus on the regret against the single best item per iteration . Thus our pairwise action set must allow repeated item pulls , unlike their setup , due to which their algorithm does not lead to sublinear regret for our objective . In another recent work , [ 10 ] did address the problem of regret minimization in continuous Dueling Bandits , however without any finite time regret guarantee of their proposed algorithms , unlike ours . A more elaborated survey is given in Appendix A . 2 Preliminaries and Problem Formulation . Notations . For any positive integer n ∈ N+ , we denote by [ n ] the set { 1 , 2 , ... , n } . 1 ( ϕ ) is generically used to denote an indicator variable that takes the value 1 if the predicate ϕ is true , and 0 otherwise . The decision space is denoted by D ⊆ Rd , where d ∈ N+ . We use 1d to denote an d-dimensional vector of all 1 ’ s . For any matrix M ∈ Rd×d , we denote respectively by λmax ( M ) and λmin ( M ) the maximum a minimum eigenvalue of matrix M . For any x ∈ Rd , ‖x‖M : = √ x > Mx denotes the weighted ` 2-norm associated with matrix M ( assuming M is positive-definite ) . 2.1 Problem Setup . We consider the stochastic K-armed contextual dueling bandit problem for T rounds , where at each round t ∈ [ T ] , the learner is presented with a context set St = { xt1 , xt2 , . . . , xtK } ⊆ D ⊂ Rd of size K which is drawn IID from some d-dimensional decision space D ( according to some unknown distribution on D , say PD ) . The learner is permitted to play a subset Xt ⊆ St of size q ≥ 2 , given a fixed q ≤ K ( see the formal setup in Sec . 2.1 ) . Clearly for q = 2 , the problem reduces to the sequential duel ( pair of items ) selection , say in this case we denote Xt = { xt , yt } . Upon this , the environment provides a stochastic subsetwise preference feedback as follows : Subsetwise-Preference Feedback Model . At any round t , upon selecting Xt , the learner receives a winner feedback ot such that : Pr ( ot = x | Xt ) = e g ( x ) ∑ y∈Xt eg ( y ) for any x ∈ Xt , where g : D 7→ [ 0 , 1 ] is a utility score function on each point in the decisions space x ∈ D. Note our preference model essentially boils down to the well studied Plackett Luce ( PL ) choice model with the individual PL score of item-x being eg ( x ) [ 21 , 35 , 32 , 30 ] . If q = 2 , then ot = 1 ( xt preferred over yt ) simply indicates the preferred arm of the duel ( xt , yt ) , such that for any x , y ∈ D , the probability x is preferred over y , denoted by Pr ( x y ) , is drawn according to ∼ Ber ( σ ( h ( x , y ) ) ) , here σ ( · ) being the sigmoid transformation ( i.e . σ ( x ) = 11+e−x for any x ∈ R ) . Analysis with linear scores . In this paper , we assume that g ( x ) = x > θ∗ , ∀x ∈ D , where θ∗ ∈ Rd is some unknown fixed vector in Rd such that ‖θ∗‖ ≤ 1 . We will henceforth denote this linear utility based ‘ subsetwise preference model ’ as SPM ( θ∗ , d , q ) . Objective : Regret Minimization . Suppose x∗t : = arg maxx∈St x > θ ∗ is the best arm ( with highest score ) of round t. Then the goal of the learner is to minimize the T -round cumulative regret RT = ∑T t=1 rt with respect to the best arm x ∗ t of each round t , where we measure the instantaneous regret rt of playing a set Xt in terms of the average score of the played duel ∑ x∈Xt x > θ∗ |Xt| . Precisely , RT = T∑ t=1 ( x∗ > t θ ∗ − ∑ x∈Xt x > θ∗ |Xt| ) . ( 1 ) Above notion of learner ’ s regret is motivated from the definition of classical K-armed dueling bandit regret introduced by [ 44 ] which is later adopted by the dueling bandit literature [ 46 , 23 , 3 , 43 , 45 , 38 , 31 ] . Here the context set at any round t is assumed to be a fixed set of K arms St = [ K ] , and at each round the instantaneous regret incurred by the learner for playing an arm-pair ( it , jt ) ∈ [ K × K ] is given by r ( DB ) t = P ( i∗ , it ) +P ( i∗ , jt ) −1 2 , i∗ ∈ [ K ] being the ‘ best-arm ’ in the hindsight ( e.g . condorcet winner [ 46 ] or copeland winner [ 23 , 41 ] ) depending on the underlying preference matrix P ∈ [ 0 , 1 ] K×K . Remark 1 ( Equivalence with Dueling Bandit Regret when q = 2 ) . It is easy to note that assuming the context set St ⊆ D to be fixed ∀t ∈ [ T ] and denoting x∗t = x∗ , our regret definition ( Eqn . ( 1 ) ) is equivalent to dueling bandit regret ( up to constant factors ) , as in our case the pairwise advantage against the best arm ( i.e . Pr ( x∗ , xt ) − 12 ) can be both upper and lower bounded as : ( 1 ) Pr ( x∗ , xt ) − 12 = ( ex ∗ > θ∗−ex > t θ ∗ ) 2 ( ex∗ > θ∗+ex > t θ ∗ ) ≤ ( x ∗−xt ) > θ∗ 2 and ( 2 ) Pr ( x ∗ , xt ) − 12 ≥ ( x∗−xt ) > θ∗ 4e . Combining above claims we get RT4e ≤ R ( DB ) T ≤ RT 2 ( analysis detail given in Appendix B.1 ) . 3 Dueling Feedback ( q = 2 ) : Algorithm and Analysis We first analyze the problem for pairwise preference feedback ( q = 2 case ) . Before proceeding to the actual algorithms , it is crucial to note that , same as generalized linear Bandits ( GLB ) [ 16 , 26 ] , both our algorithms use standard MLE techniques for maintaining a ‘ tight estimate ’ of θ∗ ( Line 5 of Alg . 1 , Line 8 of Alg . 3 ) . This is since our dueling preference feedback can be seen as a generalized linear reward over item pairs ( details in Appendix C ) . However , the regret definition of dueling bandits ( Eqn . ( 1 ) ) being very different than GLB objective , a direct application of any GLB algorithm will simply lead to O ( T ) regret in our setup : The objective of any GLB learner is to converge to the arm with highest reward , unlike ours . Thus any GLM routine would always converge to the best-worst arm pair as that would be perceived to be the duel with highest reward ( pairwise preference in our case ) . On the contrary , to have any sublinear regret , we require the learner to eventually play only the best arm in the duel , which does not have the highest pairwise-preference ( reward ) . This is the inherent complexity and primary difference of any dueling bandit problem w.r.t . GLB objective or any MAB framework per se . As a result , a sublinear MAB algorithm never works for dueling/preference bandit objectives . To circumvent the problem , we design fairly non-trivial arm-selection rules for our proposed algorithms , e.g. , Alg . 1 first needs to construct a promising-set Ct and then pick the maximum informative pair amongst them ( Line 6-7 , Alg1 ) . Alg . 2 needs to optimally maintain the set of ’ good-items ’ Gs with a careful arm-selection rule which significantly differs from any GLB approach ( Line 19 , Alg 2 ) . Consequently , we also need to resolve to new proof ideas towards analyzing their regret guarantees which remains one of the primary novelty of this work . 3.1 Algorithm-1 : Maximum-Informative-Pair Our first algorithm is a computationally efficient one with a O ( d √ T ) regret guarantee ( Thm . 3 ) , which is only suboptimal by a factor of O ( √ d ) ( as reflects from our lower bound , Thm . 10 , Sec . 4 ) . Main Idea : At any time t , the algorithm simply maintains an UCB estimate on the pairwise scores h̄ ( x , y ) : = θ̂ > ( x − y ) + η‖x − y‖V −1t for any pair of arms ( x , y ) , x , y ∈ St , where Vt = ∑t−1 τ=1 ( xτ − yτ ) ( xτ − yτ ) > , θ̂ being the maximum likelihood estimator ( MLE ) of our preference model parameter . It then builds a set of promising arms Ct : = { x ∈ St | h̄ ( x , y ) > 0 , ∀y ∈ y ∈ St \ { x } } : Arms that beats the rest in terms of their UCB score h̄ ( x , y ) . Finally it plays the most-uncertain ( least sampled ) pair ( xt , yt ) : = arg maxx , y∈Ct ‖x− y‖V −1t . Note ( xt , yt ) is the pair with highest pairwise score variance in Ct , hence ‘ maximum informative ’ . ( Detail in Alg . 1 . ) Analysis . Regret guarantee of Alg.1 ( Thm . 3 ) is based on the following main lemmas . Lemma 1 ( Self-Normalized Bound ) . Suppose { ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xt , yt ) } be a sequence of arm-pair played such that all arms x ∈ { xτ , yτ } tτ=1 belong to the ball of unit radius . Also suppose the initial exploration length t0 be such that λmin ( ∑t0 τ=1 ( xτ −yτ ) ( xτ −yτ ) > ) ≥ 1 . Then ∀ t > t0 , ∑t τ=t0+1 ‖ ( xτ−yτ ) ‖V −1τ+1 ≤ √√√√2dt log ( 4t0+td ) , where recall Vτ+1 : = ∑τ j=1 ( xj−yj ) ( xj−yj ) > . Algorithm 1 Maximum-Informative-Pair ( MaxInP ) 1 : input : Learning rate η > 0 , exploration length t0 > 0 2 : init : Select t0 pairs { ( xτ , yτ ) } τ∈ [ t0 ] , each drawn at random from Sτ , and observe the corre- sponding preference feedback { oτ } τ∈ [ t0 ] 3 : Set Vt0+1 : = ∑t0 τ=1 ( xτ − yτ ) ( xτ − yτ ) > 4 : for t = t0 + 1 , t0 + 2 , . . . T do 5 : Compute the MLE θ̂t on { ( xτ , yτ , oτ ) } t−1τ=1 : ∑t−1 τ=1 ( oτ − σ ( ( xτ − yτ ) > θ̂t ) ) ( xτ − yτ ) = 0 6 : Ct : = { x , y ∈ St | ( x− y ) > θ̂t + η‖ ( x− y ) ‖V −1t > 0 } 7 : Compute ( xt , yt ) : = arg maxx , y∈Ct ‖x− y‖V −1t 8 : Play the duel ( xt , yt ) . Receive ot = 1 ( xt beats yt ) 9 : Update Vt+1 = Vt + ( xt − yt ) ( xt − yt ) > 10 : end for Lemma 2 ( Confidence Ellipsoid ) . Suppose the initial exploration length t0 be such that λmin ( ∑t0 τ=1 ( xτ − yτ ) ( xτ − yτ ) > ) ≥ 1 , and κ is as defined in Thm . 3 . Then for any δ > 0 , with probability at least ( 1− δ ) , for all t > t0 , ‖θ∗ − θ̂t‖Vt ≤ 12κ √ d 2 log ( 1 + 2td ) + log 1δ , where recall Vt+1 : = ∑t τ=1 ( xτ − yτ ) ( xτ − yτ ) > . Theorem 3 ( Regret bound of Maximum-Informative-Pair ( Alg . 1 ) ) . Let η = 1 2κ √ d 2 log ( 1 + 2T d ) + log 1 δ , where κ : = inf‖x−y‖≤2 , ‖θ∗−θ̂‖≤1 [ σ′ ( ( x − y ) > θ̂ ) ] is the mini- mum slope of the estimated sigmoid when θ̂ is sufficiently close to θ∗ ( σ′ ( · ) being the first order derivative of the sigmoid function σ ( · ) ) . Then given any δ > 0 , with probability at least ( 1− 2δ ) , the T round cumulative regret of Maximum-Informative-Pair satisfies : RT ≤ t0 + ( 1 κ √ d 2 log ( 1 + 2T d ) + log 1 δ ) √√√√2dT log ( 4t0 + T d ) = O ( d √ T log ( T dδ ) ) , where we choose t0 = 2 ( C1 √ d+C2 √ log ( 1/δ ) λmin ( B ) ) 2 + 4λmin ( B ) , B = Ex , y iid∼PD [ ( x− y ) ( x− y ) > ] ( for some universal problem independent constants C1 , C2 > 0 ) . Proof . ( sketch ) Our choice of t0 ensures that with probability at least ( 1− δ ) , Vt0+1 is full rank , or more precisely λmin ( Vt0+1 ) ≥ 1 [ 42 ] ( see Lem . 13 , Appendix D for the formal statement ) . We next apply the two key concentration lemmas ( Lem 1 and 2 ) , upon expressing the regret definition in terms of the above concentration results . Precisely , using Lem . 2 and our ‘ most informative pair ’ based arm selection strategy , we can show at any round t > t0 , we can bound rt ≤ 4‖xt − yt‖V −1t . The results now follows from the choice of η and Lem.1 . The complete proof is given in Appendix D.1 . 3.2 Algorithm-2 : Stagewise-Adaptive-Duel ( Sta′D ) ) Our second algorithm runs with a provable optimal regret bound of Õ ( √ dT ) , except with an additional√ logK factor . When K = O ( 1 ) , the algorithm thus yields an optimal regret guarantee . Main Idea . The algorithm proceeds in stages s ∈ blog T c with the aim of tracking a set of ‘ promising arms ’ Gs per stage : At each such stage s , we maintain confidence interval on the pairwise scores of each index pair ( i , j ) pst ( i , j ) . If at any stage s , the confidence-score of any arm-pair is not estimated to the ‘ sufficient accuracy ’ , we play that pair and include it in the set of ‘ informative pairs ’ of stage φs to be further explored in following rounds . Otherwise , we sequentially eliminate the ‘ weakly-performing ’ arms which gets defeated by some other arm in terms of its optimistic pairwise score , and proceed to the next stage s + 1 to examine the surviving arms with a stricter confidence interval . Now if the pairwise scores of every index pair in the set of ‘ promising-arms ’ Gs is almost ‘ accurately estimated up to high confidence ’ , we pick the first arm xt as the one which has the maximum estimated score , followed by choosing its strongest challenger yt which beats xt with highest pairwise preference . The algorithm is given in Alg . 3 ( Appendix 3 ) . Analysis . Thm . 5 proves an optimal Õ ( √ dT ) regret bound of Alg . 3 ( matching the regret lower bound , Thm . 10 ) . It is worth pointing that the near optimal regret analysis of Stagewise-AdaptiveDuel crucially relies on the stronger concentration guarantees of the pairwise scores ( compared to the weaker concentration of Lem . 2 used earlier for Alg . 1 ) . Note , this is made possible by specifically maintaining the independent ‘ stage-wise informative samples ’ φs as also pioneered in few of the earlier works [ 8 , 13 ] for multi-armed bandits ( see our proof analysis of Lem . 4 ) . Lemma 4 ( Sharper Concentration of Pairwise Scores ) . Consider any δ > 0 , and suppose we set the parameters of Stagewise-Adaptive-Duel ( Alg . 3 ) as η = 32κ √ 2 log 3TKδ , where κ : = inf‖x−y‖≤2 , ‖θ∗−θ̂‖≤1 [ σ′ ( ( x− y ) > θ̂ ) ] , and t0 = 2 ( C1 √ d+C2 √ log ( 2/δ ) λmin ( B ) ) 2 + 4Λλmin ( B ) , where Λ = 8 κ4 ( d2 + log 3δ ) and B = E x , y iid∼PD [ ( x − y ) ( x − y ) > ] ( for some universal problem independent constants C1 , C2 > 0 ) . Then with probability at least ( 1− δ ) , for all stages s ∈ dlog T e at all rounds t > t0 and for all index pairs i , j ∈ Gs of round t : | ( xti − xtj ) > ( θ∗ − θst ) ≤ pst ( i , j ) | . Theorem 5 ( Regret bound of Stagewise-Adaptive-Duel ( Alg . 3 ) ) . Consider we set t0 and η as per Lem . 4 . Then for any δ > 0 , with probability at least ( 1− δ ) , the regret of Alg . 3 can be bounded as : RT ≤ t0 + 4η √ 2d log ( 4t0T d ) √ T log T + 2 √ T = O ( √ dT log T κ √ log ( TK δ ) log ( Td κ log 1 δ ) ) Proof . ( sketch ) Suppose we denote by φc : = { t ∈ [ T ] \ [ t0 ] | t /∈ ∪blog Tcs=1 φ s } the set of all good time intervals where all the index pairs pst ( i , j ) are estimated within the confidence accuracy 1√ T . The proof crucially relies on the concentration bound of Lem . 4 , from which we first derive : Lemma 6 . For any t > t0 , suppose the pair ( xt , yt ) is chosen at stage st , and i∗t denotes the best action of round t , i.e . xti∗t = x ∗ t = arg maxx∈St x > θ∗ . Then for any δ ∈ ( 0 , 1 ) , with probability at least ( 1−δ ) , for all t > t0 : i∗t ∈ Gst and for both x ∈ { xt , yt } , g ( x∗t ) −g ( x ) ≤ { 2√ T if t ∈ φc 4 2st otherwise . Owning to Lem . 1 and due to the construction of our ‘ stagewise-good item pairs ’ we can also show : Lemma 7 . At any stage s ∈ blog T c , with probability at least ( 1−δ ) , √ |φs| ≤ η2s √ 2d log ( 4t0T d ) . The final regret bound can be derived combining the results of Lem . 6 and 7 . ( see Appendix E.5 ) .

Summary: This paper studies the problem of preference bandits in the contextual setting, where the online learner plays $q$ arms and observes a noisy winner of the subset. The goal is to identify the best arm given a context. Two special algorithms for the $q = 2$ case are presented, along with the regret upper bound and a matching lower bound. The results are then extended to a general $q$. Corresponding experiments are presented to verify the theoretical results.