Dueling Bandits with Team Comparisons

1 Introduction . Multi-arm bandits ( MAB ) is a classical model of decision making under uncertainty . In spite of the simplicity of the model , it already incorporates the essential tradeoff between exploration and exploitation . In MAB , the learner performs actions and can only observe rewards of the actions performed . One of the main tasks in MAB is best arm identification , where the goal is to identify a near-optimal action while minimizing the number of actions executed . The MAB model has numerous practical applications , including online advertising , recommendation systems , clinical trials , and more . ( See Slivkins ( 2019 ) ; Lattimore & Szepesvári ( 2020 ) for more background ) . One weakness of the MAB model is the assumption that real-valued rewards are always available . In many applications , it is more natural to compare two actions and observe which one of them is better rather than give every single action a numerical reward . For example , recommendation systems often suggest two items and obtain only their relative preference as feedback ( e.g. , by a click on one of them ) . This leads very naturally to the well-known model of dueling bandits ( Yue et al. , 2012 ) , where the learner selects a pair of actions each time and observes the binary “ winner ” of a duel between the two . ( See Busa-Fekete et al . ( 2018 ) for a survey on extensions of the dueling bandit model ) . In this work we are interested in the case that the learner has to select two disjoint teams for a duel , which are k-sized subsets of the actions ( which we call players ) . This appears naturally in sports or online games , where the goal is to pick one of the best teams from a set of players by observing the outcomes of matches ( say , to be a school representative team , or to sponsor for tournaments ) . Examples include doubles tennis , basketball , and the online game League of Legends , where each match requires two disjoint teams of players to compete . Similar phenomena appear in working environments , where different R & D teams compete on implementing a project . Another example could be online advertisements where multiple products are bundled to a display ad and a customer ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can click on any of two presented bundles , e.g. , some online games offer in-app bundle purchases , and the information regarding sales of different bundles can improve the bundles ’ composition . Our basic model is the following . We have a universe of n players , and at each iteration the learner selects two disjoint teams for a duel and observes the winner . For any two different teams , there exists an unknown stationary probability that determines the winner of a duel between them . The requirement that dueling teams need to be disjoint is in accordance with the situation in games , where a single person can not play for both teams . The goal of the learner is to minimize the number of duels required to identify , with high probability , a Condorcet winning team , i.e. , a team which wins against any other disjoint team ( with a probability of at least 1/2 ) . We assume these probabilities are linked to a strict total order on all teams , which implies the existence of a Condorcet winning team , yet it is typically not unique . We make two minimal and natural assumptions on this total order on teams : that it is consistent to some total order among the players , and that team probabilistic comparisons hold Strong Stochastic Transitivity , a common assumption in dueling bandit settings . Our consistency assumption implies that the best team is the team of top-k players , and this also a Condorcet winning team . However , not all relations between players are deducible for the learner . In particular , even achieving accurate estimations of the latent winning probabilities between all disjoint teams might not suffice to separate the top-k players from the rest . Consider for example an instance with four players 1 ≻ 2 ≻ 3 ≻ 4 where k = 2 and the total order among the teams is lexicographical , i.e. , 12 ≻ 13 ≻ 14 ≻ 23 ≻ 24 ≻ 34 . Assume that each of the three feasible duels is won by the team containing player 1 with a fixed probability γ > 1/2 . Then , the learner has no chance of detecting the team 12 as the top-k team . However , any of the teams 12 , 13 and 14 is a Condorcet winning team . Our main target is to present algorithms for which the number of duels is bounded by a polynomial in the number of players n and the team size k , although the number of teams is exponential in k , i.e. , Ω ( ( n/k ) k ) and the number of valid duels is Ω ( 2k ( n2k ) 2k ) . Even if one were to accept an exponential number of arms , a direct reduction to the standard dueling bandits setting would not be feasible as not all pairs of teams are comparable in our model . In particular , duels of the form ( S ∪ { a } , S ∪ { b } ) , which would yield a signal regarding the relation between players a and b , are forbidden . The inherent difficulty of our endeavor comes from two limitations : ( 1 ) Not all the relations between two single players are deducible , ( see example above ) , and ( 2 ) even for pairs of players with deducible relation , having Ω ( 2k ( n2k ) 2k ) valid duels and the same amount of ( latent ) winning probabilities makes the task of deducing their relations hard . We start by giving a full characterization of the deducible pairwise relations between players , namely relations that can be detected by an unlimited amount of duels . Our characterization implies that every deducible players relation has one of two types of witnesses , which are constant-size sets of duels that prove their relation . Once we find a witness for one pair of players , it can often be transformed to a witness for other pairs of players . Building upon this characterization , we introduce a parameter ∆a , b which captures the distinguishability of any two players a and b and takes a value of 0 whenever the pair is not deducible . Assuming ∆ : = ∆k , k+1 > 0 , where k and k + 1 are kth and ( k + 1 ) th best players , we give a reduction to the classic dueling bandits problem . Combining this reduction with a high-probability top-k identification algorithm for the dueling bandits setting ( e.g. , Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) ) yields a similar sample complexity upper bound , e.g. , this implies a high-probability top-k identification algorithm for dueling teams with O ( ∆−2 ( n+ k log ( k ) ) max ( log log n , log k ) ) duels . Interestingly , it turns out that the deterministic case , i.e. , when winning probabilities are in { 0 , 1 } , constitutes a challenging special case of our problem where ∆ can be particularly small , or even 0 . To overcome this issue we design delicate algorithms which are independent of ∆ . On a high level , a preprocessing procedure first excludes as many bad players as possible . To do so , it runs a method for identifying pairwise relations between players which performs only a small number of duels , but has little control over the pair for which the relation is uncovered . For general total orders this implies an algorithm requiring O ( nk log ( k ) + 2O ( k ) ) duels . For the natural case of additive linear orders , we present a more elaborated approach for detecting a Condorcet winning team within the reduced instance , resulting in an algorithm that performs O ( nk log ( k ) + k5 ) duels . We introduce our model in Section 2 , give a characterization of deducible relations in Section 3 , and discuss the stochastic and deterministic setting in Sections 4 and 5 , respectively . In Section 6 we provide a discussion including a lower bound and a regret bound . Algorithms and ( full ) proofs are relegated to Sections A-D in the appendix , and Section E characterizes additive linear total orders . 1.1 Related Work . MAB best arm or subset identification single arm identification was initiated in Even-Dar et al . ( 2006 ) and later studied in many works including Bubeck et al . ( 2011 ) ; Kaufmann et al . ( 2016 ) ; Chen et al . ( 2017 ) . This setting was extended by Kalyanakrishnan & Stone ( 2010 ) for multiple arms identification ( i.e. , top k arms ) , using a single arm samples . Other works that address the objective of top−k identification include Chen et al . ( 2014 ) ; Zhou et al . ( 2014 ) ; Bubeck et al . ( 2013 ) . Dueling bandits The work of Yue et al . ( 2012 ) lay down the framework of non-parametric bandit feedback under total order among arms , strong stochastic transitivity , and stochastic triangle inequality assumptions and were followed by many subsequent works ( For more , see a survey , Busa-Fekete et al . ( 2018 ) . ) In particular , some subsequent works target the task of identifying the top k players in this setting Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) . Dueling bandits with sets of actions One line of dueling bandits extension consider the case where the learner selects a subset of actions and observes the outcomes of all duels between all pairs of actions in the subset ( Brost et al. , 2016 ; Sui et al. , 2017 ) , or the winner of the subset Saha & Gopalan ( 2018 ) ; Ren et al . ( 2018 ) . As a consequence , these settings give the learner strictly more information than the dueling bandits setting . In contrast , feedback in our setting reveals less information . MAB with multiple actions selection There are works in which the learner selects a ( sometimes fixed-sized ) subset of actions at each iteration , and observes either all of the individual selected arms rewards ( semi-bandit feedback ) or an aggregated form of the rewards ( full-bandit feedback ) , and the task is to detect to best arm or the top k. These include combinatorial bandits Cesa-Bianchi & Lugosi ( 2012 ) , top-k Rejwan & Mansour ( 2020 ) , linear bandit and routing Awerbuch & Kleinberg ( 2008 ) , and more . The main difference between combinatorial bandits and our setting is the feedback . Comparison models Noisy pairwise comparison models , especially for sorting and ranking , have a long history which dates backs to the 1950 ’ s ( For more , see a survey , Pelc ( 2002 ) . ) . Specifically , the mathematical problem Counterfeit coin was introduced in the form of a puzzle ( Grossman , 1945 ) : given a pile of 12 coins , determine which coins has a different weight ( and therefore counterfeit ) using balance scales while minimizing the number of measurements . The problem was followed by numerous generalizations ( see Guy & Nowakowski ( 1995 ) ) . While this problem is restricted to coins with two different weights , our setting can be seen as a variant with multiple weights . Finally , a PAC related comparison-based model was studied in Balcan et al . ( 2016 ) .

Summary: This paper considers a variation of the dueling bandits problem where teams of arms compete against each other rather than individual arms. The paper considers two different models for team-wise comparisons: (1) stochastic where each competing team wins with certain probability and the better team wins with probability >= 1/2; (2) deterministic where the better team always wins. This paper works in the pure exploration setting as the goal is to find the Condorcet winner rather than minimising regret. For the deterministic setting the paper shows that any solution for the classic dueling bandits problem can be used to solve the team-wise dueling bandits problem with bounded number of duels depending on a \emph{gap} parameter. For the deterministic setting the paper gives an algorithm to find the Condorcet winner with a bounded number of duels with no dependence on gap.