Dueling Bandits with Team Comparisons

1 Introduction . Multi-arm bandits ( MAB ) is a classical model of decision making under uncertainty . In spite of the simplicity of the model , it already incorporates the essential tradeoff between exploration and exploitation . In MAB , the learner performs actions and can only observe rewards of the actions performed . One of the main tasks in MAB is best arm identification , where the goal is to identify a near-optimal action while minimizing the number of actions executed . The MAB model has numerous practical applications , including online advertising , recommendation systems , clinical trials , and more . ( See Slivkins ( 2019 ) ; Lattimore & Szepesvári ( 2020 ) for more background ) . One weakness of the MAB model is the assumption that real-valued rewards are always available . In many applications , it is more natural to compare two actions and observe which one of them is better rather than give every single action a numerical reward . For example , recommendation systems often suggest two items and obtain only their relative preference as feedback ( e.g. , by a click on one of them ) . This leads very naturally to the well-known model of dueling bandits ( Yue et al. , 2012 ) , where the learner selects a pair of actions each time and observes the binary “ winner ” of a duel between the two . ( See Busa-Fekete et al . ( 2018 ) for a survey on extensions of the dueling bandit model ) . In this work we are interested in the case that the learner has to select two disjoint teams for a duel , which are k-sized subsets of the actions ( which we call players ) . This appears naturally in sports or online games , where the goal is to pick one of the best teams from a set of players by observing the outcomes of matches ( say , to be a school representative team , or to sponsor for tournaments ) . Examples include doubles tennis , basketball , and the online game League of Legends , where each match requires two disjoint teams of players to compete . Similar phenomena appear in working environments , where different R & D teams compete on implementing a project . Another example could be online advertisements where multiple products are bundled to a display ad and a customer ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can click on any of two presented bundles , e.g. , some online games offer in-app bundle purchases , and the information regarding sales of different bundles can improve the bundles ’ composition . Our basic model is the following . We have a universe of n players , and at each iteration the learner selects two disjoint teams for a duel and observes the winner . For any two different teams , there exists an unknown stationary probability that determines the winner of a duel between them . The requirement that dueling teams need to be disjoint is in accordance with the situation in games , where a single person can not play for both teams . The goal of the learner is to minimize the number of duels required to identify , with high probability , a Condorcet winning team , i.e. , a team which wins against any other disjoint team ( with a probability of at least 1/2 ) . We assume these probabilities are linked to a strict total order on all teams , which implies the existence of a Condorcet winning team , yet it is typically not unique . We make two minimal and natural assumptions on this total order on teams : that it is consistent to some total order among the players , and that team probabilistic comparisons hold Strong Stochastic Transitivity , a common assumption in dueling bandit settings . Our consistency assumption implies that the best team is the team of top-k players , and this also a Condorcet winning team . However , not all relations between players are deducible for the learner . In particular , even achieving accurate estimations of the latent winning probabilities between all disjoint teams might not suffice to separate the top-k players from the rest . Consider for example an instance with four players 1 ≻ 2 ≻ 3 ≻ 4 where k = 2 and the total order among the teams is lexicographical , i.e. , 12 ≻ 13 ≻ 14 ≻ 23 ≻ 24 ≻ 34 . Assume that each of the three feasible duels is won by the team containing player 1 with a fixed probability γ > 1/2 . Then , the learner has no chance of detecting the team 12 as the top-k team . However , any of the teams 12 , 13 and 14 is a Condorcet winning team . Our main target is to present algorithms for which the number of duels is bounded by a polynomial in the number of players n and the team size k , although the number of teams is exponential in k , i.e. , Ω ( ( n/k ) k ) and the number of valid duels is Ω ( 2k ( n2k ) 2k ) . Even if one were to accept an exponential number of arms , a direct reduction to the standard dueling bandits setting would not be feasible as not all pairs of teams are comparable in our model . In particular , duels of the form ( S ∪ { a } , S ∪ { b } ) , which would yield a signal regarding the relation between players a and b , are forbidden . The inherent difficulty of our endeavor comes from two limitations : ( 1 ) Not all the relations between two single players are deducible , ( see example above ) , and ( 2 ) even for pairs of players with deducible relation , having Ω ( 2k ( n2k ) 2k ) valid duels and the same amount of ( latent ) winning probabilities makes the task of deducing their relations hard . We start by giving a full characterization of the deducible pairwise relations between players , namely relations that can be detected by an unlimited amount of duels . Our characterization implies that every deducible players relation has one of two types of witnesses , which are constant-size sets of duels that prove their relation . Once we find a witness for one pair of players , it can often be transformed to a witness for other pairs of players . Building upon this characterization , we introduce a parameter ∆a , b which captures the distinguishability of any two players a and b and takes a value of 0 whenever the pair is not deducible . Assuming ∆ : = ∆k , k+1 > 0 , where k and k + 1 are kth and ( k + 1 ) th best players , we give a reduction to the classic dueling bandits problem . Combining this reduction with a high-probability top-k identification algorithm for the dueling bandits setting ( e.g. , Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) ) yields a similar sample complexity upper bound , e.g. , this implies a high-probability top-k identification algorithm for dueling teams with O ( ∆−2 ( n+ k log ( k ) ) max ( log log n , log k ) ) duels . Interestingly , it turns out that the deterministic case , i.e. , when winning probabilities are in { 0 , 1 } , constitutes a challenging special case of our problem where ∆ can be particularly small , or even 0 . To overcome this issue we design delicate algorithms which are independent of ∆ . On a high level , a preprocessing procedure first excludes as many bad players as possible . To do so , it runs a method for identifying pairwise relations between players which performs only a small number of duels , but has little control over the pair for which the relation is uncovered . For general total orders this implies an algorithm requiring O ( nk log ( k ) + 2O ( k ) ) duels . For the natural case of additive linear orders , we present a more elaborated approach for detecting a Condorcet winning team within the reduced instance , resulting in an algorithm that performs O ( nk log ( k ) + k5 ) duels . We introduce our model in Section 2 , give a characterization of deducible relations in Section 3 , and discuss the stochastic and deterministic setting in Sections 4 and 5 , respectively . In Section 6 we provide a discussion including a lower bound and a regret bound . Algorithms and ( full ) proofs are relegated to Sections A-D in the appendix , and Section E characterizes additive linear total orders . 1.1 Related Work . MAB best arm or subset identification single arm identification was initiated in Even-Dar et al . ( 2006 ) and later studied in many works including Bubeck et al . ( 2011 ) ; Kaufmann et al . ( 2016 ) ; Chen et al . ( 2017 ) . This setting was extended by Kalyanakrishnan & Stone ( 2010 ) for multiple arms identification ( i.e. , top k arms ) , using a single arm samples . Other works that address the objective of top−k identification include Chen et al . ( 2014 ) ; Zhou et al . ( 2014 ) ; Bubeck et al . ( 2013 ) . Dueling bandits The work of Yue et al . ( 2012 ) lay down the framework of non-parametric bandit feedback under total order among arms , strong stochastic transitivity , and stochastic triangle inequality assumptions and were followed by many subsequent works ( For more , see a survey , Busa-Fekete et al . ( 2018 ) . ) In particular , some subsequent works target the task of identifying the top k players in this setting Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) . Dueling bandits with sets of actions One line of dueling bandits extension consider the case where the learner selects a subset of actions and observes the outcomes of all duels between all pairs of actions in the subset ( Brost et al. , 2016 ; Sui et al. , 2017 ) , or the winner of the subset Saha & Gopalan ( 2018 ) ; Ren et al . ( 2018 ) . As a consequence , these settings give the learner strictly more information than the dueling bandits setting . In contrast , feedback in our setting reveals less information . MAB with multiple actions selection There are works in which the learner selects a ( sometimes fixed-sized ) subset of actions at each iteration , and observes either all of the individual selected arms rewards ( semi-bandit feedback ) or an aggregated form of the rewards ( full-bandit feedback ) , and the task is to detect to best arm or the top k. These include combinatorial bandits Cesa-Bianchi & Lugosi ( 2012 ) , top-k Rejwan & Mansour ( 2020 ) , linear bandit and routing Awerbuch & Kleinberg ( 2008 ) , and more . The main difference between combinatorial bandits and our setting is the feedback . Comparison models Noisy pairwise comparison models , especially for sorting and ranking , have a long history which dates backs to the 1950 ’ s ( For more , see a survey , Pelc ( 2002 ) . ) . Specifically , the mathematical problem Counterfeit coin was introduced in the form of a puzzle ( Grossman , 1945 ) : given a pile of 12 coins , determine which coins has a different weight ( and therefore counterfeit ) using balance scales while minimizing the number of measurements . The problem was followed by numerous generalizations ( see Guy & Nowakowski ( 1995 ) ) . While this problem is restricted to coins with two different weights , our setting can be seen as a variant with multiple weights . Finally , a PAC related comparison-based model was studied in Balcan et al . ( 2016 ) .

Summary: The paper introduces a new problem in which a learner can choose k disjoint teams from a set of n players and observe a winner. The objective is to pick a team that would win against any other team with probability at least 0.5 (Condorcet winner) and to do with a minimal number of duels. Deterministic and Stochastic settings are considered and upper bounds on the number of duels are obtained.