Dueling Bandits with Team Comparisons

1 Introduction . Multi-arm bandits ( MAB ) is a classical model of decision making under uncertainty . In spite of the simplicity of the model , it already incorporates the essential tradeoff between exploration and exploitation . In MAB , the learner performs actions and can only observe rewards of the actions performed . One of the main tasks in MAB is best arm identification , where the goal is to identify a near-optimal action while minimizing the number of actions executed . The MAB model has numerous practical applications , including online advertising , recommendation systems , clinical trials , and more . ( See Slivkins ( 2019 ) ; Lattimore & Szepesvári ( 2020 ) for more background ) . One weakness of the MAB model is the assumption that real-valued rewards are always available . In many applications , it is more natural to compare two actions and observe which one of them is better rather than give every single action a numerical reward . For example , recommendation systems often suggest two items and obtain only their relative preference as feedback ( e.g. , by a click on one of them ) . This leads very naturally to the well-known model of dueling bandits ( Yue et al. , 2012 ) , where the learner selects a pair of actions each time and observes the binary “ winner ” of a duel between the two . ( See Busa-Fekete et al . ( 2018 ) for a survey on extensions of the dueling bandit model ) . In this work we are interested in the case that the learner has to select two disjoint teams for a duel , which are k-sized subsets of the actions ( which we call players ) . This appears naturally in sports or online games , where the goal is to pick one of the best teams from a set of players by observing the outcomes of matches ( say , to be a school representative team , or to sponsor for tournaments ) . Examples include doubles tennis , basketball , and the online game League of Legends , where each match requires two disjoint teams of players to compete . Similar phenomena appear in working environments , where different R & D teams compete on implementing a project . Another example could be online advertisements where multiple products are bundled to a display ad and a customer ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can click on any of two presented bundles , e.g. , some online games offer in-app bundle purchases , and the information regarding sales of different bundles can improve the bundles ’ composition . Our basic model is the following . We have a universe of n players , and at each iteration the learner selects two disjoint teams for a duel and observes the winner . For any two different teams , there exists an unknown stationary probability that determines the winner of a duel between them . The requirement that dueling teams need to be disjoint is in accordance with the situation in games , where a single person can not play for both teams . The goal of the learner is to minimize the number of duels required to identify , with high probability , a Condorcet winning team , i.e. , a team which wins against any other disjoint team ( with a probability of at least 1/2 ) . We assume these probabilities are linked to a strict total order on all teams , which implies the existence of a Condorcet winning team , yet it is typically not unique . We make two minimal and natural assumptions on this total order on teams : that it is consistent to some total order among the players , and that team probabilistic comparisons hold Strong Stochastic Transitivity , a common assumption in dueling bandit settings . Our consistency assumption implies that the best team is the team of top-k players , and this also a Condorcet winning team . However , not all relations between players are deducible for the learner . In particular , even achieving accurate estimations of the latent winning probabilities between all disjoint teams might not suffice to separate the top-k players from the rest . Consider for example an instance with four players 1 ≻ 2 ≻ 3 ≻ 4 where k = 2 and the total order among the teams is lexicographical , i.e. , 12 ≻ 13 ≻ 14 ≻ 23 ≻ 24 ≻ 34 . Assume that each of the three feasible duels is won by the team containing player 1 with a fixed probability γ > 1/2 . Then , the learner has no chance of detecting the team 12 as the top-k team . However , any of the teams 12 , 13 and 14 is a Condorcet winning team . Our main target is to present algorithms for which the number of duels is bounded by a polynomial in the number of players n and the team size k , although the number of teams is exponential in k , i.e. , Ω ( ( n/k ) k ) and the number of valid duels is Ω ( 2k ( n2k ) 2k ) . Even if one were to accept an exponential number of arms , a direct reduction to the standard dueling bandits setting would not be feasible as not all pairs of teams are comparable in our model . In particular , duels of the form ( S ∪ { a } , S ∪ { b } ) , which would yield a signal regarding the relation between players a and b , are forbidden . The inherent difficulty of our endeavor comes from two limitations : ( 1 ) Not all the relations between two single players are deducible , ( see example above ) , and ( 2 ) even for pairs of players with deducible relation , having Ω ( 2k ( n2k ) 2k ) valid duels and the same amount of ( latent ) winning probabilities makes the task of deducing their relations hard . We start by giving a full characterization of the deducible pairwise relations between players , namely relations that can be detected by an unlimited amount of duels . Our characterization implies that every deducible players relation has one of two types of witnesses , which are constant-size sets of duels that prove their relation . Once we find a witness for one pair of players , it can often be transformed to a witness for other pairs of players . Building upon this characterization , we introduce a parameter ∆a , b which captures the distinguishability of any two players a and b and takes a value of 0 whenever the pair is not deducible . Assuming ∆ : = ∆k , k+1 > 0 , where k and k + 1 are kth and ( k + 1 ) th best players , we give a reduction to the classic dueling bandits problem . Combining this reduction with a high-probability top-k identification algorithm for the dueling bandits setting ( e.g. , Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) ) yields a similar sample complexity upper bound , e.g. , this implies a high-probability top-k identification algorithm for dueling teams with O ( ∆−2 ( n+ k log ( k ) ) max ( log log n , log k ) ) duels . Interestingly , it turns out that the deterministic case , i.e. , when winning probabilities are in { 0 , 1 } , constitutes a challenging special case of our problem where ∆ can be particularly small , or even 0 . To overcome this issue we design delicate algorithms which are independent of ∆ . On a high level , a preprocessing procedure first excludes as many bad players as possible . To do so , it runs a method for identifying pairwise relations between players which performs only a small number of duels , but has little control over the pair for which the relation is uncovered . For general total orders this implies an algorithm requiring O ( nk log ( k ) + 2O ( k ) ) duels . For the natural case of additive linear orders , we present a more elaborated approach for detecting a Condorcet winning team within the reduced instance , resulting in an algorithm that performs O ( nk log ( k ) + k5 ) duels . We introduce our model in Section 2 , give a characterization of deducible relations in Section 3 , and discuss the stochastic and deterministic setting in Sections 4 and 5 , respectively . In Section 6 we provide a discussion including a lower bound and a regret bound . Algorithms and ( full ) proofs are relegated to Sections A-D in the appendix , and Section E characterizes additive linear total orders . 1.1 Related Work . MAB best arm or subset identification single arm identification was initiated in Even-Dar et al . ( 2006 ) and later studied in many works including Bubeck et al . ( 2011 ) ; Kaufmann et al . ( 2016 ) ; Chen et al . ( 2017 ) . This setting was extended by Kalyanakrishnan & Stone ( 2010 ) for multiple arms identification ( i.e. , top k arms ) , using a single arm samples . Other works that address the objective of top−k identification include Chen et al . ( 2014 ) ; Zhou et al . ( 2014 ) ; Bubeck et al . ( 2013 ) . Dueling bandits The work of Yue et al . ( 2012 ) lay down the framework of non-parametric bandit feedback under total order among arms , strong stochastic transitivity , and stochastic triangle inequality assumptions and were followed by many subsequent works ( For more , see a survey , Busa-Fekete et al . ( 2018 ) . ) In particular , some subsequent works target the task of identifying the top k players in this setting Mohajer et al . ( 2017 ) ; Ren et al . ( 2020 ) . Dueling bandits with sets of actions One line of dueling bandits extension consider the case where the learner selects a subset of actions and observes the outcomes of all duels between all pairs of actions in the subset ( Brost et al. , 2016 ; Sui et al. , 2017 ) , or the winner of the subset Saha & Gopalan ( 2018 ) ; Ren et al . ( 2018 ) . As a consequence , these settings give the learner strictly more information than the dueling bandits setting . In contrast , feedback in our setting reveals less information . MAB with multiple actions selection There are works in which the learner selects a ( sometimes fixed-sized ) subset of actions at each iteration , and observes either all of the individual selected arms rewards ( semi-bandit feedback ) or an aggregated form of the rewards ( full-bandit feedback ) , and the task is to detect to best arm or the top k. These include combinatorial bandits Cesa-Bianchi & Lugosi ( 2012 ) , top-k Rejwan & Mansour ( 2020 ) , linear bandit and routing Awerbuch & Kleinberg ( 2008 ) , and more . The main difference between combinatorial bandits and our setting is the feedback . Comparison models Noisy pairwise comparison models , especially for sorting and ranking , have a long history which dates backs to the 1950 ’ s ( For more , see a survey , Pelc ( 2002 ) . ) . Specifically , the mathematical problem Counterfeit coin was introduced in the form of a puzzle ( Grossman , 1945 ) : given a pile of 12 coins , determine which coins has a different weight ( and therefore counterfeit ) using balance scales while minimizing the number of measurements . The problem was followed by numerous generalizations ( see Guy & Nowakowski ( 1995 ) ) . While this problem is restricted to coins with two different weights , our setting can be seen as a variant with multiple weights . Finally , a PAC related comparison-based model was studied in Balcan et al . ( 2016 ) .

Summary: This paper studies a variant of dueling bandits with teams. There are n players, and your goal is to select the “best” team of k of these players (more specifically, you want to select a team of k of these players that doesn’t lose to any team of size k that you can form from the remainder of the n-k players). Each round you can pick two teams of size k, watch them compete, and see who wins. The probabilities that one team beats another are chosen in such a way to be consistent with an underlying ordering of the players, in addition to satisfying some other assumptions (e.g. strong stochastic transitivity).  The authors study this problem in both the stochastic setting and the deterministic setting (where all the probabilities are guaranteed to be 0/1). In the stochastic setting, they can identify an optimal team whp with O((n + klog(k))/Delta^2), where Delta is some notion of “gap” between the performance of the kth player and the (k+1)th player. They accomplish this by first showing how to compare two players (roughly, you do something like pick a random S of size (k-1) and compare S + {a} to S + {b}, but something a little more complex than this). They then use this to reduce the problem to the well-studied problem of top K identification under pairwise comparisons.   In the deterministic case (where Delta is usually 0, so the previous bounds don’t apply), they give an algorithm which identifies a “best” team with O(kn log (k) + 2^{O(k)}) comparisons. Furthermore, when the order is an “additive total order” (i.e. generated by just adding values corresponding to specific members), they give an algorithm that only uses O(kn log(k) + k^5). These algorithms are complex combinatorial algorithms -- the main idea is showing that it is possible to identify useful comparisons somewhat quickly (especially under additive orders).