Identification of the Generalized Condorcet Winner in Multi-dueling Bandits

1 Introduction . The standard multi-armed bandit ( MAB ) problem describes a sequential decision scenario , in which one of finitely many choice alternatives must be selected in each time step , resulting in the observation of a numerical reward of stochastic nature . One important and extensively studied variant of the MAB setting is the dueling bandits problem , where a duel consisting of two arms is chosen in each time step and one of the duelling arms is observed as the winner [ 4 ] . Recently , the multi-dueling bandits setting has been introduced [ 7 , 40 , 31 ] as a generalization with multiple practically relevant applications , such as algorithm configuration [ 13 ] or online retrieval evaluation [ 36 ] . Instead of pairs of arms , in this generalization a set consisting of k ≥ 2 arms can be chosen in each time step . These arms compete against each other and determine a single winner , which is observed as feedback by the learner . The outcomes of the ( multi- ) duels in the ( multi- ) dueling bandit scenario are typically assumed to be of time-stationary stochastic nature in the sense that whenever arms a1 , . . . , ak compete against each other , then ai wins with some underlying ( unknown ) ground-truth probability P ( ai| { a1 , . . . , ak } ) . One often targeted learning task in the context of multi-armed bandits and its variants is the problem of identifying the best among all arms . While for standard MABs , the canonical definition of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . “ best arm ” is the arm with highest expected reward , the picture is less clear for its variants . In the realm of dueling bandits , any arm that is likely to win ( i.e. , with probability > 1/2 ) in each duel against another arm is called the Condorcet winner ( CW ) . This notion dates back to the 18th century [ 8 ] and also appears in the social choice literature [ 16 , 17 ] , where the data is typically assumed to be available in the form of a list containing total rankings over all alternatives from different voters . In practice , the Condorcet winner does not necessarily exist due to the presence of preferential cycles in the probabilistic model in the sense that ai is likely to win against aj , aj against ak , and ak against ai . For the theoretical analysis of the best-arm-identification problem , this issue is overcome in the literature either by the consideration of alternative optimality concepts such as Borda winner or Copeland winner , which are guaranteed to exist , or by simply assuming the existence of the CW . In this paper , we focus on finding a generalized variant of the CW in the multi-dueling bandits setting under the assumption that it exists . There have been several suggestions for generalizations of the CW in social choice . For example , a weighted variant is introduced in [ 30 ] , where the weights control the relevance given to the ranking positions of the alternatives , while in [ 25 ] the notion of a k-winner is defined as an alternative that ( in some appropriate sense ) outperforms all other arms among any k alternatives . In contrast to our work , these papers focus on offline learning tasks and suppose full rankings over all alternatives to be given . In this paper , we adapt the notion of generalized Condorcet winner ( GCW ) as in [ 1 ] , i.e. , a GCW is an arm ai that outperforms each arm aj in every query set S containing both ai and aj , in the sense that P ( ai|S ) ≥ P ( aj |S ) . Regarding the dueling bandits setting as the multi-dueling setting where the allowed multi-duels S are exactly those with |S| = 2 , the GCW is indeed a generalization of the Condorcet winner . We analyze the sample complexity of ( probabilistic ) algorithms that are able to identify the GCW with high probability under the assumption of mere existence as well as more restrictive assumptions . We provide upper and lower bounds for this task , which depend on the desired confidence , the total number m of alternatives , the size k of allowed query sets as well as the underlying unknown preference probabilities P ( ai|S ) . We start in Section 2 with a brief literature overview on the multi-dueling bandits scenario . Section 3 introduces the basic formalism and a precise definition of the considered GCW identification problem . It also gives a rough , simplified overview of the sample complexity bounds obtained in this paper . In Section 4 , we discuss the special case m = k , in which the GCW identification problem essentially boils down to the task of finding the mode of a categorical distribution . We provide solutions to this problem and prove their sample complexity to be optimal up to logarithmic factors in the worst-case sense . Section 5 focuses on lower bounds for the general case m ≥ k , and in Section 6 , we discuss several upper bounds . In Section 7 , we empirically compare the algorithms discussed before , prior to concluding in Section 8 . For the sake of convenience , detailed proofs of all theoretical results presented in the paper are deferred to the supplemental material . 2 Related Work . Initially , the multi-dueling bandit problem was studied intensively in the case of pairs as actions of the learner , which is also better known as dueling bandits [ 42 ] . The extension to the scenario considered in this paper , where more general sets as pairs of arms are selectable as an action , has been the focus of recent work . Part of these works model the feedback process by essentially tracing it back to the dueling bandit case [ 7 , 40 , 31 ] . The majority of papers , however , assume latent utility values for the arms and model the feedback process using a random utility model ( RUM ) [ 3 ] based on these utility values . Thanks to the latent utility values , an ordering of the arms is obtained quite naturally , which in turn makes it easy to define an objective such as the optimal arm or the top-k arms . Under these assumptions , the PB-MAB problem was investigated with respect to various performance metrics such as the regret [ 32 , 5 , 1 ] or the sampling complexity in an ( , δ ) -PAC setting [ 33 , 34 , 35 ] . In [ 1 ] a different approach is taken by generalizing the concept for the naturally optimal arm in the dueling bandit case , namely the Condorcet winner ( CW ) , under the term generalized Condorcet winner ( GCW ) . The optimal arm defined in this way coincides with the optimal arm if latent utility values for the arms and a RUM for the feedback process are assumed . While in [ 1 ] the problem for finding this GCW is investigated in a regret minimization scenario , we are interested in the minimum sampling complexity . In light of this , the work by [ 35 ] is the most related to ours , although the authors assume a PL model ( a special case of a RUM ) . If we restrict the learner ’ s actions to pairs of arms in our more general setting , i.e. , the dueling bandits case , the GCW and the CW coincide . This special case of our problem setting has been dealt with by [ 26 ] , [ 20 ] and [ 29 ] . Finally , it remains to mention that there are a number of similar problem scenarios , namely the Stochastic click model ( SCM ) [ 43 ] , the dynamic assortment problem ( DAS ) [ 9 ] and the best-of-kbandits [ 39 ] . However , all these scenarios take into account other specific aspects in the modelling such as the order of the arms in the action subset ( SCM ) , known revenues associated with the arms ( DAS ) or a so-called “ no-choice option '' ( all three ) . Accordingly , these problem scenarios are fundamentally different from our learning scenario ( see also Sec . 6.6 in [ 4 ] for a more detailed discussion ) . The same is true for combinatorial bandits [ 10 ] , which also allow subsets of arms as actions , but differ fundamentally in the nature of feedback ( quantitative vs. qualitative feedback ) . 3 The GCW Identification Problem . For adequately stating our results , we introduce in the following some basic terminology and notations used throghout this paper . For the sake of convenience , Table 1 summarizes the most frequently used notations . 3.1 The Notion of a GCW . If not explicitly stated otherwise , we suppose throughout the paper the total number of arms m , the query set size k ∈ { 2 , . . . , m } , a desired confidence 1 − γ ∈ ( 0 , 1 ) and a complexity parameter h ∈ ( 0 , 1 ) to be arbitrary but fixed . We write [ m ] : = { 1 , . . . , m } and [ m ] k : = { S ⊆ [ m ] | |S| = k } . Parameter spaces of categorical distributions . For any subset of size k , i.e. , S ∈ [ m ] k , define ∆S : = { p = ( pj ) j∈S ∈ [ 0 , 1 ] |S| | ∑ j∈S pj = 1 } as the set of all possible parameters for a categorical random variable X ∼ Cat ( ( pj ) j∈S ) , i.e. , P ( X = j ) = pj for any j ∈ S. For p ∈ ∆S , we write mode ( p ) : = arg maxj∈S pj and in case |mode ( p ) | = 1 we denote by mode ( p ) — with a slight abuse of notation — also the unique element in mode ( p ) . Let us define for h ∈ ( 0 , 1 ] the sets ∆hS : = { p ∈ ∆S | ∃i ∈ S s.t . pi ≥ maxj∈S\ { i } pj + h } , and with this ∆0S : = ⋃ h∈ ( 0,1 ) ∆ h S . These sets are nested in the sense that ∆ h S ⊆ ∆h ′ S ⇔ h ≥ h′ . If p ∈ ∆S is fixed , the value h ( p ) : = max { h ∈ [ 0 , 1 ] |p ∈ ∆hS } is well-defined and we have p ∈ ∆hS iff h ≤ h ( p ) . Obviously , the equivalence |mode ( p ) | = 1⇔ p ∈ ∆0S holds for all p ∈ ∆S . Probability models on [ m ] k. A family P = { P ( · |S ) } S∈ [ m ] k of parameters P ( · |S ) ∈ ∆S , S ∈ [ m ] k , is called a probability model ( short : PM ) on [ m ] k. We write PMmk for the set of all probability models on [ m ] k and define the following subsets of PMmk : PMmk ( ∆ 0 ) : = { P = { P ( · |S ) } S∈ [ m ] k ∣∣ ∀S ∈ [ m ] k : P ( · |S ) ∈ ∆0S } , PMmk ( ∆ h ) : = { P = { P ( · |S ) } S∈ [ m ] k ∣∣ ∀S ∈ [ m ] k : P ( · |S ) ∈ ∆hS } , PMmk ( PL ) : = { { P ( · |S ) } S∈ [ m ] k ∣∣ ∃θ ∈ ( 0 , ∞ ) m ∀S ∈ [ m ] k : P ( i |S ) = θi/ ( ∑j∈S θj ) } . Note that PMmk ( PL ) denotes the set of all probability models P consistent with a Plackett-Luce ( PL ) model [ 27 , 23 ] . Let h ( P ) : = maxh∈ [ 0,1 ] { P ∈ PMmk ( ∆h ) } = minS∈ [ m ] k h ( P ( · |S ) ) , then it is easy to see that P ∈ PMmk ( ∆h ) iff h ≤ h ( P ) . An element i ∈ [ m ] is called a generalized Condorcet Winner ( short : GCW ) of P if ∀S ∈ [ m ] k with i ∈ S , ∀j ∈ S : P ( i |S ) −P ( j |S ) ≥ 0 and we write GCW ( P ) for the set of all GCWs of P. With this , we define the following subsets of PMmk related to the concept of the GCW : PMmk ( ∃GCW ) : = { P = { P ( · |S ) } S∈ [ m ] k ∣∣GCW ( P ) 6= ∅ } , PMmk ( ∃GCW ∗ ) : = { P = { P ( · |S ) } S∈ [ m ] k ∣∣ |GCW ( P ) | = 1 } , PMmk ( ∃hGCW ) : = { { P ( · |S ) } S∈ [ m ] k ∣∣ ∃i : ∀S ∈ [ m ] k , j ∈ S \ { i } : P ( i |S ) −P ( j |S ) ≥ h } . Clearly , it holds that PMmk ( ∃GCW ∗ ) = ⋃ h > 0 PM m k ( ∃hGCW ) and every probability model P ∈ PMmk ( ∃GCW ) has at least one GCW , while for a probability model P ∈ PMmk ( ∃GCW ∗ ) the GCW is unique . ( ∃GCW ) ( ∃GCW∗ ) ( ∃hGCW ) ( PL ) ( ∆0 ) ( ∆h ) The figure on the right illustrates the relationships between the introduced subsets of PMmk . For the sake of convenience , we write simply ( X ) instead of PMmk ( X ) , where X ∈ { ∃GCW , ∃GCW∗ , ∃hGCW , ∆0 , ∆h , PL } . This convention will be used several times in the course of the paper .

Summary: This work studies the BAI problem in the multi-dueling bandits.  It proposes algorithms with theoretical guarantee and provides related lower bounds. There are detailed comparisons to existing works. Moreover, it compare the proposed algorithm with other existing algorithms numerically.