Title: On Weak Regret Analysis for Dueling Bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dueling bandits

Weak regret

Preference-based learning

Bandits theory
Abstract: We consider the problem of $K$-armed dueling bandits in the stochastic setting, under the sole assumption of the existence of a Condorcet winner. We study the objective of weak regret minimization, where the learner doesn't incur any loss if one of the selected arms is a Condorcet winner—unlike strong regret minimization, where the learner has to select the Condorcet winner twice to incur no loss. This study is particularly motivated by practical scenarios such as content recommendation and online advertising, where frequently only one optimal choice out of the two presented options is necessary to achieve user satisfaction or engagement. This necessitates the development of strategies with more exploration. While existing literature introduces strategies for weak regret with constant bounds (that do not depend on the time horizon), the optimality of these strategies remains an unresolved question.  This problem turns out to be really challenging as the optimal regret should heavily depend on the full structure of the dueling problem at hand, and in particular on whether the Condorcet winner has a large minimal optimality gap with the other arms. Our contribution is threefold: first, when said optimality gap is not negligible compared to other properties of the gap matrix, we characterize the optimal budget as a function of $K$ and the optimality gap. Second, we propose a new strategy called \wrtinf that achieves this optimal regret and improves over the state-of-the-art both in $K$ and the optimality gap. When the optimality gap is negligible, we propose another algorithm that outperforms our first algorithm, highlighting the subtlety of this dueling bandit problem.  Finally, we provide numerical simulations to assess our theoretical findings.

On Weak Regret Analysis for Dueling Bandits

Abstract

We consider the problem of K -armed dueling bandits in the stochastic setting, under the sole assumption of the existence of a Condorcet winner. We study the objective of weak regret minimization, where the learner doesn't incur any loss if one of the selected arms is a Condorcet winner-unlike strong regret minimization, where the learner has to select the Condorcet winner twice to incur no loss. This study is particularly motivated by practical scenarios such as content recommendation and online advertising, where frequently only one optimal choice out of the two presented options is necessary to achieve user satisfaction or engagement. This necessitates the development of strategies with more exploration. While existing literature introduces strategies for weak regret with constant bounds (that do not depend on the time horizon), the optimality of these strategies remains an unresolved question. This problem turns out to be really challenging as the optimal regret should heavily depend on the full structure of the dueling problem at hand, and in particular on whether the Condorcet winner has a large minimal optimality gap with the other arms. Our contribution is threefold: first, when said optimality gap is not negligible compared to other properties of the gap matrix, we characterize the optimal budget as a function of K and the optimality gap. Second, we propose a new strategy called WR-TINF that achieves this optimal regret and improves over the state-of-the-art both in K and the optimality gap. When the optimality gap is negligible, we propose another algorithm that outperforms our first algorithm, highlighting the subtlety of this dueling bandit problem. Finally, we provide numerical simulations to assess our theoretical findings.

1 Introduction

We consider an instance of the problem of sequential and active learning from comparisons - namely dueling bandits. It can be modeled as a sequential game where, at each time, a learner presents to a user a pair of two items and collects feedback, which is a noisy indication of the user's preference between the two items. If neither of the presented items aligns with the user's top choice, the learner incurs a loss. Preference-based learning has gained importance recently as it reflects human decisionmaking processes, which often rely on relative rather than absolute evaluations. This approach is notably effective in systems that involve human interaction, where feedback is provided in a qualitative form [7]. In the dueling bandit setting, initially presented by [19], the compared items

are called 'arms' and there are K of them. This setting is structured as an ongoing sequential game with time horizon T , where in each round t ≤ T , the learner selects two arms (items) indexed by i, j ∈ [ K ] , and receives the result of a duel between these arms as feedback. The result of each duel is encoded by 1 if the first arm - in our case arm i - beats the second one - in our case arm j - and 0 otherwise. This result follows a Bernoulli distribution with unknown parameter q i,j . Specifically, this paper focuses on the stochastic scenario where the parameters governing the duels are assumed to be constant throughout the game, albeit unknown to the learner. The literature on the stochastic dueling bandits is very rich and contains various settings that differ from our paper either in optimal arm characterization or the definition of rewards.

In contrast to the classic multi-armed bandit problem, defining the optimal arm in dueling bandits is not straightforward. This led to the introduction of various definitions of winners within the literature as detailed in the surveys [2, 16, 17]. Our study focuses on situations where there is an arm k ∗ ∈ [ K ] that, on average, defeats all other arms: formally, q k ∗ ,j > 1 / 2 for any j ∈ [ K ] \ { k ∗ } . This arm is termed the Condorcet winner while we refer to it as optimal in the rest of the paper. In the context of dueling bandits, particularly the cumulative regret minimization problem, most of prior works either made the assumption of the existence of a Condorcet winner [24, 23, 10, 8, 4, 14, 15] or the stronger assumption of the exitence of a total order between arms [18, 20, 4].

Once the concept of the optimal arm is established, the next step is to define the objective. Rather than identifying the best arm, our goal is to minimize the cumulative loss. To this end, we must determine the loss incurred each round based on the two arms selected by the learner. The dueling bandit literature distinguishes between two primary types of losses: strong loss and weak loss, as described by [18]. With strong loss, the learner must select the Condorcet winner twice to avoid any loss (noting that the feedback in this case is equivalent to a fair coin flip). In contrast, weak loss requires only one of the selected arms to be the Condorcet winner. Formal definitions of weak and strong regrets are provided in Section 2. In many practical scenarios, such as recommendation systems and online advertising [6, 3], minimizing weak regret aligns more closely with the learner's objectives than strong regret minimization. For example, consider a situation where the learner operates as a service provider, presenting two options to a client who then chooses their preferred option. In this framework, the learner should incur a loss only if neither option matches the client's preference, encouraging exploration and maximizing information gain. While previous research in dueling bandits has primarily focused on minimizing strong regret, developing optimal strategies for minimizing weak regret remains an unresolved issue despite prior works [4, 12], as highlighted in the survey [2]. Further details on the technical distinctions between these two objectives will be discussed in the next sections.

In this paper, we focus on minimising the weak regret. We provide a lower bound for this problem in a specific regime where the Condorcet winner beats largely the other arms. We provide an algorithm that matches it. Nevertheless, it is not optimal in all regimes, and we highlight this by providing another algorithm that performs better in some interesting regimes.

2 Problem setting

We consider K arms. Let Q = ( q i,j ) 1 ≤ i,j ≤ K ∈ [0 , 1] K × K be the matrix of preference probabilities where the probability of arm i beating arm j in a duel corresponds to q i,j . We assume that q j,i = 1 -q i,j and q i,i = 1 / 2 for all i, j ∈ [ K ] . Define ∆ i,j := q i,j -1 2 . Notably, the sign of ∆ i,j indicates the relative preference between arms i and j (specifically, i is preferred over j if ∆ i,j > 0 ). The quantity ∆ i,j characterizes the hardness of distinguishing which of the arms ( i, j ) is preferred to the other. We denote ∆ := (∆ i,j ) the gap matrix. The only assumption made in this paper is regarding the existence of a Condorcet winner, which we denote k ∗ for the remainder of this paper:

Assumption 2.1. Existence of a Condorcet winner: There exists an arm k ∗ ∈ [ K ] such that:

We consider that at each time t = 1 , 2 , . . . , the learner chooses two arms ( I t , J t ) based on past information and receives the output of a duel between the chosen arms. More formally, the output is a sample from a Bernoulli distribution with parameter q I t ,J t , independent of everything else after conditioning on ( I t , J t ) . We consider that after each round t the learner incurs a loss given by:

which we term the weak instantaneous loss, following [18]. Another concept of instantaneous loss that is often considered in the literature is the strong instantaneous loss [18, 1, 24], where at round t the learner incurs the loss ℓ ( s ) t := (∆ k ∗ ,I t + ∆ k ∗ ,J t ) / 2 . Note that when it comes to the weak instantaneous loss, in contrast to the strong instantaneous loss, the learner does not incur any loss if at least one of the two chosen arms is the Condorcet winner, I t = k ∗ or J t = k ∗ , while for the strong instantaneous loss, both arms need to be the Condorcet winner in order for the learner to not incur a loss. We finally define the weak expected cumulative regret up to time T by for the weak instantaneous loss by R ( w ) T := ∑ T t =1 E [ ℓ ( w ) t ] , which we term weak regret. Similarly, we define the strong regret as R ( s ) T := ∑ T t =1 E [ ℓ ( s ) t ] .

3 Literature review and our contributions

3.1 Related work

When it comes to the stochastic dueling bandit literature, much of the prior work has been devoted to the goal of minimizing the strong regret, under the assumption of a total order between the arms [19, 18] or only under the assumption of the existence of a Condorcet winner [24, 23, 10, 8, 12]. We detail nevertheless those results here as, since the strong regret upper bounds the weak regret, all algorithms and upper bounds that are available for the strong regret also hold for the weak regret. In [8], an instance-dependent lower bound for strong regret was established:

̸

where O k = { i ∈ [ K ] | q i,k > 1 / 2 } . This work also introduces an algorithm that asymptotically matches this lower bound as T → + ∞ . However, in finite-horizon, their regret bound has a quadratic dependence on the number of arms K . Deriving bounds that scale linearly with K has been the subject of several works [23, 10], In particular, [14] devised a reduction to a standard (but adversarial) multi-armed bandits problem. They obtained guarantees on the strong regret which are of the order ∑ k = k ∗ log( T ) / ∆ k ∗ ,k . This regret bound turns out to match (1) in scenarios where the Condorcet winner is also the arm that is best for eliminating all other sub-optimal arms, namely where ∆ k ∗ ,k = max i ∆ i,k . In more general cases, the last upper bound of [14] does not match the lower bound given in (1).

̸

̸

Weak regret itself was introduced in [18] to model in a more refined way some recommender systems applications. As mentioned, it is upper bounded by the strong regret so that all described algorithms and associated regret upper bounds would also hold for the weak regret. However, a distinction was made in [4] regarding the fundamental nature of these two problems. While the problem-dependent optimal order of the strong regret scales as log T (see above) - which is aligned with classical results in stochastic bandits - there exist some algorithms whose problem-dependent weak regret is upper bounded by a quantity that does not depend on T - which is in sharp contrast with classical results on stochastic bandits. Specifically, [4] introduced an algorithm called WS-W, which, under the sole assumption of the existence of a Condorcet winner, achieves an upper bound on weak regret of the order K 2 / min i = j ∆ 2 i,j . More recently, in [12], the Beat The Winner (BTW) algorithm was introduced. BTW adopts a round-based approach where the best arm so far keeps being challenged through batches of duels by candidate arms. Assuming only the presence of a Condorcet winner, this algorithm achieves an upper bound on weak regret of the order K 2 + K/ min i = k ∗ ∆ 4 k ∗ ,i . Finally, under the additional and arguably the strong assumption of the existence of a total order between arms, the upper bound can be proven to be of order ( K log K ) / min i = j | ∆ i,j | 5 . In summary, the dependency on the optimal regret on both K and on the matrix ∆ still remains largely unknown.

̸

From a technical standpoint, developing optimal strategies in the weak regret framework underlies different challenges than the ones for the strong regret. This complexity arises because losses in the strong regret framework are linear in the problem parameters (the gaps matrix entries (∆ i,j ) 1 ≤ i,j ≤ K ): ℓ ( s ) t = (∆ k ∗ ,I t +∆ k ∗ ,J t ) / 2 , whereas in weak regret, the loss is determined as the minimum gap with the client's preference: ℓ ( w ) t = min { ∆ k ∗ ,I t , ∆ k ∗ ,J t } , which breaks linearity. As a result, classical reduction methods as the one used in [1, 14, 15] are not directly applicable for weak losses.

̸

3.2 Main contributions

In this paper, we address the following fundamental question:

(i) What is the best possible weak regret one can achieve in terms of K and the gaps (∆ k ∗ ,i ) i = k ∗ to the Condorcet winner?

̸

̸

(ii) Beyond that, is it possible to improve the regret by leveraging over the unknown entries of the matrix ∆ ? As a simple toy example, assume that the gaps ∆ k ∗ ,k are small and that, some gaps ( ∆ i,j ) for i, j = k ∗ are much higher. In that regime, it is perhaps more beneficial to explore the O ( K 2 ) duels between all arms to better discard sub-optimal arms than simply to directly look for the Condorcet winner. This informal argument suggests the optimal guarantees depend in an intricate way on the number of arms K and the gaps (∆ i,j ) and that there is a complex trade-off between directly aiming for the Condorcet winner and further exploration for better elimination.

̸

To address the first question, we provide a lower bound on the weak regret, which, to the best of our knowledge, is the first of its kind for this problem. We demonstrate that in certain cases, where in particular the gaps between the Condorcet winner and the sub-optimal arms are larger than the gaps between sub-optimal arms, the bound K/ min i = k ∗ ∆ k ∗ ,i is not improvable (see Section 5).

We introduce and analyze two new procedures. First, we provide in Section 4.1 an algorithm WR-TINF (Weak Regret-Tsallis INF) whose weak regret is bounded by

̸

̸

̸

This can be upper-bounded by K/ (min i = k ∗ ∆ k ∗ ,i ) . This improves over the state-of-the art [4, 12] (where bounds are respectively of the order of K 2 / min i = j ∆ 2 i,j and K 2 + K/ min i = k ∗ ∆ 4 k ∗ ,i ) both in the dependency with respect to K and the gaps. Also, we do not require the strong stochastic transitivity assumption, required in [12] 2 . Conversely, the bound K/ min i = k ∗ ∆ k ∗ ,i turns out to be impossible to improve in general -see Section 5.

̸

Second, we introduce in Section 4.2 the algorithm WR-EXP3-IX (Weak Regret EXP3-IX), which, from an heuristic viewpoint aims at eliminating sub-optimal arms by looking at duels between sub-optimal arms. For any ∆ , its weak regret is at most of the order of

̸

̸

where j ∗ ( i ) ∈ arg max j ∆ j,i and ∆ ∗ = min k = k ∗ ∆ k ∗ ,k . In the case, where the gaps ∆ j ∗ ( i ) ,i are larger (up to log-terms) than ∆ k ∗ ,i √ K , the regret guarantee (3) for WR-EXP3-IX becomes smaller than (2) for WR-TINF. Up to our knowledge, WR-EXP3-IX is the first algorithm in weak regret minimization that builds upon the complete structure of the gaps matrix ∆ to lower the regret.

̸

To further discuss the difference between the performances of both procedures, let us consider a toy model where, for some positive constants, ∆ cw and ∆ sub, we have, for any i = k ∗ , ∆ k ∗ ,i = ∆ cw, that is the gap between the Condorcet winner and the sub-optimal arms. Besides, for any i = k ∗ , there exists j ∗ ( i ) such that ∆ j ∗ ( i ) ,i = ∆ sub. We distinguish two main regimes: (a) If ∆ cw / ∆ sub ≥ 1 / √ K , then the weak regret of WR-TINF is the better one and is of the order of K/ ∆ cw . (b) If ∆ cw / ∆ sub ≤ 1 / √ K , then a transition occurs. To show that an arm k is not the Condorcet winner, then it now becomes beneficial to identify arms that provide the most evidence for the suboptimality of k . Here, WR-EXP3-IX achieves the better guarantee which is (up to log terms) of the order of K 2 (∆ cw / ∆ 2 sub ) .

̸

The presented algorithms use different techniques: we develop WR-TINF using an adaptation of the standard reduction technique (discussed in Section 4.1). We extend the idea of using a best-of-both worlds procedure as a base algorithm to sample each of the two arms I t and J t . However, since only one of the sampled arms should be optimal, we modify the sampling distribution prescribed by the

̸

̸

base algorithm to induce more exploration. The second procedure, WR-EXP3-IX, uses a different approach. Given the value of the left arm I t (selected in a round-robin manner), we use the EXP3-IX algorithm [11] to select the right arm J t . Then after a fixed number of rounds, the choice of J t (given the value of I t ) concentrates around the arm with highest probability of defeating it. We leverage the fact that when I t is the Condorcet winner, the gaps are positive, while for sub-optimal arms the minimal gap is negative.

4 Upper Bounds

This section presents two algorithms with guarantees on weak regret. Recall that we present two strategies since we identified two regimes as discussed in Section 3.2. Each of the algorithms we present is optimal in one of the regimes and none of them require prior knowledge on the problem parameters.

The first algorithm, WR-TINF, is built upon an adaptation of the reduction technique to a standard multi-armed bandit problem. Its upper bounds depend on the gaps between sub-optimal arms and the Condorcet winner (∆ k ∗ ,k ) k ∈ [ K ] . As demonstrated in the results of Section 5, this algorithm is optimal for some regimes.The second procedure, WR-EXP3-IX, aims for the task of identifying, for each arm, the arm that can eliminate it most rapidly (i.e., the arm with the largest gap). While this strategy results in a quadratic dependence on K , we argue that it outperforms WR-TINF for some instances.

4.1 Algorithm 1: Weak Regret Tsallis-INF

We adopt a previously explored approach [1, 15, 14], where the dueling bandit problem is converted into two separate multi-armed bandit problems - one for each arm pulled. This reduction was originally applied in the context of strong regret. However, adapting this approach to weak regret requires a more nuanced approach.

The idea of reducing a dueling bandit problem to a standard one was first introduced in [1] where it was termed Sparring in the context of minimizing strong regret. The high-level idea of this technique is to view the problem of selecting the the arm pair ( I t , J t ) as two individual multi-armed bandit (MAB) problems. The choice of I t (resp. J t ) can be performed by the first (resp. second) player, following which they incur a loss denoted ℓ -1 ,t ( I t ) := X t ( I t , J t ) (resp. ℓ +1 ,t ( J t ) := 1 -X t ( I t , J t ) ), where X t ( I t , J t ) ∼ Ber ( q I t ,J t ) . Here the subscript -1 (resp. +1 ) refers to the first (resp. second) player. It is easy to show that the regret of each player R ± 1 ,T satisfies the following identity, where R ( s ) T is the strong regret of the dueling bandits problem:

The last identity reveals that the dueling bandits problem can be addressed using a 'black-box' strategy, where each player is allowed to use a standard Multi-Armed Bandit (MAB) algorithm. In [1], the authors selected the EXP3 algorithm, which provides guarantees suitable for worst-case scenarios. It's important to note that achieving problem-dependent bounds is not possible when the players use stochastic MAB procedures such as Upper Confidence Bounds algorithms, as the losses experienced by the first player, for example, are not stationary. In a later work, [14] implemented a best-of-both-worlds MAB algorithm, specifically the online mirror descent with the Tsallis-INF regularizer [22]. This approach is effective because, from the perspective of the first player, the loss distribution, although variable, is not entirely arbitrary. This is due to the second player's strategy of minimizing their own regret, which involves concentrating on sampling distributions that approximate those associated with the optimal choice, corresponding to the Condorcet winner.

Adopting the reduction above to solve the weak regret dueling bandit problem seems however insufficient due to several reasons. First, Equation (4) shows that minimizing the strong regret, and minimizing the regrets of individual players is equivalent. Second, the weak regret can be significantly smaller than the strong regret. This is because selecting the Condorcet winner just once is sufficient to suffer zero instantaneous weak regret while leaving the second arm free to explore and gain information about the problem. This is not the case in the strong regret minimization where both selected arms have to be Condorcet winners to incur zero instantaneous strong regret. This suggests that the algorithms that are optimal for strong regret cannot be expected to be optimal for weak regret.

̸

̸

For the task of minimizing weak regret, we follow the intuition presented in [14], which involves using a best-of-both-worlds procedure consisting of online mirror descent with the Tsallis-INF regularizer. However, as previously argued, minimizing weak regret necessitates increased exploration, requiring an adjustment to the sampling scheme, denoted as p t = ( p t,i ) 1 ≤ i ≤ K , suggested by these strategies. Our algorithm first performs an internal step where two arms, I ′ t and J ′ t , are sampled independently from the distribution p t over [ K ] , without playing them. We consider two cases: If I ′ t = J ′ t , we infer that p t lacks sufficient exploration. In this case, we use the left arm I t for exploitation by sampling it from p t , and the right arm J t for exploration. This approach reduces to a decoupled exploration-exploitation problem in standard multi-armed bandits, as studied in [13]. The authors in [13] developed a strategy where exploitation is carried out using p t , while exploration follows a distribution r t with r t,i ∝ p 2 / 3 t,i . They further demonstrated that this strategy, using Tsallis entropy regularizers with a power of 2 / 3 , provides instance-specific bounds independent of the time horizon T . In the second case, if the internal step results in I ′ t = J ′ t , we consider that p t adequately encourages exploration, and both I t and J t are sampled from p t . In summary, given that I ′ t and J ′ t are sampled from p t , we employ the following strategy:

̸

where r t = ( r t,k ) k is a distribution over the set [ K ] defined as follows: for each k ∈ [ K ] :

Observe that arms with small probability p t,k , have a higher chance of being sampled under r t . Hence, the distribution r t encourages more exploration, which will be beneficial for the weak regret. Finally the losses fed to the online mirror descent procedure are estimated using the importance weight estimators:

where, q t,k = p t,k if I ′ t = J ′ t and q t,k = r t,k if I ′ t = J ′ t . We dedicate Section B in the appendix to develop guarantees on the resulting modified online mirror descent with Tsallis regularizer using the sampling scheme described above.

Algorithm 1 WR-TINF

Input : α , Learning rates ( η t )

init: ˆ L 0 = 0 .

for t = 1 , . . . do compute:

where S K -1 is the K -dimensional simplex

Sample I ′ t and J ′ t independently following p t ′ ′

if I t = J t then

Sample I t following p t and J t following r t in (6).

else

Sample I t and J t independently following p t end if

Play ( I t , J t ) , for each k ∈ [ K ] compute ˆ ℓ t ( k ) using (7) and update: ˆ L t ( k ) = ˆ L t -1 ( k ) + ˆ ℓ t ( k ) end for

Theorem 4.2. Consider Algorithm 1 with α = 2 / 3 and η t = 2 K -1 / 6 / √ t . For any T ≥ 1 , the weak regret satisfies:

̸

̸

where c is a numerical constant and ∆ ∗ = min k = k ∗ ∆ k ∗ ,k .

We obtain an upper-bound on weak regret of the order of ˜ O ( √ K/ ∆ ∗ √∑ k = k ∗ 1 / ∆ k ∗ ,k ) . In the setting where all the gaps for the Condorcet winner are constant, the upper bound above translates to O ( K/ ∆ cw ) . The last optimality result is shown in Theorem 5.1.

4.2 Algorithm 2: Weak Regret-EXP3-IX

This algorithm uses the Implicit Exploration strategy (EXP3-IX) [11], which is adapted specifically for the dueling bandits problem. At its core, the algorithm selects one arm (left arm I t ) for exploitation and another (right arm J t ) for exploration. Given I t , J t is selected following the EXP3-IX procedure. We chose the EXP3-IX algorithm (restated in Section D.2 of the Appendix), particularly the version without a fixed horizon for technical reasons, namely its bounds on cumulative loss that hold with high probability. To clarify the notation used: in each round, we observe the result of the duel between I t and J t , denoted by X t ( I t , J t ) . The variable X t ( i, j ) represents the duel outcome between arm i and arm j in round t .

The algorithm operates across multiple stages, where each stage n ≥ 1 is defined by a threshold value B = 2 n -1 , updated through a doubling technique. At every stage, given B , we consistently select the left arm as I t = i , and consider a standard Multi-Armed Bandit problem where the choices are the duels between arm i and the other arms in [ K ] \ i . Specifically, these choices relate to the variables X t ( i, j ) -1 2 : j ∈ [ K ] \ i . Recall that E [ X t ( i, j ) -1 2 ] = ∆ i,j , and the optimal arm, which minimizes cumulative loss, is j ∗ ( i ) = arg min j ∈ [ K ] \ i ∆ i,j . The cumulative loss after executing EXP3-IX for this specified problem over u rounds is denoted by S ( i, n, u ) .

where τ represents the round at which the procedure starts. We continue the procedure until the value of S ( i, n, u ) reaches the threshold -B √ u . When this threshold is met, we transition to the next arm, i +1 , and address the duels involving this new arm. A stage is completed once all arms have met this stopping criterion, allowing the algorithm to advance to the next stage, n +1 .

̸

The underlying rationale of the algorithm is as follows: consider stage n , by design of the algorithm, if i is a sub-optimal arm, then after a constant number of rounds, the process S ( i, n, u ) mimics a random walk characterized by a negative drift of ∆ i,j ∗ ( i ) < 0 . We demonstrate that S ( i, n, u ) typically reaches the threshold -B √ u when u is approximately of the order max { K,B } / ∆ 2 j ∗ ( i ) ,i . In contrast, the process S ( k ∗ , n, u ) , which is linked to the Condorcet winner, has a positive drift. We show that the probability of the last process never meeting the threshold -B √ u for some u ≥ 1 is less than min { 1 , exp( -B 2 ) log(1 / ∆ ∗ ) } , where ∆ ∗ = min k = k ∗ ∆ k ∗ ,k . Consequently, there is a high probability that, at some stage, the algorithm will be trapped in a loop where the left arm is the Condorcet winner leading to zero regret when considering weak regret.

Theorem 4.3. Under the assumption of the existence of a Condorcet winner, the weak regret of Algorithm 2 satisfies:

̸

where for each k = k ∗ : j ∗ ( k ) ∈ arg max j ∆ j,k , ∆ ∗ = min k = k ∗ ∆ k ∗ ,k and c = c ′ max { 1 , log log log( K ∨ 16) } with c ′ being an absolute constant.

̸

̸

The complete proof is presented in Section D of the supplementary material.

Comparison with WR-TINF: Intuitively, WR-EXP3-IX is designed to outperform WR-TINF when the gaps between sub-optimal arms are more important than the gaps with the Condorcet winner. In

̸

Algorithm 2 WR-EXP3-IX

̸

this case, as argued in Section 3.2, the algorithm should explore the K 2 duels to detect sub-optimal arms. More rigorously, consider an example where the gaps satisfy: For all i = k ∗ : ∆ k ∗ ,i = ∆ cw and max j = k ∗ ∆ j,i = ∆ sub, where ∆ cw and ∆ sub are positive constants. Then the upper bound in Theorem 4.3 is of order K 2 ∆ cw / ∆ 2 sub . The last bound is sharper than the bound for WR-TINF (which is of order K/ ∆ cw), when we have ∆ sub / ∆ cw > 1 / √ K .

̸

5 Lower Bound

In this section, we provide a lower bound on the largest weak regret of any algorithm, when confronted with a given set of dueling bandit problems, which we will discuss below.

Let ∆ cw ∈ (0 , 1 / 4) denote a positive number. For a dueling bandits problem, define the class of problems D (∆ cw ) by the set of matrices M representing the gaps (∆ i,j ) ij such that M is skewsymmetric and there exists some k ∗ ∈ [ K ] (representing the Condorcet winner) such that:

̸

̸

The introduced class of matrices D (∆ cw ) includes many natural instances, such as when the gaps satisfy the general identifiability assumption. This assumption states that for each sub-optimal arm j , the arm with the highest probability to beat j is the Condorcet winner k ∗ : i.e., k ∗ ∈ arg min i ∈ [ K ] ∆ j,i . It has been considered in prior works such as [21] and more specifically it is implied by strong stochastic transitivity assumption (Section 3.1 of [2]).

Theorem 5.1. Fix K ≥ 6 , ∆ cw ∈ (0 , 1 / 4) . The weak regret of an algorithm A satisfies:

when T ≥ c ′ K/ ∆ 2 cw . Here c and c ′ are numerical constants.

The result in Theorem 5.1 proves that Algorithm 1 is optimal for the considered instance, particularly highlighting that linear scaling with K is optimal in this case. In the lower bound, we assumed uniform gaps between the Condorcet winner and the sub-optimal arms (equal to ∆ cw). A potential improvement would be to develop a lower bound that depends on all the gaps with the Condorcet winner (∆ k ∗ ,i ) i ∈ [ K ] . Additionally, a more general lower bound should discard the general identifiability assumption. As previously argued, if the gaps between sub-optimal arms are large compared to the gaps with the Condorcet winner, it becomes easier to explore the K 2 duels to detect the sub-optimality of the arms and focus decision-making on the Condorcet winner.

6 Experiments

In this section, we perform a numerical evaluation of WR-EXP3-IX and WR-TINF algorithms in three different scenarios that favor different algorithms according to the prior theoretical results. As a benchmark for our experiments, we utilize the state-of-the-art algorithm for weak regret, WS-W [4]. Additionally, we include one of the best-performing algorithms for strong regret, Versatile-DB

Weak Regret

500

400

300

200

100

0

4000

0

WR-TINF

Versatile-DB

WR-EXP3-IX

WS-W

2000

6000

8000

10000

Time

(a) Weak regret for small problem (Scenario 1)

0

20000

40000

60000

80000

100000

Time

0

5000

10000

15000

20000

Strong Regret

WR-TINF

WR-EXP3-IX

Versatile-DB

WS-W

(c) Strong regret for moderate problem (Scenario 2)

0

0

WR-TINF

Versatile-DB

WR-EXP3-IX

WS-W

20000

40000

60000

80000

100000

4000

3500

3000

2500

2000

1500

1000

500

0

Time

(b) Weak regret for moderate problem (Scenario 2)

12000

10000

8000

6000

4000

2000

0

20000

WR-TINF

Versatile-DB

WR-EXP3-IX

WS-W

10000

30000

40000

50000

Time

(d) Weak regret for large problem (Scenario 3)

[14], to demonstrate that optimizing for strong regret does not necessarily translate into optimal weak regret performance. For each of the experiments, we plot the mean regret over 20 iterations together with 0 . 2 and 0 . 8 quantiles. All the experiments in this section use theoretical values of parameters for the algorithms. The runtime of each algorithm and iteration is in terms of minutes on a personal computer.

Scenario 1: weak regret under SST (Figure 1a). We consider here: K = 30 , T = 10000 , q i,j = 1 -q j,k ∗ = 0 . 8 for i, j ∈ [ K ] such that i < j . In this scenario, we have a small number of arms and the SST holds - the arm with the lower index always wins with probability 0 . 8 . This favors WS-W and WR-EXP3-IX algorithms. On the other hand, WR-TINF is a explores less, this results in larger regret for small K while the algorithm shines as K grows.

Scenario 2: Strong and weak regret comparison without SST (Figures 1b and 1c). We consider here: K = 150 , T = 100000 , q k ∗ ,i = 0 . 9 for every i ∈ [ K ] \ { k ∗ } , q i,j = 0 . 9 (resp. q i,j = 0 . 1 ) for i, j ∈ [ K ] \ { k ∗ } such that i < j and ( i + j ) ≡ 0 mod 2 (resp. ( i + j ) ≡ 1 mod 2 ). In this scenario, we have a moderately large number of arms and SST does not hold - each arm, except for the Condorcet winner, wins against approximately K/ 2 (every other index) other arms with probability 0 . 9 and loses to the other arms with probability 0 . 1 . This should still favor WR-EXP3-IX algorithm but lack of ordering makes WS-W algorithm perform slightly worse. Algorithm WR-TINF slightly closes the gap in weak regret thanks to the increased number of arms. This can be seen in Figure 1b. For completeness of comparison, we also plot strong regret of the algorithms, see Figure 1c. Naturally, algorithms WS-W and WR-EXP3-IX suffer linear strong regret since they never play the same arm twice. However, WR-TINF performs well even with extra exploration, needed for weak regret, compared to Versatile-DB.

Scenario 3: large number of arms, no SST (Figure 1d). We consider: K = 400 , T = 50000 , q k ∗ ,i = 0 . 9 for every i ∈ [ K ] \ { k ∗ } , q i,j = 0 . 9 (resp. q i,j = 0 . 1 ) for i, j ∈ [ K ] \ { k ∗ } such that i < j and ( i + j ) ≡ 0 mod 2 (resp. ( i + j ) ≡ 1 mod 2 ). The same setup without SST as in Scenario 2 but with a larger K . Better scaling with K gives algorithm WR-TINF an edge over algorithms WS-W and WR-EXP3-IX while WS-W still suffers from the lack of SST.

Weak Regret

Weak Regret

Remark 6.1. On the variance of the WS-W algorithm: WS-W is a round-based procedure where the selected arms, "winner and challenger," duel in batches of iterations. The length of each batch increases with the number of duels won by the selected arms so far. When an arm loses, it is replaced by a contender chosen from the remaining arms. Once the set of candidate arms is exhausted, the process is repeated. In numerical experiments, particularly with a large number of arms (Scenario 3 in the simulations section), we observe that in some unfortunate cases, especially in the early stages, the CW may lose its duels. This results in a large number of iterations before it is picked again as a contender, leading to very high weak regret for the procedure. Although such outcomes are infrequent, they significantly impact the empirical variance of the weak regret of WS-W.

7 Conclusion and limitations

̸

In this work, we addressed the problem of weak regret analysis under the assumption of a Condorcet winner. We showed that, it is impossible in general to achieve a weak regret smaller than K/ (min i = k ∗ ∆ k ∗ i ) and we introduced the procedure WR-TINF which achieves this bound. The second algorithm, WR-EXP3-IX, employs a more aggressive exploration strategy by querying the K 2 duels. We show that in some cases, this approach, despite inducing a quadratic dependence on K can outperform WR-TINF, because it better adapts to the gaps between suboptimal arms. This is the first work in duelling bandit with weak regret that establishes how that the full matrix ∆ can be leveraged in the regret.

This work gives rise to several open questions. While WR-TINF is optimal in certain instances, developing algorithms that fully adapt to the underlying problem parameters remains a significant challenge.

Acknowledgments and Disclosure of Funding

The work of E.M. Saad and N. Verzelen has been partially supported by grant ANR-21-CE23-0035 (ASCAI,ANR). The work of A. Carpentier is partially supported by the Deutsche Forschungsgemeinschaft (DFG)- Project-ID 318763901 - SFB1294 "Data Assimilation", Project A03, by the DFG on the Forschungsgruppe FOR5381 "Mathematical Statistics in the Information Age - Statistical Efficiency and Computational Tractability", Project TP 02 (Project-ID 460867398), by the Agence Nationale de la Recherche (ANR) and the DFG on the French-German PRCI ANR-DFG ASCAI CA1488/4-1 "Aktive und Batch-Segmentierung, Clustering und Seriation: Grundlagen der KI" (Project-ID 490860858).

References

Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning , pages 856-864. PMLR, 2014.

Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preferencebased online learning with dueling bandits: A survey. The Journal of Machine Learning Research , 22(1):278-385, 2021.

Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large-scale validation and analysis of interleaved search evaluation. ACMTransactions on Information Systems (TOIS) , 30(1):1-41, 2012.

Bangrui Chen and Peter I Frazier. Dueling bandits with weak regret. In International Conference on Machine Learning , pages 731-739. PMLR, 2017.

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999.

Marco De Gemmis, Leo Iaquinta, Pasquale Lops, Cataldo Musto, Fedelucio Narducci, Giovanni Semeraro, et al. Preference learning in recommender systems. Preference Learning , 41(4155):48, 2009.

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS) , 25(2):7-es, 2007.

Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret lower bound and optimal algorithm in dueling bandit problem. In Conference on learning theory , pages 1141-1154. PMLR, 2015.

Gregory F Lawler. Introduction to stochastic processes . Chapman and Hall/CRC, 2018.

Chang Li, Ilya Markov, Maarten De Rijke, and Masrour Zoghi. Mergedts: A method for effective large-scale online ranker evaluation. ACM Transactions on Information Systems (TOIS) , 38(4):1-28, 2020.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems , 28, 2015.

Erol Peköz, Sheldon M Ross, and Zhengyu Zhang. Dueling bandit problems. Probability in the Engineering and Informational Sciences , 36(2):264-275, 2022.

Chloé Rouyer and Yevgeny Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Conference on Learning Theory , pages 3227-3249. PMLR, 2020.

Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning , pages 19011-19026. PMLR, 2022.

Aadirupa Saha and Shubham Gupta. Optimal and efficient dynamic regret algorithms for non-stationary dueling bandits. In International Conference on Machine Learning , pages 19027-19049. PMLR, 2022.

Yanan Sui, Masrour Zoghi, Katja Hofmann, and Yisong Yue. Advancements in dueling bandits. In IJCAI , pages 5502-5510, 2018.

Xinyi Yan, Chengxi Luo, Charles LA Clarke, Nick Craswell, Ellen M Voorhees, and Pablo Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 567-577, 2022.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538-1556, 2012.

Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1201-1208, 2009.

Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 241-248. Citeseer, 2011.

Julian Zimmert and Yevgeny Seldin. Factored bandits. Advances in Neural Information Processing Systems , 31, 2018.

Julian Zimmert and Yevgeny Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research , 22(1):1310-1358, 2021.

Masrour Zoghi, Shimon Whiteson, and Maarten de Rijke. Mergerucb: A method for large-scale online ranker evaluation. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining , pages 17-26, 2015.

Masrour Zoghi, Shimon A Whiteson, Maarten De Rijke, and Remi Munos. Relative confidence sampling for efficient on-line ranker evaluation. In Proceedings of the 7th ACM international conference on Web search and data mining , pages 73-82, 2014.

8 Appendix / supplemental material

A Preliminary results for Algorithm 1

A.1 Preliminary Lemma:

Lemma below upper-bounds the weak regret of Algorithm 2.

Lemma A.1. Let an algorithm A ′ playing in each round the pair of arms ( I ′ t , J ′ t ) observing only the feedback of the duel between arms ( I t , J t ) sampled following the scheme in (5) . Let A be an algorithm playing in each round the pair ( I t , J t ) and observing the feedback X t ( I t , J t ) . let R ( s ) T, A ′ denote the strong regret of A ′ and R ( w ) T, A the weak regret of A . We have:

Proof. For t ∈ [ T ] observe that following Algorithm 1, we have:

̸

̸

̸

We used in the second line the fact that conditionally to the event { I ′ t = J ′ t } , ( I t , J t ) are sampled independently from p t (same distribution as ( I ′ t , J ′ t ) ), and conditionally to the event { I ′ t = J ′ t } , I t is sampled from p t . The result follows by summing over t ∈ [ T ] .

A.2 Additional Lemmas:

Lemma below states that the reduction proved in Theorem 2 of [14] is still valid in our setting. Recall the notation:

where ℓ ′ -1 ,t ( k ) = X t ( J ′ t , k ) and ℓ ′ +1 ,t ( k ) = X t ( I ′ t , k ) .

Lemma A.2. Theorem 2 of [14] The expected strong regret of algorithm A ′ satisfies:

̸

Moreover, we have:

̸

̸

Proof. We have

We used in the second line the fact that: ℓ ′ -1 ,t + ℓ ′ +1 ,t = X t ( I ′ t , J ′ t ) + X t ( J ′ t , I ′ t ) = 1 .

Furthermore, observe that for any t ∈ [ T ] :

where we used the fact that I ′ t and J ′ t are sampled independently from the same distribution. Therefore we have summing over t :

Now let us prove the last identity of the lemma. We have

̸

We used in the last equality (9), and the fact that I ′ t ∼ J ′ t in the fourth.

Recall the notation: ℓ ′ t ( k ) := X t ( k, J ′ t ) , which corresponds to the loss of the learner playing k when the environment chooses J ′ t . Recall the expression of the importance weights estimator introduced in (7) in the main body:

̸

where,

̸

We provide the following expressions of the probabilities used in the definition above. we have that I ′ t and J ′ t are sampled independently following p t = ( p t,k ) k ∈ [ K ] :

̸

Lemma A.3. We have for any k ∈ [ K ] :

where E t -1 [ . ] corresponds to the expectation at round t given the past information.

Proof. We have

̸

̸

̸

where we used in the third line the fact that conditionally to I ′ t = J ′ t , we have I t ∼ J ′ t .

Calculating Term 2: We have

The conclusion follows by summing the obtained expressions.

B Analysis for the modified OMD with Tsallis regularizer

B.1 The Setting:

The online mirror descent with Tsallis regularizer in the standard coupled exploration and exploitation case was analyzed in [22] and in the decoupled exploration and exploitation setting in [13]. In this section, we develop guarantees in the case where exploration and exploitation are partially coupled via the sampling scheme that we employ, which is restated in Algorithm 4. Note that to be compatible with our setting for dueling bandits, we need to make some modifications to the game protocol for the problem of regret minimization. More precisely, we assume that in each round t instead of choosing a sequence of numbers, the environment chooses a sequence of distributions for losses. The incurred and observed losses are sampled independently from the sequence chosen by the environment. Note that this change doesn't affect the definition of the pseudo-regret, which is the quantity of interest here, since the definitions involve expectations. We present in Algorithm 3 the game protocol of this game.

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

Algorithm 3 Game Protocol

The player chooses two distributions p t = ( p t,k ) k ∈ [ K ] and q t = ( q t,k ) k ∈ [ K ] over [ K ] .

Concurrently, the environment chooses K random distributions with support in [0 , 1] denoted ( L t,k ) k ∈ [ K ] .

The player plays an arm A t sampled following p t and incurs an unseen loss ℓ t,A t sampled from L t,A t .

The player samples an arm B t following q t and observes an independent fresh sample ¯ ℓ t,B t from the distribution L t,B t .

end for

We define the pseudo-regret with respect to k as follows:

We denote by k ∗ := arg min k E [ ∑ T t =1 ℓ t,k ] .

We consider the stochastically constrained adversarial setting where we have for some positive numbers (∆ k ) :

̸

B.2 The Algorithm:

Following [22], we consider a follow-the-regularized leader (FTRL) approach using

as regularizers where ( η t ) is a sequence of positive numbers and α ∈ (0 , 1) .

More specifically, following the analysis of decoupled exploration and exploitation in [13], we focus on the case where α = 2 / 3 . We introduce the following loss estimators:

Recall that following Game Protocol 3: ¯ ℓ t,k and ℓ t,k are independent and follow the same distribution L t,k .

We define our exploration distribution q t by:

where E t is some internal event specified by the learner based on past information and some internal randomization. In Algorithm 4 below, E t corresponds to the event { A t = B ′ t } where A t and B ′ t are independent and sampled from [ K ] following p t . E c t denotes the complementary to E t . ( q (1) t,k ) k ∈ [ K ] and ( q (2) t,k ) k ∈ [ K ] are probability distributions defined by:

We define the probability ( q (1) t,k ) k ∈ [ K ] by: q (1) t,k := p t,k . Define the probability q (2) t,k as follows:

Algorithm 4 Partially coupled Tsallis-INF

Input : (Ψ t ) t =1 , 2

̸

end for

Theorem B.2. Suppose the regret satisfies the self-constraining condition (10) . The pseudo-regret of Algorithm 4 with α = 2 / 3 , η t = 2 K -1 / 6 √ t , Ψ t ( w ) = -1 η t ∑ i w α i -αw i α (1 -α ) , satisfies:

̸

where ∆ ∗ = min k = k ∗ ∆ k and c is a numerical constant.

̸

Proof. Following previous works, we decompose the expected regret into the stability and penalty terms using the potential Φ t defined by:

where S K -1 is the set of probability weights on [ K ] . Let ˆ L t = ∑ T t =1 ˆ ℓ t , where ˆ ℓ t is defined by (11). We have

Recall that we have:

̸

Therefore:

̸

Let us bound each term separately.

Bounding the term corresponding to the penalty: Term 2 Let T 0 ≥ 1 . Wehave using Lemma B.4

̸

̸

̸

where we used in the third line Jensen's inequality on the concave function x → x 2 / 3 , giving: ∑ i = k ∗ E [ p 2 / 3 t,i ] ≤ K 1 / 3 and the fact that ∑ T 0 t =1 1 √ t ≤ 2 √ T 0 .

Bounding the term corresponding to the stability: Term 1 We have using Lemma B.3

̸

̸

̸

̸

where we used the definition of η t from line 1 to line 2.

̸

̸

̸

̸

̸

̸

̸

̸

̸

The Terms 1.2 and 1.3 can be upper bounded using the same derivation as in the previous calculations for Term 1, where we obtained (15) from (14):

̸

Let us bound Term 1.1. Let ¯ T := ⌈ 256 K ∆ 2 ∗ ⌉ . We have

̸

̸

̸

Let t ≥ ¯ T , recall that ∑ K i =1 p t,i = 1 , therefore using Jensen's inequality for the concave function x → x 1 / 3 , we have: ∑ i = k ∗ p 1 / 3 t,i ≤ K 2 / 3 ( ∑ i = k ∗ p t,i ) 1 / 3 = K 2 / 3 (1 -p t,k ∗ ) 1 / 3

̸

̸

where the the second line follows from the fact that t ≥ ¯ T , and the last line from the fact that p t,k ∗ ∈ [0 , 1] . We conclude that

̸

̸

where we used Lemma F.4 to obtain the last line. Using ∑ ¯ T t =1 t -3 / 4 ≤ 4 ¯ T 1 / 4 , we have

̸

Next we use the definition of ¯ T , then Jensen's inequality for the concave function x → √ x to have:

̸

We conclude combining (16) and (17) that:

̸

̸

̸

̸

̸

Conclusion: Combining the bounds in (18) and (15) we have:

̸

We conclude that

̸

̸

̸

where S i ( T ) := ∑ T t = T 0 +1 1 ∆ 2 i t 3 / 2 . Let us bound the quantities S i ( T ) . We have for any i = k ∗ :

Next, we re-inject the bound above on inequality (20) and obtain

̸

We take

Therefore:

̸

Finally, we use the bounds (21) and (22) in (19) and conclude that for some numerical constant c we have:

̸

̸

̸

̸

̸

̸

̸

B.3 Some Lemmas:

The following lemma is an adaptation of part 2 of Lemma 6 in 13 that bounds the stability term. Lemma B.3. We have

̸

̸

̸

Proof. We adapt the arguments presented in the proof of Lemma 6 in [13]. We use Lemma B.5, where we choose x = 1 { B t = k ∗ } ¯ ℓ t,k ∗ . When B t = k ∗ , we have x = 0 and the expression is maximized for ˜ p i = p t,i , since the losses are non-negative, and ∇ Ψ ∗ t is monotonically increasing. When B t = k ∗ , we have ˆ ℓ t,k ∗ -x = 0 -x ≥ -1 and we can apply Lemma B.6 and bound ˜ p i 4 / 3 by (3 / 2) p 4 / 3 t,i . We conclude that

̸

Let us bound each term in the expression above. For the first term we have:

̸

̸

We have:

Similarly, we show that

Therefore:

̸

For the second term in (23)we have:

̸

Finally, we inject the bounds (24) and (25) into (23), rearranging the terms we obtain

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

Upper bounding Term 1: We have:

Upper bounding Term 2: We have:

̸

̸

where we used in the last line Jensen's inequality for the concave function x → x 2 / 3 .

̸

̸

̸

̸

̸

̸

̸

̸

̸

The following lemma is a direct consequence of the second part of Lemma 7 in 13 where we take α = 2 / 3 and β = K -1 / 6 . It provides a bound on the penalty term:

Lemma B.4 (Part 2 of Lemma 7 in 13) . For η t = 2 K -1 / 6 √ t , the penalty term satisfies:

̸

Proof. Our algorithm uses the Tsallis-inf framework introduced by [22] and [13]. We use a learning rate η t = 2 β √ t , with β = K -1 / 6 . Moreover, the loss estimator we use defined by (11) is unbiased. Therefore the statement of Lemma 7 from [13] applies. The expression in the lemma follows by taking α = 2 / 3 and β = K -1 / 6 .

LemmaB.5. Lemma 10 in [13] Let p t = ∇ Φ t ( -˜ L t -1 ) for ˜ L t = ˜ L t -1 + ˜ ℓ t , where ˜ ℓ t is an unbiased estimate of ℓ t . For any x ≥ 0 , the instantaneous stability of the pseudo-regret of Algorithm 4 satisfies:

Proof. Recall that our loss estimators (11) are unbiased and the played arm A t is selected following the same rule as in [13]. Therefore the statement of Lemma 10 in [13] applies.

Lemma B.6. Lemma 11 in [13] Let p ∈ S K -1 and ˜ p = ∇ Ψ ∗ t ( ∇ Ψ t ( p ) -ℓ ) . If η t ≤ 1 / 4 then for all ℓ i ≥ -1 it holds that ˜ p 4 / 3 i ≤ 3 2 p 4 / 3 i .

C Proof of Theorem 4.2

Proving Theorem 4.2 amounts to combining the previous results. We have using Lemma A.1:

Furthermore, using Lemma A.2, we have:

Now let us show how the problem of upper-bounding the regret above related to the analysis of the modified OMD with Tsallis-INF regularizer in Algorithm 4 developed in Section B.

The regret R ′ -1 ,T is with respect to a learner playing with the following strategy: In each round t ∈ [ T ]

The learner samples I ′ t from p t .

The learner plays I ′ t and incurs ℓ t,I ′ t = X t ( I ′ t , J ′ t ) where J ′ t is independently sampled from p t .

The learner samples J t using:

-from p t if an event with probability 1 -∥ p t ∥ 2 holds.

-from r t if an event with probability ∥ p t ∥ 2 holds.

The learner observes the feedback X t ( J t , I t ) where I t is independently sampled using p t .

Comparing the strategy above with the game protocol 3 and Algorithm 4 of Section B, we conclude that: A t plays the same role as I ′ t , and J t plays the same role as B t . Moreover, the losses observed X t ( k, I t ) corresponding to ¯ ℓ t,k are independent but follow the same distribution as the actual losses X t ( k, J ′ t ) corresponding to ℓ t,k .

Finally in order to apply Theorem B.2, we need to check whether the regret R ′ -1 ,T satisfies the necessary self-bounding condition (condition considered in Section B). The last requirement is a direct consequence of Lemma A.2.

Therefore, the bound in Theorem B.2 applies and gives the result.

D Proof of Theorem 4.3

We restate here the theorem:

Theorem D.1. Under the assumption of the existence of a Condorcet winner, the weak regret of Algorithm 2 satisfies:

̸

where for each k = k ∗ : j ∗ ( k ) ∈ arg max j ∆ j,k , ∆ ∗ = min k = k ∗ ∆ k ∗ ,k and c = c ′ max { 1 , log log log( K ∨ 16) } with c ′ an absolute constant.

̸

̸

Proof. Notation : Let F t = σ ( ( I 1 , J 1 ) , X 1 ( I 1 , J 1 ) , . . . , ( I t , J t ) , X t ( I t , J t ) ) . Following Algorithm 2, we run EXP3-IX Algorithm for each arm k in each phase (a phase corresponds to a fixed value of the parameter B ), during the n th increment of the value B (phase number n ) we have B = -2 n -1 . When we fix arm k as a left arm and run EXP3-IX to choose the right arm for t rounds, we denote by S ( k, n, t ) the obtained cumulative loss:

More rigorously, the sum above is over rounds between τ + 1 and τ + t , where τ is a variable independent of the considered cumulative loss. We consider the notation above to simplify the proof.

We define, for each k ∈ [ K ] and n ≥ 1 , by τ ( n ) k as the stopping time, with respect to ( F t ) , corresponding to the round where ( S ( k, n, t )) t hits the level -2 n -1 √ t for the first time:

We call phase, the time interval in Algorithm 2 between two increments of the variable B . More formally, phase number n corresponds to the number of rounds t such that τ ( n ) K < t ≤ τ ( n +1) K .

For k ≤ K and a positive n , let E ( n ) k denote the following event

We use the following τ (0) K = 0 , τ ( n ) 0 := τ ( n -1) K for each n ≥ 1 . The expected regret incurred at phase n satisfies:

Therefore the weak regret of Algorithm 2 satisfies:

Define by U k the quantity:

Using Lemma D.2. We have for some numerical constant c > 0

We have

Let N = log 4 (1 ∨ log(1 ∨ log 2 (1 / ∆ ∗ ))) , therefore we have exp(2 2 N ) ≥ log 2 (1 / ∆ ∗ ) . Therefore, using Lemma D.3, we have:

Moreover, we have:

Finally we plug (27) and (28) into (26) and use the fact that K log( K ) N + log(1 / ∆ ∗ ) ≤ 2 K log( K/ ∆ ∗ ) max { 1 , log log log( K ∨ 16) } , to conclude.

D.1 Auxiliary Lemmas

We recall the notation: for each k ∈ [ K ] , let j ∗ ( k ) ∈ arg min j ∈ [ K ] \{ k } ∆ k,j . Therefore, j ∗ ( k ) represents the arm with the largest chance to beat arm k . Moreover, let S ( k, n, t ) denote the cumulative loss obtained when running EXP3-IX algorithm in phase n for arm k . For n ≥ 1 , let τ ( n ) k denote the stopping time corresponding to the round where the process S ( k, n, t ) hits the level -2 n -1 √ t :

Lemma D.2. Let k ∈ [ K ] \ { k ∗ } , then we have for any n > 0 :

where c is an absolute constant.

Proof. Fix k ∈ [ K ] \ { k ∗ } and n ≥ 1 . Suppose that { τ ( n ) k -1 < + ∞} holds. Recall the definition of S ( k, n, t ) :

To ease notation we will focus only on the rounds where the fixed arm k was chosen as a left arm in phase n . Recall that at each phase when we consider a new arm, we run EXP3-IX from scratch, therefore the obtained cumulative loss process is independent from the past. Denote by Y j,s the sample received when choosing j as a right arm at round s , Y j,s has the same distribution as the variable X ( k, j ) -1 / 2 , hence E [ Y j,u ] = ∆ k,j . In each round u , the chosen arm is denoted A u . Therefore we have:

Let j ∗ ∈ arg min j { ∆ k,j } ( j ∗ = j ∗ ( k ) , we just dropped the dependence on k ). Let

∆ is negative because k is not the Condorcet winner (remember we fixed k ∈ [ K ] \ { k ∗ } ). Define the (random) regret for this problem after t rounds as follows:

Our objective is to upper-bound the expectation of the stopping time τ ( n ) k . To develop such a bound we use the identity:

In the remainder of this proof all events are assumed to hold conditionally to { τ ( n ) k -1 < + ∞} . Let us bound the probabilities in the rhs: Fix m ≥ n and let

where ∆ is defined by (29). Recall that conditional to { τ ( n ) k -1 < + ∞} the event { τ ( n ) k > N m } implies in particular that { ∑ N m t =1 Y A t ,t > -2 n -1 √ N m } . Therefore:

Let us upper bound the probability of the last event. We have

where x is any constant larger than 1 . Using the definition of N m (recall that ∆ < 0 ), we have -N m ∆ ≥ -2 m / ∆ . Therefore

Now let us take:

Define ¯ n as the smallest integer in [2 n +3 , + ∞ ) such that

Hence 2 ¯ n ≥ K . Moreover, for m ≥ ¯ n , we have: x ≥ 1 . Furthermore:

where we used m ≥ ¯ n ≥ 2 n +3 .

Using (31):

Using Theorem D.5 to bound the first term (recall that by definition of ¯ n , for m ≥ ¯ n : x ≥ 1 ), and Hoeffding's inequality to bound the second term we obtain:

Let us upper-bound the r.h.s of the inequality above. We have for m ≥ ¯ n :

Using (30) we have that N m ≤ 2 m ∆ 2 , therefore 1 √ N m ≥ | ∆ | 2 m/ 2 we get

where we used the definition of N m . The second term in (34) can be bounded for m ≥ ¯ n following:

Combining (34), (35) and (36) we conclude that:

We conclude that for m ≥ ¯ n :

We have:

Recall that by definition of ¯ n in (32), we have: 2 ¯ n ≥ K . Therefore:

Next we use the bound

We conclude using the definition of N m and ¯ n that

where c is an absolute constant.

Lemma D.3. We have for any n ≥ 2 , k ∈ [ K ] :

̸

where ∆ ∗ = min k = k ∗ ∆ k ∗ ,k .

Proof. Let n ≥ 2 and k ∈ [ K ] . We have that E ( n ) k implies the event: E ( n -1) k ∗ +1 . Therefore:

where we used in the last line Lemma F.3.

D.2 Deviation guarantees for the regret of EXP3-IX

We recall below the algorithm EXP3-IX from [11]

Algorithm 5 EXP3-IX

Define the random regret R T by: R T := ∑ T t =1 ℓ I t ,t -min i ∈ [ K ] ∑ T t =1 ℓ i,t . We restate below the main result from [11].

Theorem D.4. Theorem 1 in [11]. Fix an arbitrary δ ∈ (0 , 1) , set η t = 2 γ t = √ log( K ) Kt for all t , then EXP3-IX guarantees with probability at least 1 -δ :

We have the following corollary

Corollary D.5. Fix x ≥ 1 , and consider EXP3-IX algorithm with η t = 2 γ t = √ log( K ) Kt , then:

Proof. Let x ≥ 1 , take:

Therefore:

We have with probability at least 1 -δ ≥ 1 -2 exp( -x √ log( K )) :

E Proof of Theorem 5.1

We restate the theorem for the lower bound, then we proceed with the proof. Let ∆ cw denote a positive number. For a dueling bandits problem, we denote by M = ( M i,j ) 1 ≥ i,j ≤ K the matrix such that M i,j = ∆ i,j . Define the class of problems D (∆ cw ) by the set of matrices M representing the gaps (∆ i,j ) ij such as M is skew-symmetric and there exists some k ∗ ∈ [ K ] such that

̸

̸

where k ∗ ∈ [ K ] denotes the Condorcet winner.

Theorem E.1. Fix K ≥ 6 , ∆ cw ∈ (0 , 1 / 4) . The weak regret of an algorithm A satisfies:

when T ≥ c ′ K/ ∆ 2 cw . Here c and c ′ are numerical constants.

Proof. Let M (0) denote a matrix such that M (0) 1 ,i = ∆ cw for any i > 1 and M (0) i,j = 0 for i, j = 1 . Let P 0 denote the probability distribution associated with the dueling bandits' problem with matrix M (0) (where arm 1 is the Condorcet winner).

̸

̸

̸

Let k = 1 , let M ( k ) denote the matrix defined by: for all u = k M ( k ) k,u = ∆ cw = -M ( k ) u,k , for all i = k M ( k ) 1 ,i = ∆ cw = -M ( k ) i, 1 , otherwise for i = k, 1 and j = k, 1 : M ( k ) i,j = 0 . Let P k denote the probability distribution associated with the dueling bandits' problem with matrix M ( k ) (where arm k is the Condorcet winner).

̸

̸

̸

For any u ∈ [ K ] , let N u denote the total number of rounds where arm u was queried. For u, v ∈ [ K ] , let N u,v denote the total number of rounds where arms u and v were dueled.

Information theoretic tool: Without loss of generality, we assume that the player follows a deterministic strategy A . Let us introduce the following notation: let Z t = ( ( I t , J t ) , X t ( I t , J t ) ) denote the information disclosed to the player at time t . Let Z t = ( Z 1 , . . . , Z t ) denote the entire information available to the player after t rounds.

Lemma E.2. Assume that ∆ cw ≤ 1 / 4 . Let F ( Z T ) denote a fixed function of the player observations, taking values in [0 , B ] . Then for any k ∈ [ K ] \ { 1 } and any player strategy A ,

where N u = ∑ K v =1 N u,v and N u,v denotes the number of rounds where arms u and v were duelled.

Proof. Recall that for any function G bounded by R , we have: | E X ∼ P [ G ( X )] -E X ∼ Q [ G ( X )] | ≤ 2 R TV ( P , Q ) , where TV ( ., . ) denotes the total variation distance. Therefore, by shifting F by -B/ 2 , we have:

by Pinsker's inequality, where KL(.) denotes the Kullback-Leibler divergence and KL(x,y) for x, y ∈ (0 , 1) denotes the Kullback-Leibler divergence between two Bernoulli distributions with means x and y .

Next we use the chain rule for relative entropy (Theorem 2.5.3 in 5):

Observe that we have for each t ∈ [ T ] :

̸

where in the last line we used that KL ( x, y ) ≤ ( x -y ) 2 / [ y (1 -y )] . Summing over t ∈ [ T ] leads to the desired result.

Recall that the weak regret for the problem P k is given by:

Applying Lemma E.2 with F ( Z T ) = ∆ cw N k , we have:

Averaging over k ∈ [ K ] \ { 1 } and using Jensen's inequality:

Observe that ∑ K u =1 N u ≤ 2 T , therefore the inequality above gives:

Let P ∗ denote the problem where we choose k uniformly at random from [ K ] \ { 1 } , then we proceed to the game where gaps are given by M ( k ) . Hence we have:

We conclude that

Therefore,

Recall that we assume that K ≥ 6 . In the above inequality, the supremum is achieved for T ′ of the order of K/ ∆ 2 cw , which is achievable as long as T is at least of this order. This leads to the desired result when K/ ∆ 2 cw is higher than some numerical constant c ′′ . In the extreme situation, where K/ ∆ 2 cw is smaller or equal to c ′′ , the lower bound (38) could be negative for any T ′ ≥ 1 . In that case, we can use the trivial bound E ∗ [ R T ] ≥ E ∗ [ R 1 ] ≥ c ′ ∆ cw as, with probability bounded away from zero, the first duel does not contain the Condorcet winner.

F Technical Results:

Lemma F.1. Let q ∈ (0 , 1) , we have:

Proof. We have:

Lemma F.2 (Doob's maximal inequality: Section 5.6 from Lawler) . Let ( X i ) be a sequence of independent Bernoulli variables such that for any i ≥ 1 : E [ X i ] = p i . Let S t = ∑ t i =1 ( X i -p i ) . We have for any t ≥ 1 , a > 0 :

Proof. We have S t is a martingale with respect to the filtration associated to the process ( X i ) . Therefore using Doob's maximal inequality: Section 5.6 [9], for any b > 0 :

where we used the fact that for each i ∈ [ n ] : E [exp( b ( X i -E [ X i ]))] ≤ exp( b 2 / 8) . The conclusion follows by minimizing the upper bound for b > 0 .

Lemma F.3. Let ( X t ) be a sequence of independent Bernoulli variables such that for some ∆ ∈ (0 , 1 / 2) , for each t we have: E [ X t ] ≥ 1 2 +∆ . Let B ≥ 1 and define the stopping time:

Then we have:

Proof. Let S t = ∑ t i =1 ( E [ X i ] -X i ) . We have

Since we have for each t ≥ 1 : S t ≤ t , and ∆ > 0 . We have:

Let N 0 = ⌊ log 2 ( B 2 ) ⌋ . We have

Pluging the last bound in the previous display gives:

To get the last bound, we bounded the sum by an integral and use the change of variable u = ( B ∆) 2 2 t / 4 .

Lemma F.4. Lemma 8 in [13] Let α ∈ (0 , 1) , c > 0 and d ∈ (0 , 1] , we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we introduce in a non-technical way the main setting of the paper (duelling bandits and minimisation of weak regret), and expose our main contributions also in a non-technical way. We detail this after we introduce the setting.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This work primarily concentrates on theoretical aspects. We explored the optimality and limitations of the guarantees in Section 3.2. Additionally, discussions on the guarantees for WR-TINF and WR-EXP3-IX are provided in Sections 4.1 and 4.2, respectively. Limitations and directions for potential improvements are presented in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumption made is the existence of a Condorcet winner Assumption 2.1. All the proofs are provided in the appendix: The proof of the upper bound for WR-TINF (Theorems 4.2) is provided in Section C, the proof of the upper bound for WR-EXP3-IX (Theorem 4.3) is provided in Section D. The proof of the lower bound (Theorem 5.1) is provided in Section E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Algorithms introduced in this paper, together with the theoretical values of parameters, are fully described in sections 4.1 and 4.2. Section 6 provides all the details needed to reproduce the simulations presented in our paper. The code is provided as well.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In our experiments, we used data generated synthetically. The description of the distributions of the duels considered is provided in Section 6. The code is provided as well.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 6 describes all the necessary details to understand the experimental results and their connection to the theoretical guarantees of the presented algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results in Figure 1 compare algorithms and their average performance over multiple iterations together with 0.2 and 0.8 quantiles.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The algorithms for dueling bandits use a small amount of resources and can be run on personal computers.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The NeurIPS Code of Ethics was respected.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of theoretical Machine Learning. There are no potential relevant societal consequences of our work, we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original owners of assets are authors.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an analysis of weak regret in the context of dueling bandits, addressing the challenges posed by the non-linearity of weak loss. The authors introduce two algorithms: WR-TINF, which employs a reduction scheme to multi-armed bandits (MAB) and improves upon state-of-the-art methods, and WR-EXP3-IX, which exhibits varying performance in different instances. Additionally, the paper provides a lower bound on the weak regret that aligns with the regret bounds of the WR-TINF algorithm.\n\nSTRENGTHS:\n* **Originality**: The paper combines known methods like a reduction scheme to MAB [3] and decoupled exploration and exploitation in MAB [4] in a novel way to solve the largely unexplored weak regret problem. The use of the MAB Exp3-IX [5] in this setting is also novel.\n* **Quality**: The theoretical analysis is rigorous, with clear proofs supporting the claims about the regret upper and lower bounds.\n* **Clarity**: The paper contains intuitive discussions explaining the motivation behind both algorithms and the different instances they are suited to. Experimental evaluation helps validate these principles. \n* **Significance**: This work provides interesting results for dueling bandits in the weak regret setting, which is much less explored compared to its strong regret counterpart. Specifically, it improves upon previous results in this setting [1,2] to become the new state-of-the-art, and provides the first time-independent lower bound for dueling bandits. For these reasons, and despite the weaknesses described below, I believe accepting it will contribute to the community. \n\n[1] Chen, Bangrui, and Peter I. Frazier. \"Dueling bandits with weak regret.\" International Conference on Machine Learning. PMLR, 2017.\\\n[2] Peköz, Erol, Sheldon M. Ross, and Zhengyu Zhang. \"Dueling bandit problems.\" Probability in the Engineering and Informational Sciences 36.2 (2022): 264-275.\\\n[3] Saha, Aadirupa, and Pierre Gaillard. \"Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences.\" International Conference on Machine Learning. PMLR, 2022.\\\n[4] Rouyer, Chloé, and Yevgeny Seldin. \"Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits.\" Conference on Learning Theory. PMLR, 2020.\\\n[5] Neu, Gergely. \"Explore no more: Improved high-probability regret bounds for non-stochastic bandits.\" Advances in Neural Information Processing Systems 28 (2015).\n\nWEAKNESSES:\n**Minor Issues**:\n* line 53: 'in both selections'-> 'twice' will sound better. \n* line 55: remove 'that'.\n* line 58: 'tan'->'than'.\n* line 85: the sentence that begins here is too long. \n* line 89: remove 'by'.\n* line 109: the reference in this line predates the ones in the previous paragraph, so 'then' should not be used. \n* line 118: According to section 3.1.8 in [1], the improved regret for WS-S uses $\\Delta_{k*,i}$ instead of $\\Delta_{i,j}$.\n* line 123: the upper bound of which algorithm is not specified. If this is WS-S, I think the power in the denominator should be 6 instead of 5 (section 3.1.8 in [1]).\n* line 134: it seems  $\\Delta_{k*,i}$ can be used here instead of  $\\Delta_{k*,i}^2$.\n* line 152: I did not find an SST assumption within the paper referenced in this line. \n* line 164: perhaps it should be $\\Delta_{j*(t),i}$ instead of $\\Delta_{i,j*(t)}$.\n* line 183: add 'and'.\n* line 197: 'bandits'->'bandit'.\n* line 201: $X_t$ is not defined yet. \n* line 254: 'initial' should not be used here. \n* line 268: $X_s(i,J_s)$ instead of $X_s(k,J_s)$.\n* line 277: 'have'->'has'.\n* line 278: 'ever'->'never'. 'last process' not explained. \n* line 301: 'su'->'sub'. Also, as far as I understand, matrices in the family defined have a CW and satisfy the general identifiability assumption, but the reverse is not necessarily true, so this sentence is slightly confusing (perhaps the same holds with SST). \n* line 309: space after '.'.\n* lines 328-329: it is not clear what 'robust and conservative' means. \n* line 346: 'WS-W and WR-EXP3-IX algorithms'->'algorithms WS-W and WR-EXP3-IX'.\n* Many more corrections should be done in the Appendix (for example line 537 goes out of margin). \n\n**Other Issues**:\n* As rightfully acknowledged by the authors, the major issue that limits the scope of this paper is the fact that the lower bound does not fully describe the complexity of the weak-regret setting. The lower bound is proved for a narrow sub-family of instances, so while it does hold for the whole family of instances with a CW, there are other sub-families that might have a different lower bound. This is exemplified through the two proposed algorithms: while WR-TINF matches the lower bound, WR-EXP3-IX which does not match it may be better for sub-families that are different than the ones used to prove the lower bound. The fundamental issue here is that a lower bound should depend on all problem parameters. \n* Similarly, since each one of the proposed algorithms performs better for different sub-families of instances, some prior knowledge about the environment is needed to choose which one is better. In other words, it seems none of the algorithms is optimal for the class of dueling bandits with a CW. \n* I think that the Introduction, Contributions, and Related Works sections are too long with some repetitions that can be removed - the paper only begins introducing the main methods on page 5. \n* The analysis of WR-TINF is similar to analysis methods used in previous works [2,3], but as I mentioned above I think the combination is novel. \n\n\n[1] Bengs, Viktor, et al. \"Preference-based online learning with dueling bandits: A survey.\" Journal of Machine Learning Research 22.7 (2021): 1-108.\\\n[2] Saha, Aadirupa, and Pierre Gaillard. \"Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences.\" International Conference on Machine Learning. PMLR, 2022.\\\n[3] Rouyer, Chloé, and Yevgeny Seldin. \"Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits.\" Conference on Learning Theory. PMLR, 2020.\n\nQUESTIONS:\n* line 122: Why is SST mentioned here instead of the less strict assumption of a total order of arms? If this refers to WS-S, the improved regret bound should hold for the latter (section 3.1.8 in [1]).\n* WR-TINF uses a partial decoupling technique between exploration and exploitation, unlike the full decoupling used in the MAB case [2]. Can you explain why this is necessary? It would be simpler to just adopt full decoupling and not introduce the 'fake' draws $I_t^{'},J_t^{'}$. Is there some intuition behind this or is this the only way the reduction schemes work?\n* As far as I understand, in the doubling scheme for WR-EXP3-IX, learning starts from scratch at each stage. I suppose this is due to independence issues like in standard doubling schemes, but it may harm performance. Did you try running experiments without forgetting previous stages?\n* It's not clear to me in what way this algorithm mimics a random walk (line 275), could you explain this issue? Also, in line 277 could it be that the CW has a zero drift instead of a positive one? (as no other arm wins against it). \n* In Figure 1a, why does WS-S perform so well despite the fact that its weak regret bounds (even for the SST case) are worse than those of WR-TINF?\n* In Figures 1b,d why are the quantiles for WS-S so high?\n\n[1] Bengs, Viktor, et al. \"Preference-based online learning with dueling bandits: A survey.\" Journal of Machine Learning Research 22.7 (2021): 1-108.\\\n[2] Rouyer, Chloé, and Yevgeny Seldin. \"Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits.\" Conference on Learning Theory. PMLR, 2020.\n\nLIMITATIONS:\nThere is no potential negative societal impact in this work. Limitations were discussed in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce two new algorithms for weak regret minimization in the setting of $K$-armed dueling bandits in order to demonstrate how the optimal strategy changes depending on how the victory probabilities of the Condorcet winner compare to the victory probabilities of the arm most likely to beat each non-Condorcet winner. The first algorithm, WR-TINF, applies online mirror descent with the Tsallis regularizer but chooses one of the arms in each duel in a way that achieves exploration while reserving the other arm for exploitation. The second algorithm, WR-EXP3-IX, applies the classic EXP3-IX algorithm repeatedly in stages where each stage focuses on a single arm with the goal of avoiding the stopping condition when the focus is on the Condorcet winner and hence being trapped in that stage where the Condorcet winner is played on every round. A lower bound on the worst-case weak regret which applies under a specific class of dueling bandit problems is presented to show that WR-TINF indeed matches that lower bound in terms of the dependence on $K$ and on the gap for the Condorcet winner (assuming that gap is constant across all arms it can duel). Numerical experiments are conducted to demonstrate the superiority of either WR-TINF or WR-EXP3-IX depending on the setup for the gaps.\n\nSTRENGTHS:\n# Originality\nThe authors primarily make use of techniques that have been applied to classic or dueling MAB problems in the past, but apply them to the under-explored problem of weak regret minimization which requires modifications to the approach and analysis. This makes the paper a sufficiently novel contribution to the dueling bandit literature. The authors differentiate their work from existing works primarily in how they exploit the entries of the gap matrix. The authors cite papers which have applied the Tsallis-inf and EXP3-IX algorithms to the classic MAB problem or the dueling bandit problem under strong regret, as well as the papers which make up the state of the art for weak regret minimization. In this way, they do a good job of citing relevant related works.\n# Quality\nIt is evident that significant effort was made by the authors to show their work in the proofs. Furthermore, the authors are careful to explain the rationale behind their algorithmic design choices. In my judgment, this has led to a technically sound paper. As for the transparency with respect to the limitations of the work, the authors are honest about the fact that neither of the algorithms introduced will perform optimally in all cases, and that further work is required to find an algorithm which exhibits the benefits of both WR-TINF and WR-EXP3-IX at once. I consider this reasonable since the regret bound for WR-TINF already serves as a major theoretical contribution.\n# Clarity\nI found the authors’ explanations easy to understand aside from one point which I mention in the questions section. I did not see any mathematical errors in the main content of the paper, and I saw only a few typos in the writing. Overall, very well presented.\n# Significance\nThe authors argue that understanding weak regret is valuable because in many practical scenarios where recommendation systems are used, the satisfaction of the user depends only on the option they choose, which we expect to be the option they prefer the most. Although uncertainty in preferences and the user’s own desire to explore can violate the assumption that the user will always choose the preferred option out of a selection, the notion of weak regret is generally closer to reality than that of strong regret, which consistently penalizes the recommendation system for options which the user does not choose. Hence, algorithms for weak regret minimization are worthy of exploration. Furthermore, the lower bound for weak regret which establishes the order-optimality of WR-TINF for a certain class of dueling bandit problems will be useful for researchers doing work in this area to judge what directions for future improvement are viable.\n\nWEAKNESSES:\n# Originality\nI see no issues in terms of the originality of the work.\n# Quality\nI see no quality issues in the theoretical analysis of the algorithms. However, I would have liked to have seen the Modified Beat-the-Winner algorithm from [12] featured as a benchmark in the experiments, since the experimental results in [12] suggest that it outperforms WS-W.\n# Clarity\nSome of the steps in the proof of Lemma D.2 are hard to follow; in particular, for equation 34 it would help to see a few of the intermediate steps after applying the definition of $N_m$.\n\nI will also point out the following typos in the paper:\n- line 58 – “tan” should be “than”\n- line 287 – “Comparaison” should be “Comparison”\n- line 301 – “su-optimal” should be “sub-optimal”\n- line 308 - should be “linear scaling with K is optimal in this case” (you are missing the \"is\")\n- line 557 – “having” should be “have”\n- line 668 - “we the proof” should be “with the proof”\n# Significance\nBecause this paper is primarily applying known techniques to a problem setting where they have not been fully exploited yet, its significance is somewhat limited to that problem setting. However, this is a perfectly valid approach in cases where the state-of-the-art for that problem setting can be advanced significantly.\n\nQUESTIONS:\nIn the explanation of algorithm 2 at the start of page 8, I don’t see why $\\Delta_{sub}$ wouldn’t always be greater than or equal to $\\Delta_{cw}$. If $\\Delta_{sub}$ is the maximum gap that any arm can have over arm $i$ and $\\Delta_{cw}$ is the gap for a specific arm, namely the Condorcet winner, how would it be possible for their ratio to be smaller than 1? The example given in lines 161 to 169 makes more sense to me because $\\Delta_{sub}$ is defined such that it clearly does not depend on $\\Delta_{cw}$. This seems like a miscommunication rather than a technical oversight; maybe you meant for $\\Delta_{sub}$ to be the max over all arms excluding $k^*$?\n\nLIMITATIONS:\nSince this paper presents foundational research, there are no societal impacts to consider. The main limitation with the algorithms that the authors present is addressed in Section 7 of the paper, as mentioned in the strengths section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses weak regret minimization. The authors demonstrate a lower bound result in terms of gaps between the Condorcet winner and the sub-optimal arms. Furthermore, they propose the WR-TINF algorithm, which achieves this optimal regret when the optimality gap is sufficiently large. Additionally, they introduce the WR-EXP3-IX algorithm, which outperforms WR-TINF when the optimality gap is negligible.\n\nSTRENGTHS:\n1. The authors proposed two algorithms focusing on minimizing weak regret\n\n2. The authors provided both upper and lower bounds rigorously.\n\n3. Both algorithms have special advantages. For instnce, the former algorithm is optimal in some instances. The latter algorithm is more powerful when the optimality gap is negligible.\n\nWEAKNESSES:\n1. Too long and comlicated structure of proofs \n\n2. some typos and unclear notations (with equations out of range)\n\nQUESTIONS:\n1. Lemma B.4 in Line 545 used the learning rate $\\eta_t=\\frac{2\\beta}{\\sqrt{t}}$ but the equation (8) in Algorithm 1 used $\\frac{1}{\\eta_t}$. Is  this  corrct?\n\n2. tan in line 58 should be than. Is this right?\n\n3. In Algorithm 1 WR-TINF, else statement seems to be unnecessary because $I_t$ and $J_t$ are already selected.\n\n4. If $J_t$ is sampled by $r_t$ in Algorithm 1, and still $I_t$ equals to $J_t$, It is curious whether $J_t$ is sampled again or not.\n\n5. If the Condorcet winner is unique and there is a large optimality gap, I suspect that most of the second arms sampled by $r_t$ will still be the Condorcet winner after a sufficient number of rounds. If so, the WR-TINF algorithm may become sublinear in terms of strong regret (i.e., its exploration ability may eventually disappear). Please seek the authors' opinion on this matter. i.e., I am curious whether WR-TINF is still effective to the strong regret.\n\n6. The equation in Line 537 is outside the scope of the paper.\n\n7. the mearning of $\\lceil * \\rceil$ is unclear in Line 507.\n\nLIMITATIONS:\nThe authors clearly specified their paper's limitation (and solving the limitation will be difficult).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}