Non-Stationary Dueling Bandits Under a Weighted Borda Criterion
Authors: Joe Suk
Arpit Agarwal
Venue: Accepted by TMLR ()

Abstract: In $K$-armed dueling bandits, the learner receives preference feedback between arms, and the regret of an arm is defined in terms of its suboptimality to a winner arm. The non-stationary variant of the problem, motivated by concerns of changing user preferences, has received recent interest (Saha and Gupta, 2022; Buening and Saha, 2023; Suk and Agarwal, 2023). The goal here is to design algorithms with low dynamic regret, ideally without foreknowledge of the amount of change.

The notion of regret here is tied to a notion of winner arm, most typically taken to be a so-called Condorcet winner or a Borda winner. However, the aforementioned results mostly focus on the Condorcet winner. In comparison, the Borda version of this problem has received less attention which is the focus of this work. We establish the first optimal and adaptive dynamic regret upper bound $\tilde{O}(\tilde{L}^{1/3} K^{1/3} T^{2/3})$, where $\tilde{L}$ is the unknown number of significant Borda winner switches.

We also introduce a novel weighted Borda score framework which generalizes both the Borda and Condorcet problems. This framework surprisingly allows a Borda-style regret analysis of the Condorcet problem and establishes improved bounds over the theoretical state-of-art in regimes with a large number of arms or many spurious changes in Condorcet winner. Such a generalization was not known and could be of independent interest.

Published in Transactions on Machine Learning Research (12/2024)
Non-Stationary Dueling Bandits Under a Weighted Borda Criterion
Joe Suk Columbia University
joe. www.scluubbbbacccmmm edu
Arpit Agarwal"
agarpi com
Columbia University
Reviewed on OpenReview: https: // openreview. net/ formm? id= XXXX
Abstract
In K-armed dueling bandits, the learner receives preference leedback ween arms, and the regret of an arm defined in terms of its suboptimality to a winner arm. The non-stationary variant of the problem, motivated by concerns of changing user preferences, has reccived recent interest (Saha & Gupla, 2022; Buening & Saha, 2023; Suk & Agarwal, 2023). The goal here is to design algorithms with low dynamic regret, ideally without foreknowledge of the amount of change.
The notion of regret here is tied to notion of winner arm, most typically taken to be so-called Condorcet winner or a Borda winner. However, the aforementioned results mostly focus on the Condorcet winner. In comparison, the Borda version of this problem has received less attention which is the focus of this work. We establish the first optimal and adaptive dynamic regret upper bound (1////3333/3 where the unknown number of significant Borda winner switches.
We also introduce a novel weighted Borda score framework which generalizes both the Borda and Condorcet problems. This framework surprisingly allows a Borda-style regret analysis of the Condorcet problem and establishes improved bounds over the theoretical state-of-art in regimes with large number of arms many spurious changes in Condorcet winner. Such generalization was not known and could be of independent interest.
1 Introduction
In K-armed dueling bandits problem, learner relics relative feedback between arms, opposed to the reward feedback model of the more well-studied multi-armed bandit problem (see Sui el. al., 2018; Bengs et al., 2021, for surveys). This problem has application in information retrieval, recommendation systems, etc, where rolative foodback is casy to elicit, while real-valued foedback is difficult to obtaín or interpret. For example, the availability of implicit user feedback comparing the output of two information retricval algorithms allows one to automatically tune the parameters of these algorithms using the framework of dueling bandits (Radlinski al., 2008; Liu, 2009).
Formally, at round E [T], the learner pulls a pair of arms and observes relative feedback between these arms indicating which was preferred. The feedback is drawn randomly according to a pairwise mean preference matrix and the regret measured in terms of the sub-optimality of choices to 'winner' arm. Unlike classical MAB, there are different contending notions of winner arm in ducling bandits, and the undorlying theory depends ically (CW) (Urvoy et al., 2013; Ailon et al., 2014; Zoghi et al., 2014; 2015b; Komiyama et al., 2015) which is an arm that 'stochastically beats' every other arm. An alternative line of works focuses on the Borda winner
the chosen notion. Most. early work in dueling bandits considered the Condorcel winner
"The author is currently at FAIR, Meta. Work done while the author was Columbia University.

---

Published in Transactions on Machine Learning Research (12/2024)
(BW), ) an arm maximizing the probability of defeating a uniformly at random chosen comparator (Urvoy et al., 2013; Jamieson et al., 2015; Ramamohan et al., 2016; Falahatgar et al., 2017; Lin & Lu, 2018; Heckel et al., 2018; Saha et al., 2021).
Both Borda and Condorcet carry their own notions of suboptimalityyregret While the Condorcet winner may not always exist, the Borda winner is always well-defined. On the other hand, the Borda winner may not satisfy the independence of clones property (Schulze, 2011), i.e. the Borda winner may change by adding identical copies of an arm to the model. Thus, it's difficult to prefer one winner notion over the other.'
In the more challenging non-stationary dueling bandit problem, preferences may change over time. Saha & Gupta (2022) first studied this problem and provided an algorithm that achieves a nearly optimal Condorcet dynamic regret of O(VScKT), where Se is the number of changes in Condorect Winner. For the Borda problem, they showed dynamic regret bound of 0(S
////2//////// in terms of Sg switches + + K-1//////////
B in Borda winner. While the Condorcet rate of VT is minimax optimal, up to log terms, the reported Borda rate is suboptimal as confirmed by our lower bound (Theorem 7). Furthermore, the aforementioned procedures (for both winner settings) are not adaptive, requiring knowledge of the number of switches. Moreover, the dependence on Sc, SB is pessimistic, as these may count insignificant changes (e.g., if there are Q(T) spurious changes in CW) which do not properly capture the difficulty of non-stationarity.
Addressing these limitations, Buening & Saha (2023) propose for the CW setting notion of significant CW switches (only counting those switches in CW which truly pose challenging non-stationarity). Under the classical strong stochastic transitivity (SST) and stochastic triangle inequality (STI) conditions on the preferences (Assumption 1), they achieve a parameter-free dynamic regret bound of OKK in terms of Se such switches. A subsequent work (Suk & Agarwal, 2023) achieved the optimal dependence on K of Ö STT)). Naturally, one may ask whether an analogous result is possible for the Borda problem, leading to our first question (answered in the affirmative by our Theorem 2):
Question #1. Can we attain adaptive and optimal Borda dynamic regret in terms of "Borda notion" of signifieant winner switches?
Now, turning back to the Condorcet problem, we again note that the (cKT) rate was only established under SST and STI, which assumes a linear ordering on arms and monotonicity celiooniiiitttraniiiii ttraniitivityy conditions on preferences. While such assumptions are well-studied in carlier ducling bandit works (Yue & Joachims, 2011; Yue el al., 2012), they're re arguably unrealistic as arm preferences may not even be totally ordered in application (e.g., cyelic preferences in a tournament). Despite this, Suk & Agarwal (2023) showed the VScKT rate is in fact unattainable without SSTNSTI.
Buening & Saha (2023) show, outside of SSTNSTI, a dynamic regret upper bound of Ö(K in terms of the coarser count S Sc of total CW switches. However, when compared to the optimal stationary regret rate VKT, it's unclear if the dependence of K in this result is optimal and also whether an intermediate notion of significant non-stationarity (counting between Sç and S switches in CW) can be learned adaptively. In other words, one hopes for a regret rate of Ö(VSmm in terms of some count Sint E [Sc, S] capturing the learnable significant non-stationarity, thus resolving the gap in the findings of Suk & Agarwal (2023) and Buening& & Saha (2023).
Question #2. Can we learn count of enhanced changes for switching Condorcet dueling bandits and improve the K /ST rate?
In fact, we argue this is a difficult problem. Even in the stationary dueling bandit problem, the minimax optimal regret rate of VKT is only known to be achievable using sparring reduction to adversarial bandit. algorithms (Dudik et al., 2015; Balsubramani et al., 2016; Saha & Gaillard, 2022). On the other hand, the adaptive procedures of Buening & Saha (2023): Suk & Agarwal (2023) crucially rely on stochastic elimination-style algorithms. Thus, it's unclear how to simultaneously attain a sharp dependence on K and handle unknown non-stationarity.
Thee celebrated Arrow's impossibility theorem from social choice theory (Arrow, 1950) shows no single winner notion satisfies class of "reasonable" axiomatic requirements. measured to a time-varying sequence of Condorcet winners. a These assume the preference matrix admits a total ordering on arms which obey monotonicity and transitivity conditions.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Condorcer Winner (CW) Class
GIC
SST
STI
Figure 1: GIC SST, STI
Our second contribution is to make progress in answering the above question. In particular, we introduce a new count Sapprox of approzimate CW changes and show adaptive Condorcet regret bound of cl/3 K under a weaker general adentifiability condition (GIC) (Condition 1). GIC mandates that there's an arm stochastically dominating any other arm with the largest margin, and is weaker than SSTOSTI (see Figure for comparison). This yields an improvement over the KvST rate of Buening & Saha. (2023) under GIC when the number of arms is large or when Sapprox < S.
Both our discussion for non-stationary Borda and Condorcet dueling bandits can also go through alternative non-stationarity measure Vr measuring the total variation of preferences over time. For the Borda setting, we give the first optimal dynamic regret bound in terms of Vr (Theorem 2). For the Condorcet setting, we give the first adaptive dynamic rogret bound, which was only shown proviously under the restrictive SSTOSTI assumption (Buening& & Saha, 2023; Suk & Agarwal, 2023).
Both our answers to Questions #1 and 2 involve a new unified regret minimization framework, called weighted Borda scores, for dueling bandits. This framework generalizes both the Condorcet and Borda problems and, in particular, allows for the Condorcet regret to be recast as a Borda-like regret. Such a relationship was not known before and may be of independent interest. We note that, within this framework, showing a Condorcet regret bound involves additional complications beyond the Borda-style regret analysis as the reference weight (a notion generalizing the uniform weight of the Borda score; ef. Section 6) on arms is unknown and must be efficiently estimated. Further details the challenges of this Condorcet regret analysis found in Section 7.
A more expansive account of relatod works on ducling bandits deferrod Appendix J.
1.1 Tabular Summary Contributions
Notation
Non-Stationati Measure
SB
Borda winner switches
5p
Significant BW switches (Definition 1)
Sc
Condorcet winner switches
Šc
Significant CW switches (Bucning & Saha, 2023)
Sapprox VT
Approximate CW switches (Definition 4)
Total Variation in preferences (Saha & Gupta, 2022)
Figure Glossary of Non-Stationarity Measures
2 Setup Non-stationary Dueling Bandits
We consider K Kaaredd dueling bandits with horizon T. At round t E [2'], the pairwise preference matrix is dcnoted by P, E [0, K where (t.j)-th entry P,(i.) encodes the likelihood of observing a preference for arm ? in a direct comparison with arm j. In the stationary dueling bandit problem, the preference matrices the non-stationary problem, the preference matrices P may
P. P P are unchanging time, whereas change arbitrarily from round round. At round the learner selects a pair of actions (ie,).) [K] [K] and observes the feedback Oc( je) Ber((P ji)) where P(().. is the underlying preference of arm i over je. We next outline the two main formulations of dueling bandits (Borda and Condorcet).
3

---

Published in Transactions on Machine Learning Research (12/2024)
Model
Regret Upper Bound
Adaptive?
Borda
Saha & Gupta (2022)
None
Sy K 1/3///6/66 Sg K KL/3/2/3
No
Saha & Cupta (2022)
None
V. 1-1K--1715 + V 15/22/110//
No.
Our Work (Theorem 2)
None
min
m S K1/37213 V k1/4/5/4 + K2/312/3
Yes
Lower Bound (Theorems and 8)
None
min(Sp (811' K K1/37213, V "//"1//47p k1/478/4 + K1/372/3
Condorcet
Saha & Gupta (2022)
C.W.
VScKT
No
Saha & Gupta (2022)
C.W.
V K1/372/3 VKT
No
Buening & Saha (2023)
SSTOSTI min K K Sc, V / KVT)
Yes
Suk & Agarwal (2023)
SSTOSTI min SCKT, V k1/372/3 + VKT:
Yes
Buening & Saha (2023)
C.W. KvSc!
Yes
Our Work (Theorem 5)
GIC min Sapprox K K21á74/3 V K1/27//4 K2/a1//3
177
Yes
Lower Bound (Saha & Gupta, 2022)
C.W. min KT V
K ///2//3 VKT]
Table I: Comparison of dynamic regret upper bounds. See Subsection 5.2 for detailed comparison of our
K2/3 Sapprox rate vs. the K /ScT rate of Buening & Saha (2023). See Figure for a comparison of the model classes CW, GIC, SSTSSTI. Our regret upper bound for the Borda model is tight (Theorem 7).
Borda Criterion. Let (a) = *
Σ Pa(a,d) a') be the Borda score of arm a Then, ap
a'E[K]
argmax,e]K] (a) is the Borda winner (BW) at round t. Then, the Borda dynamic regret
T Duaaa
Regret B
(1)
We let (a) = br((a) ba(a) be the instantaneous dynamic Borda regret of arm at round t.
Condorcet Criterion. Here, one assumes the existence of Condorcet winner (CW) of such that Pi(af a) 1/2 for all arms [K]. Then, the Condorcet dynamic regret is defined as
Pr in) + Pi(af, -1
Regret C
2
2.1 Non-Stationarity Measures
Following the discussion in Section 1, key in works non-stationary (dueling) bandits is measure of non-stationarity which captures the difficulty of the problem. Speaking plainly, the higher the amount of non-stationarity the more difficult the problem is, and so larger regret rates are expected. It is thus crucial that such a measure of change properly captures the difficulty of the problem. However, the only other work dealing with Borda dynamic regret (Saha Gupta, 2022) relies on coarse non-stationarity measures, such as the aggregate number L Σt= 1(P + P--1} or magnitude Vi = Σ max [P Pa- aa)a of a.a'E
changes (see Figure 2).
To contrast, in non-stationary MAB and Condorcet dueling bandits, it is now recognized (Suk & Kpotufe, 2022; Buening & Saha, 2023; Suk & Agarwal, 2023) that lighter non-slationary measures, so-called significant shifts, which capture only those changes which are detrimental to performance can in fact be adaptively learned. The main intuition bohind a significant shift is that, for any notion of regret (c.g., MAB, Condorcet, Borda), one can track when each arm accrues significant regret (i.e., surpassing the minimax rate) and define a significant shift. as occurring when every arm has accrued significant regret. Thus, our Definition has a similar form Definition 1 of Suk & Kpotufe (2022) or as the corresponding notion for Condorcet winner dueling bandits (Buening & Saha, 2023). The key difference here is that use the Borda notion of regret and compare to the Borda minimax regret rate 2/3 vs. 1/2)..
4

---

Published in Transactions on Machine Learning Research (12/2024)
Definition 1. Define an arm a as having significant Borda regret on 82] if
Σ (a) K2/3. (82 - -81)2/3
(2)
Define significant BW switches (abbrev. SBS) as follows: let To = and the (i + 1)-th sig. shift T4t1 is recursively defined as the smallest t TÜ such that for each arm a E [K], 3[51, 32] C [Ti, t such that arm a has significant regret over 82]. We refer to the interval of rounds [T;, T++1) as an SBS phase'. Let SB be the numberr of SBS phoses elopsed in T rounds. By convention, lel T T +
Remark 1. If there is a fized Borda winner at = a², then it does not incur signéficant regret. Thus, the number SB of SBS is smaller than the number Sg of Borda winner switches, meaning Definition is tighter measure of non-stationarity.
Notation. As common works on non-stationary bandils (see Appendia J), we 'll often conflate counts of changes (e.g., SB) with the number of phases induced, appearing in regret rates, which always one greater than the corresponding count. Note that this only offects constants in the regret rates.
3 Dynamic Regret Lower Bounds
We briefly summarize the minimax dynamic regret rates for the Borda and Condorcet problems. Full statements and proofs are deferred to Appendices B and C. We note the arguments are fairly standard and mostly follow previous lower bound constructions for Borda (Theorem 16 Saha et al., 2021) and non-stationary dueling bandits (Theorems 5.1, 5.2 Saha & Gupta, 2022).
Borda Dynamic Regret As the minimax Borda regret rate over n stationary rounds is K//3 (Saha et al., 2021), it follows that the minimax dynamic regret rate over L stationary phases of length T/L is (T/L//T/ = ///33/2//////// (Theorem 7). In fact, the number of SBS phases (Definition 1) may replace L here. To our knowledge, this establishes the first lower bound on Borda dynamic regret.
Condorcet Dynamic Regret Here, since the minimax regret rate over stationary problems is of order v VKT, the lower bound on Condorcet dynamic regret of order VLKT (Theorem 5.1; Saha & Gupta, 2022). Once again, L here may be replaced by a tighter measure of non-stationarity (Suk & Agarwal, 2023).
4 Algorithmic Design
To minimize the weighted Borda dynamic regret, we rely on estimating (a w) for weights w belonging to a reference set of weights W. In what follows, we'll discuss the procedures in terms of fixed W. In the end, W can be flexibly specialized for each of the Borda and Condorcet regret problems (see Definition 4).
Following the high-level idea of prior works on non-stationary dueling bandits (Suk & Agarwal, 2023; Buening & Saha, 2023), we'll first design a base algorithm which works well mildly non-stationary environments where there is no approximate winner change (Definition 4) and then randomly schedule different instances of this base algorithm to allow for detection of unknown non-stationarity.
One key deviation from the aforementioned prior non-stationary works is that we avoid using a successive elimination base algorithm. This is necessary since accurate estimation of the (weighted) Borda score by (a, w), calls for some baseline uniform exploration, which rules out any hard elimination strategy.
Thus, we first design a new base algorithm, which mixes elimination with lime-varying exploration.
4.1 Base Algorithm Soft Elimination with WBS
Estimating Weighted Borda Scores. At round t, we estimate the weighted Borda scores via importance weighting for weight w E W:
1( a.j. = a'} Or a'j (ie, (e(a) (c
b, w) = E
(3)
We conflate intervals b), [a, b] for a, b E N with the set of natural numbers contained therein.
5

---

Published in Transactions on Machine Learning Research (12/2024)
where Qi is the play-distribution over arms at time t. Let (a', = =a((a, w) - ba w) denote the induced estimator the weighted Borda gap.
Eliminating Arms
We evict arm a from the condidate orn set As at round if for some (si, 82] S [1, d:
max d'En
E (a'. Clog(T) F([s1. s2)).
(4)
In the above C > is universal constant free of any problem-dependdet parameters whose value can be determined from the analysis and the eviction threshold F(J) for interval determined using the estimation error bounds on Σ (a, w) (see Definition in Appendix E for exact formulas for F()))-
These error bounds scale with the inverse play-probability min, (a), calling for a careful design of the play distribution qc(,), namely that which balances exploration and proper elimination.
Novel Exploration Schedule. We choose dr as mixture exploring the candidate set A and uniformly exploring all arms:
de ~ (1 (1) Uni[(A) + * Unif(W),
(5)
where WW) is a further uniform mixture of playing according to the distributions in W. The learning rate 7/ must then be carefully set ensure both sufficient exploration (to reliably estimate the WBS) and safe regret. As before, the values of T) (see Definition 4) will depend on the Borda vs. Condorcet setting.
However, prior works on stationary Borda dueling bandits must set Th using knowledge of the horizon T (Jamieson et al., 2015; Saha et al., 2021; Wu et al., 2023). In our non-stationary problem, this amounts to setting Te based on knowledge of the underlying non-stationarity (Saha Gupta, 2022).
Thus, a key difficulty in targeting the regret rate of Theorem 2 without knowledge of non-stationarity is that the optimal oracle learning rale Y) depends on the unknown phase lengths (Ti+1 тi for Borda scores; Pi+1 - Pi for Condorcet). To circumvent this, we employ a time-varying learning rate π) which depends on the current number of rounds elapsed. To our knowledge, such an idea has only been used in works on adversarial bandit for analyzing EXP3 (Seldin al., 2013; Maillard, 2011).
4.2 Non-Stationary Meta- Algorithm
Algorithm 1: BOSSE( (start, ma): (Weighted) BOrda Score Soft Elimination
Input:
Input set of weights W, learning rate profile y = {7)T=1, eviction threshold F(+).
Initialize: t. +- fstart; active arm set Atrart +- [K].
while { 7 do
Set exploration rate for this round: f- Te-be Update play distribution q: let de A be the mixture (1 -nm))) Unif(AA +m Un]f(WW..
Sample Jr ~ i.i.d..
Receive feedback Oe(ie.ji) ~ Ber( (e)(
Increment + f+
Evict arms with large weighted Borda gaps:
A x- Ae\ (a E [K] 7(81, 82] S [bstrrt, f), w E W s.t. (4) holds with a E A.};
Aglobal + Aglobal \ fa E [K] 3[s1, s2] S [te, t), w W s.t. (4) holds with a E n?,,, A.}: if 3m such that Bt. then
10
/ Lines 10-13 only for use with METABOSSE */ // Set maximum replay length. // Replay interrupts.
Let m = max(m E 42,4.... (:44..,,,[0o(T))) 2[og(T)]) Bt,. > 00. Run BOSSE(t,m). Restart criterion: Aginbal then RETURN.;
11
12
13
14
if t Estart +mo then RETURN.

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm 2: Meta-BOSSE
Input: horizon T, input set of weights W, learning rate profile y = (Yr)e=> eviction threshold F(.). Initialize: round count & f-11. 1.
Episode Initialization (setting global variables ty: Aglobal, Be.m):
lg 1. Aglobal < [K] For each m = 2, 2,,,..., 20og((T) and 8 = 1 ,....,?? T:
11 1e indicates start of f-th episode.
// Global active arm set.
Sample and store Be,m ~ Bernoulli (1778.(0-
// Set replay schedule.
Run B
if t < T then restart from Line (i.e. start a new episode).
For the non-stationary setting, we use hicrarchical algorithm METABOSSE (Algorithm 2) to schedule multiples copies of the base algorithm BOSSE m) (Algorithm 1) at random times lstart and durations m.
Going into more detail, METABOSSE proceeds in episodes, starting cach episode by running a starter instance of BOSSE. A running base algorithm may further activates its own base algorithuns of varying durations (Line of Algorithm 1), called replays according to a random schedule decided by the Bernoulli's Bs, (see Line 6 of Algorithm 2). We refer the base algorithm playing at round
the active base algorithm.
The condidate arm set Ar is pruned by the active base algorithm at round I, and globally shared between all running base algorithms. In addition, all other variables, i.e. the &-th episode start time te, round count t, and replay schedule (Bx, ),m, are shared between base algorithms.
In sharing these global variables, any replay can trigger new episode: every time an arm is evicted by a replay, it is also evicted from the global arm set Aglobal, essentially the active arm set for the entire episode. A new episode is triggered when Aglohal becomes empty, i.e., there is no safe arm left to play.
For further intuition on the hicrarchical schedule and management of base algorithms, defer the reader to Section 4 of Suk& & Kpotufe (2022). We focus here on highlighting the novelties of this algorithm.
Each Active Base Alg. Chooses Own Learning Rate Each base algorithm BOSSE(fatare m) determines its own time-varying learning rate using its starting round hatrt- Then, the global learning rate ve at round t is set by the base algorithm active at round t (Line 3 of Algorithm 1). Furthermore, the history of global learning rates } (globally accessible by any base algorithm) is used to determine the eviction thresholds (16) and (17) over intervals s2] of rounds where multiple base algorithms may be active.
This ensures reliable detection of critical segments [$1, of time where arm has large (weighted) Borda regret. For such a critical segment, a core argument of the analysis is that an ideal replay, i.c., an instance of BOSSE($1, m) for m = 82 81, is scheduled with high probability. However, such an ideal replay must be scheduled for a duration commensurate with $2 - $1, and hence must also use a commensurate learning rate.
A Different Replay Scheduling Rate. In order to properly detect critical segments while also safeguarding T2/3 regret, a different replay scheduling rate (Line 12 of Algorithm 1) is required. For this, we find that instantiating a base algorithm of length m at time in episode [te, t441) with probability m m-133 (s -2/3 2/3 balances regret minimization and detection of changes in winner.
Remark 2. For the Borda setting, the total complexity of our algorithm is O(KT2) versus OKT)) for DEX3 (Saha & Gupta, 2022), but the latter algorithm requires knowledge of non-stationarity. For the Condorcet setting, we have a complexity of O(K2T²) which matches that of ANACONDA (Buening & Saha, 2023), the only other adoptive algorithm with regret guarantees for Condorcet winner. We emphasize, regardless of setting, it remains open whether higher runtime can be avoided while getting adaptive and optimal regret

---

Published in Transactions on Machine Learning Research (12/2024)
5 Dynamic Regret Upper Bounds
5.1 Borda Dueling Bandits
Our goal then to establish a dynamic regret upper bound which depends optimally on the number of SBS, and does not require knowledge of the underlying non-stationarity.
Intuitively, if one knows the SBS Tú, then, in each SBS phase [T, T=-1), onc can achieve a tight regret bound of order K-// (T++1 - 2/33//3 by learning the last safe arm, or the last arm to incur significant Borda regret in said phase. We show that in fact such rate can be attained, over all SBS phases, without any knowledge of the non-stationarity.
Adaptive and Optimal Regret Upper Bound.
Theorem Algorithm 2 with the fixed weight specification (see Definition 4) satisfies:
Sn Σ 1=0
E Regret PJ So
k2/3
(T++1 - (T=+1-T;)
Corollary 3. By Jensen's inequality, the regret bound of Theorem 2 is upper bounded by .2//3///3... .5/33 Furthermore, relating SBS to total variation Vr, regret bound of order v-/4 73/44 K1/1 + K1/3 T2/3 also holds (see Appendix H.5).
In particular, the upper bound of Theorem 2 matches the lower bound of Section 3 (sce Theorem 7). The K k1/312/35) rate also improves the previously best-known upper bound (Saha & Gupta, 2022, Theorem 6.1), which was suboptimal, non-adaptive, and relied on a coarser notion of non-stationary (i.c., counting all changes in Borda winner).
5.2 Condorcet Dueling Bandits
Significant Notions of Change Outside of SSTOSTI. Following the discussion of Section 1, was previously shown in Suk & Agarwal (2023) that analogue of Definition 1 for Condorcet winner switches can't be learned adaptively outside the SST and STI preference model assumptions.
Assumption 1. There is total ordering arms, > such that Vi Z ž k:
i iz.j + ((1,))2113
P((ik)) 2 (ax( k)} (SST).
P.(i,)) + (STI).
In particular, such conditions convenient for relating uncompared arms through an inferred ordering, and turn out to be fundamental to detecting unknown changes in CW.
However, this impossibility result leaves open whether other tighter notions of non-stationarity can be learnod outside of SSTSSTI, and at rates faster than the stale-of-the-art K K Condorcet dynamic regret, in terms of Se changes in winner, achieved by Buening & Saha (2023).
We show this indcod possible in a broad class preference models, called GIC, outside of SSTOSTI. Condition 1. (General Identifiability Condition) At each round t, there exists an arm ot such that oj E argmax Pr(a, for all orms a' E [K].
While GIC requires the CW to beat every other arm with the largest margin, it is far broader than SSTOSTI in not requiring any ordering non-winner arms, allowing for cycles arbitrary preference relations. The GIC was previously studied in utility dueling bandits by Zimmert & Seldin (2018) (see also Bengs et al., 2021, Scction 3.1). Unlike those prior works, we do not require the winner arm ot to be unique.
We also note that Condition implies the Condorcet condition, but is weaker than SST while incomparable with STL See Figure 1 for full comparison.
8

---

Published in Transactions on Machine Learning Research (12/2024)
We next show that, under GIC, a tighter notion of non-stationarity than the count Se of winner switches can be learned with improved regret rates. In particular, we count changes in approzimale winner, which are arms exhibiting similar preferences to other arms as the true winner. The threshold for similarity is determined by the minimax regret rate which this case is K2/37223 (Theorem 11). We count an approximate winner change only when there is no longer stable approximate winner.
Definition 4 (Approximate Winner Changes). Define Çi recursively as follows: let Ço = 1 and let Sit1 be the smallest 1 > Çi such that there does not exist an approximate winner arm à such that
K2
1/3
Vs E (G, [K]:|P((0,,0) a,(ā,a))s -
(6)
Let Sapprox = 1 max : St < T} be the total count of approzimate winner phases.
Note that (6) always holds so long as the winner az does not change. Thus, we have Sapprox S S. Example 1 (Environment where Sapprox 1, but S = -(T)), Consider non-stationary environment with three arms 1,3 whose preference structure allernales belween the two malrices:
1/2 1/2 1/2-T-1 1/2-7-1/3
1/2+T-* 1-133
1 1/2 T-1/3 P = 1/2+7-1/8
1/2
1/2 - T-1/3
0
(112-
p+ =
1/3
1/2
1/2
1/2 - 7-133
1/2-7-1/3
1/2 - 1/3
1/2
1
1/2
Then, under environment P (resp. PT), ) the arms ordered > 2 3 (resp. 3 2 1). In either environment, 2 is upprozimale winner arm meaning the T-rownd non-slationary evecroonmen has S = (TT) winner switches but no approrimate winner switches.
Condorcet Dynamic Regret Bound. Our regret upper bound is then as follows.
Theorem 5. Under Condition 1, Algorithm 2 with unknown weight specification (see Definition 4) satisfies:
E[Regret < (0min[33/3 2/3 $1/3 approxe 1//2. v2/4 T3/4 + K2/3 (2/33))
We now compare this regret bound to the KVST rate of Buening & Saha (2023). In particular, we have:
T Saprrxx
T T
min(K$T ti Spprrox Kt x + k S KVST K min
max
(7)
S
S
s
We highlight two regimes, under GIC, where this comparison is favorable to rate:
Many spurious winner changes $ $ T: if there are = changes in winner, then (7) always holds regardless of the values of Vr, Sapprex K, T and thus captures regimes where sublinear regret is possible while the K /ST rate is vacuous. The superiority of our regret rate is most evident when Sapprox or VT are small, which is possible if the majority of $ = T winner changes are spurious to performance.
Large number of arms: viewed another way, (7) states that if the number of arms K is larger than some threshold determined by the discrepancy in the non-stationarity measures S vs. Sapprox or S vs Vr, then Theorem 5's regret rate is superior to the KVST rate.
We also note that the K2/2 v2/4 7// rate of Theorem 5 is the first adaptive Condorcet dynamic regret bound in terms of total variation, prior works only achieved regret upper bounds under SSTOSTI (Buening & Saha, 2023; Suk & Agarwal, 2023).

---

Published in Transactions on Machine Learning Research (12/2024)
6 A New Unified View of Condorcet and Borda Regret
Continuing the discussion of Subsection 1.1, our regret upper bounds (Theorems 2 and 5) are shown by appcaling to new framework which generalizes the Borda and Condorcet regret minimization tasks.
Key to this is the new idea of a weighted Borda score (WBS), which measures the preference of an arm a over a reference distribution or weight of arms w E A where A denotes the probability simplex on [K]. More prociscly, we define WBS with respect weight w = (w* ,...,,. as
ba w) = Σ Pi(a,a') a') w". d'E[K]
Notions of weighted Borda winner (maximizer of WBS) and weighted Borda dynamicc regrel (analogous to (1)) follow suit. Taking w to be [n]] this recovers the Borda score and dynamic regret.
If a Condoreet winner of exists, then taking to be the point-mass weight w(a,) of also allows to capture the Condorcet regret. Indeed, one observes that the CW of maximizes the score ba ((a))), and the regret in terms of weighted Borda scores of arm w.r.t. weight w(af) becomes
be C w(a ((al) Pi(o; 2'
which is precisely the Condorcet regret at round t.
Recasting the Condorcet regret objective as a Borda-like regret quantity will be key to bypassing the neod for SSTOSTI, while still capturing an enhanced measure of non-stat ionaait.
Changing and Unknown Weights (Key Difficulty). The challenge with this reformulation of the Condorcet problem is that the reference weight w(af') is unknown since the CW of is. Furthermore, in the more difficult non-stationary problem, the identity of af may change at unknown times meaning so can the weight ( This makes it difficult to even estimate the weighted Borda score by(a, we(af)) compared to the usual Borda task where the weight w is fixed over time.
We will show that such difficulties are resolved by tracking all point-mass weights and carefully amortizing the regret analysis to periods of time where estimating a fixed point-mass weight suffices.
7 Proof Outlines and Novelties
To simplify discussion, we'll first outline the proof for the regret upper bound for Borda dueling bandits (Theorem 2) and then subsequently discuss the necessary modifications for the Condorcet setting (Theorem 5).
7.1 Proof Outline for Theorem
We follow general proof strategy seen in prior works on non-stationary (dueling) bandits (Suk & Kpotufe, 2022; Buening & Saha, 2023; Suk& & Agarwal, 2023). The full proof can be found in Appendix H for the generalized weigted Borda problem with fixed weight (introduced in Section 6).
Estimators Concentrate around True Gaps. Let 8 (a', a) be the estimated weighted Borda score induced by (3) for the uniform weight w ~ [n]} We first assert using Freedman's incquality (Lemma 15) that our estimated (cumulative) weighted Borda scores Σte Dema, 8," (a', a) concentrate about the true values E:=., 8,(('', a)). This mcans arm climination criterion (4) is valid in the sense that arm being evicted means it has significant regret (2).
Thus, with high probability, each episode [te, le+1) aligns with the SBS phases (Definition 1) in the sense that a restart only occurs if an SBS has occurred within the episode (Lemma 20). Thus, suffices to bound the regret in each episode (le, le+1) by the minimax regret rate over each overlapping SBS phase:
Σ
K1/3 (TH41 7/)2/3.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Decomposing the per-Episode Regret Let of denote the last safe arm at round t, or the last arm to incur significant Borda regret in the unique SBS phase T++1) containing round i, per Definition 1. Furthermore, let ag denote the last global arm of episode (e,te41) or the last arm to be evicted from Aglobal in said episode. Then, we can decompose the per-episode regret:
he-1-1
fe-1-1 Σ
fe41-1
BE41-1
Σ (1)) 1=te
+ +
B (a.)
+
Σ
(a i) + 6f (a) +
Σ t=te
(ai
t=
t=te
We next handle each of the three sums on the RHS above. They are respectively:
Per-Episode Regret of the Last Safe Arm: by definition of the safe arm and SBS (Definition 1) this arm has safe regret K (3(31 - T2)2// on each SBS phase [Ts, T++1).
Per-episode Regret of the Played Arms to Last Global Arm: this part of the analysis is analogous to Appendix B.1 of Suk & Kpotufe (2022), Appendix A.2 of Buening & Saha (2023), or Appendix C.3 of Suk & Agarwal (2023). First, note the play distribution q (5) defaults to uniform exploration at a frequency which is safe for K1/3, (te+1 - t22 regret. Thus, it suffices to bound regret when l explores the arms in the candidate arm set A.. Now, dg, by definition, any arm a' retained at all rounds in episode l, and 6f (d', is) = Eme[k] 6 (a', a) 1( = a). Thus, it suffices to bound the relative regrets 5 a) 1ii a}, by carefully considering intervals of time where a is also retained. In particular, we first bound this sum on an interval from t, to (a), defined as the round at which a is evicted from Agiobal, and then intervals within replays of BOSSE which bring a back into A,- Regret on the first kind of interval is bounded using our elimination criterion (4) and concentration. Bounding the regret on the second kind of interval uses the key fact that sufficiently few replays are scheduled.
Per-Episode Regret of the Last Global Arm to the Last Safe Arm: here, we'll use bad segment analysis similar to Appendix B.2 of Suk & Kpotufe (2022) Appendix C.4 of Suk & Agarwal (2023). Roughly, a bad segment [s1, will be such that Σ.=., 8 (at, at) Z (82 - 2/)3/3 K1/3 On such bad segment [ 82], a perfect replay is roughly defined as a replay whose scheduled duration Then, the key fact is that the randomized scheduling of replays (Line 12 of Algorithm 1) ensures that perfect replay is scheduled for some bad segment (thus evicting at from Aglohal before too many bad segments elapse, giving a way of bounding Σ..,, 8 (at ae) with high probability.
overlaps most of s2). s2).
7.2 Challenges of Condorcet Regret Analysis for Proving Theorem
Leaving the details of the analysis to Appendix I, we instead focus here on highlighting the key challenges showing our Condorcet dynamic regret upper bound.
Recasting Condorcet Regret as a weighted Borda Regret. As discussed in Section 1, we emphasize our main analysis novelty is in reformulating the Condorcet regret as a weighted Borda-like rogret. This allows us to take the analysis route of prior works on non-stationary MAB (Suk Kpotule, 2022) without suffering from difficulties endemic to preference feedback, as seen in Buening & Saha (2023); Suk & Agarwal (2023). To our knowledge, this is the first work on dueling bandits to make use of such a trick.
Keeping Track of All Unknown Weights. As mentioned carlier in Section 6, a crucial difficulty with making use of the weighted Borda framework for Condorcet dueling bandits is that the reference weights w are unknown. Furthermore, the space of all possible weights A is combinatorially large and thus one cannot. hope to maintain estimates for all weights in A without an intractable union bound. We show in fact that such accurate estimation can be bypassed because the worst-case regret is always attained at point-mass weight some arm [K] due to regret being linear in the weight vector.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Keeping Track of Unknown Changes in Weights. An added difficulty in the non-stationary setting is that the unknowns weights may change at unknown times. This makes it challenging to bound the regret. over intervals of rounds [2] which clapse multiple approximate winner changes Ps and hence for which we require score estimation w.r.t. a changing weight sequence {we]r- However, even Tollowing the above proposal for tracking point-mass weights, we note the space of all sequences of point-mass weights is combinatorially large. This prohibits estimating all E: (a, Ws) for changing point-mass weights w..
Instead, we carefully divide up the regret analysis into segments of time [s1, s2] S [pt. Pi+1) lying within an approximate winner phase (pt,) where it suffices to bound the regret using a single weight (as it turns out, that of the current approximate winner a²). This involves doing a separate analysis of when an arm is detected as having significant regret for each arm a E [K] and each base algorithm.
8 Conclusion and Future Questions
We've achieved the first optimal and adaptive dynamic Borda regret upper bound. Additionally, we've introduced the weighted Borda framework which revealed new preference models where faster Condorcet regret rates are attainable, adaptively, in terms of new tighter measures of non-stationarity. In the Condorcet setting, it's still unclear if the bounds of Theorem 5 or the KvSc7 rate of Buening & Saha (2023) are tight for the GIC class and, more challengingly, whether the KvScT rate can be improved outside of the GIC class. Outside of the Condorcet Winner assumplion, tracking changing von-Neumann winners in non-stationary dueling bandits appears a challenging, but quite interesting direction.
More broadly, on the theoretical side, future work also independently study this weighted Borda framework (with generic known or unknown weights) and characterize the instanco-dependent regret rates, which remain unclear even in stationary settings.
It'd also be interesting explore deseadee- or logarithmic, regret bounds for the non-stationary ducling bandit problem. We note that even in the non-stationary MAB problem, it's been shown that logarithmic instance-dependent dynamic regret bounds are unattainable in general without knowledge of non-stationarity (Lattimore & Szepesvári, 2020, Theorem 31.2). Thus, it remains open what is an appropriate instanco-dependent dynamic regret rate for this problem.
Future work can also implement Algorithm 2 on real datasets and compare with other algorithms for non- stationary dueling bandits. One challenge here is the high computational cost (O(KT2) computations over T rounds) which makes our procedure somewhat impractical. We note recent work (Li et al., 2024) explores ways to improve the computational complexity while preserving optimal regret guarantees in non-stationary bandits.
References
Yasin Abbasi-Yadkori András György, and Nevena Lazič. A new look at dynamic regret for non-stationary stochastic bandits. Journal of Machine Learning Research, 24(288):1 1-37, 2023. URL https: http///mm/rr////eeeee/l .df.
papers/var //tppr////ouuue222220030///22222
Arpit Agarwal, Nicholas Johnson, and Shivani Agarwal. Choice bandits. In NewrIPS, 2020. URL https: http://///0///....004/4444aa.ee.
// //proceedings wwwiinnssnnnnurre www.nnuur.sccc.
Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing Dueling Bandits to Cardinal Bandits. In Proceedings of the 31st International Conference on Machine Learning, 2014. URL https://arxxv..rr pdf/ / 1405.40540 3396.
Kenneth J. Arrow. A difficulty in the concept of social welfare. Journal of Political Economy, 58(4): 328 346, 1950. ISSN 00223808, 1537534X. URL https://www. stat. uchiccgo.ococo... www. www.aa.-eeee--eeeeeeemeeetineee. mathofranking/ren http/arrow.ppp. pdf.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal computing, 32(1):48-77, 2002. URL http://rob.. schapire net/papers/ AuerCeFrSc01.pds pdf.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best arm with an unknown number of distribution changes. 14th European Workshop on Reinforcement Learning (EWRL), 2018. URL https://ewrl wwwderrwwdddds....ooommm wttp///pcccoatettt////oodaaaeeee 1ttpl//a.22211/////....222222111 _a .df..
Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit with unknown URL https:
number of distribution changes. Conference Learning Theory, pp. 138 158, 2019.
//proceedings .ml.ediiinnim.........mmm...m..e mlr. www.mml.rr..99e-11 httss//v9//aeere/0 aataaa/aerrraaa......
A Akshay Balsubramani, Zohar Karnin, Robert E. Schapire, and Masrour Zoghi. Instance-deeendent regret. bounds for dueling bandits. In 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pp. 336-360. PMLR, 23-26 Jun 2016. URL https://proceeddngg mlr. pdf.
Gábor Bartók, Dean P. Foster, Dávid Pál, Alexander Rakhlin, and Csaba Szepesvári. Partial monitor- ing classification, regret bounds, and algorithms. Mathematics of Operations Research, 39(4)9967-99) 2014. ISSN 0364765X, 15265471. URL 0663.
ine. informs orgg /two//////////////////////////// /moor//o....//////////////////// 2014.
Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. J. Mach. Learn. Res., 22(1), 2021. URL attps://wwwy jmlr.org/ papers/volume22) 18--566/18-544 pdf.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Optimal exploration-expllitatiio in a multi-armed- bandit problem with non-stationary rewards. Stochastic Systems, 9(4):319-337, 2019. URL https://pubsonlinee informs.
inwfrmss.rg//oo//eeeeeee.ooe org/doi/epdf/100 1///1/000118//////// 201.e 20..201190000 .0333.
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandit algorithms with supervised learning guarantees. ATSTATS, 2011. URL httpps://proceedingss mlr. allrpprss//
pdf.
Jasmin Brandt, Björn Haddenhorst, Viktor Bengs, and Eyke Hüllermeier. Finding optimal arms in non- stochastic combinatorial bandits with semi-bandit foedback and finite budget. Advances in Neural Infor-
newriiinnnnnnuuussooeeeeeere..rs wwwwips..cppaappeeeeeee wwtppppp_fiiaaeeeeeeeeeeepeeeepe wwl.ffllespppppee htlles/paper/002/// mation Processing Systems, 2022. URL Conaper--Coffeeence pdf.
Thomas Kleine Buening and Aadirupa Saha. Anaconda: An improved dynamic regret algorithm for adaptive non-stationary dueling bandits. In Francisco Ruiz, Jennifer Dy, and Jan-Willens van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 3854-38 3878. PMLR, 25-27 Apr 2023. URL https: http://2000..leeeeebeeeaaa////// www.eeeneebbennnnaa..a.... pdf.
//proceedings. mlr
Yifang Chen, Chung- Weing-Weiiii Lee, Haipeng Luo, and Chen-Yo Yu Wei. new algorithm for non-stationary contextual bandits: efficient, optimal, and faramcter-free. In 32nd Annual Conference on Learning Theory, 2019.
URL https::/proccedddngg mlr. httpe/eddnng.......eees///////
http:///99//ee/99//e9ee 1eten/ee///bbbbbbeeeee pdf.
Miroslav Dudik, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual Dueling Bandits. In Proceedings of the 28th Conference on Learning Theory, 2015. URL http://
proceedings.co mlr. wttps/sssr.rreeee/000000 http////////di......m
Moein Falahatgar, Yi Hao, Alon Orlitsky, Venkatadheeraj Pichapati, and Vaishakh Ravindrakumar. Maxing and ranking with few assumptions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Aduances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https://proceddng. www.igg...uiis.... ceaappp.ccpppppprrrrro..... c1lles/paper22017/E1. abbcccoaaateeeef77cceee001666663
.dff.
Pratik Gajane, Tanguy Urvoy, and Fabrice Clérot. A relative exponential weighing algorithm for adversarial utility-based dueling bandits. In Francis R. Bach and David M. Blei (eds.), Procecdings of the 32nd International Conference on Machine Learning, 1CML 2015, Lille, France, 6-11 July 2015, volume 37 of
13

---

Published in Transactions on Machine Learning Research (12/2024)
JMLR Workshop and Conference Proceedings, pp. 218-227. JMLR.org, 2015. URL https://proceediing mlr. wttpppesss////jaaaaaaaaaaa .df..
Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In Proceedings of the 22nd International Conference on Algorithmic Learning Theory, pp. 174-188. ALT 2011, Springer, 2011. URL
hwt..eecccccaaa-0002133222211 document.
Björn Haddenhorst, Viktor Bengs, and Eyke Hüllermeier. Identification of the generalized condorcet winner in multi-ducling bandits. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 25904 25911. Cur- ran Associates, Inc., 2021. URL https:/ http::///p..ceeinnee...m wew.eediiinnnn.uiip...... cc/paper cc/paper_files/ppppeeeeeeeeeeeee /tt.s///apae//22//////. pdf.
d9de6a1 24t.13111113000000/7000022222222
Reinhard Heckel, Max Simchowitz, Kannan Ramchandran, and Martin Wainwright. Approximate ranking from pairwise comparisons. In Proceedings of the Twenty- First International Conference on Artificial Intelligence and Stalistics, volume 84 of Proceedings of Muchine Learning Research, pp. 1057-1066 PMLR, 09 11 Apr 2018. URL https:/ amltiinnnnmmm...........mm. mlr.
Lennard Hilgendorf. Duelling bandits with weak regret in adversarial environments. Master's thesis, University of Copenhagen, 2018. URL https: htaps:///axxixe http:://wwwvv.......ae wwt/odgg////ooooooooooo.oo/... 1112//2.f/111//.///////// 1812. 04152.
Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse Dueling Bandits. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, 2015. URL htttp://arxiv..or org/pdf/1522.. 00133.
S. Jia, Qian Xie, Nathan Kallus, and P. Frazier. Smooth non-stationary bandits. In International Conference on Machine Learning, 2023. URL https://ddl acm. org/doi/10 0555//10055555311400000 2t155//33618000...1117..
Levente Kocsis and Csaba Szepesvári. Discounted ucb. 2nd PASCAL Challenges Workshop, 2006. URL
https://wwx.rri.fff http://ww.......ssa..eeeeeeeeeee wtt.f///aaa///lieeeeeeeeee
Patrick Kolpaczki, Viktor Bengs, and Eyke Hüllermeier. Non-stationary dueling bandits. ar preprint
arXiv:2202.0000 2022. URL haaps:///axiize org/ wwggpdf//202// addf/2200000335.
Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem. In Proceedings of the 28th Conference on Learning Theory, 2015.
URL https: //arxiv/..///////a/eaa wwwpsiii.org/ppoeeo http/ddf/5566 02550000000000.
Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Copeland Dueling Bandit Problem: Regret Lower Bound, Optimal Algorithm, and Computationally Efficient Algorithm. In Proceedings of the 39rd International Conference on Machine Learning, 2016. URL https:// mlr, mww.press/verr press/v48/
komiyama16.ppl
Ramakrishnan Krishnamurthy and Aditya Gopalan. On slowly-varying non-stationary bandits. arXiv preprint: 22916.pdff.
arXiv:2110.120 12916, 2021. URL https: http//awwww........a//a/aaa
Tor Lattimore and Csaba Szepesvári. Bandit Algoritms. Cambridge University Press, 2020. URL https: //tor-lattimore www.amorr..oomddaeeeeeeeeeeeeeee wwt.coomoooo//cccc http///wwwlld/ooooooooooo pdf.
Kuan-Ta Li, Ping-Chur Hsieh, and Yu-Chih Huang. Diminishing exploration: A minimalist approach to piecewise stationary multi-armed bandits, 2024. URL https://arxiv http://xxxxvooooppp/////a/ /2410.
Chuang-Chit Chinh Lin and Chi-Jen Lu. Efficient mechanisms for peer grading and dueling bandits. In Jun Zhu and Ichiro Takeuchi (eds.), Proceedings of The 10th Asian Conference on Machine Learning, volume 95 of Proceedings of Machine Learning Research, pp. 740-755. PMLR, 14-16 Nov 2018. URL https:// proceedings.o mlr.
Tie- Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr., 3(3)222 331, mar 2009. ISSN 1554-0669. doi: 10.1561/1500000066 URL http://didawikinf di dii.nippi. it ht/tt//le/eeeeeeeeeeeeee /feechet.h./// php/ maggssteeleenntommtttc////////// larli__eeeaannnnnoee ttr.ttoorrarr.........
14

---

Published in Transactions on Machine Learning Research (12/2024)
Odalric-Ambrym Maillard. APPRENTISSAGE SÉQUENTIEL Bondits, Statistique et Renforcement. Phd thesis, Université des Sciences et Technologie de Lille - Lille I, October 2011. URL attts://theses. hall /atpac/eeeal-000454000ooooaoaaee
Filip Radlinski, Madhu Kurup, and Thorsten Joachims. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM '08, pp. 43-52 New York, NY, USA, 2008. Association for Computing Machinery. ISBN 595999999. doi: URL https://wwwt http:///wwwwccccocellcceee
cs. cornell. wtt.://oooooelleeeepeeep//////// http://pppppe///ppp...ccoooo///a
1a11__8bpppl
Siddartha Ramamohan, Arun Rajkumar, and Shivani Agarwal. Dueling Bandits : Beyond Con- dorcet Winners to General Tournament. Solutions. In Advances in Neural Information Process- ing Systems 29, 2016. URL https: https////ooceedgssneeeeeee wttppp...e....pppppprrr... cttpaperr_iiles/pppeeeeeeeeeeeee /ttp///////apee///////eeee Paper, pdf.
Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning, pp- 19011-19026. PMLR, 2022. URL https::/proceediingg
mlr httt:p//cccccceee.....eeesss////
pdf.
Aadirupa Saha and Shubham Gupta. Optimal and efficient dynamic regret algorithms for non-stationary dueling bandits. In International Conference on Machine Learning, ICML, 2022, 17-23 July 2022, Ballimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 19027- 19049. PMLR, 2022. pdf.
URL https: /ptp:///e/occco.......o. mlr. pttppp//1/////aaaa2/////a////
Aadirupa Saha, Tomer Koren, and Yishay Mansour. Adversarial ducling bandits. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, TCML, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9235 9244. PMLR,
2021. URL https: http:://pprccedingg mlr. ptttsssmm...p..eseee. pttpp//eee/eeeeeeeeeeee prress/v13/sahaalaa/sssss
Markus Schulze. A new monotonic, clone-independent, reversal symmetric, and consssect-consistent single- winner election method. Social Choice and Welfore, 36 2011. ISSN 01761714, 1432217X. URL
attps://linke http://iiikk.riingrr... springer. wwt.coomaaaaa http://ar..aa//0111////006000--0
Yevgeny Seldin, Csaba Szepesvári, Peter Auer, and Yasin Yadkori. Evaluation and analysis of the performance of the exp3 algorithm in stochastic environments. In Mare Peter Deisenroth, Csaba Szepesvári, and Jan Peters (eds.), Proceedings of the Tenth Furopean Workshop on Reinforcement Learning, volume 24 of Proceedings of Machine Learning Reseorch, pp. 103 116, Edinburgh, Scotland, 30 Jun 01 Jul 2013. PMLR. URL attps://proceett
mlr. https///////oo111a//aaaol///
pdf.
Yanan Sui, Masrour Zoghi, Kalja Hofmann, and Yisong Yue. Advancements in dueling bandits. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventhh Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 19-19, 2018, Stockholm, Sweden, pp. 5502-5510. ijcai.org, 2018. URL https: //www. ijcai. aeeai.org/proceeeeo /rg.proceeting//2188/0000 0070//////777.7../.1/ paf.
Joe Suk and Arpit Agarwal. When can we track significant preference shifts in dueling bandits? Advances in Neural Information Processing Systems (NewrIPS), 2023. URL https: haaps:///rxxiii /arxiv.vvvv../ owg/pot//////o/ooooooooooooo 06595. pdf.
Joe Suk and Samory Kpotufe. Tracking most significant arm switches in bandits. Conference on Learning Theory (COLT), 2022. URL https://proceedingg mlr. wtw.t..rress/v/////// stt///sukk2as2222 pdf.
Tanguy Urvoy, Fabrice Clerol, Raphael Feraud, and Sami Naamane. Generic Exploration and K-armed Voting Bandits. In Proceedings of the 30th International Conference on Machine Learning, 2013. URL
https://prooceeding: mlr. prres//28/urvoyy1
Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10 10, 2016, Barcelona, Spain, pp. 649 657, 2016. URL https: //txxv//a///////aa org/pdf/160// 07101.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Yue Wu, Tao Jin, Hao Lou, Farzad Farnoud, and Quanquan Gu. Borda regret minimization for generalized linear dueling bandits, 2023. URL https://arre www.vvvorgggeeeee http/ddf/300.. www...088.... pdf.
Lirong Xia. Generalized scoring rules: a framework that reconciles borda and condorcet. SIGecom Fach.,, 12 (1):42 48, jun 2013. URL attps://www sigecom. org/ http//chhaaeeeooeeeee//////////e _etmmee_11//iiaa.e11 pdf.
Lirong Xia and Vincent Conitzer. Generalized scoring rules and the frequency of coalitional manipulability. In Proceedings of the 9th ACM Conference on Electronic Commerce, EC '08, pp. 109 118, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581699, URL https:/ /www. CB. cmm.cmm.ee emu htte://eeeaaaied_soreeeese
pdf.
Yisong Yue and Thorsten Joachims. Beat the mean bandit. Proceedings of the 28th International Conference on Machine Learning, 2011. URL https://www.. cs. cornell......o.o http://ppppe/t//ppiiaaaaaaiiooia
yue_joachims 1at...a.pp......
Yisong Yue, Josof Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed ducling bandits problem. Journal of Compuler and System Sciences, 78(5):1538 1556, 2012. ISSN 0022-0000. URL https //www. cs. cornell. weameee..ddu/eppoeee httppp/pp///pppiibaaaaaooo//ppep aaaaaaalaappppa.. pdf. JCSS Special Issue: Cloud Computing 2011.
Julian Zimmert and Yevgeny Seldin. Factored bandits. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https: http://prreeeeig...neeeee neurips. httppps.ccppppptteeeeeeeeeeeeeee /tt.e///paarrr/2////ii/ 22GG1110cBB1T300000CCCCC01--- paf.
Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten de Rijke. Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem. In Proceedings of the 31st International Conference on Machine Learning, 2014. URL https: ptttps://axxi..oo/ wwwps.org/pp......e.. http//////..0/.. 3393.
Masrour Zoghi, Zohar Karnin, Shimon Whiteson, and Maarten de Rijke. Copeland Ducling Bandits In Advances in Neural Information Processing Systems 28, 2015a. URL https://arxiv.orr/pttt http://df/1155000031111
Masrour Zoghi, Shimon Whiteson, and Maarten de Rijke. MergeRUCB: A method for largo-scale online ranker evaluation. In Proceedings of the 8th ACM International Conference Web Search and Data Mining, 2015b. URL https: h/ww://wwww//..c cs. ox ac uk/people//hhmmn. http:///eee....oo.....eeee////pe htttesso//pbbs//ghhiwsomm pdf.
16

---

Published in Transactions on Machine Learning Research (12/2024)
A
Setting up the Weighted Borda Problem
Recall from Section 6 that the weighted Borda score (WBS) be(a, w) is defined as
ba w) = Σ e'E(K]
Pi(a, w
with respect to weight w = (..",,.., xx) E A K. Our goal is to minimize the analogous notion of dynamic regret with respect to the quantities by (a, w) in two situations: (1) the setting of known weights where the vector w is known to the learner and (2) the setting of unknown and changing weights where w, changes over time I and are unknown to the learner. These two scenarios will respectively capture the Borda and Condorcet regret problems. To avoid confusion, throughout this appendix we'll sometimes refer to the usual Borda score ba,a, Uni{{[K]}) as the uniform Borda score (resp. regret, winner).
Throughout this appendix, we'll rewrite the definitions, theorems, and prools of the body in terms of this general framework (with arbitrary weights we) in order to unify presentation across the Borda and Condorcet settings. We first summarize the koy results.
Note that our investigation into characterizing the minimax regret rates for this problem go beyond what is needed for showing the main results presented in the body of the paper.
A.1 Summary of Results in Appendix for Weighted Borda Problem
Environment
Model Assumption
Reference Weights
Regret Lower Bound
Upper Bound
Fixed Winner
None
Known W. w
K K77$7273
K K7337
Unknown
12(T)
GIC (Condition 1)
Unknown W. W
K2
Non-Stationary
None
Known {weht
K2// 1/3
T2/3
K1/3 11/3
T2233
Known
Known
GIC (Condition 1)
Unknown, aligned [w.]c
K 2/3
L Unknown
1/3
175
2/3
Unknown
2/3
Table 2: Summary of Results
(a) In Appendix B.1, we first establish a minimax regret rate of 22/31//3 for a known reference weight under the stationary setting. Interestingly, the minimax regret the same as that of the usual Borda regret, thus showing the generic known weight problem no harder than the Borda problem.
(b) Then, in Appendix B.2, we establish the minimax regret rate of K1/3/1/3 Known T2/3 in the non-stationary dueling bandit problem where the underlying preference model and (known) reference weights are allowed to change, and LKnown counts the significant changes in weighted winner (Definition 1).
1/3
(c) We next consider the unknown weight setting and show in Appendix C.1 that it's impossible for a single algorithm to simultaneously obtain sublinear regret for both the uniform and Condorcet winner reference weights, thus ruling out getting both T2/3 Borda regret and vT Condorcet winner regret.
(d) Then, in Appendix C.2, we show via lower bound that, even under GIC, a higher dependence on of 22/3 is unavoidable. This in comparison the 1 dependence in the known weight setting).
This implies lower dynamic regret lower bound of K2/3 1/3 Lunknown T2/3 terms Lunknown sigréfficant winner switches w.r.t. unknown weights (Definition 2), which can be found in Appendix C.3.
(e) Finally, we establish matching upper bounds on dynamic regret for both the known and unknown weight settings in, respectively, Appendices H and I. As a reminder, our algorithm is adaptive in that it does not require any knowledge of the non-stationarity. As result, we show Theorem 2 and Theorem 5.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A.2 Generalized Notions Related to WBS
Weighted Borda Scores and Regret. For ease of presentation, the remainder of the appendix, we reparametrize the WBS ba w) be in terms of the gaps in preferences d'(a, = Pi(a, - 2. Or,
ba w) = E óe a') w
a'E(K]
Note the resulting regret notions and rates will not change under this new formulation. Now, define the WBS winner as aj (w) = argmaxne(Ks (a, w) and the WBS dynamic regret as
Regret (ondd((()))))) (We) )
(2 (a)(()), w.) 6e w.) belic: we))
2
In the case of a known weight We = w, we'll simplify the notation as Regret(w).
Let of (a', a) = ba(a', w) ba w) be the gap in WBS between arms a and a' at time L. Let (a( = [ 87 a) be the absolute gap in WBS of arm a.
Non-Stationarity Mcasures. We now introduce genoralizations of significant Borda winner switches (Definition 1) for arbitrary weights.
Definition 1 (Significant Winner Switches w.r.t. Known Weightings (SKW)). Fir weight w. Define an arm a as having significant weighted Borda regret over [81, 82] if
(a) 2 K2/3 (s2 - 81 21/2/3.
(8)
Define significant winner switches w.r.t. the known weight w (abbreviated as SKW) as follows: the 0-th sig. shift is defined as To = and the (i+ 1)-th sig. shift T441 is recursively defined as the smallest > T such that for each arm a E [K], S[] S [Tr, t] such that arm a has significunt weighted Borda regret over s2]. We refer to the interval of rounds [Ta, T(41) as an SKW phase. Let LKnown be the number of SKW W phases elapsed in T rounds..
Remark 3. Desinition and subsequent resulls (Theorem 7 and Theorem 12) for a known weight w can be trivially generalized to the setting where there's 's a fized and known sequence of weights {we)?= For simplicity of presentation, we focus on the fixed weight =.ww, which preserves the essence of the theory.
We introduce a similar generalization of Definition 4 for the weighted Borda problem with unknown weights. In particular, we define a notion of significant shifts which tracks when an SKW occurs for any weight w. Definition 2 (Significant Winner Switches w.r.t. Unknown Weightings (SUW)). Define an as having significant worst-case weighted Borda regret over [s1, if
WEAR Σ (a) K2/3 (82 81 2//3
(9)
We then define weighted significant winner switches Po, Pl.... in an analogous manner to Definition 1. Let LUnknown be the number of weighted SUW phases.
Remark 1. Under GIC, Lunknown is less than the number of changes in the winner at (Condition 1).
Although SUW are a stronger notion of shift than SKW (i.e., Tunkknon 2 [.known); the SUW can be learned in a more general setting where one aims to minimize dynamic regret w.r.t. any sequence changing weights [w.)T-i which are aligned with the SUW, defined as follows.
Definition 3 (Aligned Sequence of Weights). We say a sequence [ of weights is aligned with SUW if, for each SUW phase [Pt. Ps+1), the weight w, does not change for rounds t lying in (Pt.P:))).
In particular, mentioned in Section 6, setting w, the Condorcet winner weight w(aa) recovers the Condorcet regret problem. Quite importantly, in this setting, the weights w, are unknown and so their changepoints are also unknown since the SUW are unknown.
while Te, LKaown depend the fixed weight w, we'll drop the dependence as needed for sake of presentation
18

---

Published in Transactions on Machine Learning Research (12/2024)
B Regret Lower Bounds for WBS with Known Weights
B.1 Stationary Regret Lower Bound
We first show that, the stationary sctting, the problem of minimizing weighted Borda regret w.r.t. any fixed and known weight is hard as the uniform Borda problem (i.e., K1/3. T2/3 minimax regret). Theorem 6 (Lower Bound on Regret in Stochastic Setting). For any algorithm and any reference weight w, there ezists preference matric P, with K 2 3, such that the expected regret w.r.t. We is:
E E[Regret (w)] 2 92K 1/3 ....... T2/3).
Proof. We construct an environment similar to the lower bound construction for the uniform Borda score (Saha et al., 2021, Lemmn 14). As in said result, it will in fact suffice to construct an environment forcing a regret lower bound (w...t. weight w) of order (minn E, K/E²]) for E (0, 0.05). Taking & := (K (T) 1/3 will then give the desired result, since (K/T)a/3 < 0.05 for large enough value of T/K. For T/K smaller than 2/3 K = T (22/3K11 K2/3 to conclude.
constant, may take small enough that / (
Suppose K is even; if K is odd, we may reduce to a dueling bandit problem with K - arms by setting the gaps 5(a, d') to zero for some fixed arm a E [K] and any other arm a' [K]. First observe that for any reference weight w, there must exist a set of arms I of size at most K/2 such that its mass under w is {K/2 + 1...., K} has mass at least
w(I) 2 1/2. This follows since either the set of arms {1,..., K/2) 1/2 under the distribution w(-). We then partition the K arms into good set G := [K] and bad set I Without loss of generality, suppose g = (1....,K/) K/2}. We'll consider K/2 + 1 different environments Ex//2 where in & for a E [K/2], arm a will maximize the weighted Borda score b(: ((:w).
Specifically, we set the preference matrix P = P, follows:
Environment Eg: Let the preference matrix P have entries
0.5 a, a, E or ar e
P(a,a')
0.9 aEG,''EB
In this environment, the weighted Borda score is
wBB) aEg -0.4 . w()) a
b(a, w)
where w(-) denotes the distribution on A [K] induced by weight w. We remind the reader here that the WBS b(a, w) are defined in terms of the preference gaps 8e(a', a) and hence can take negative values.
The alternative environments E.....,EjJ2 will be small perturbations of & where an arm E G will have the highest weighted Borda score by a margin of e.
Environment E, for E [K/2]: Let the preference matrix P be identical to that of enviromment E except in the entries P(a, 0.9 + for a' B. Thus, in this environment the weighted Borda scores are identical to that of Eg except for b(a, w) (0.4 +e) w(B), meaning arm is the winner w.r.t. weight
Let Ec, (t) denote the expectation and probability measure of the induced distributions on observations and decisions under environment Eg. Now, the expected regret on environment &g is lower bounded by
Σ Ee, (1()))))))) # a))]= ( T
T
TTEE,, Ee, E (a. (0.00)))
Eg, Regret w)] N
Letting U be a uniform prior over {1..... K/2}, we have the expected regret over a random environment drawn from U lower bounded by:
T
Ex-u [Regret(w]]
26. 6. T- TTEE EnEu
EE,
Σ
(((r,)t)) (a, (a.)))
(10)
19

---

Published in Transactions on Machine Learning Research (12/2024)
Let Nr(a) = E H((.)))) = (a, a)} be the (random) arm-pull count of arm a. Then, since Nr(a) - T for all values of a, by Pinsker's inequality (see Saha & Gaillard, 2022, proof of Lemma C.1), we have
(L, 2
Es, [NT((a) - [Nr(Nr(a)) & <V
where Pa is the induced distribution ion on the history of observations and decisions under enviromment Egg Taking a further expectation over a E U and using Jensen's inequality, we obtain:
Ea-2 Re, [Nr(a)]] S
1
Σ
Eco [N) +T
2 K
Σ
KI,E,,
(11)
K/2
OE K/2)
(/2)...//))
We now aim to bound this last KL term Letting H, be the history of randomness, observations, and decisions till round t: He == {a} U {(tar Je, (a)), for 1 and Ho == {a}. Let Ps denote the marginal distribution over the round data (b,. (t()).)) under the realization of environment & By the chain rule for KT. we have
K
KL(Po, Pa)
Σ KLPP' H4-1, P1 H4-1) 1=1
Σ
Σ
Pe, (fi.))) {a, i}) (Le(Be)),.)) Ber(0.9 + =)).
t=1 k=2++
(12)
Next, observe the following bound on the KL. between Ber(0.9) and Ber(0.9 + e):
0.9
0.1
KL(Ber(0.9), (0. + +)) s 0.9 log 0.9 +
+ 0.1 log
0.1 -
By elementary calculations, the above is less than 10- for any E 0..0.
Now, suppose the algorithm incurs regret at most T on all enviromments Eas lest we be done. Then, the total number of times bad arm in B is played must be at most e. T/0.44 or for all f0, 1.... K/2 1}:
T Σ
e.T
Ex
0}
s
0.4
Plugging the above two steps into (12) yields
Σ
KL(Po. <
20
Σ
Σ
K
Eeo
Σ
T
M(in) = i3)
5
50 3..
K
K
K
K/2)
K=K/2+1
Then, plugging the above into (11) yiclds
50g3
Be (AT((a)))))) s
Σ
Eco
[T))+ + +TV
T.
K/2 aE K/()...K/2)
K
Thus, plugging the above steps into (10) gives
T
50
EaEuEE [Regret(w)) 2 T
+
+TV
T
K/2
K
Now, suppose 50 < 1/50. Then, the above RHS is lower bounded by T/2500 for K 3.
It remains to handle the case of S0 > 1/50 = T 2 2500 Suppose, for contradiction, that for all such K
K
K
T we have regret at most 2500"27 Then, the regret over just the first To = 2500 rounds must also be at most 25002-2
K
K To/2000. However, this contradicts the previous case's lower bound for T = To. E
20

---

Published in Transactions on Machine Learning Research (12/2024)
B.2 Dynamic Regret Lower Bound
Now, for the known weight setting, the k1/3T2/33 lower bound of Theorem 6 can naturally be extended to dynamic regret lower bound of order L Known T2/3 K1/3 over LKnown SKW phases. The proof techniques are routine, similar to arguments already used for showing dynamic regret lower bounds for Condorcet regret (Saha & Gupta, 2022, Section 5), and will not be done in detail here to avoid redundancy.
Theorem 7 (Fixed Weight Dynamic Regret Lower Bound in Terms of SKW). For L. E (T), let P(L.) be the class of environments with SKW LKuowul (w) I L. For any algorithm, we have that the mánimaz dynamic regrel w.r.l. known weighl w over class P(L) is lower bounded by
Ee Re[Regret(w)] 23 SEP(L)
sup
Proof. (Sketch) We can take the lower bound construction of Theorem over a horizon of length T/L to force a regret of (T/S(T/ Repeating this construction with a new randomly chosen winner arm every T/L rounds forces a total regret K2/3 22/32//33 while satisfying IKnown(W) S
Theorem 8 (Fixed Weight Dynamic Regret Lower Bound in Terms Total Variation). For V E [0,7]nk let P(V) be the class of environments with total variation VT most V. For any algorithm, we have the minimar dynamic regret w.r.t. known weight w is lower bounded by
sup Eg (egRegee(w))) 2 D2V'/1///// K 1/4 K k1/33213) SEP(V)
Proof. (Sketch) The argument analogous to the proof of Theorem 5.2 in Saha & Gupta (2022). First, assume T1/4, 13/4, K-1/4 > 1. Then, using the previous lower bound construction forcing regret of order the specialization L x /1/4, v3/4, K-1/4 which ensures the total variation Vi
//3 2233, K1/3, at most.
L- (K L/T) 14/3 K1/3 T <V
and forces a regret lower bound of order
1/ T2/3, K2/3 73/4, K2/4 V1/4
If /44 v3/4. K-1/4 < 1, then 1 T3/4 1/44 < 1133, 2233 which means it suffices to establish a regret lower bound order K1/3, T2/3 which is already evident from Theorem
Remark 2. Note the optimal dependence on VT, T, and K in Theorem differ from the minimax Condorcet dynamic regret rate of /V/3//22/331/3 k//3, This arises from the different stationary minimar regret rates 2T2/3 versus T2/2) of Condorcet versus Borda regret.
C Regret Lower Bounds for WBS with Unknown Weights
C.1 Hardness of Learning all Weighted Winners
We next turn our attention to minimizing stationary regret w.r.t an unknown, but fixed, weight w, which the following theorem asserts hard.
Theorem 9 (Lower Bound on Regret with Unknown Weight). There exists a stochastic problem e 00, 1 Kxk such that for any algorithm, there is reference weight A such that:
E(Regret > T/45.
Proof. Consider the following stationary preference matrix:
1/2
3/5
3/5)
P
2/5
1/2
1
2/5
0
1/2
21

---

Published in Transactions on Machine Learning Research (12/2024)
In this environment arm is the Condoreet winner while arm 2 is the Borda winner w.r.t. the uniform reference weight. For the Condorcet weight w(1), both arms 2 and have gap of 1/10. On the other hand, the Condorcet winner has a gap in Borda scores of 1/15 to arm 2. However, any algorithm must play one of arms 1, 2, or 3 at least. T/3 times in expectation. No matter which arm is played T/3 times, there a weight for which its Weighted Borda regret is at least T/45.
Remark 3. fact, the lower bound of Theorem holds even for an algorithm knowing the preference matris P. The task of minimizing weighted Borda regret for on unknown reference weight can also be cast as partial monitoring problem with fired feedback P but unknown matrir of losses decided by the adversary' 's weight. It's straightforward to verify that this game is not globally observable, meaning the minimor expected regrel is (QT) by the classification of finile partial monitoring games (Barlók el al., 2014).
C.2 Stationary Regret Lower Bound under GIC
Under GIC (Condition 1), we next focus on deriving regret lower bounds stochastic envirouments where there is a fixed winner arm a" = az across all rounds. Even under GTC, the best rate we can achieve for all weights is 72/3, In particular, long as an algorithm attains optimal uniform Borda regret it cannot achieve VT Condorcet regret.
Theorem 10 (Lower Bound on Adaptive Regret with GIC). Fir any algorithm salisfying
[[eegre(UUn((((]]]] S -
for all K =3 3 armed dueling bandit instances. Then, there ezists a stochastic problem E [o, 1]3x3 satisfying Condition such that for the specialization w Wa and sufficiently large T, we have:
([Regret(w). /0T2211/C))
Proof. Consider the preference matrix for
40 11/3/3
1/2
1/2
0.9 +
1/2
1/2
0.9
p'
1/2
1/2
0.9
,P²=
1/2
1/2
0.9
+
0.1
-
0.1
1/2
0.1
0.1
-
1/2
Set 7' large enough so that e < 0.05.
We first sketch out the intuition behind the proof. In P1, arm both the Condorcet winner and the Borda winner, while in p2 arm 2 is. The argument will go as follows: first, the algorithm must identify the winner arm and whether we're in pd or pa since otherwise it pays more than C' T2/3 uniform Borda regret. The only way to identify the winner arm, however, is to play arm 3 at least 2(T2/3) times. This forces 2/T2/33 Condorcet winner regret.
Specifically, for a E [1,2) let a = (1,2))(a) denote the other potential winner Then, have
T
3T s 4
Ep-
C T2/3 = Ep
n (ie.te) # 0)
Then, using a Pinsker inequality argument similar to the proof of Theorem 6,
Ep- [egget w.)) 2 8 Ep-
Σ 1[13, a (ir, # #0]
T Σ
2 8
T
Epa
1 (ie = je a0)
1=1
T Σ 1fi = J. =0 - TV
KL(Pa-,PM Pa)
2
T -
Bpa-
2
22

---

Published in Transactions on Machine Learning Research (12/2024)
where once again Pe. Pa- denote the corresponding induced distributions on all random variables, We next bound the above KT. divergence by chain rule and our earlier bound on (T.(Ber(O.9), Ber(0.9 + e)):
KL(Pa-,Pal 10 Ep,
(a.3)) 1(64.)) = fa= .3)}
Next, note that arm 3 has a uniform Borda gap of at least 0.4 2/3 in either p or P This means we must have
T E = 31) + = far ,31 v
C. E.
Ep
0.4 0/4-2/3
0.4-8/3 8/33
Thus, our KL bound from earlier becomes 26 8
Plugging this into our earlier inequality, we have regret lower bound of
<( (T-
(T-(3
3T
1 33
75
(T))
+ T
By similar argument to the proof of Theorem 6, the above is order Q(min 1/1/(²)) by analyzing the two cases 8² T (75/8) > 8/75 and e²_ T (75/8) s 8/75.
Now, plugging in our value of E, this last rate is of order T2/3 min
As consequence, Theorem 10 prohibits using Condorcet regret minimizing algorithms in this setting, and hence new algorithmic techniques required for efficiently learning winners under GIC.
We next investigate the dependence on K in the minimax regret rate. We show that, to achiove optimal regret for all (unknown) weights adaptively, we need to suffer a higher dependence on of K2/3, compared to the known and fixed weight setting with dependence K1/3 (see Theorem 6).
Theorem 11 (Lower Bound Adaptive Regret under GIC). Fir any algorithm. Then, for K > 3, there erists dueling bandit problem P E [0, [| satisfying Condition such that there is fixed weight E AK with:
IF [RRegee(w)] 2 ( (223m//////////// ,T)).
Proof. We first sketch the argument. We'll consider a variant of the environments Eg, E Ek/2 introducod in the proof of Theorem 6. Recall there is a good set = (1...., K/2) and bad set B == {K/2+ 1, .,..., K} of arms. In every environment, the good arms (resp, bad arms) have gap of zero when compared to each other. In Eo, each good arm has a gap of 0.4 to each bad arm. Now, we introduce further refinement of this construction and define the environment E for a E G and a' E B as identical to the environment Eo except P(a,a') 0.9 + s. Now, the sample complexity of finding the winner arm a in enviromment E,,a' will be K² whence we'll pay a regret of min(s- K? =²)). Setting e x 22/3T-1/13 1/3 then forces 22/3 T2/3 regret. Let U be uniform prior over pairs of arms (a, a') for E G and a' E B. Then, we have
T
Da Re,,,
T -T
Beg,ar
Σ
((1,,))) (a, ((.
Now, by Pinsker's incquality we have
Eaa Be, Ben, [Nr((a)] < (K/2)2 aEG,n'E6
Σ
Es, (N)) +T
Σ
KL Ea,a))
(K/22)
eEG,a'EB
where recall Ny(a) = Σt (((rs))) = (a, a)). Nexl, we bound this KL in an identical way to the calculations of the proof of Theorem 6:
T - (a,
Ea,ar) S 10 e2 Ee,
Σ
[ll.dd =
23

---

Published in Transactions on Machine Learning Research (12/2024)
Now, suppose the algorithm incurs regret at most E T on environment &p lest we be done. This means it cannot pull arms a' E B more than 7 g/0.4 times in total, or
T 0.4
Eg
Σ BE
Σ
160o (a,d'))
Thus,
100 100-e3
Σ
KL,En, Ea.a.) S
K
K
eE9,a'EB
Plugging this KL bound into our earlier Pinsker bound, obtain a regret lower bound of
(
(T-
4T
100
.
u Ee (Regret(w(d)))))
+TV
K2
K²
By analogous argument to the proof of Theorem the above is lower bounded by Mminn( T, k.-kk==2))
C.3 Dynamic Regret Lower Bounds
By analogous arguments as in the previous section, except now using the stationary lower bound construction of Theorem 11 as base template, we have the dynamic regret is lower bounded by L/3 T2/3 2 for environments SUW count Tukknown S T.
For total variation budget V, claim the regret lower bound is of order V 73/4, / K1/2 + K2/3 72/3 In particular, note that if V = 0(K2/3T-1/3), then /// = O(K2/372/3) to use the stationary regret bound of Theorem 11. Otherwise, we have V = - and L K-llp/////////// = (I(1) so that regret lower bound of order 2/332/3322/3 2(V22/4r3/// holds. At the same time L (K2/T) < V so that the constructed environment has total variation at most V.
that it sulfices
D Formal Statement of Regret Upper Bounds
D.1 Known Weights.
We first state formally the upper bound dynamic regret w.r.t. known weight w in terms of the SKW (Definition 1). In particular, we show matching upper bounds, to log terms of the lower bounds of Appendix B.2. As a reminder, taking w to be the uniform weight, we recover the Borda dynamic regret and so the below results generalize Theorem 2 and Corollary
Theorem 12. Algorithm 2 with the fixed weight specification (see Definition 4) w.r.t. fixed weight w satisfies:
TiKooen (w)
E E[Regre((w)) s
Σ
K1/3, (Ter TT(3))2/3
Corollary 13. By Jensen's inequality, the regret bound of Theorem 12 is further upper bounded by K1/372/3 /1/ Known- Furthermore, relating SKW to total variation, a regret bound of order V T3/4 K1/4 + K1/S /2/3 also holds in terms of lotal variation quantity VT (see Appendia H.5 for proof).
D.2 Unknown Weights.
We next state the analogous result for the unknown weight setting. Theorem 14. Algorithm 2 with the unknown weight specification (see Definition 4) satisfies for all sequences of aligned weights (wehe (see Definition 3):
[[Regret (w 50(min(K' 2/3 Unkkowwn K 1/2 1 1/4////44 + 22/3/2/31).
24

---

Published in Transactions on Machine Learning Research (12/2024)
Remark 4. Interestingly, Theorem 14 holds even without GIC (Condition 1). However, without GIC, Definition 2 may not be meaningful notion of non-stationarity as (8) may be triggered even in stalionary environments.
As warmup to proving the above dynamic regret upper bounds, we'll 'l first show regret upper bound in casier fized winner environments where there is a fixed winner arm either (1) with respect to known weight or (2) with respect to all unknown weights under Condition 1.
These "simpler" analyses will reappear and serve a core template for the complicated dynamic regret analyses of Appendix H. (known weight) and Appendix 1 (unknown weight).
Even before doing this, however, we'll establish some preliminary notation and concentration bounds on estimation error, which will be used all our regret analyses.
E Regret Analysis Preliminaries and Estimation Bounds
Throughout the regret upper bound analyses 61s 62, will denote positive constants not depending on T or any distributional parameters. We first. recall version of Freedman's inequality, which has become standard in adaptive non-stationary bandit analyses (Suk & Agarwal, 2023; Buening & Saha, 2023; Suk & Kpotufe, 2022).
Lemma 15 (Theorem 1 of Beygelzimer et al. (2011)). Let Xi,.. X,,..., X,, E IR be a martingale difference sequence with respect to some filtration Fi....... Assume for all t that X, S R o.s. and that Σ [[ s Vn 0.8. for some constant Vn only depending on n. Then for any 8 E (0,1) and E [0, 1/R,, with probability at least 5, we have:
(x (VV, log(1/0) +
(olo)))))+llg(18))) (llg((16)).
We next apply Lemma 15 to bound the estimation error of our estimates ba w) (see (3) in Subsection 4.1) Proposition 16. Let & be the event that for all rounds $1 82 and all arms E As for all t [81, 82), and weight w E W:
K (1E
w² Is
Σ blla, w)
Σ E () 1=81
s 10 1) log(T[WI)
Σ
K
max Th sE(e1.02]
(a()))
(13)
where F = (FeYT=1 is the canonical filtration generated by observations and randomness of elapsed rounds. Then, ε occurs with probability at least 1/T2.
Proof. First, note for any arm a E [K] and weight w, the random variable & (a, w) - Eid, (a, [1 is martingale difference bounded above by maxiglsi nh !! Then, in light of Lemma 15, suffices compute the variance of (a()) We have for arm a active at round t (i.c., a E At):
lffe = = = a'} (O,(a, a²) - 1/2) Waa
E 0 (a) 14-1)
(a(a) (a()
a'E[K]
[( jt) = (a.
E
w
(a) (a')
a'ElK]
2 (a) (a))))
2
< K
(a()
25

---

Published in Transactions on Machine Learning Research (12/2024)
where the last inequality follows from the fact that (a(a) 2 1/K by virtue of E A. Then, the result follows from Lemma 15 and taking union bounds over arms weight. w E W, and rounds 81,82-
We next establish separate applications of this concentration bound for the setting of known weights and unknown weights. For these settings, respectively set W [w] for known weight and W {War a [K]} for unknown weights.
Known Reference Weight. First, in the case of known and fixed reference weight W {W), we can observe:
w
qo(a) 2 To Wa
K
Wa
n
K 175
(14)
(a))
Unknown Reference Weight. For an unknown reference weight, and when W = {Wa, E [K]} we have the algorithm plays, w.p. at round t, from Un]][[K]} so that:
K
(( 2 m/KK
K
(15)
(a)))
Now, combining (14) and (15) with (13) yields the following parameter specifications for use in the known vs. unknown weight setting.
Definition 4 (Parameter Specifications). We define two parameter specifications for use in Algorithom (for known us. unknown weights). We define the known weight specification w.r.l. weight w E AK via
W [w], "t min(K 1/3 1/13,,1,
F([s1, s2)) = K1/3, (82 v
K
Σ
17
K
77
(16)
sE sE|si ,82]
The unknown weight specification is defined via W {Wara a E [K] (i.e., the point-mass weights), Y = mm/n t- 1/3 1}, and
F((81, = K2/3, (82 81) 2/3
K2
1 + K max 1/ sE($1.82
(17)
We next establish an elementary helper lemma which asserts that the intervals of rounds [$1, #2] over which we evict an arm (4) must be least Q(K)) rounds in length for the fixod weight specification and least 2(K2) rounds length for the unknown weight specification. This will serve useful in further simplifying the concentration bound of (13) throughout the regret analysis, needed.
Lemma 17. On event E,, letting F([11, s2]) be as in (16) with the fized weight specification, have that the eviction criterion (4) holding over interval ($1, s2] implics
$2 51 2 K/8.
Letting F[[51, 82]) be as in (17) with the unknown weight specification, we have
51 k2/S
Proof. By concentration (Proposition 16) and since the weighted Borda gaps 6t (a', a) are bounded above by 1, eviction of an arm a over [s1, 82] under the fixed weight specification implies
2. (s2 - $1)
82 31 +1
(a) 2 1/3 (82 - (62-31)37 2/3
$=81
The above implies 82 s1 2 K/8. For the unknown weight specification, repeat the above argument with 1//3 replaced by 2/3
26

---

Published in Transactions on Machine Learning Research (12/2024)
F Stationary Regret Upper Bound for Known Weight
Theorem 18 (Regret Upper Bound for Known Weight). Algoritio 1 with the fixed weight specification (see Definition 4) w.r.t. fixred weight w, satisfies in any environment with a fixed winner a" = aj (w):
E[Regret(w)] & Ö(K1/3 T2/3).
Proof. We first note the regret bound is vacuous for < K; so, assume T K.
WT.OG suppose arms are evicted in the order 1, .... K at respective times 4 S 1/ S... S lg (if arm a is not evicted, let tu = T-1. 1. Then, we can first decompose the regret depending on whether we play an active arm or explore other arms with probability Y/:
T
T
E
2b. w) belfe, w) - bel w)
+
Σ
Σ a.a'E.d
(2-bi(a", ,.bb((a,,,)) w) be(a, w) - ba(a', w)) (:)a)
1=1
I=
Using the fixed specification for (K/t) from Definition 4, we have first term on the RHS above (our exploration cost) is of order K1/3,, T2/3
Now, in light of our concentration bound Proposition 16, it suffices to bound the regret the hiob-probability good event &, since the total regret is negligible outside of this event. For the second expectation, using our fixed-weight concentration bound (14) we get that the regret of playing arm as a candidate is at most (on the good event &
(V
la be(a", w) - be(aa, () I 10 (c - 11) log(()
h--1 K. max -1
1 (
K Σ t=1
-1
1/1
(18)
AA
1.
Note that |Ad 2 K since we assumed WLOG that arm the a-th to be evicted. Additionally, plugging in the specialization for Nhe according to the fixed weight specification of Definition 4, we have that
4-1
t-
1/3
K
Σ Th Sks
+
K2 tc/K3113 (la (la-1)4/3,
K
K max [E[1.6-1]
K2/3 (ta - 1)2/3,
Next, by Lemma 17, we must have la - 2 K/8 so that K2/3 (la - 1)1/3 13 of order K1/3, 12/3. By the same reasoning, K2 is of order K2/3 (ta 1)4/3_
Then, plugging the above two displays into (18) and summing the inequality over arms gives:
t bi(a", w) ba w)
Co log(T) (T) KK/3 12/3
2 E 11
A
K K+1-0
Upper bounding each tu by T, we obtain a regret bound of order (K) log(T)) K K1/3/2/3
G Regret Upper Bound under Fixed Winner for Unknown Weight
Theorem 19 (Adaptive Regret Upper Bound for Unknown Weights). Suppose Condition 1 holds and there is a winner orm a" ot fixed across time. Then, Algoritiu with the unknown weight specification (see Definition 4) satisfies for all weights A
E[Regret(ww) w)] < Ō(K2/3 22/3).
27

---

Published in Transactions on Machine Learning Research (12/2024)
Proof. We first show that it suffices to bound regret w.r.t. the point-mass weights Wa E W for a E [K]. This is true since the regret is linear in the reference weight w:
Eb((a" w) be(ee, w) Ba'nw E Es((at.a') d((
S max Σ be(a", Wa) be( w.). K
Now, in light of our concentration bound (14) from Proposition 16, it suffices to bound the regret on the high-probability good event & since the total regret is negligible outside of this event.
We then follow the recipe of the proof of Theorem 19 (see Appendix F) except now using the concentration bound of Proposition 16 with (15) and the unknown weight specification = K2//////2 First, note that it's clear the exploration cost Σ/= Ve for the unknown weight specification is order K2/3, T2/33
Now, suppose WT.OG that the arms are evicted in the order 1,2,..., Proposition 16 gives us a bound with respect to any point-mass weight in W:
, K at times 4 S l2 S ... S lx- Then,
(1
] max be(a", w(a')) - be(a, w(a')) < 10- (e- 1e-1)e1)) log(T)
K2 Σ
K
max "16 ) (19)
['E(K]
A
nE(1,tu -1] 1]
Crucially, note that the fixed winner arm a" is never evicted by Condition 1. Now, have
4-1
K2 4-1 t1/3
K2
Σ T1
S K*
K2/3 K +,3K4/3. (ta 1ta-19 1)4/3
K - max
Th = K1/3 (to 1)133
Now, by Lemma 17, we have since 82 $1 2 K2 /2/8, the K term in the first display above is of order K4/3, (to 1)4/3.
Then, plugging the above two displays into (19) and summing over arms a E [K], we have:
K ta bi(a", w) ba w)
c4 log(T) K2/3 2/3
2E
1{
IA
K+1-c a
As before, upper bounding each la by T, we obtain regret bound of order log(K) log(T)) K2/3 T2/3, 0
H Dynamic Regret Analysis for Known Weights
Proof of Theorem The broad outline of our regret analysis will be similar prior works on adaptive non-stationary (dueling) bandits (Suk & Agarwal, 2023; Buening & Saha, 2023; Suk & Kpotufe, 2022). Our first goal is to show that episodes (t21) align with the SKW phases, in the sense that new episode is triggered only when an SKW has occurred.
Recall from Algorithm 2 that [le, le41) represents the f-th episode. As a notation, we suppose there are T total episodes and, by convention, we let t, := T + if only episodes occurred by round T.
Episodes Align with SKW W Phases Lemma 20. On event E,, for each episode [te, te-1) with te41 S T (L.e. an episode which concludes with a restart), there erists on SKW T E [1e, le41).
Proof. We first note that ba(a, w) is an unbiased estimator for (a, w) in the sense that Blba(a, w) = ba(a, w) if E A.. Then, by concentration (Proposition 16) and our eviction criteria (4), we have that arm a being evicted from A, over the interval (81, 82) using the eviction threshold (16) implies
$2
la w) ba w) co(52 (1)(22-s)2222 2/) K1/3
28

---

Published in Transactions on Machine Learning Research (12/2024)
This means arm incurs significant weighted Borda regret w.r.t. w over [s1, Since new episode is triggered only all arms are evicted from Aglobal, there must exist an SKW each episode [Me, (e-1)- ]
H.1 Decomposing the Regret
Let af denote the last safe arm at round i, or the last arm to incur significant weighted Borda regret w.r.t. w in the unique phase [Ts, T++1) containing round t, per Definition 1. Furthermore, let ag denote the last global arm of episode (41)) the last arm to be evicted from Aglobal in said episode. Then, the per-episode regret:
decompose
-
Te+s
Te+1 -1
Σ
of (i) + 6,7%
= E
ot (at))
+B
Σ
oe (ac, + (ae:
+
Σ
87 (a ae)
(20)
We next handle each of the expectations on the RHS separately. Our goal will be to show that each of the expectations above of order
E
{{E}}
Σ
K1/3 (f4+1 - T/)3/3
+ T
(21)
sE LKnown Akknenn1 30
Admitting this goal, summing the regret over episodes while using Lemma 20 to ensure each SKW phase [Ti, T2+1) only intersects at most two episodes will yield the desired total regret bound. This will follow in a nearly identical manner to Section 5.5 of Suk & Kpotufe (2022).
Now, the three expectations on the RHS of (20) are respectively:
The per-episode dynamic regret of the last safe amm (analyzed in Appendix H.2)
The per-episode regret of the played arms (i the last global arm (analyzed in Appendix H.3)
The per-cpisode regret of the last global arm to the last safe arm (analyzed in Appendix H.4).
H.2 Bounding the Per-Episode Regret of the Safe Arm
By the definition of SKW (Definition 1), the safe arm at is fixed for E [n, Te+1) and has dynamic regret upper bounded by K (s2 $1 2/33 on any subinterval (s1, s2] I [Ti, T++1). In particular, letting [$1, s2] be the intersection [T;, T441) n [te, te+1) (which is necessarily an interval), we have that the dynamic regret of of episode [te, t(41) is at most order (21).
H.3 Bounding Epi-Eoe Regret of Active Arms to Last Global Arm
This will follow similar argument
regret analysis in fixed winner environments (Theorem 18). Recall that the global learning rate 7) is the learning rate Nt-tu set by the base algorithm which is active at round t. Note that γ) is random variable which depends on the scheduling of base algorithms in episode [tz, fe11.
We first observe that the play distribution qe plays an arm according to weight w with probability de and plays chosen from As uniformly at random with probability 1 v/- Let a be random draw from the distribution de be the play distribution at round t, which is measurable w.r.t. F. Then, may rewrite the regret as
E
E.g. [E[1(1 (411 b.(a, w) le, te [t) [0a)
Note that 1(t < tetil and by (a, w) are independent conditional on te and qr. Next, observe that:
Elbi(a, wilte, qr] = mh Baawwbbr(a w)lte. gd + (1 mp) En- (A. [b(aa w) Lt, gr).
29

---

Published in Transactions on Machine Learning Research (12/2024)
Plugging in the results of the above to our regret formula, we obtain:
4-1-1
1++1 -1
16+11
E
Σ
(ae, in) + 6t (c
S
M
+ +
EniUui((AA 1or (ae,
(22)
Bounding the Regret of Extra Exploration. We bound the first expectation on the above RHS. Recall that St-tu denotes the learning rate "Ir set by a base algorithm initiated at round seart Now, we can coarsely bound the sum of the ne's by the sum of all possible ?t- weighted by the probabilities of the scheduling of each base algorithm. So, we have
T++
7++1 Σ
Letare +-
S
M--l
+
Yt batart
BOSSE
t= lstart
Recall in the above that the Bernoulli By.m (see Line 6 of Algorithm 2) decides whether BOSSE(s, m)) is scheduled.
The first sum on the RHS above is order K1/3, (fe+1 4 4x)2/3, The analogous sum in the second expectation the RHS above is order K1/3, m223 Then, it suffices to bound
T
E
Σ
E 1ftstare < te+13 K1/3 m 2/3
m
Conditioning on te, and noting that 1 Be and 1fatart < tet1} are independent conditional on te, we have that by tower rulc:
Σ
E B mn [t]] B[1 < (1-1))11) K1/3 m22
Astari 11
Plugging in E[B m m|t]]
m (1ear -2/2/3 (from Line 12 of Algorithm 2) and taking sums over and s,
the above becomes order log(T") K/33 (te+1 - -t)3/33 This is of the right order with respect to (21) by the sub-additivity of the function x + 2/3
Bounding the Regret of Active Arms. Next, we bound the second expectation on the RHS of (22). This follows similar argunent to Appendix (the proof of Theorem 18). Supposing WLOG that the arms are evicted from Aglobal in the order 1, 2,..., K at respective times t .... S t then we have the second expectation on the RHS of (22) can be written
K 4+111 0 (at, a)
(ar,
E
1fa A+}
(23)
For the first double sum above, we repeat the arguments Appendix F using the episode-specific learning rates (ndlte 14-1111 Crucially, we note that, no matter which base algorithm BOSSE( m) is active round E [te, te1,, 2 K 1/3 (t (t-tt) -1/33 Thus, we have
--
K
Σ T
<
K2
+ CB K2/3 (19 4/3
t=
-1
< 2/3 (t) - 1/3
max 1E
Then, using Lemma 17 and following the arguments of Appendix F:
(ar,a)
1(8)
Σ
S 07 log(T) K1/3 (le41 2/3
At
30

---

Published in Transactions on Machine Learning Research (12/2024)
For the second double sum in (23), our aim is to bound the regret of playing arm on the rounds t E [22, te+1) where a replay is active and, thus, reintroduces arm a to Ar after it has been evicted from Aglobal- For this, we rely on a carcful decompasition of the rounds in [tg. te+1) when a E A based on which replay reintroduces the arm a to the active set. We'll first require some definitions, repeated from the analyses of Suk Agarwal (2023); Suk & Kpotufe (2022).
We first note that the various base algorithms scheduled in the process of running METABOSSE have an sncestor-parent-Cne
structure determined by which instance of BOSSE calls another. Keeping this in mind, we now set. up the following terminology (which all w.r.t. a fixed arm a): Definition 5.
(1) For each scheduled and activated BOSSE(s, m); let the round M(s, m)) be the minimum of two quantilies: (a) the last round in [s, s m] when arm is retained by BOSSE(s, m) and all of its children, and
(b) the last round that BOSSE(s, m) is active and not permanently interrupted. Call the interval [s, (s.M(smm the active interval of BOSSE(s, m).
(1t) Call replay BOSSE(s, m) proper if there is other scheduled replay BOSSE(s', m') such that [s, + m] C (s', + m') where BOSSE(s', ,m') will become active again after round + m. In other words, a proper replay is not scheduled inside the scheduled range of rounds of another replay. Let PROPER(TE: te+1) be the set of proper replays scheduled to start before round te+1-
(i0) Call scheduled replay BOSSE(s, m. subproper if it is non-proper and if each of its ancestor replays (i.e,, previously scheduled replays whose durations have not concluded) BOSSE(s', m²) satisfies M(s', m() <s In other words, a subproper replay either permanently interrupls ils parent or does nol, but is scheduled after its porent (and all its ancestors) stops playing arm a. Let SUBPROPER te+1) be the set of all subproper replays scheduled before round tę+1-
Equipped with this language, we now show some basic claims which essentially reduce analyzing the complicated hierarchy of replays to analyzing the active intervals of replays in PROPER((e, 4441) U SUBPROPER(te, te+1). te+1).
Proposition 21. The active intervals
(s, BOSSE(s, m) E PROPER(te,te)) U USSBPRoPEE(ttt444)
are mutually disjoint.
Proof. Clearly, the classes of replays PROPER(te,te4) and SUBPRoPER(te f(+1) are disjoint. Next, we show the respective active intervals (s, (s, m)] and [s', Ms', m')] of any two BOSSE(s, m), BOSSE(s', mm) e PROPER((e, 1441) SUbPropeEn SUBPROPER((e, 1441) are disjoint. There three cases here:
Proper replay vs. subproper replay: a subproper replay can only be scheduled after the round (ss, m) of the most recent proper replay BOSSE(s, m) (which is necessarily an ancestor). Thus, the active intervals of proper replays and subproper replays are disjoint.
Two distinct proper replays: two such replays can only permanently interrupt each other, and since M (s, m) always occurs before the permanent interruption of BOSSE(s, m), we have the active intervals of two such replays are disjoint.
Two distinct subproper replays: consider two non-proper replays
BOSSE m). BOSSE(s', m!) E SUsPRoPER(te
with s' s. The only way their active intervals intersect is if BOSSE(s, m) is an ancestor of BOSSE(s', m'). Then, if BOSSE(S', m') is subproper, we must have s' > M M(s, m), which means that. |s', . (s', m()) and [s, (s, m)] are disjoint.
31

---

Published in Transactions on Machine Learning Research (12/2024)
Next, we claim that the active intervals [s, (s, m)] for BOSSE(s, m) E PROPER(te: te+1) U SUBPRoPeR((s: (4+1) contain all the rounds where a is played after being evicted from Aglobal- To show this, we first observe that for cach round & when a replay is active, there namely the proper replay scheduled most recently. Nexi, nole that any round > 1 where arm a A, must either belong the active interval [s, (s, m]] of the unique proper replay BOSSE(s, m) associated round i, else satisfies t (s, m) in which case unique subproper replay BOSSE(s', m') E SUBPROPER(te, fe+1) is active at round / and not yet permanently interrupted. Thus, it must be the case that E [s', M m()].
unique proper replay associated to t,
At the same time, every round I E [s, (s, m)) for a proper or subproper BOSSE(s, m) is clearly a round where arm a is once again active, i.c. a E Ae, and no such round accounted for twice by Proposition 21. Thus,
{t (1g,te41 a As)
U
[s, M (s,
Then, we can rewrite the second double sum in (23) as:
K
M(s.m)
0f (ac.a)
Σ
Bemm
[A
BOSSE(s,m)ee ( /ttp://eeeeetStttooetoeeeee (USuoProruttts
Further bounding the sum over / above by its positive part, we can expand the sum over BOSSE(s, m) E PROPER(te, te+1) U SUBPROPER(t, te+1) to be over all scheduled BOSSE(s, m)), or obtain:
K
M(a.m)
a,(a,a)) Aee
Bamm
1fa Ail
(24)
a=1 BOSSE(s.m)
L=
where the sum is over all replays BOSSE(s, m), i.e. E (tc + (t.++,..... +...,,,, te+1 13 and m (2, 12,4.... ".....,,,0000( 2[ex(T)] It
then remains to bound the contributed relative regret of each BOSSE(s, m) in the interval [s V 12. M(s, m)), which will follow similarly to the previous steps. Fix m and suppose + 1 s (s,m) since otherwisc BOSSE(s, m) contributes no regret in (24).
Note that the global learning de for E [8 V E2, m)) must satisfy TH 2 K1/3 m-3 since any child base algorithm BOSSE(s', m') of BOSSE(s, m) will use larger learning rate Y 2 k-s-
Then, following similar reasoning as before, i.e. combining our concentration bound (13) with the eviction criterion (4), we have for fixed arm a:
M(s.m) Σ t=SVE
8 (aE,
Cs colog(T): (K1/3, m/33 A m)
1(
S
IAd
Minre]s.M(smm)) [Acl
Plugging this into (24) and switching the ordering of the outer double sum, obtain (we also overload the notation (s, m, a) to avoid ambiguity which arm we're bounding the regret of):
K
Bam log(T) (K1/33 1/3 2/3 A m)
min (s. ((s.m.a) Ail
BOSSE(s.m)
Following previous arguments, the above innermost sum over a is most log(K) since the k-th arm as in [K] to be evicted by BOSSE(s, m)) satisfies minng(s,M(ssm.ar |A, 2 K K-11k
Now, let R(m) == cs log(K) log(T): (K1/3 m/3 A m) which is the bound we've obtained so far the relative regret for single BOSSE(s, m). Now, plugging R(m) into (24) gives:
K 4+11
8w (aeea)
1(81)
1fa E A:)
$ FE
E
Σ
Ban R(m)
Ael
o=1
BOSSE(a.m)
T
= E
Σ
Σ
E Ba.m 1fs < te+il | tel R(m)
Ls= m
32

---

Published in Transactions on Machine Learning Research (12/2024)
Next, we observe that Bem and 1fs < te+1h are independent conditional on te since 1{s < t&+1} only depends on the scheduling and observations of base algorithms scheduled before round 8. Thus, recalling that (P P(Bs.m= =) = m 1/3, (s te) 2/3 -2/3,
B { < (411 [e] = B [e] [B < (411 [4] [ (s < te+13 tel.
2/3 (s - 44)2/8
Then, plugging the above display into expectation from before and unconditioning, obtain:
fea1-1 [lgg(T)
E
Σ s=f++1
2n/3, (s +t)2/33 R(2")
1
I cy (o) log)(T) Eeg,er [x1/3 (te+1 - tt)
(25)
Note that in the above, we used the fact that the episode length te+1 - to dominates K by Lemma 17 to bound term of order K2/3, (te+1 - t2) 13//3/ by K1/3 (te+1 - t,)2/3.
H.4 Bounding per-Episode Regret of the Last Global Arm to the Last Safe Arm
Now, it remains to bound E Etar- 67 (af, an) For this, we'll use bad segment analysis similar to Suk & Agarwal (2023); Buening & Saha (2023); Suk & Kpotule (2022). Roughly a bad segment [21, will be such that E. 0% (af, ae) (s2 s1)2/3 k On such a bad segment [s1, s2], a perfect replay is roughly defined as a replay whose scheduled duration overlaps most of [$1, #2]. Then, the key fact is that the randomized scheduling of replays (Line 12 of Algorithm 1) ensures that perfect replay is scheduled for some bad segment (thus evicting ag from Aglobal before too many bad segments elapse, giving us a way of bounding Σ>.,, 8th (af, az) with high probability.
We next formally define such notions. In what follows, bad segments will be defined with respect to a fixed arm a and conditional on the episode start time 4G- In particular, this will hold for = as which will ultimately be used to bound 87" (af, ae) across the episode [te, te+1). The following definition only depends on fixed arm and the episode start time te and, conditional these quantities, are deterministic given the environment.
Definition 6. Pir the episode start time te, and let Te41) be any phose intersecting [tz, T). For any arm a, define rounds St,(a), $;,1 (a), 81,((a)... E [te V Ti, T++1) recursively as follows: let s1,0(a) == t V Ta and define (i)((a) as the smallest round in (Si.-1(()) (a),, Tx41) such that arm a salisfies for some fixed Cio > 0:
si.sla)
( a) 2 c10 log(T) K//3, (s.)(a)-s.g-((())) s1)-1(a)))//
(26)
(=)))-1(0)
if such round sa)(a) exists. Otherwise, we let the st)(a) T441 1. We refer to any interval (s)))) as critical segment, and as bad segment (W.r.t. arm a) if (26) above holds.
Remark Arm af is fixed within any critical segment (s) $1. (a)) S [Ta T411) since an SKW does not occur inside [T,, T++1).
The following fact, which may be considered an analogue of Lemma. 17, will serve useful.
Fact. 22. A bad segment (s,)(a), (a)+1(a)) satisfies sa)+1(a) $1 (a) 2 K/8.
Now, note a bad segment [st.] (a), (a)) only contributes order K1/3 to af. At the same time, we claim that a well-timed replay (sce Definition 7 below) rumning from sa)(() to (a)+1(() will in fact be capable of evicting arm using (4). This will allow us to reduce the problem to studying the number and lengths of bad segments which elapse before one is detected by such a replay.
regret of a
We next define perfect replay.
Definition 7. Let (a) == sipla)+ denote the approzimate midpoint of (a), s (a)). Given bad segment (a), (()+)())), define perfect replay w.r.t. [sis(a), s1.+1(()) as call of poosSE(tstrtt.. m) where tatatt E (a), (a)( and m 2 (a)+1(a) (a,((a)
33

---

Published in Transactions on Machine Learning Research (12/2024)
Next, we analyze the behavior of a perfect replay on the bad segment (st,(()), We first invoke an elementary lemma:
Lemma 23. (x + y) - 2//3 2 /2/3122 for real numbers 2 > 0.
Proof. This follows from noting the derivative of the function y + (ar + y)) x2/33 /2/3/2 /2 in the domain
y 2 x is positive since
0
1
x v)
2/3 - 2//3 - ///3/2 /2- =
By
3 y)1/38
3 yyy133"
The above positive since
(2/v + 1)1/3 2 21/3 2 3 (x + y) 3- 3-y'13
>
>
Thus, y (x+ - - 2/3 (2 > 0f
is an increasing function in y. Next, since (2x) - 2//33 22/3/22 /2- x [f0===-1/>00f=>>00000 0, we have that the desired inequality must always be true for 2 > 0. ]
Proposition 24. Suppose the good event & holds (ef. Proposition 16). Let [si,(a), s1)+1(a)) be bad segment. with respect to arm a. Then, if a perfect reploy with respect to [8t5(a), (1)+ is scheduled,, arm a will be evicted from Aglobal by round (1)+1(a).
Proof. We first observe that by Lemma 23 and Definition 6:
$1.211 (a)
(a)
(a)(a)-1
(a) = Σ
of (at.
Σ
(at a) 2 610 log(T) K1/3, (Bis+i(a) - (a)s()))
(a)
(=)
Next, note that any perfect replay BOSSE (start, m) will not evict of since otherwise it incurs significant regret within SKW phase [Tr, Ti41) (sce also the proof of Lemma 20). The same applics for any child base algorithm of a perfect replay.
Next, we argue that arm a must be evicted from Aglobal at some round in [á:,) (t))+1(a)). If & A, for some round & (a), s1.+1 (a)] we are already done. Otherwise, suppose A,, for all t (B.j(a), and so we must have (a a)]FF-1] = 8t (at, a) for all such rounds Now, if a perfect replay is scheduled, then we must have a global learning rate V) 2 K1/3 (s).)+((a) (a) for all t. E (a), (a)) since any child base algorithm of a perfect replay can only set a larger learning rate by Definition
Now, noting [a), (a)) and the second half of the bad segment [s, 8i.+i (a)) have commensurate lengths up to constants, this means, if a perfect replay w.r.t. [st,(a),, s1)+1(a)) is scheduled, by similar calculations to carlier (and using Fact 22) we must have
$4,4+1 (a)
K
Σ
Us
+ K
max sE(a..
ne S 1/3 (( - 81,5 (a))2/33
- (o)
Then, by our eviction criterion (4) and concentration, we have that arm will be evicted over | (a), s(.+1(a) (a) for large enough constant Ciu in Definition 6.
It remains to show that, for any arm a, perfect replay is scheduled w.h.p. before too much regret is incurred on the elapsed bad segments w.r.t. a. In particular, this will hold for the last global arm ag, allowing us to bound the remaining expectation E E 8%" (at ae)].
To show this, we'll define bad round s(a) > te which will roughly be the latest time that arm can be evicted by a perfect replay before there is too much regret. We'll then show that a perfect replay with respect to some bad segment indeed scheduled before the bad round reached.
First, fix an arm a and an episode start time te.
34

---

Published in Transactions on Machine Learning Research (12/2024)
Definition 8. (Bad Round) For fized round tę and arm a, the bad round s(a) > ty is defined as the smallest round which satisfies, for some fixed 61 > 0:
Σ ((s - - 012 log)))(T) (s(a) tz)2/3, (i.5)
(27)
where the above sum is over all pairs of indices (i,j) N x N such that (sts(a), st,+i(a)) is bad segment (see Definition 6) with (a)) < s(aa).
Our goal is then to then to show that arm a is evicted by some perfect replay scheduled within episode (t+1) with high probability before the bad round s(a) occurs.
For each bad segment (s,,(a), s1)+1(a)), recall that (a)(a) is the approximate midpoint between s1 and (1)+1(a) (see Definition 7). Next, let maj" == 2 where E N satisfies:
2 > St.((() > 2n-1
Plainly, mi, is dyadic approximation of the bad segment length. Next, recall that the Bernoulli Bt.m decides whether BOSSE(I, m) is scheduled at round (see Line 6 of Algorithm 2). If for some t E [sig(a), (a)] Br, = 1, i.e. a perfect replay is scheduled, then a will be evicted from Aglobal by round (1)+1(a) (Proposition 24). We will show this happens with high probability via concentration on the sum
(L.p(o)
x(a
Σ
Σ
Beamus
(a)))) (=))(())
Note that the random variable X(a, ()) only depends on the replay scheduling probabilities ( given fixed arm a and episode start time te, since the bad round s(a) is also fixed given these quantities. This means that (a,, te) is independent sum of Bernoulli random variables Br. conditional on ty. Then, multiplicative Chernoff bound over the randomness of (a,, te), conditional on tt yields
P px((att))
E(X(a,t))
ltp] tel
[[X (,)
[te
S exp
2
8
The above RHS error probability is bounded above above by 1/T by observing:
([X te) tel 2 Σ Σ
(a)(a)
1
(s.g41(a) - (a)s()))2/3
C12 2
log (Tg(T),
1/3 (a) mij
(1 - 12) 22/3
2 (i.J) (s(a) t2)2/3
(t.))
for C12 > 0 large enough, where the last incquality follows from (27) in the definition of the bad round s(a) (Definition 8). Taking a further union bound over the choice of arm a [K] gives us that X(a, (2) > 1 for all choices of arm (define this as the good event Et())) with probability at least 1 K/T3. Thus, under event. E2(tz), all arms a will be evicted before round s(a) with high probability.
Recall on the event & the concentration bounds of Proposition 16 hold. Then, on E, n Ele,, letting a = az in the preceding arguments we must have t(+1 - S s(ae) Thus, by the definition of the bad round s(ar) (Definition 8), we must. have:
Σ
(sij+i si (a=))) & 012 log(T) (le+1 1e)2/3.
(28)
Thus, by (26) in the definition of bad segments (Definition 6), over the bad segments (si,(az) s1.f+((a))) which elapse before the end of the episode 44+1 - 1, the regret as to af is most order log2(T) (T)
1/3 (le+1 2/3
Over each non-bad critical segment (ae), sis+1 (ae)), the regret of playing arm os to of is at most log(T)- 1/3 (T+41 2/3 and there is at most non-bad critical segment per phase [T, 7+1) (follows from Definition 6).
35

---

Published in Transactions on Machine Learning Research (12/2024)
So, we conclude that on event & (2t(t)):
4+111
of (at, ar) S cia log (T)
Σ
K (T44 T:)2/3.
xEPHAsIS(tetee1)
Taking expectation, we have by conditioning first on tt and then on event &₁ (E2(t)):
T---11 -1
4-1-1
oy (ai. ae)
E
Σ
S Bie
E 11 ((i)n
Σ
or (af, ar) 4 1
T E., E [1fE U & [4]
t=
2K
c13 (T)
E 1[8 (E2(1e)
Σ
K1/3 (TH+1 - 7:)3/3 1
+
72
(te
$ c13 (ogllog(())
1{E1}
E
K1/3
(Tet1 - 2/3
+ T
where in the last step bound 1(8 n (x() s 1{{ and apply tower law again. This concludes the proof.
H.5 Total Variation Regret Rates for Known Weights (Proof of Corollary 13)
Re-Deftning Total Variation in Terms of Weighted Borda Scores. is defined as:
Recall the total variation quantity
VT Σ max (o, a') - 52-1(a,
0,
where (e(aaa)) = Pr(a,a') is the gap in dueling preferences. We first note that this can be rewritten Et= WEE [bx(a, w) ( w)l since by Jensen:
WEAK (b, w) ba, w)l max Eg' (ieeww/ll((aaa a') 81-1(a,a') max (5,(a, - (a)
wEAK
a.a
Note the other directions of the above inequalities are clear by taking w(a'). Thus, we may redefine the total variation in terms of weighted Borda scores:
VT
Σ max 1b,(a, w) (a- w)). WEAK
Proof of Corollary 13. First, we bound the total variation (ITATg)) over an SKW phase [Te, 7411). Consider the arm a (w). By Definition 1, there must exist round E [7,, Ti41) such that
K
1/3
be(a; (w), w) br(o (w)) (w), w)
Ti+1
T
Adding b (at,,, (w), w) b (a (w), w) 2 to the RHS we obtain that
K
1/3
Tit1
Now, summing over SKW phases, we have by Hölder's inequality:
Krawwa
1/3
1/4
3/4
E K1/3,
(T++1 - 2/33 < Σ Ti+1
K2/3 Σ (T++1 Tr) - ..).K(/3
+ K1/3 T2/3
1=0
Thus, obtain total dynamic regret bound of V2/1 T3311 K1/1 K1/3 T2/3
36

---

Published in Transactions on Machine Learning Research (12/2024)
Dynamic Regret Analysis for Unknown Weights
1.1 Analysis Overview for Theorem 14
The proof of Theorem 14 will broadly follow the same outline as the regret analysis for known weights, but with replacements of the eviction threshold by (#2 81) K2/3 (see the unknown weight specification in Definition 4), and bounding the variance of estimation by quantities of this order. The key difficulty for unknown weights is that the evaluation weights w, may change at the unknown SUW shifts P:-
Importantly, Algorithm 2 can only estimate aggregate gaps De=e, 6. (a', a) over intervals [st with respect to fixed and unchanging weight w. Thus, the main novelty in this analysis is to carefully partition the regret analysis along intervals 82] lying within a phase [pi, (iti) In fact, such strategy is already inherent. to the bad segment argument done in Appendix H.4 bound Eier1 87 (a), ae) for a known and known weight w. So, we'll repeat similar such bad segment analysis, but for different. arms and even for different. base algorithms.
This strategy is similar to the approach taken in the Condorcet winner dynamic regret analysis of Buening & Saha (2023, see Appendix A.3 therein), who rely on such a tactic to avoid decomposing the regret using triangle inequalities as done in the original non-stationary MAB analysis of Suk & Kpotufe (2022). Our need for this strategy is different, as we must constrain ourselves to only being able to detect that an arm is bad over segments [81,*2] of rounds where we can use a single reference weight w.
We first establish the analogue of Lemma 20 from Appendix H for the unknown weight setting.
1.2 Episodes Align with SUW Phases
Lemma 25. On event Er, for each episode te-) with te+1 T (d.e., an episode which concludes with restart), there erists on SUW shift P E [1z, (e+,))
Proof. This follows in an analogous manner as Lemma 20, where we note that signifieant SUW regret occurs if, for some weight E AK, we have Ees, 8 (a) K2/3 (82 - $1 )2/3 Thus, an arm being evicted from Aglobal implies it has significant SUW regret meaning a new episode is triggered only when an SUW has occurred.
In what follows, we redefine the last safe arm at as the last arm to become unsafe in the sense of Definition 2 in the unique SUW phase [pt, Pi+1) containing round
1.3 Generic Bad Segment Analysis
We'll first define a generic good event &, over which the bad segment-type argument (as seen in Appendix H.4) holds for any arm a and any episode start time tg. Before we can define such an event, we'll generically redefine the necessary mathematical objects of Appendix H.4 for the unknown weight setting. We note that everything that follows in this subsection is independent of any observations or decisions made by Algorithm 2 and depend only the possible random choices of replay schedules (see Line 12 of Algorithm 2), which may be instantiated independently and obliviously to the algorithm's 's actual behavior.
In what follows, fix a starting round hnit E [T] and an arm a E [K]. Define the random variables Bl s,m
Ber(m- 1/3 (s f for m = 2, ,,, [[oo((T) and == fmit +1...., T, which are the replay schedulers of Line 12 in Algorithm 2 if lmit was the start of an episode. Also, fix a round lstan from which we will begin defining bad segments; the variable fstart. will serve useful when we analyze bad segments for different base algorithms BOSSE( m). The following definitions will then be relative to a fixed fiit, fstart, and arm a. Definition 9. (Generic Bad Segment) Fir an SUW phase [pt, Pi+1) intersecting [Istart; T). Define rounds $1,0 81,1, 81,2 ... E |sttatt Vp Ps+1) recursively follows: let 8 == batar Vp, and define 8ig the smallest round in (8i.j-1, Po41) such that orm a salisfies for some fixed C14 > 0:
max d' d'E(K)
Σ
(ldd) a) 2 C14 log(T) K2/3 (8i. 11/-1)2/3
(29)
1=
37

---

Published in Transactions on Machine Learning Research (12/2024)
if such a round St,j crists. Otherwise, we let the $i,j := Pi+1-1 We refer to any interval 115) as critical segment, and as a bad segment if (26) above holds.
Remark 5. Nole that (29) mimies the nolion (9) of signifieant worse-case weighted Borda regret in Definition 2. However, we need only concern ourselves with checking for bad regret w.r.t. point-mass weights w(a') for a' [K], as that will suffice for the analysis.
Definition 10. (Generic Bad Round) Define the bad round s(a, lstarts finit) > latart the smallest round which satisfies, for some fized C13 > 0:
Σ (55)
- - -1g)
cis (oslog(T) (s(a, lstart linie) - liiit) )2/3. )
(30)
where the above sum over all pairs of indices (i.j) N x N such that [st. s1,+1) is a bad segment (see Definilion 6) with sij+1 < s(a, lstrr, linit).
Now, define the independent sum of Bernoulli's
((a lstart: Muit) =
Σ
(A.i)s8 Stacts
where maj is the dyadic approximation w.r.t. (ai,(a), s1.j+( (a)) in the same sense as defined in Appendix H.4.
We are now prepared to define the good event Es, where all bad segment arguments hold, based on the following lemma.
Lemma 26. Let E3 be the event over which for all a E [K] and rounds laart, linit [T] with htart 2 knit. ((a, lstart limit) > 1. Then, over the randomness of all possible replay schedules, TP(E3) > > 1-7-3
Proof. This follows from repeating the multiplicative Chernoff bound as used in Appendix H.4 for each ((a fstart, )m)) and then taking union bounds over arms a E [K] and rounds fnit E [T]. We note that crucially all potential replay schedules {B } Aass are independent across different 8, m, Anit
Next, we show that, under event E3, the regret of a fixed arm a to of on the active interval [s, M(s m, a)] (see Definition in Appendix H.3) of any scheduled base algorithm BOSSE(s, m) will be boundable by running a customized bad segment analysis using the notions defined above.
Notation 27. (Aclive Interval of First Ancestor Base Algorithm) For the first ancestor base algoritlem BOSSE(t,, T + te) (i.e., that which is instantiated first in episode [te, (t)) per Line 7 of Algorithm 2), we '11 let (te, T + te, a) denote the last round when a is retained by BOSSE(te, T 1 ft) and all of its children.
Definition 11. For an interval of rounds 1, Let PHASES(I) = {i [LUnknown] : 1i+ n + 0}, i.e. denote those phases intersecting I. Let S(I) == [PHASES(I) be the number of intersecting phoses and let L(L,i) = [[.. P1+1) n n|] be the intersection's length for phase (pt.Pt+1).
Lemma 28. (Generic Bad Segment Anolysis for BOSSE(s, m)) For any scheduled BOSSE(s, m,)), letting M(s m, m)) be its active interval, we have on event Eg N
(s, T:
((s.m.))
Σ 6the (oi, I c16 log2(T) K2/3 K2/3. (s- 1(S(T) > 1)
Σ
K2/3 L(T, 2)/33
ie (E
Proof. For (kuit, (start) = (1e, 8), we can define perfect replays with respect to the generic bad segments of Definition 9 analogously Definition 7. We next show an analogue of Proposition 24 for such perfect replays. In particular, for each bad segment [sig s1,4t),, we have for some point-mass weight w,
86.+1
Σ (at. a) 2
C17 log(T)) K2/3 (si.+1 - BL,))
2
Next, the key points hold if perfect replay w.r.t. (1,4,81,441) ) is scheduled:
38

---

Published in Transactions on Machine Learning Research (12/2024)
Arm of (which is constant for E [ 81,+1) S [ps, P++1)) is not evicted from A, since it does not incur significant SUW regret.
The global learning rates Th for E [3 S1,+1] satisty 1/t 2 K2/3 (s - $1.) 1/3 since any child base algorithm only increase the learning rate of its parent base algorithm.
11/))1/3
By similar calculations to the proof of Theorem 19 (see Appendix G) and using the fact that a generic bad segment, as defined in (29) must have length at least K2/8 /8 (analogous to Lemma 17), we have
81,5+1
K
Σ S
16 +
max (1
C18 K2/3 ($1J+1 St,))/3.
Combining the above points with our concentration bound (15) and eviction criterion (4) give that a will be evicted from Ar by round 8i,+1. By Lemma 28, we have that perfect replay with respect to arm a, Istart and hnit will be scheduled before the bad round s(a, s, (y) on event Ez. By (30), we then must have:
Σ (s1J+1 - s c1s log(T) (M m, - t2)3/3 (3,5)
where the sum is over pairs of indices (i,j) representing these generic bad segments. Thus, the desired regret bound follows by similar arguments to Appendix H.4. Note that bounding the regret over the bad segments of multiple phases [pi, Pi+1) needed only if the number of intersecting phases S(I) > 1. Otherwise, we can avoid bad segment analysis and follow the proof steps of Appendix F to directly bound the regret order K2//3 2/3
1.4 Decomposing the Regret Along Different Base Algorithms
Equipped with the generic bad segment analysis for any base algorithm BOSSE(s, m), we're now ready to bound the per-episode regret. It will suffice to decompose the regret along active intervals [s, M (s, m, a)] of different base algorithms BOSSE(s, m) a similar fashion Appendix H.3, within cach of which we can plug in the regret bound of Lemma 28 and then carefully integrate with respect to the randomness of replay scheduling.
We may first decompose the regret as
t+++ -1
4+ -1
E
5thh + (ir)
E
W.
(at)
+E
8th (af.4) + %
t=
The first expectation the above RHS is bounded of the right order by Definition and earlier arguments. For the second expectation on the above RHS, we further condition on whether we're in exploration mode or playing from the candidate set Ac. Following the steps of Appendix H.3, we have that suffices to bound
te11-1
(at a)
E
1fa E Ai
AAl
aE[K]
In fact, following the same decomposition of rounds into active intervals of different base algorithms, further upper bound the above by
K
M(s.m.)) Σ
0:
(aff a)
)
E
M
B
A
BOSSE(s,M)
I=S
Note that in the above BOSSE(t,, T + te) using Notation 27 and for which we the convention B = 1.
all base algorithms including the "first ancestor" base algorithm
39

---

Published in Transactions on Machine Learning Research (12/2024)
Next, we plug in the guarantee of Lemma 28. Let [(s, a, := [s, (s, m, a)] be the active interval of BOSSE (8, m). Then, it remains to bound (hiding the added log terms from Lemma 28 for ease of notation):
K2/3, (s 1/)2/8 ((S((ss > 13 LiEPnAes(((ssmm.)) 2 minrg/(s.m..) |Ae
K2/3 [(T(s, m, m, a).
E
Bam
(O.SE(s.m)
[E[K]
We can further upper bound L([s, ((s, m, a)). by L[[s,s + ml, 1), the sum over phases [ps, Pi+1) in PHASES([s, (s, m. a)l) by a sum over PHASESS(s, ml), and the [[S([(s, m. a) 13 by a [[S + >
} obtain:
K2/3. (s (8-14) (8-4))8/.1(((66 + m]) > 1) Eis PiEPNsss((AAee-m) K2/3, . I([s, + m], 1)2/3
E
Σ
B&m
min,es(s,m.a) AA
BOSSE(s.m)
WE[K]
S (kg(K) by previous arguments which turns the sum over Next, we note that Ede(K) Min A.l [K] in the above display to log(K) factor.
Now, define the function
G(s, == 2/3 (s- -8-tt to)2/3 1[8([s, + ml) > 13 +
Σ
K2/3 L[ss, + m], 8)2/3,
iE AEPHASIS((A,+**m
Then, following the same chain of arguments as in Appendix H.3, we have that it suffices to bound
1441-1
G(s,m)
m/3 (s 22)333
We split this up into two expectations based on the two terms in the definition G(s, m):
f2-1-1
L([s, + ml, 2)23
E
Σ Σ
Σ
+E
Σ
Σ Σ
m-/3
1/3 (s t/)2/3
(EPINSIS((4.
m2
suitt
m
(EPuAses([s.et*i)
For the first expectation above, we have Empp-iss m 1/3 < log(T) (p++1 s)-1/3 Then, summing over
this becomes Σou (pt+1 1/3 < (pi+1 2/33 Thus, the first expectation in the above display is at 1/3 most order log( Σ (t (pi+1 - 2/3
For the second expectation, we use Jensen's 's inequality to bound
Σ I[is, s + m), 1)2/3 m²/3 S([s, +
(EPHAsEs((s.o4m))
where recall from Definition 11 that S(I) counts the number of phases [pt, Pt+1) intersecting interval Now, plugging this into our earlier expectation gives
m/3 1/33.S((s.. s m/)
E
(s (s-tz) 22)3
sate
Then, coarsely bounding S([s. s ml) by S(Ite te41)) and summing over m and s, we obtain the above is most order log(T) S([te, (441)) (le+1 2/33/3
Combining the above steps, we obtain a per-episode regret bound of
t++1 1
Sw (at, a)
E
11a
S
At
[E[K]
t=te
C19 log(K) (K) ) log
1{8}} k
S((te,tezz)) 2/3
Σ
(p 2/3
+ T
iePunsEs((t..ce.))
(31)
40

---

Published in Transactions on Machine Learning Research (12/2024)
Now, we will sum the above over episodes [te, te+1).
1.5 Summing Regret Over Episodes
In Terms of SUW. We first show the total dynamic regret bound of order K2/3 T2/3 11/3 Theorem 14. By Hölder's inequality, summing over episodes gives:
Unknown in
1/3 T2/33 & (I.uknoow + f.)1/3 72/3.
Σ S(Ite (+11))/3 (te+1 12)2/8 S Σ S(ite, (ott)))))))
where L represents the number of realized episodes by Algorithm 2 (i.e., episodes [1e, le+1) where le < <T + 1).
Since Lemma 25 gives that i < LUnknown; the above is of order 11/3 -22/3
Unkknown
Next, again since Lemma 25 implies each phase intersects at most two episodes, we have that summing Exepusss((eeden (pi+1 p;)a/3 over l gives upper bound of Unknown -72/3 by Jensens' inequality.
Total Variation Regret Bound. Next, we show the total variation regrel bound of order V 73/4 K1/2 + K2/3 T2/3, We'll follow a similar argument to that Appendix H.5 for known weight, except taking care to handle the extra (((t,, (te+1 - -/)2// term in (31).
We first bound the total variation Vp 1) over SUW phase (pi, Consider the winner arm "a By Definition 2, there must exist a round (Pi,Pet1) and weight w A such that:
K²
1/3
be(a,, w) (a w)
Pi+1 P
This implies VIp 2 (K2/(PH4 - p By an analogous argument to the SKW total variation regret analysis (with the only modification being the power of K), we have:
Lunknwwn
x2/3-(p141 pp) s V//4 T3/4. K2/2 + +K2/3 12/33
Next, we bound the total variation V(e.t14)) over an episode [Le, t2+1). Let S(ite, ((+1)) be the number of phases [ps, Pi+1) properly contained in episode [4x, (e+1) ) or such that [pi, S (le, le+1). Then, we have the bound
K2
Viterees) 2
Σ
Σ
Pi+1 Pi
(l ()teeeee))
Next, we have that since r x 11/3 is convex, we have the above RHS can be further lower bounded by Jensen's inequality:
K²
1/3
2 R2/3 $((te
Σ
Pi+1
(te+1 1/3
((etett))))))
Alternatively, we may also lower bound Viteftes)) by the same argument that we made for an SUW phase [ps since [te, tet1) is period of rounds where every arm incurs significant regret some period [2] C [le, (e+1) w.r.t. some weight Thus, we also have
K²
1/3
Vieeees) 2
te41 tt
Now, by Hölder's inequality we have
b
1 Σ
K2/3 S(1le, , le+ 1)
1/4
Σ k2/3 Se/t.S((tt,teeipp t(1))) (te+1 - t2)2/3 < K3/2 2/2 T3/1.
+ 22/3.
(te+1 - 1/3
1=1
1=1
41

---

Published in Transactions on Machine Learning Research (12/2024)
Upper bounding S([te, ((ft))t)))))// I C2n (S((te, te+1)) + +2//3) using the inequality (a+b)4/3 <2-(a"/3 we can upper bound the above RHS by order:
1/4
K1/2. T3/4
Σ Vieeeera)
+ KK2/.. & <K . 28/4 + +K233.
1.6 Proof of Theorem 5
Although our main regret upper bound for CW (Theorem 5) does not directly follow from Theorem 14, the analysis will follow nearly identical structure while substituting the SUW phases [pt. (itP)))) with the approximate winner phases ((+1)) (Definition 4).
First, we transform the Condorcet dynamic regret to a weighted Borda dynamic regret. Let w, = w(()(()) where de is the approximate winner arm of the unique approximate winner phase [s, (1+1) containing round Let (1 = + Now, the Condorcet dynamic regret may be re-written using Definition 4:
E Regret
E
(P((a (4) + .1)) - 1)
2
Sapprox
< E
(Pr(ār, ic) + P.(ā,, - -1) + Σ
K2/3 (Git1 Ci)2/3
1=0
S Supprox
E
(ō,(ār, ae) -5, āe) + de 5e ae) 6e(ic, ael)
Σ K2/3 ((i+1 - (6-+1-))2//
S
Spproo
< E
6 (i+) + 6 (i.)
Σ
K2/3 (Gi+1 - -2)
t=
It remains bound the expectation on the above RHS by the sum (up log terms) in the same display and then show
S Spprrox
Σ K2/3 (G++1 G,)2// < min ( 2/3 upproxe K2/21//413/ + 22/372/3)
The key facts will be crucial in showing these claims.
Fact 29. An SUW cannot occur within an approrimate winner phase. other words, supposing ā; the approzimate winner arm of phose [Si, 1i+1) (such that (6) holds for ā. and for all E [Gi, (4+1), E [K]) then must have that à, satisfies for all [$1, 82] S++1):
8 (ū.) < R2/3, (82 - 81132/3
max WEAK YEAK K
Proof. Note that (6), Jensen's inequality, and GIC (Condition 1) implies for any weight w A K and E [Gr C:41):
K2
1/3
6tHh (a:) 10 (a", a.)l < Ee P. a)ll ) s
Thus, à; does not incur significant worst-case weighted Borda regret over any interval [S1, s2] C [6i G+11). 0
Fact 30. The total variation in an approzimate winner phase is at least
K²
1/3
ViGiSon)) 2
Ci+1
Ce
42

---

Published in Transactions on Machine Learning Research (12/2024)
Proof. Fix phase Iss: S441) and consider the winner arm 2a at round Si+1. By Definition 4, there must exist round E Isi, (+41 such that for some
[K]:
( G++1-1+ -G
(
k²
1/3
<6 (arl
(vll) (a). - (ag) acot)) S (iRcEL))
where the second inequality follows from GIC.
Fact 31 (Analogue of Lemma 25). On event ε for each episode te+1) with tet1 ST (ie.. an episode which concludes with restart), there erists on approzimate winner change Çi E [1z, (e,2441)..
Proof. This follows from Fact 29, as an SUW cannot occur within an approximate winner phase. This means that since a restart implies an SUW has occurred, an approximate winner change must have also occurred. r
Next, using Fact 31, we bound the regret EI 8M (ix) + 6," (it) using the generic bad segment analysis of Appendix L.3 except replacing the SUW phases [pt, pa+1) with the approximate winner phasos [Gi, S-1)- Fact 29 ensures that the analogue of Lemina 28 will hold as the approximale winner arm 0g of phase Si cannot be evicted by a perfect replay corresponding to generic bad segment in phase [Si, (++1). Then, the proof steps of Appendix 1.4 and Appendix L.5 follow mutatis mutandis while using Fact 30 to get the total variation bound.
J Additional Related Work
Dueling bandits. The stochastic dueling bandit problem was first proposed by Yue & Joachims (2011); Yue et al. (2012), which provided algorithm achieving inslance-dependent O(K log 7) regret under the SSINSTI condition. Urvoy et al. (2013) studied this problem under the broader Condorcet winner condition and achievod an instance-dependdent O(K2 log T) regret bound, which was further improved by Zoghi et al. (2014) and Komiyama et al. (2015) to O(K²= + K log T). Finally, Saha & Gaillard (2022) showed it is possible to achieve an optimal instance-dependent bound of O(K T) and instance-independent bound of O(VK") under the Condorcet condition. These works all assume stationary environment.
Works on adversarial dueling bandits (Gajane et al., 2015; Saha et al., 2021) allow for changing preferences and thus are closer to this work. However, these works focus on the static regret objective against the 'best" arm hindsight and whereas we consider the dynamic regret.
Other than the carlier mentioned works on dynamic regret minimization in ducling bandits, Kolpaczki et al. (2022) st studies weak dynamic regrel minimization but uses procedures requiring knowledge of non-st non-stationarily.
Borda Regret Minimization. The only works studying Borda regret (in stochastic or adversarial settings) are Saha el al. (2021); Saha & Gupta (2022); Wu et al. (2023). Of these, only Saha & Gupla (2022) establishes dynamic Borda regret bounds, which require knowledge of the underlying non-stationarity, and are suboptimal in light of our optimal regret bound (Theorem 2). Hilgendorf (2018) studies weak Borda regret where the learner only incurs the Borda regret of the better of the two arms paid.
Other Notions of Winner. Other alternative notions of winner and objectives, beyond Condorcet and Borda, have been proposed, such Copeland winner (Zoghi et al., 2015a; Komiyama et al., 2016; Wn & Liu, 2016) and von Neumann winner (Dudik et al., 2015; Balsubramani et al., 2016).
There have also been generalized notions Borda (Brandt et al., 2022), and Condorcet winners (Agarwal et al., 2020; Haddenhorst et al., 2021) to combinatorial settings with subset comparisons.
Related notions to our weighted Borda winner (sce Appendix A) appear in earlier social choice theory literature (Xia & Conitzer, 2008; Xia, 2013).
Non-stationary multi-armed bandits. Switching multi-armed bandits with was first considered in the so-called adversarial setting by Auer et al. (2002), where version of EXP3 was shown to attain optimal dynamic regret / VLT when given knowledge of the number L of changes in the rewards. Later works
43

---

Published in Transactions on Machine Learning Research (12/2024)
showed similar guarantees in this problem for procedures inspired by stochastic bandit algorithms (Garivier & Moulines, 2011; Koesis & Szepesvári, 2006). Recently, Auer et al. (2018; 2019); Chen et al. (2019) established the first adaptive and optimal dynamic regret guarantees, without requiring parameter knowlodge of the number of changes.
Alternative characterizations of the change in rewards in terms of a total variation quantity was first introduced in Besbes et al. (2019) with minimax rates quantified therein and adaptive rates attained in Chen et al. (2019). There have also been characterizations of non-stationarity in terms of drift parameters (Jia et al., 2023; Krishnamurthy & Gopalan, 2021). Yet another characlerization, in terms of the nunber of best arm switches S was studied in Abbasi-Yaddes Yadkori et al. (2023), establishing an adaptive regret rate of VSKT. Around the same time, Suk & Kpotufe (2022) introduced the aforementioned notion of significunt shifts for the switching bandit problem and adaptively achieved rates of the form VŠKT in terms of significant shifts in the rewards, which serves as the inspiration for our Definition 1.
44


Reviews:
[{"review_id": "Mc3PB5GUub", "reviewer": "Reviewer_meZc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The introduction of the significant Borda Winner switches (SBS) is meaningful, as there are lots of works in the nonstationary K-armed bandits literature who quantify the level of nonstationarity as the number of big distribution switches. The proposed notion, SBS, fills the gap in the nonstationary dueling bandits within the Borda winner setup.\n- The proposed General Identifiability Condition further generalizes the SST and STI conditions presented in the Condorcet winner literature. Under this relaxed condition, the final upper bound is more refined and is superior to the previous ones under certain conditions.\n\n**Weaknesses and Questions**\n\nWeaknesses:\n- The intuition for the definition of the SBS phase/Significant Borda regret, and approximate winner phase is not provided in detail.\n- The usefulness of the approximate winner arm needs to be justified by concrete examples.\n\nQuestions:\n- While the introduction of SBS is desired, the definition of the significant Borda regret is not clear to me.\n\t- According to equation (2), if the Borda winner gradually decrease the winning probability and the rest suboptimal arms gradually increase the winning probability such that (2) will not hold for any arm, then it may happen that the Borda winner has changed but there is not any SBS over the whole time horizon. Is this property desired?\n- In Definition 4, is this requirement really useful? \n\t- When the top two arms have close winning probabilities against the rest arms and the winning probability of the top two arms are swapped frequently. It seems that although the Condorcet winer changes, the number of approximate winner phases does not increase.\n\t- As indicated by equations (3) and (4), the number of approximate winner phases is small only when the number of arms is large, which can even be linear in $T$. \nTherefore, I am not perfectly convinced by the utility of this definition.\n- Can the authors provide some insights for the choice of the threshold, in particular the orders, Definition 1 and 4? It seems that these orders are picked so that the expected regret in each phase can match that in the stationary setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the instance-independent bound for non-stationary dueling bandits under the Condorcet winner and Borda winner setup. It establishes the a dynamic regret upper bound, which depends on the (unknown) number of significant switches. Another contribution is a weighted version of the Borda score. The winner under the weighted Borda score can particularize to the Condorcet winner and Borda winner with the point-distribution and uniform distribution, respectively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses and questions section. In particular, the definitions of the SBS phases and approximate winner phases need more discussions and justifications.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x51h3BDeeL", "reviewer": "Reviewer_YQxQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- This work provides a comprehensive research on the problem of non-stationary dueling bandits, contributing to understanding on both Borda winner and Condorect winner, including both upper and lower bounds. The obtained results are of their importance based on my understanding: providing the first adaptive upper bound for Borda winner and a deep understanding on the non-stationary measurement for the Condorect winner.\n\n- The writing of this work is well-organized. I had only basic knowledge of dueling bandits before, while I found the reading experience smooth and entertaining. \n\n- The proposed unified view of Condorect and Borda regret, together with the BOSSE algorithmic framework, would be of independent interest based on my understanding. The unified view could contribute to a more strong connection between studies on these two targets in dueling bandits while the algorithm design can serve as an important base for further extensions.\n\nWeakness:\n- The main paper has omitted the majority of technical details, which is understandable. However, with that, the overall reading seems to lack of a deep dive into the key innovation/technical contributions. I conjecture that there would be some key steps (besides the unified view) for the upper and lower bound analysis.\n\n- Similar to the first point, the algorithmic contributions of BOSSE is only touched very briefly in the main paper, which makes me hard to judge its contribution. While it does share similarity with the phased scheme in non-stationary and adversarial designs, I believe it holds its own merit that should be highlighted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the problem of non-stationary dueling bandits from the perspective of both Borda winner and Condorcet winner. For Borda winner, this work establishes the first optimal and adaptive dynamic regret upper bound. For the Condorect winner, this work introduces a new non-stationary measurement (less stringent than the previously studied one), under which an upper bound that improves the SOTA general performance is achieved. Both of these advances are originated from a unified weighted Borda framework and a novel soft elimination algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major (from weakness):\n- Certain technical details in the main paper to facilitate readers to understand the key steps/main challenges of the proof. Currently, there are some verbal discussions, which I found useful, while I would encourage the authors to have a bit more technical details in the main paper.\n\n- More discussions on the BOSSE algorithm. I would suggest the authors to provide more discussions on BOSSE to highlight its novelty/difference from the previously adopted schemes in non-stationary bandits.\n\nMinor:\n- Lower bounds can also be put in Table 1 for a complete view.\n\n- I would love to hear the authors' comments on the possibility of doing some simulated experiments. While I understand the difficulty to do so due to the non-stationary environment and algorithmic complexity, I would lean to see some experiments. If not possible, please explicitly mention this as a limitation of this work worth further exploration.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EZnfSvJ8vO", "reviewer": "Reviewer_VcZ5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. I really appreciate the authors insight to combine the two notions of Borda regret and Condorcet regret. It is good works which unify different notions. With this the paper provides a new tool to analyze Condorcet models using Borda frameworks.\n\n2. The paper provides an algorithm which works with non-stationary preference matrices by performing controlled exploration on all arms. The exploration is controlled using learning rate parameter $\\eta_t$ which is time dependent. The regret analysis works by bounding the variation of the importance sampling estimator which gives insights into who to select the learning rate of the algorithm (Appendix E)\n\n\nWeaknesses:\n1. The paper is extremely difficult to parse.\n\n   a . There are a lot of forward references, for example the regret of the algorithm described before defining the algorithm. \n\n   b. The SST$\\cap$STI condition is not clearly explained in the introduction again requiring to move forward. Even there the complete names should have been provided and some details/intuitions would have helped.\n\n   c.  What is ordering of the arms, that is again not clearly defined.\n\n2. Lack of motivation: It would be helpful if the authors explained why there exists two school of thoughts Borda/Condorcet best arms and where is one useful?\n\n3. Some prototype algorithm implementation even on toy example would make the paper stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of non-stationary dueling bandits where agent dealing with a bandit framework where the agent can play two arms and gets the feedback about which arm is better. An interesting contemporary setup where this framework could be useful is collecting feedback by presenting different LLM outputs to users and see which is the preferred output. For this framework the authors considered two parallel preference models of Borda and Condorcet and show their equivalence using a weighted Borda approach.\n\nFor the weighted Borda approach, the paper considers a non-stationary preference matrix and provide BOSSE algorithm which works by injecting uniform exploration controlled with time-dependent learning rate $\\eta_t$. For the proposed algorithm, the authors derive regret bounds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses mentioned.", "broader_impact_concerns": "The paper is theoretical and does not have broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The setting and the results are novel. Together with the reviewers, I believe this contribution would be a good addition to the literature on dueling bandits.
Decision: Accept (poster)