Title: When Can We Track Significant Preference Shifts in Dueling Bandits?
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-stationary

multi-armed bandits

dueling bandits

preference-based learning
Abstract: The $K$-armed dueling bandits problem, where the feedback is in the form of noisy pairwise preferences, has been widely studied due its applications in information retrieval, recommendation systems, etc. Motivated by concerns that user preferences/tastes can evolve over time, we consider the problem of _dueling bandits with distribution shifts_. Specifically, we study the recent notion of _significant shifts_ (Suk and Kpotufe, 2022), and ask whether one can design an _adaptive_ algorithm for the dueling problem with $O(\sqrt{K\tilde{L}T})$ dynamic regret,
where $\tilde{L}$ is the (unknown) number of significant shifts in preferences. We show that the answer to this question depends on the properties of underlying preference distributions. Firstly,  we give an impossibility result that rules out any algorithm with $O(\sqrt{K\tilde{L}T})$ dynamic regret under the well-studied Condorcet and SST classes of preference distributions. Secondly, we show that $\text{SST}\cap \text{STI}$ is the largest amongst popular classes of preference distributions where it is possible to design such an algorithm. Overall, our results provides an almost complete resolution of the above question for the hierarchy of  distribution classes.

When Can We Track Significant Preference Shifts in Dueling Bandits?

Abstract

The K -armed dueling bandits problem, where the feedback is in the form of noisy pairwise preferences, has been widely studied due its applications in information retrieval, recommendation systems, etc. Motivated by concerns that user preferences/tastes can evolve over time, we consider the problem of dueling bandits with distribution shifts . Specifically, we study the recent notion of significant shifts [Suk and Kpotufe, 2022], and ask whether one can design an adaptive algorithm for the dueling problem with O ( √ K ˜ LT ) dynamic regret, where ˜ L is the (unknown) number of significant shifts in preferences. We show that the answer to this question depends on the properties of underlying preference distributions. Firstly, we give an impossibility result that rules out any algorithm with O ( √ K ˜ LT ) dynamic regret under the well-studied Condorcet and SST classes of preference distributions. Secondly, we show that SST ∩ STI is the largest amongst popular classes of preference distributions where it is possible to design such an algorithm. Overall, our results provides an almost complete resolution of the above question for the hierarchy of distribution classes.

1 Introduction

The K -armed dueling bandits problem has been well-studied in the multi-armed bandits literature [Yue and Joachims, 2011, Yue et al., 2012b, Urvoy et al., 2013, Ailon et al., 2014, Zoghi et al., 2014, 2015a,b, Dudik et al., 2015, Jamieson et al., 2015, Komiyama et al., 2015, 2016, Ramamohan et al., 2016, Chen and Frazier, 2017, Saha and Gaillard, 2022, Agarwal et al., 2022]. In this problem, on each trial t ∈ [ T ] , the learner pulls a pair of arms and observes relative feedback between these arms indicating which arm was preferred. The feedback is typically stochastic, drawn according to a pairwise preference matrix P ∈ [0 , 1] K × K , and the regret measures the 'sub-optimality' of arms with respect to a 'best' arm.

This problem has many applications, e.g. information retrieval, recommendation systems, etc, where relative feedback between arms is easy to elicit, while real-valued feedback is difficult to obtain or interpret. For example, a central task for information retrieval algorithms is to output a ranked list of documents in response to a query. The framework of online learning has been very useful for automatic parameter tuning, i.e. finding the best parameter(s), for such retrieval algorithms based on user feedback [Liu, 2009]. However, it is often difficult to get numerical feedback for an individual list of documents. Instead, one can (implicitly) compare two lists of documents by interleaving them and observing the relative number of clicks [Radlinski et al., 2008]. The availability of these pairwise comparisons allows one to tune the parameters of retrieval algorithms in real-time using the framework of dueling bandits.

However, in many such applications that rely on user generated preference feedback, there are practical concerns that the tastes/beliefs of users can change over time, resulting in a dynamically

SST

STI

SST∩STI

changing preference distribution. Motivated by these concerns, we consider the problem of switching dueling bandits (or non-stationary dueling bandits), where the pairwise preference matrix P t changes an unknown number of times over T rounds. The performance of the learner is evaluated using dynamic regret where sub-optimality of arms is calculated with respect to the current 'best' arm.

Saha and Gupta [2022] first studied this problem and provided an algorithm that achieves a nearly optimal (up to log terms) dynamic regret of ˜ O ( √ KLT ) where L is the total number of shifts in the preference matrix, i.e., the number of times P t differs from P t +1 . However, this result requires algorithm knowledge of L . Alternatively, the algorithm of Saha and Gupta [2022] can be tuned to achieve a dynamic regret rate (also nearly optimal) ˜ O ( V 1 / 3 T K 1 / 3 T 2 / 3 ) in terms of the total-variation of change in preferences V T over T total rounds. This is similarly limited by requiring knowledge of V T .

On the other hand, recent works on the switching MAB problem show it is not only possible to design adaptive algorithms with ˜ O ( √ KLT ) dynamic regret without knowledge of the underlying environment [Auer et al., 2019], but also possible to achieve a much better bound of ˜ O ( √ K ˜ LT ) where ˜ L ≪ L is the number of significant shifts [Suk and Kpotufe, 2022]. Specifically, a shift is significant when there is no 'safe' arm left to play, i.e., every arm has, on some interval [ s 1 , s 2 ] , regret order Ω( √ s 2 -s 1 ) . Such a weaker measure of non-stationarity is appealing as it captures the changes in best-arm which are most severe, and allows for more optimistic regret rates over the previously known √ KLT ∧ ( V T K ) 1 / 3 T 2 / 3 .

Very recently, Buening and Saha [2022] considered an analogous notion of significant shifts for switching dueling bandits under the SST ∩ STI 1 assumption. They gave an algorithm that achieves a dynamic regret of ˜ O ( K √ ˜ LT ) , where ˜ L is the (unknown) number of significant shifts . However, their algorithm estimates Ω( K 2 ) pairwise preferences, and hence, suffers from a sub-optimal dependence on K .

In this paper we consider the goal of designing optimal algorithms for switching dueling bandits whose regret depends on the number of significant shifts ˜ L . We ask the following question:

Question. Is it possible to achieve a dynamic regret of O ( √ K ˜ LT ) without knowledge of ˜ L ?

We show that the answer to this question depends on conditions on the preference matrices. Specifically, we consider several well-studied conditions from the dueling bandits literature, and give an almost complete resolution of the achievability of O ( √ K ˜ LT ) dynamic regret under these conditions.

1.1 Our Contributions

We first consider the classical Condorcet winner (CW) condition where, at each time t ∈ [ T ] , there is a 'best' arm under the preference P t that stochastically beats every other arm. Such a winner arm is a benchmark in defining the aforementioned dueling dynamic regret . Our first result shows that, even under the CW condition, it is in general impossible to achieve O ( √ K ˜ LT ) dynamic regret.

Theorem 1. (Informal) There is a family of instances F under Condorcet where all shifts are non-significant, i.e. ˜ L = 0 , but no algorithm can achieve o ( T ) dynamic regret uniformly over F .

Note that in the case when ˜ L = 0 , one would ideally like to achieve a dynamic regret of O ( KT ) . The above theorem shows that, under the Condorcet condition when ˜ L = 0 , not only is it impossible

√

√

to achieve O ( KT ) regret, it is even impossible to achieve O ( T α ) regret for any α < 1 . Hence, this rules out the possibility of an algorithm whose regret under this condition is sublinear in ˜ L and T .

The proof of the above theorem relies on a careful construction where, at each time t , the preference P t is chosen uniformly at random from two different matrices P + and P -. These matrices have different 'best' arms but there is a unique safe arm in both. However, it is impossible to identify this safe arm as all observed pairwise preferences are Ber( 1 2 ) over the randomness of the environment. Moreover, the theorem gives two different constructions (one ruling out SST and one STI) which together rule out all preference classes outside of SST ∩ STI. Our second result shows that the desired regret √ K ˜ LT is in fact achievable (adaptively) under SST ∩ STI.

Theorem 2. (Informal) There is an algorithm that achieves a dynamic regret of ˜ O ( √ K ˜ LT ) under SST ∩ STI without requiring knowledge of ˜ L .

Figure 1 gives a summary of our results. Note that in stationary dueling bandits there is no separation in the regret achievable under the CW vs. SST ∩ STI conditions, i.e. O ( √ KT ) is the minimax optimal regret rate under both conditions [Saha and Gaillard, 2022]. However, our results show that in the non-stationary setting with regret in terms of significant shifts, there is a separation in adaptively achievable regret.

Key Challenge and Novelty in Regret Upper Bound: To contrast, the recent work of Buening and Saha [2022] only attains ˜ O ( K √ ˜ LT ) dynamic regret under SST ∩ STI due to inefficient exploration of arm pairs. Our more challenging goal of obtaining the optimal dependence on K introduces key difficulties in algorithmic design. In fact, even in the classical stochastic dueling bandit problem with SST ∩ STI, most existing results that achieve O ( √ KT ) regret require identifying a coarse ranking over arms to avoid suboptimal exploration of low ranked arms [Yue et al., 2012a, Yue and Joachims, 2011]. However, in the non-stationary setting, ranking the arms meaningfully is difficult as the true ordering of arms may change (insignificantly) at all rounds. Our main algorithmic innovation is to bypass the task of ranking arms and instead directly focus on minimizing the cumulative regret of played arms. This entails a new rule for selecting 'candidate' arms based on cumulative regret that may be of independent interest.

1.2 Related Work

Dueling bandits. The stochastic dueling bandits problem and its variants have been studied widely (see Sui et al. [2018] for a comprehensive survey). This problem was first proposed by Yue et al. [2012b], who provide an algorithm achieving instance-dependent O ( K log T ) regret under the SST ∩ STI condition. Yue and Joachims [2011] also studied this problem under the SST ∩ STI condition and gave an algorithm that achieves optimal instance-dependent regret. Urvoy et al. [2013] studied this problem under the Condorcet winner condition and achieved an instance-dependent O ( K 2 log T ) regret bound, which was further improved by Zoghi et al. [2014] and Komiyama et al. [2015] to O ( K 2 + K log T ) . Finally, Saha and Gaillard [2022] showed that it is possible to achieve an optimal instance-dependent bound of O ( K log T ) and instance-independent bound of O ( √ KT ) under the Condorcet condition. More general notions of winners such as Borda winner [Jamieson et al., 2015], Copeland winner [Zoghi et al., 2015a, Komiyama et al., 2016, Wu and Liu, 2016], and von Nuemann winner [Dudik et al., 2015] have also been considered. However, these works only consider the stationary setting whereas we consider the non-stationary setting.

There has also been work on adversarial dueling bandits [Saha et al., 2021, Gajane et al., 2015], however, these works only consider static regret against the 'best' arm in hindsight and whereas we consider the harder dynamic regret. Other than the two previously mentioned works [Gupta and Saha, 2022, Buening and Saha, 2022], the only other work on switching dueling bandits is Kolpaczki et al. [2022], whose procedures require knowledge of non-stationarity and only consider the weaker measure of non-stationarity L counting all changes in the preferences.

Non-stationary multi-armed bandits. Multi-armed bandits with changing rewards was first considered in the adversarial setup by Auer et al. [2002], where a version of EXP3 was shown to attain optimal dynamic regret √ KLT when properly tuned using the number L of changes in the rewards. Later works established similar (non-adaptive) guarantees in this so-called switching bandit problem via procedures inspired by stochastic bandit algorithms [Garivier and Moulines, 2011, Kocsis

and Szepesvári, 2006]. More recent works [Auer et al., 2018, 2019, Chen et al., 2019] established the first adaptive and optimal dynamic regret guarantees, without requiring knowledge of the number of changes. An alternative parametrization of switching bandits, via a total-variation quantity, was introduced in Besbes et al. [2014] with minimax rates quantified therein and adaptive rates attained in Chen et al. [2019]. Yet another characterization, in terms of the number of best arm switches S was studied in Abbasi-Yadkori et al. [2022], establishing an adaptive regret rate of √ SKT . Around the same time, Suk and Kpotufe [2022] introduced the aforementioned notion of significant shifts and adaptively achieved rates of the form √ K ˜ LT in terms of ˜ L significant shifts in rewards.

2 Problem Formulation

We consider non-stationary dueling bandits with K arms and time-horizon T . At round t ∈ [ T ] , the pairwise preference matrix is denoted by P t ∈ [0 , 1] K × K , where the ( i, j ) -th entry P t ( i, j ) encodes the likelihood of observing a preference for arm i in a direct comparison with arm j . The preference matrix may change arbitrarily from round to round. At round t , the learner selects a pair of actions ( i t , j t ) ∈ [ K ] × [ K ] and observes the feedback O t ( i t , j t ) ∼ Ber( P t ( i t , j t )) where P t ( i t , j t ) is the underlying preference of arm i t over j t . We define the pairwise gaps δ t ( i, j ) := P t ( i, j ) -1 / 2 .

Conditions on Preference Matrix. We consider two different conditions on preference matrices: (1) the Condorcet winner (CW) condition and (2) the strong stochastic transitivity (SST) and stochastic triangle inequality (STI), formalized below.

Definition 1. (CW condition) At each round t , there is a Condorcet winner arm, denoted by a ∗ t , such that δ t ( a ∗ t , a ) ≥ 0 for all a ∈ [ K ] \{ a ∗ t } . Note that a ∗ t need not be unique.

Definition 2. ( SST ∩ STI condition) At each round t , there exists a total ordering on arms, denoted by ≻ t , and ∀ i ⪰ t j ⪰ t k :

(b) δ t ( i, k ) ≤ δ t ( i, j ) + δ t ( i, k ) (STI).

It's easy to see that the SST condition implies the CW condition as δ t ( i, j ) ≥ δ t ( i, i ) = 0 for any i ≻ t j . Hence, the highest ranked item under ≻ t in Definition 2 is the CW a ∗ t . We emphasize here that the CW in Definition 1 and the total ordering on arms in Definition 2 can change at each round, even while such unknown changes in preference may not be counted as significant (see below).

Regret Notion. Our benchmark is the dynamic regret to the sequence of Condorcet winner arms:

Here, the regret of an arm i is defined in terms of the preference gap δ t ( a ∗ t , i ) between the winner arm a ∗ t and i , and the regret of the pair ( i t , j t ) is the average regret of individual arms i t and j t . Note the this regret is well-defined under both Condorcet and SST ∩ STI conditions due to the existence of a unique 'best' arm a ∗ t , and is non-negative due to the fact that δ t ( a ∗ t , i ) ≥ 0 for all i ∈ [ K ] .

Measure of Non-Stationarity. We first recall the notion of Significant Condorcet Winner Switches from Buening and Saha [2022], which captures only the switches in a ∗ t which are severe for regret. Throughout the paper, we'll also refer to these as significant shifts for brevity.

Definition 3 (Significant CW Switches) . Define an arm a as having significant regret over [ s 1 , s 2 ] if

We then define significant CW switches recursively as follows: let τ 0 = 1 and define the ( i +1) -th significant CW switch τ i +1 as the smallest t > τ i such that for each arm a ∈ [ K ] , ∃ [ s 1 , s 2 ] ⊆ [ τ i , t ] such that arm a has significant regret over [ s 1 , s 2 ] . We refer to the interval of rounds [ τ i , τ i +1 ) as a significant phase . Let ˜ L be the number of significant CW switches elapsed in T rounds.

Notation. To ease notation, we'll conflate the closed, open, and half-closed intervals of real numbers [ a, b ] , ( a, b ) , and [ a, b ) , with the corresponding rounds contained therein, i.e. [ a, b ] ≡ [ a, b ] ∩ N .

3 Hardness of Significant Shifts in the Condorcet Winner Setting

We first consider regret minimization in an environment with no significant shift in T rounds. Such an environment admits a safe arm a ♯ which does not incur significant regret throughout play. Our first result shows that, under the Condorcet condition, it is not possible to distinguish the identity of a ♯ from other unsafe arms, which will in turn make sublinear regret impossible.

Theorem 3. For each horizon T , there exists a finite family F of switching dueling bandit environments with K = 3 that satisfies the Condorcet winner condition (Definition 1) with ˜ L = 0 significant shifts. The worst-case regret of any algorithm on an environment E in this family is lower bounded as

Proof. (sketch; details found in Appendix B) Letting ϵ ≪ 1 / √ T , consider the preference matrices:

In P + , arm 1 is the Condorcet winner and 1 ≻ 2 ≻ 3 , whereas in P -, 3 is the winner with 3 ≻ 2 ≻ 1 . Let an oblivious adversary set P t at round t to one of P + and P -, uniformly at random, inducing an environment where arm 2 remains safe for T rounds. Then, any algorithm will, over the randomness of the adversary, observe O t ( i t , j t ) ∼ Ber(1 / 2) no matter the choice of arms ( i t , j t ) played , by the symmetry of P + , P -. Thus, it is impossible to distinguish arms, which implies linear regret by standard Pinsker's inequality arguments. In particular, even a strategy playing arm 2 every round fails as arm 2 is unsafe in another (indistinguishable) setup with arms 1 and 2 switched in P + , P -.

SST and STI Both Needed To Learn Significant Shifts. The preferences P + , P -in the above proof violate STI but satisfy SST, whereas another construction using preferences P + , P -which violate SST but satisfy STI also works in the proof (see Remark 2 in Appendix B). This shows that sublinear regret is impossible outside of the class SST ∩ STI (visualized in Figure 1).

Remark 1. Note the lower bound of Theorem 3 does not violate the established upper bounds √ LT and V 1 / 3 T T 2 / 3 scaling with L changes in the preference matrix or total variation V T [Gupta and Saha, 2022]. Our construction in fact uses L = Ω( T ) changes in the preference matrix and V T = Ω( T ) total variation. Furthermore, the regret upper bound √ ST , in terms of S changes in Condorcet winner, of Buening and Saha [2022] is not contradicted either, for S = Ω( T ) .

4 Dynamic Regret Upper Bounds under SST/STI

Acknowledging that significant shifts are hard outside of the class SST ∩ STI, we now turn our attention to the achievability of √ K ˜ LT regret 2 in the SST ∩ STI setting. Our main result is an optimal dynamic regret upper bound attained without knowledge of the significant shift times or the number of significant shifts . Up to log terms, this is the first dynamic regret upper bound with optimal dependence on T , ˜ L , and K .

Theorem 4. Suppose SST and STI hold (see Definition 2). Let { τ i } ˜ L i =0 denote the unknown significant shifts of Definition 3. Then, for some constant C 0 > 0 , Algorithm 1 has expected dynamic regret

and using Jensen's inequality, this implies a regret rate of C 0 log 3 ( T ) √ K · ( ˜ L +1) · T .

In fact, this regret rate can be transformed to depend on the Condorcet winner variation introduced in Buening and Saha [2022] and the total variation quantities introduced in Gupta and Saha [2022] and inspired by the total-variation quantity from non-stationary MAB [Besbes et al., 2014]. The following corollary is shown using just the definition of the non-stationarity measures.

Corollary 5 (Regret in terms of CW Variation) . Let V T := ∑ T t =2 max a ∈ [ K ] | P t ( a ∗ t , a ) -P t -1 ( a ∗ t , a ) | be the unknown Condorcet winner variation. Using the same notation of Theorem 4: Algorithm 1 has expected dynamic regret

5 Algorithm

At a high level, the strategy of recent works on non-stationary multi-armed bandits [Chen et al., 2019, Wei and Luo, 2021, Suk and Kpotufe, 2022] is to first design a suitable base algorithm and then use a meta-algorithm to randomly schedule different instances of this base algorithm at variable durations across time. The key idea is that unknown time periods of significant regret can be detected fast enough with the right schedule. In order to accurately identify significant shifts, the base algorithm in question should be robust to all non-significant shifts. In the multi-armed bandit setting, a variant of the classical successive elimination algorithm [Even-Dar et al., 2006] possesses such a guarantee [Allesiardo et al., 2017], and serves as a base algorithm in Suk and Kpotufe [2022].

5.1 Difficulty of Efficient Exploration of Arms.

In the non-stationary dueling problem, a natural analogue of successive elimination is to uniformly explore the arm-pair space [ K ] × [ K ] and eliminate arms based on observed comparisons [Urvoy et al., 2013]. The previous work [Theorem 5.1 of Buening and Saha, 2022] employs such a strategy as a base algorithm. However, such a uniform exploration approach incurs a large estimation variance of K 2 , which enters into the final regret bound of K √ T · ˜ L . Thus, smarter exploration strategies are needed to obtain √ K dependence.

In the stationary dueling bandit problem with SST ∩ STI, such efficient exploration strategies have long been known: namely, the Beat-The-Mean algorithm [Yue and Joachims, 2011] and the Interleaved Filtering (IF) algorithm [Yue et al., 2012a]. We highlight that these existing algorithms aim to learn the ordering of arms, i.e., arms are ruled out roughly in the same order as their true underlying ordering. This fact is crucial to attaining the optimal dependence in K in their regret analyses, as the higher ranked arms must be played more often against other arms to avoid the K 2 cost of exploration.

However, in our setting, adversarial but non-significant changes in the ordering of arms could force perpetual exploration of lowest-ranked arms. This suggests that learning an ordering should not be a subtask of our desired dueling base algorithm. Rather, the algorithm should prioritize minimizing its own regret over time. Keeping this intuition in mind, we introduce an algorithm called SW itching I nterleaved F il T ering (SWIFT) (see Algorithm 2 in Section 5.2) which directly tracks regret and avoids learning a fixed ordering of arms.

A new idea for switching candidate arms. A natural idea that is common to many dueling bandit algorithms (including IF) is to maintain a candidate arm ˆ a which is always played at each round, and serves as a reference point for partially ordering other arms in contention. If the current candidate is beaten by another arm then a new candidate is chosen, and this process quickly converges to the best arm. Since the ordering of arms may change at each round, any such rule that relies on a fixed ordering is deemed to fail in our setting. Our procedure does not rely on such a fixed ordering over arms, but instead tracks the aggregate regret ∑ t δ t ( a, ˆ a t ) of the changing sequence of candidate arms { ˆ a t } t to another fixed arm a . Crucially, the candidate arm ˆ a s is always played at round s and so the history of candidate arms { ˆ a s } s ≤ t is fixed at a round t . This fact allows us to estimate the quantity ∑ t s =1 δ s ( a, ˆ a s ) using importance-weighting at √ K · t rates via martingale concentration. An algorithmic switching criterion then switches the candidate arm ˆ a t to any arm a dominating the sequence { ˆ a s } s ≤ t over time, i.e., ∑ t s =1 δ s ( a, ˆ a s ) ≫ √ K · t . This simple, yet powerful, idea immediately gives us control of the regret of the candidate sequence { ˆ a t } t which allows us to bypass the ranking-based arguments of vanilla IF and Beat-The-Mean. It also allows us to simultaneously bound the regret of a sub-optimal arm a against the sequence of candidate arms ∑ t s =1 δ s (ˆ a s , a ) .

5.2 Switching Interleaved Filtering (SWIFT)

SWIFT at round t compares a candidate arm ˆ a t with an arm a t (chosen uniformly at random) from an active arm set A t . Additionally, SWIFT maintains estimates ˆ δ t (ˆ a t , a ) of δ t (ˆ a t , a ) which are used to (1) evict active arms a ∈ A t and (2) switch the candidate arm ˆ a t +1 for the next round.

Estimators and Eviction/Switching Criteria. Let A t be the active arm set at round t . Let

which is an unbiased estimator of the gap δ t (ˆ a t , a ) when a ∈ A t . We evict an active arm a from A t at round t if for some constant C > 0 3 and rounds s 1 < s 2 ≤ t :

where ˆ δ s ( a, ˆ a s ) := -ˆ δ s (ˆ a s , a ) . Next, we switch the next candidate arm ˆ a t +1 ← a to another arm a ∈ A t at round t if for some round s 1 < t :

SWIFT is formally shown in Algorithm 2, defined for generic start time t start and duration m 0 so as to allow for recursive calls in our meta-algorithm framework.

5.3 Non-Stationary Algorithm ( METASWIFT )

For the non-stationary setting with multiple (unknown) significant shifts, we run SWIFT as a base algorithm at randomly scheduled rounds and durations.

Our algorithm, dubbed METASWIFT and found in Algorithm 1, operates in episodes , starting each episode by playing a base algorithm instance of SWIFT . A running base algorithm activates its own base algorithms of varying durations (Line 8 of Algorithm 2), called replays according to a random schedule decided by the Bernoulli's B s,m (see Line 6 of Algorithm 1). We refer to the (unique) base algorithm playing at round t as the active base algorithm .

Global Variables. The active arm set A t is pruned by the active base algorithm at round t , and globally shared between all running base algorithms. In addition, all other variables, i.e. the ℓ -th episode start time t ℓ , round count t , schedule { B s,m } s,m , and candidate arm ˆ a t (and thus the quantities δ t (ˆ a t , a ) ) are shared between base algorithms. Thus, while active, each Base-Alg can switch the candidate arm (4) and evict arms (3) over all intervals [ s 1 , s 2 ] elapsed since it began.

Note that only one base algorithm (the active one) can edit A t and set the candidate arm ˆ a t at round t , while other base algorithms can access these global variables at later rounds. By sharing these global

variables, any replay can trigger a new episode: every time an arm is evicted by a replay, it is also evicted from the master arm set A master, tracking arms' regret throughout the entire episode. A new episode is triggered when A master becomes empty, i.e., there is no safe arm left to play.

6 Regret Analysis

6.1 Regret of METASWIFT over Significant Phases

Now, we turn to sketching the proof of Theorem 4. Full details are found in Appendix C.

Decomposing the Regret. Let a ♯ t denote the last safe arm at round t , or the last arm to incur significant regret in the unique phase [ τ i , τ i +1 ) containing round t . Then, we can decompose the dynamic regret around this safe arm using SST and STI (i.e., using Lemma 8 twice) as:

where we recall that a t ∈ A t is the other arm played (Line 3 of Algorithm 2). Next, the first sum on the above RHS is order ∑ ˜ L i =1 √ K · ( τ i -τ i -1 ) as the last safe arm a ♯ t does not incur significant regret on [ τ i , τ i +1 ) . So, it remains to bound the last two sums on the RHS above.

Episodes Align with Significant Phases. We claim that a new episode is triggered only if there a significant shift occurs (Lemma 11). This follows from our eviction criteria (3) with Freedman's inequality for martingale concentration (Lemma 9). Then, acknowledging episodes roughly align with significant phases, we turn our attention to bounding the remaining regret in each episode.

Bounding Regret of an Episode. Let t ℓ be the start of the ℓ -th episode of METASWIFT . Then, our goal is to show for all ℓ ∈ [ ˆ L ] (where ˆ L is the random number of episodes used by the algorithm):

̸

where the RHS sum above is over the significant phases [ τ i -1 , τ i ) overlapping episode [ t ℓ , t ℓ +1 ) . Summing over episodes ℓ ∈ [ ˆ L ] will then yield the desired total regret bound by our earlier observation that the episodes align with significant phases (see Lemma 11).

Bounding Regret of Active Arms to Candidate Arms. Bounding ∑ t ℓ +1 -1 t = t ℓ δ t (ˆ a t , a t ) follows in a similar manner as Appendix B.1 of Suk and Kpotufe [2022]. First, observe by concentration (Lemma 9) the eviction criterion (3) bounds the sums ∑ s 2 t = s 1 δ t (ˆ a t , a ) over intervals [ s 1 , s 2 ] where a is active. Then, accordingly, we further partition the episode rounds [ t ℓ , t ℓ +1 ) into different intervals distinguishing the unique regret contributions of different active arms from varying base algorithms, on each of which we can relate the regret to our eviction criterion. Details can be found in Appendix C.3.

Bounding Regret of Candidate Arm to Safe Arm. The first sum on the LHS of (5) will be further decomposed using the last master arm a ℓ which is the last arm to be evicted from the master arm set A master in episode [ t ℓ , t ℓ +1 ) . Carefully using SST and STI (see Lemma 13), we further decompose δ t ( a ♯ t , ˆ a t ) as:

The sum C above was already bounded earlier. So, we turn our attention to B and A.

Bounding B. Note that the arm a ℓ by definition is never evicted by any base algorithm until the end of the episode t ℓ +1 -1 . This means that at round t ∈ [ t ℓ , t ℓ +1 ) , the quantity ∑ t s = t ℓ ˆ δ s ( a ℓ , ˆ a s ) is always kept small by the candidate arm switching criterion (4). So, by concentration (Proposition 10), we have ∑ t ℓ +1 -1 s = t ℓ ˆ δ s ( a ℓ , ˆ a s ) ≲ √ K ( t ℓ +1 -t ℓ ) .

Bounding A The main intuition here, similar to Appendix B.2 of Suk and Kpotufe [2022], is that well-timed replays are scheduled w.h.p. to ensure fast detection of large regret of the last master arm a ℓ . Key in this is the notion of a bad segment of time : i.e., an interval [ s 1 , s 2 ] ⊆ [ τ i , τ i +1 ) lying inside a significant phase with last safe arm a ♯ where:

For a fixed bad segment [ s 1 , s 2 ] , the idea is that a fortuitously timed replay scheduled at round s 1 and remaining active till round s 2 will evict arm a ℓ .

It is not immediately obvious how to carry out this argument in the dueling bandit problem since, to detect large ∑ t δ t ( a ♯ , a ℓ ) , the pair of arms a ♯ , a ℓ need to both be played which, as we discussed in Section 5.1, may not occur often enough to ensure tight estimation of the gaps.

Instead, we carefully make use of SST/STI to relate δ t ( a ♯ , a ℓ ) to δ t (ˆ a t , a ℓ ) . Note this latter quantity controls both the eviction (3) and ˆ a t switching (4) criteria. This allows us to convert bad intervals with large ∑ t δ t ( a ♯ t , a ℓ ) to intervals with large ∑ t δ t (ˆ a t , a ℓ ) . Specifically, by Lemma 13, we have that (7) implies

Then, we claim that, so long as a base algorithm Base-Alg ( s 1 , m ) is scheduled from s 1 running till s 2 , we will have ∑ s 2 t = s 1 δ t (ˆ a t , a ℓ ) ≳ √ K · ( s 2 -s 1 ) which implies a ℓ will be evicted. In other words, the second sum dominates the first and third sums in (8). We repeat earlier arguments to show this:

By the definition of the last safe arm a ♯ , ∑ s 2 t = s 1 δ t ( a ∗ t , a ♯ ) < √ K · ( s 2 -s 1 ) .

Meanwhile, ∑ s 2 t = s 1 δ t ( a ♯ , ˆ a t ) ≲ √ K · ( s 2 -s 1 ) by the candidate switching criterion (4) and because a ♯ will not be evicted before round s 2 lest it incurs significant regret which cannot happen by definition of a ♯ .

Combining the above two points with (8), we have that ∑ s 2 t = s 1 δ t (ˆ a t , a ℓ ) ≳ √ K · ( s 2 -s 1 ) , which directly aligns with our criterion (3) for evicting a ℓ . To summarize, a bad segment [ s 1 , s 2 ] in the sense of (7) is detectable using a well-timed instance of SWIFT , which happens often enough with high probability. Concretely, we argue that not too many bad segments elapse before a ℓ is evicted by a well-timed replay in the above sense and that thus the regret incurred by a ℓ is bounded by the RHS of (5). The details can be found in Appendix C.5.

7 Conclusion

We consider the problem of switching dueling bandits where the distribution over preferences can change over time. We study a notion of significant shifts in preferences and ask whether one can achieve adaptive dynamic regret of O ( √ K ˜ LT ) where ˜ L is the number of significant shifts. We give a negative result showing that one cannot achieve such a result outside of the SST ∩ STI setting, and answer this question in the affirmative under the SST ∩ STI setting. In the future, it would be interesting to consider other notions of shifts which are weaker than the notion of significant shift, and ask whether adaptive algorithms for the Condorcet setting can be designed with respect to these notions. Buening and Saha [2022] already give a O ( K √ ST ) bound for the Condorcet setting, where S is the number of changes in 'best' arm. However, their results have a suboptimal dependence on K due to reduction to 'all-pairs" exploration.

References

Yasin Abbasi-Yadkori, Andras Gyorgy, and Nevena Lazic. A new look at dynamic regret for non-stationary stochastic bandits. arXiv preprint arXiv:2201.06532 , 2022.

Arpit Agarwal, Rohan Ghuge, and Viswanath Nagarajan. Batched dueling bandits. In Proceedings of the 39th International Conference on Machine Learning , pages 89-110, 2022.

Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing Dueling Bandits to Cardinal Bandits. In Proceedings of the 31st International Conference on Machine Learning , 2014.

Robin Allesiardo, Raphaël Féraud, and Odalric-Ambrym Maillard. The non-stationary stochastic multi-armed bandit problem. International Journal of Data Science and Analytics , 3(4):267-283, 2017.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The Nonstochastic Multiarmed Bandit Problem. SIAM Journal on Computing , 32(1):48-77, 2002.

Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best arm with an unknown number of distribution changes. 14th European Workshop on Reinforcement Learning (EWRL) , 2018.

Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 138-158. PMLR, 2019.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with nonstationary rewards. Advances in Neural Information Processing Systems , 27:199-207, 2014.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics , pages 19-26, 2011.

Thomas Kleine Buening and Aadirup Saha. Anaconda: An improved dynamic regret algorithm for adaptive non-stationary dueling bandits. arXiv preprint arXiv:2210.14322 , 2022.

Bangrui Chen and Peter I. Frazier. Dueling Bandits with Weak Regret. In Proceedings of the 34th International Conference on Machine Learning , 2017.

Yifang Chen, Chung-Wei Lee, Haipeng Luo, and Chen-Yu Wei. A new algorithm for non-stationary contextual bandits: Efficient, optimal, and parameter-free. In Proceedings of the 32nd Conference on Learning Theory , 99:1-30, 2019.

Miroslav Dudik, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual Dueling Bandits. In Proceedings of the 28th Conference on Learning Theory , 2015.

Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems. Journal of Machine Learning Research , 7:1079-1105, 2006.

Pratik Gajane, Tanguy Urvoy, and Fabrice Clérot. A relative exponential weighing algorithm for adversarial utility-based dueling bandits. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 218-227. JMLR.org, 2015.

Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory , pages 174-188. Springer, 2011.

Shubham Gupta and Aadirupa Saha. Optimal and efficient dynamic regret algorithms for nonstationary dueling bandits. In International Conference on Machine Learning , pages 19027-19049. PMLR, 2022.

Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse Dueling Bandits. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics , 2015.

Levente Kocsis and Csaba Szepesvári. Discounted ucb. 2nd PASCAL Challenges Workshop , 2006.

Patrick Kolpaczki, Viktor Bengs, and Eyke Hüllermeier. Non-stationary dueling bandits. arXiv preprint arXiv:2202.00935 , 2022.

Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem. In Proceedings of the 28th Conference on Learning Theory , 2015.

Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Copeland Dueling Bandit Problem: Regret Lower Bound, Optimal Algorithm, and Computationally Efficient Algorithm. In Proceedings of the 33rd International Conference on Machine Learning , 2016.

Tie-Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr. , 3(3):225-331, mar 2009. ISSN 1554-0669. doi: 10.1561/1500000016.

Filip Radlinski, Madhu Kurup, and Thorsten Joachims. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM Conference on Information and Knowledge Management , CIKM '08, page 43-52, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781595939913. doi: 10.1145/1458082.1458092.

Siddartha Ramamohan, Arun Rajkumar, and Shivani Agarwal. Dueling Bandits : Beyond Condorcet Winners to General Tournament Solutions. In Advances in Neural Information Processing Systems 29 , 2016.

Aadirupa Saha and Pierre Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning , pages 19011-19026. PMLR, 2022.

Aadirupa Saha and Shubham Gupta. Optimal and efficient dynamic regret algorithms for nonstationary dueling bandits. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 19027-19049. PMLR, 2022.

Aadirupa Saha, Tomer Koren, and Yishay Mansour. Adversarial dueling bandits. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9235-9244. PMLR, 2021.

Yanan Sui, Masrour Zoghi, Katja Hofmann, and Yisong Yue. Advancements in dueling bandits. In Jérôme Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , pages 5502-5510. ijcai.org, 2018.

Joe Suk and Samory Kpotufe. Tracking most significant arm switches in bandits. In Conference on Learning Theory , pages 2160-2182. PMLR, 2022.

Tanguy Urvoy, Fabrice Clerot, Raphael Feraud, and Sami Naamane. Generic Exploration and Karmed Voting Bandits. In Proceedings of the 30th International Conference on Machine Learning , 2013.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 32nd International Conference on Learning Theory , 2021.

Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 649-657, 2016.

Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th International Conference on Machine Learning , 2011.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k -armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538-1556, 2012a.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538-1556, 2012b. ISSN 0022-0000. doi: https://doi.org/10.1016/j.jcss.2011.12.028. JCSS Special Issue: Cloud Computing 2011.

Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten de Rijke. Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem. In Proceedings of the 31st International Conference on Machine Learning , 2014.

Masrour Zoghi, Zohar Karnin, Shimon Whiteson, and Maarten de Rijke. Copeland Dueling Bandits. In Advances in Neural Information Processing Systems 28 , 2015a.

Masrour Zoghi, Shimon Whiteson, and Maarten de Rijke. MergeRUCB: A method for large-scale online ranker evaluation. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining , 2015b.

20000

20000

17500

17500

15000

15000

Regret

12500

12500

Regre

Dynamic

10000

Dynamic

10000

7500

7500

5000

5000

RANDDUEL

RANDDUEL

IF

IF

2500

METASWIFT

2500

METASWIFT

ANACONDA

ANACONDA

0

10000

20000

30000

40000

50000

0

10000

20000

30000

40000

50000

Time

Time

A Experiments

Code can be found at https://github.com/joesuk/nonstationary-duel .

Synthetic Environments. We used a geometric BTL model where the arms are linearly ordered and the i -th best arm beats the j -th best arm with probability P ( i ≻ j ) = 2 -i 2 -j +2 -i . At each changepoint, the ordering of arms was randomly permuted, with a total T = 50 , 000 rounds with K = 10 arms. Regret was computed over N = 50 trials of each environment with standard confidence bands shown.

Algorithms. We considered four algorithms: (1) METASWIFT (Algorithm 1, (2) the ANACONDA algorithm of Buening and Saha [2022] (3) Interleaved Filtering (which we abbreviate as IF) as specified by Yue et al. [2012b], and a baseline (3) RANDDUEL which naively plays a pair of arms selected uniformly at random every round.

Parameters. Parameters associated with each of the algorithms (e.g., the constants in displays (3) and (4), analogous quantities in ANACONDA, and IF's eliminination threshold) were tuned using cross validation on randomly generated geometric BTL environments with number of changepoints varying from 0 to 1000 . For fairness, all algorithms were given the chance to tune parameters on the same environments before testing.

The first graphic in Figure 2 shows the regret curves in a stationary environment with S = 0 changes. The second graphic shows the regret curves in a non-stationary environment with S = 4 changes. Exact mean and standard deviations on final regret are given in Table 1. These do support the theoretical message that METASWIFT performs better than the existing ANACONDA algorithm in non-stationary environments due to more efficient exploration of arms (demonstrated through √ K versus K dependence in the theoretical bounds). Moreover, we also observe that the IF algorithm which is designed for stationary environments can have almost linear regret in non-stationary environments.

B Proof of Theorem 3

Consider the following preference matrices for some ϵ > 0 (to be chosen later):

In environment P + , arm 1 is the Condorcet winner and we have 1 ≻ 2 ≻ 3 . In environment P -, arm 3 is the winner with 3 ≻ 2 ≻ 1 .

Consider a uniform mixture U of the preference matrices P + and P -, Let E be a (random) sequence of T environments sampled i.i.d. from U , with P t := ( E ) t being the sampled environment at round t .

First, it is straightforward to verify in every such switching dueling bandit E , arm 2 does not incur significant regret over any interval of rounds [ s 1 , s 2 ] ⊆ [1 , T ] , for ϵ < 1 / √ T . Thus, every such E exhibits zero significant shifts.

Next, in what follows, we use E E [ · ] to denote an expectation over both the randomness of U ⊗ T and the algorithm's feedback and decisions. If there exists a realization of E such that the algorithm gets expected regret at least T/ 8 , then we are already done. Otherwise, we have the expected regret over the random environment E is bounded above by T/ 8 . Next, define the arm-pull counts N ( T, a ) := ∑ T t =1 1 { i t = a } +1 { j t = a } for each arm a . Then, we relate these arm-pull counts to the regret:

where we use the tower law in the third inequality (note that i t , j t are independent of ( E ) t ). Thus, in expectation over both the model noise and randomness of E , arms 3 and 1 cannot be played more than T/ 2 times without causing linear regret.

Since ∑ 3 a =1 E E [ N ( T, a )] = 2 T , we conclude that E E [ N ( T, 2)] ≥ 3 T/ 2 . We will next show that arm 2 is statistically indistinguishable from arm 3 . To do so, we consider an analogous environment which is identical to E except the identities of arms 2 and 3 are switched. Specifically, let E ′ be a random sequence of T environments sampled i.i.d. from a uniform mixture of Q + and Q -, which are respectively P + and P -with switched entries for arms 2 and 3 .

We next claim E E [ N ( T, 2)] = E E ′ [ N ( T, 2)] . Admitting this claim, it immediately follows that the algorithm has expected regret (over the randomness of E ′ ) at least (using an analogous chain of inequalities as above):

In particular, there exists a realization of E ′ within the prior on environments on which the regret is at least 3 T/ 8 .

It remains to show E E [ N ( T, 2)] = E E ′ [ N ( T, 2)] . This will follow from Pinsker's inequality and showing that the KL divergence between E and E ′ is zero.

Wefirst observe that the dueling observations O t ( i, j ) at each round t ∈ [ T ] are identically a Ber(1 / 2) R.V. for all pairs of arms i, j in both E and E ′ , since a uniform mixture of a Ber(1 / 2 + ϵ ) and a Ber(1 / 2 -ϵ ) is a Ber(1 / 2) , while so is the uniform mixture of a Ber(1) and a Ber(0) .

Then, since N ( T, 2) ≤ 2 T , by Pinsker's inequality [see Gupta and Saha, 2022, proof of Lemma C.1], we have:

where P and P ′ are the induced distributions over the randomness U ⊗ T , and the history of observations and decisions in T rounds by E and E ′ . Let H t be the history of randomness, observations, and decisions till round t : H t = { ( u s , i s , j s , O s ( i s , j s )) } s ≤ t where u s ∼ Ber(1 / 2) decides whether P + / Q + or P -/ Q -is realized at round t . Let P t and P ′ t denote the respective marginal distributions over the round t data ( u t , i t , j t , O t ( i t , j t )) . Then, repeatedly using chain rule for KL and then conditioning on the played arms ( i t , j t ) (whose identities are fixed given H t -1 ) at round t , we get:

■

Remark 2. The constructed environments P + , P -in the proof of Theorem 3 satisfies SST but violates STI. A similar construction which violates SST (but satisfies STI) can also be used in the proof. Let

and let P -be the same preference matrix with arms 2 and 3 switched. Note that 3 ≻ 2 ≻ 1 in P + and 2 ≻ 3 ≻ 1 in P -. Here, arm 1 is the 'safe' arm as it always has a gap of ϵ while arms 2 and 3 randomly alternate between being the best arm and the worst arm with a gap of 1 / 2 . Thus, both the STI and SST assumptions are required to get sublinear regret in mildly adversarial environments.

Due to these observations we have the following corollaries.

Corollary 6. For each horizon T , there exists a finite family F of switching dueling bandit environments with K = 3 that satisfies the SST condition with ˜ L = 0 significant shifts. The worst-case regret of any algorithm on an environment E in this family is lower bounded as

Corollary 7. For each horizon T , there exists a finite family F of switching dueling bandit environments with K = 3 that satisfies the STI condition with ˜ L = 0 significant shifts. The worst-case regret of any algorithm on an environment E in this family is lower bounded as

C Full Proof of Theorem 4

Throughout the proof c 1 , c 2 , . . . will denote positive constants not depending on T or any distributional parameters. First, we observe the regret bound is vacuous for T < K ; so, assume T ≥ K . Recall from Line 3 of Algorithm 1 that t ℓ is the first round of the ℓ -th episode. WLOG, there are T total episodes and, by convention, we let t ℓ := T +1 if only ℓ -1 episodes occurred by round T .

Next, we establish an elementary lemma which will help us leverage the STI and SST assumptions.

C.1 Decomposing the Regret

Lemma 8. For any three arms b, c , under SST ∩ STI : δ t ( a ∗ t , c ) ≤ 2 · δ t ( a ∗ t , b ) + δ t ( b, c ) .

Proof. If b ⪰ t c , this is true by STI. Otherwise, δ t ( a ∗ t , c ) ≤ δ t ( a ∗ t , b ) ≤ δ t ( a ∗ t , b )+ δ t ( a ∗ t , b ) -δ t ( c, b ) by SST.

Using Lemma 8 twice, we have the regret can be written as

Following the discussion of Section 6, it remains to bound ∑ T t =1 δ t ( a ♯ t , ˆ a t ) and ∑ T t =1 δ t (ˆ a t , a t ) in expectation. For this, we need to relate our estimators ˆ δ t (ˆ a t , a ) to the true gaps δ t (ˆ a t , a ) .

C.2 Relating Estimated Gaps to Regret

We first recall a version of Freedman's martingale concentration inequality, identical to the one used in Suk and Kpotufe [2022], Buening and Saha [2022].

Lemma 9 (Theorem 1 of Beygelzimer et al. [2011]) . Let X 1 , . . . , X n ∈ R be a martingale difference sequence with respect to some filtration {F 0 , F 1 , . . . } . Assume for all t that X t ≤ R a.s. and that ∑ n i =1 E [ X 2 i |F i -1 ] ≤ V n a.s. for some constant V n only depending on n . Then for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have:

ˆ δ (ˆ a , a )

We next apply Lemma 9 to bound the estimation error of our estimates t t , found in (2). Proposition 10. Let E 1 be the event that for all rounds s 1 < s 2 and all arms a ∈ [ K ] :

for an appropriately large constant c 1 , and where F := {F t } T t =1 is the canonical filtration generated by observations and randomness of elapsed rounds. Then, E 1 occurs with probability at least 1 -1 /T 2 .

Proof. The random variable ˆ δ t (ˆ a t , a ) -E [ ˆ δ t (ˆ a t , a ) |F t -1 ] is a martingale difference bounded above by K for all rounds t and all arms a, a ′ . Note here that the identity of the candidate arm ˆ a t is fixed conditional on the observations of the previous rounds F t -1 . The variance of this difference is:

Then, the result follows from Lemma 9 and taking union bounds over arms a and rounds s 1 , s 2 .

Since the contribution to the expected regret is small outside of the high-probability good event E 1 , going forward we will assume as necessary that (9) holds for all arms a ∈ [ K ] and rounds s 1 , s 2 . The next result asserts that episodes roughly correspond to significant shifts in the sense that a restart (Line 8 of Algorithm 1) occurs only if a significant shift has been detected.

Lemma 11. On event E 1 , for each episode [ t ℓ , t ℓ +1 ) with t ℓ +1 ≤ T (i.e., an episode which concludes with a restart), there exists a significant shift τ i ∈ [ t ℓ , t ℓ +1 ) .

Proof. We have that

Thus, by concentration (Proposition 10) and the eviction criteria (3) with large enough constant C > 0 , we have that an arm a being evicted over interval [ s 1 , s 2 ] implies ∑ s 2 t = s 1 δ t (ˆ a t , a ) > √ K · ( s 2 -s 1 ) . By the SST condition, this means that

This means, over the course of episode [ t ℓ , t ℓ +1 ) , every arm a ∈ [ K ] incurs significant regret meaning a significant shift must take place between rounds t ℓ and t ℓ +1 -1 .

Following the outline of Section 6, we now turn our attention to bounding the regrets δ t ( a ♯ t , ˆ a t ) and δ t (ˆ a t , a t ) over a single episode [ t ℓ , t ℓ +1 ) .

C.3 Bounding E [ ∑ t ℓ +1 -1 t = t ℓ δ t (ˆ a t , a t )] : Regret of Active Arms to Candidate Arm

We first decompose the total sum of regrets E [ ∑ T t =1 δ t (ˆ a t , a t )] based on which arm a t chooses within the active set A t . Using tower law, we have

Splitting the above RHS back along episodes, we obtain the sum E [ ∑ t ℓ +1 -1 t = t ℓ ∑ a ∈A t δ t (ˆ a t , a ) / |A t | ] .

Next, we condition on the good event E 1 on which the concentration bounds of Proposition 10 hold. Additionally, we divide up the rounds t into those before arm a is evicted from A master and those after. Suppose arm a is evicted from A master at round t a ℓ ∈ [ t ℓ , t ℓ +1 ) . In particular, this means arm a ∈ A t for all t ∈ [ t ℓ , t a ℓ ) . Thus, it suffices to bound:

Suppose WLOG that t 1 ℓ ≤ t 2 ℓ ≤ · · · ≤ t K ℓ . Then, for each round t < t a ℓ all arms a ′ ≥ a are retained in A master and thus retained in the candidate arm set A t . Thus, |A t | ≥ K +1 -a for all t ≤ t a ℓ .

Then, the first double sum in (10) can be bounded by combining our eviction criterion (3) with our concentration bounds Proposition 10. Since arm a is not evicted from A t till round t a ℓ , on event E 1 we have for some c 2 > 0 :

Then, using the fact that |A t | ≥ K +1 -a for all t ∈ [ t ℓ , t a ℓ ) , we have:

Then, summing the above R.H.S. over all arms a , we have on event E 1 :

Next, we handle the second double sum in (10). We first observe that if arm a is played after round t a ℓ , then it must due to a scheduled replay. The difficulty here is that replays may interrupt each other and so care must be taken in managing the relative regret contribution ∑ t δ t (ˆ a t , a ) (which may be negative if a ≺ ˆ a t ) of different overlapping replays.

Fixing an arm a , our strategy is to partition the rounds when a is played by a replay after round t a ℓ according to which replay is active and not accounted for by another replay. This involves carefully designating a subclass of replays whose durations while playing a span all the rounds where a is played after t a ℓ . Then, we cover the times when a is played by a collection of intervals corresponding to the schedules of this subclass of replays, on each of which we can employ the eviction criterion (3) and concentration like before.

For this purpose, we define the following terminology (which is all w.r.t. a fixed arm a ):

Definition 4.

(i) For each scheduled and activated Base-Alg ( s, m ) , let the round M ( s, m ) be the minimum of two quantities: (a) the last round in [ s, s + m ] when arm a is retained by Base-Alg ( s, m ) and all of its children, and (b) the last round that Base-Alg ( s, m ) is active and not permanently interrupted. Call the interval [ s, M ( s, m )] the active interval of Base-Alg ( s, m ) .

proper replays

arm a evicted

Figure 3: Shown are replay scheduled durations (in gray) with dots marking when arm a is reintroduced to A t . Black segments indicate the period [ s, M ( s, m )] for proper and subproper replays. Note that the rounds where a ∈ A t in the left unlabeled replay's duration are accounted for by the larger proper replay.

(ii) Call a replay Base-Alg ( s, m ) proper if there is no other scheduled replay Base-Alg ( s ′ , m ′ ) such that [ s, s + m ] ⊂ ( s ′ , s ′ + m ′ ) where Base-Alg ( s ′ , m ′ ) will become active again after round s + m . In other words, a proper replay is not scheduled inside the scheduled range of rounds of another replay. Let PROPER ( t ℓ , t ℓ +1 ) be the set of proper replays scheduled to start before round t ℓ +1 .

(iii) Call a scheduled replay Base-Alg ( s, m ) subproper if it is non-proper and if each of its ancestor replays (i.e., previously scheduled replays whose durations have not concluded) Base-Alg ( s ′ , m ′ ) satisfies M ( s ′ , m ′ ) < s . In other words, a subproper replay either permanently interrupts its parent or does not, but is scheduled after its parent (and all its ancestors) stops playing arm a . Let SUBPROPER ( t ℓ , t ℓ +1 ) be the set of all subproper replays scheduled before round t ℓ +1 .

Equipped with this language, we now show some basic claims which essentially reduce analyzing the complicated hierarchy of replays to analyzing the active intervals of replays in PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( t ℓ , t ℓ +1 ) .

Proposition 12. The active intervals

are mutually disjoint.

Proof. Clearly, the classes of replays PROPER ( t ℓ , t ℓ +1 ) and SUBPROPER ( t ℓ , t ℓ +1 ) are disjoint. Next, we show the respective active intervals [ s, M ( s, m )] and [ s ′ , M ( s ′ , m ′ )] of any two Base-Alg ( s, m ) and Base-Alg ( s ′ , m ′ ) ∈ PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( t ℓ , t ℓ +1 ) are disjoint.

Proper replay vs. subproper replay: a subproper replay can only be scheduled after the round M ( s, m ) of the most recent proper replay Base-Alg ( s, m ) (which is necessarily an ancestor). Thus, the active intervals of proper replays and subproper replays are disjoint.

Two distinct proper replays: two such replays can only permanently interrupt each other, and since M ( s, m ) always occurs before the permanent interruption of Base-Alg ( s, m ) , we have the active intervals of two such replays are disjoint.

Two distinct subproper replays: consider two non-proper replays Base-Alg ( s, m ) , Base-Alg ( s ′ , m ′ ) ∈ SUBPROPER ( t ℓ , t ℓ +1 ) with s ′ > s . The only way their active intervals intersect is if Base-Alg ( s, m ) is an ancestor of Base-Alg ( s ′ , m ′ ) . Then, if Base-Alg ( s ′ , m ′ ) is subproper, we must have s ′ > M ( s, m ) , which means that [ s ′ , M ( s ′ , m ′ )] and [ s, M ( s, m )] are disjoint.

Next, we claim that the active intervals [ s, M ( s, m )] for Base-Alg ( s, m ) ∈ PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( t ℓ , t ℓ +1 ) contain all the rounds where a is played after being evicted from A master. To show this, we first observe that for each round t when a replay is active, there is a unique proper

replay associated to t , namely the proper replay scheduled most recently. Next, note that any round t > t a ℓ where arm a ∈ A t must belong to the active interval [ s, M ( s, m )] of the unique proper replay Base-Alg ( s, m ) associated to round t , or else satisfies t > M ( s, m ) in which case a unique subproper replay Base-Alg ( s ′ , m ′ ) ∈ SUBPROPER ( t ℓ , t ℓ +1 ) was active and not yet permanently interrupted by round t . Thus, it must be the case that t ∈ [ s ′ , M ( s ′ , m ′ )] .

At the same time, every round t ∈ [ s, M ( s, m )] for a proper or subproper Base-Alg ( s, m ) is clearly a round where a ∈ A t and no such round is accounted for twice by Proposition 12. Thus,

Then, we can rewrite the second double sum in (10) as:

Recall in the above that the Bernoulli B s,m (see Line 6 of Algorithm 1) decides whether Base-Alg ( s, m ) is scheduled.

Further bounding the sum over t above by its positive part, we can expand the sum over Base-Alg ( s, m ) ∈ PROPER ( t ℓ , t ℓ +1 ) ∪ SUBPROPER ( t ℓ , t ℓ +1 ) to be over all Base-Alg ( s, m ) , or obtain:

where the sum is over all replays Base-Alg ( s, m ) , i.e. s ∈ { t ℓ + 1 , . . . , t ℓ +1 -1 } and m ∈ { 2 , 4 , . . . , 2 ⌈ log( T ) ⌉ } . It then remains to bound the contributed relative regret of each Base-Alg ( s, m ) in the interval [ s ∨ t a ℓ , M ( s, m )] , which will follow similarly to the previous steps. Fix s, m and suppose t a ℓ +1 ≤ M ( s, m ) since otherwise Base-Alg ( s, m ) contributes no regret in (11).

Then, following similar reasoning as before, i.e. combining our concentration bound (9) with the eviction criterion (3), we have for a fixed arm a :

Plugging this into (11) and switching the ordering of the outer double sum, we obtain (now for clarity overloading the notation M ( s, m, a ) to also depend on the reference arm a ):

We claim the above innermost sum over a is at most log( K ) . For a fixed Base-Alg ( s, m ) , if a k is the k -th arm in [ K ] to be evicted by Base-Alg ( s, m ) or any of its children, then min t ∈ [ s,M ( s,m,a k )] |A t | ≥ K +1 -k . Thus, our claim follows follows from ∑ K k =1 1 K +1 -k ≤ log( K ) .

Let R ( m ) := c 2 log( K ) log( T ) √ Km ∨ K 2 which is the bound we've obtained so far on the relative regret for a single Base-Alg ( s, m ) . Then, plugging R ( m ) into (11) gives:

Next, we observe that B s,m and 1 { s < t ℓ +1 } are independent conditional on t ℓ since 1 { s < t ℓ +1 } only depends on the scheduling and observations of base algorithms scheduled before round s . Thus,

Plugging this into our expectation from before and unconditioning, we obtain:

Then, it suffices to bound √ K ( t ℓ +1 -t ℓ ) ∨ K 2 . First, we claim that every phase [ τ i , τ i +1 ) is length at least K/ 4 . Observe by our notion of significant regret, that an arm a incurring significant regret on the interval [ s 1 , s 2 ] means

Thus, each significant phase (Definition 3) must be at least K/ 4 rounds long meaning τ i +1 -τ i = ( τ i +1 -τ i ) ∨ K/ 4 . This will allow us to remove the ' ∨ K 2 ' in (12). In particular, since the episode length t ℓ +1 -t ℓ in (12) can be upper bounded by the combined length of all significant phases [ τ i , τ i +1 ) interesecting episode [ t ℓ , t ℓ +1 ) , (12) gives us the desired bound.

C.4 Bounding E [ ∑ t ℓ +1 -1 t = t ℓ δ t ( a ♯ t , ˆ a t )] : Regret of Candidate Arm to Safe Arm

We first invoke an elementary lemma based on SST and STI to further help us decompose the regret.

Lemma 13. For any three arms a, b, c , under SST ∩ STI :

where a ∗ t is the winner arm.

Proof. We handle all the different orderings:

(a) a ≻ t b, c : this already follows from Lemma 8 since then δ t ( a, c ) ≤ 2 · δ t ( a, b ) + δ t ( b, c ) .

(b) c ≻ t a ≻ t b : δ t ( a, c ) ≤ 0 ≤ δ t ( a, b ) and δ t ( a ∗ , b ) ≥ δ t ( c, b ) by SST. Summing these together gives the result.

(c) b ≻ t a ≻ t c : δ t ( a, c ) ≤ δ t ( b, c ) and δ t ( a ∗ t , a ) ≥ δ t ( b, a ) by SST. Summing these together gives the result.

(d) b, c ≻ t a : δ t ( a ∗ , a ) dominates the first two terms on the desired inequality's RHS.

Then, using Lemma 13, we further decompose the regret about the last master arm a ℓ defined in Section 4, which is the last arm to be evicted from A master in episode [ t ℓ , t ℓ +1 ) . We have

As said earlier, the sum ∑ t ℓ +1 -1 t = t ℓ δ t ( a ∗ t , a ♯ t ) is of the right order. Meanwhile, the sum ∑ t ℓ +1 -1 t = t ℓ δ t ( a ℓ , ˆ a t ) is bounded using our candidate arm switching criterion (4). If ˆ a t = a ℓ for every round t ∈ [ t ℓ , t ℓ +1 ) we are already done. Otherwise, let m ℓ be the last round that a ℓ is not the candidate arm ˆ a t . Then, we must have that since arm a ℓ is not evicted until round t ℓ +1 -1 :

Then, by concentration (Proposition 10) and the fact from earlier that each phase [ τ i , τ i +1 ) is at least K/ 4 rounds (so that ' ∨ K 2 ' can be removed in the above), we have that ∑ t ℓ +1 -1 t = t ℓ δ t ( a ℓ , ˆ a t ) is of the right order.

Then, turning back to (13), it remains to bound the regret of a ℓ to a ♯ t over the episode [ t ℓ , t ℓ +1 ) .

C.5 Bounding E [ ∑ t ℓ +1 -1 t = t ℓ δ t ( a ♯ t , a ℓ )] : Regret of Last Master Arm to Safe Arm

First, following the outline of Section 4, we recall the definition of the last safe arm a ♯ t at round t which is the last arm to incur significant regret in the unique phase [ τ i , τ i +1 ) containing round t .

We next formally define a bad segment, alluded to in Section 4. In what follows, bad segments will be defined with respect to a fixed arm a and conditional on the episode start time t ℓ . We will then show that, with respect to any arm a , not too many bad segments will elapse before a is evicted from A master . In particular, this will hold for a = a ℓ which will ultimately be used to bound δ t ( a ♯ t , a ℓ ) across the episode [ t ℓ , t ℓ +1 ) .

Definition 5. Fix the episode start time t ℓ , and let [ τ i , τ i +1 ) be any phase intersecting [ t ℓ , T ) . For any arm a , define rounds s i, 0 ( a ) , s i, 1 ( a ) , s i, 2 ( a ) . . . ∈ [ t ℓ ∨ τ i , τ i +1 ) recursively as follows: let s i, 0 ( a ) := t ℓ ∨ τ i and define s i,j ( a ) as the smallest round in ( s i,j -1 ( a ) , τ i +1 ) such that arm a satisfies for some fixed c 4 > 0 :

if such a round s i,j ( a ) exists. Otherwise, we let the s i,j ( a ) := τ i +1 -1 . We refer to any interval [ s i,j -1 ( a ) , s i,j ( a )) as a critical segment , and as a bad segment (w.r.t. arm a ) if (14) above holds.

Note that the above definition only depends on the arm a and the episode start time t ℓ and that, conditional on these variables, they are fixed in the environment. Observe also that the arm a ♯ t is fixed within any critical segment [ s i,j -1 ( a ) , s i,j ( a )) ⊆ [ τ i , τ i +1 ) since a significant shift does not occur inside [ τ i , τ i +1 ) .

Now relating this notion of a bad segment to our goal of bounding regret, a given bad segment [ s i,j ( a ) , s i,j ( a )) only contributes order √ K · ( s i,j ( a ) -s i,j -1 ( a )) to the regret of a to a ♯ t . At the same time, we claim that a well-timed replay (see Definition 6 below) running from s i,j -1 ( a ) to s i,j ( a ) is capable of evicting arm a . This in turn allows us to reduce the regret bounding to studying the number and lengths of bad segments which elapse before one is detected by such a replay.

We first define such a well-timed and perfect replay.

Definition 6. Let ˜ s i,j ( a ) := ⌈ s i,j ( a )+ s i,j +1 ( a ) 2 ⌉ denote the approximate midpoint of [ s i,j ( a ) , s i,j +1 ( a )) . Given a bad segment [ s i,j ( a ) , s i,j +1 ( a )) , define a perfect replay w.r.t. [ s i,j ( a ) , s i,j +1 ( a )) as a call of Base-Alg ( t start , m ) where t start ∈ [ s i,j ( a ) , ˜ s i,j ( a )] and m ≥ s i,j +1 ( a ) -s i,j ( a )

Next, we analyze the behavior of a perfect replay on the bad segment [ s i,j ( a ) , s i,j +1 ( a )) . Going forward, we will use the simpler notation a ♯ i to denote the last safe arm of a phase [ τ i , τ i +1 ) , known in context.

Proposition 14. Suppose the good event E 1 holds (cf. Proposition 10). Let [ s i,j ( a ) , s i,j +1 ( a )) be a bad segment with respect to arm a . Fix an integer m ≥ s i,j +1 ( a ) -s i,j ( a ) . Then, if a perfect replay with respect to [ s i,j ( a ) , s i,j +1 ( a )) is scheduled, arm a will be evicted from A master by round s i,j +1 ( a ) .

Proof. Suppose event E 1 (i.e., our concentration bound (9)) holds. We first observe that by elementary calculations and the definition of the rounds s i,j ( a ) , we have (in an identical fashion to Lemma 4 of Suk and Kpotufe [2022]):

where ˜ s i,j ( a ) is the midpoint of [ s i,j ( a ) , s i,j +1 ( a )) as defined in Definition 6. The above will come in handy in showing arm a is evicted over the second half of the bad segment [˜ s i,j ( a ) , s i,j +1 ( a )] .

Next, following the intuition given in Section 4, in order to relate δ t ( a ♯ i , a ) to δ t (ˆ a t , a ) , we again use SST and STI via Lemma 13 on inequality (15):

We next show that ∑ s i,j +1 ( a ) t =˜ s i,j ( a ) δ t ( a ♯ i , ˆ a t ) and ∑ s i,j +1 ( a ) t =˜ s i,j ( a ) δ t ( a ∗ t , a ♯ i ) on the above LHS are small.

First, it is clear that any perfect replay Base-Alg ( t start , m ) will not evict a ♯ i since otherwise it incurs significant regret within phase [ τ i , τ i +1 ) (see the earlier Lemma 11). At the same time, by the candidate arm switching criterion (4) and concentration:

Meanwhile, by the definition of significant regret (Definition 3),

Thus, for sufficiently large c 4 > 0 in the definition of bad segments (Definition 5), we have that the above two inequalities can be combined with (16) to yield:

If arm a is evicted from A master before round s i,j +1 ( a ) , then we are already done. Otherwise, using the fact that E [ ˆ δ t (ˆ a t , a ) |F t -1 ] = δ t (ˆ a t , a ) for any round t ∈ [˜ s i,j ( a ) , s i,j +1 ( a )] with a ∈ A t , we have that arm a will be evicted at round s i,j +1 ( a ) using the above inequality and concentration.

It remains to show that, for any arm a , a perfect replay is scheduled w.h.p. before too much regret is incurred on the elapsed bad segments w.r.t. a . In particular, this will hold for the last master arm a ℓ , allowing us to bound the remaining term E [ ∑ t ℓ +1 -1 t = t ℓ δ t ( a ♯ t , a ℓ )] . The argument will be identical to that of Appendix B.2 of Suk and Kpotufe [2022].

First, fix an arm a and an episode start time t ℓ . Then, define the bad round s ( a ) > t ℓ as follows:

Definition 7. (bad round) For a fixed round t ℓ and arm a , the bad round s ( a ) > t ℓ is defined as the smallest round which satisfies, for some fixed c 6 > 0 :

where the above sum is over all pairs of indices ( i, j ) ∈ N × N such that [ s i,j ( a ) , s i,j +1 ( a )) is a bad segment with s i,j +1 ( a ) < s ( a ) .

Our goal is then to then to show that arm a is evicted by some perfect replay scheduled within episode [ t ℓ , t ℓ +1 ) with high probability before the bad round s ( a ) occurs. Going forward, to simplify notation we will drop the dependence on the fixed arm a in some variables.

For each bad segment [ s i,j ( a ) , s i,j +1 ( a )) , recall that ˜ s i.j ( a ) is the approximate midpoint between s i,j ( a ) and s i,j +1 ( a ) (see Definition 6). Next, let m i,j := 2 n where n ∈ N satisfies:

Plainly, m i,j is a dyadic approximation of the bad segment length. Next, recall that the Bernoulli B t,m decides whether Base-Alg ( t, m ) is scheduled at round t (see Line 6 of Algorithm 1). If for

some t ∈ [ s i,j ( a ) , ˜ s i,j ( a )] , B t,m i,j = 1 , i.e. a perfect replay is scheduled, then a will be evicted from A master by round s i,j +1 ( a ) (Proposition 14). We will show this happens with high probability via concentration on the sum

Note that the random variable S ( a, t ℓ ) only depends on the replay scheduling probabilities { B s,m } s,m given a fixed arm a and episode start time t ℓ , since the bad round s ( a ) is also fixed given these quantities. This means that S ( a, t ℓ ) is an independent sum of Bernoulli random variables B t,m i,j , conditional on t ℓ . Then, a Chernoff bound over the randomness of S ( a, t ℓ ) , conditional on t ℓ yields

The above RHS error probability is bounded above above by 1 /T 3 by observing:

for c 6 > 0 large enough, where the last inequality follows from (17) in the definition of the bad round s ( a ) (Definition 7). Taking a further union bound over the choice of arm a ∈ [ K ] gives us that S ( a, t ℓ ) > 1 for all choices of arm a (define this as the good event E 2 ( t ℓ ) ) with probability at least 1 -K/T 3 . This means arm a will be evicted before round s ( a ) with high probability.

Recall on the event E 1 the concentration bounds of Proposition 10 hold. Then, on E 1 ∩E 2 ( t ℓ ) , letting a = a ℓ in the preceding arguments we must have t ℓ +1 -1 ≤ s ( a ℓ ) Thus, by the definition of the bad round s ( a ℓ ) (Definition 7), we must have:

Thus, by (14) in the definition of bad segments (Definition 5), over the bad segments [ s i,j ( a ℓ ) , s i,j +1 ( a ℓ )) which elapse before the end of the episode t ℓ +1 -1 , the regret of a ℓ to a ♯ t is at most order log 2 ( T ) √ K · ( t ℓ +1 -t ℓ ) .

Over each non-bad critical segment [ s i,j ( a ℓ ) , s i,j +1 ( a ℓ )) , the regret of playing arm a ℓ to a ♯ i is at most log( T ) √ τ i +1 -τ i since there is at most one non-bad critical segment per phase [ τ i , τ i +1 ) (follows from Definition 5).

So, we conclude that on event E 1 ∩ E 2 ( t ℓ ) :

Taking expectation, we have by conditioning first on t ℓ and then on event E 1 ∩ E 2 ( t ℓ ) :

where in the last step we bound 1 {E 1 ∩ E 2 ( t ℓ ) } ≤ 1 {E 1 } and apply tower law again. This concludes the proof. ■

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the non-stationary multi-armed dueling bandit under the SST and STI conditions. The paper first shows a hardness result that SST+STI are necessary for sub-linear regret, while merely SST or STI cannot guarantee sublinear regrets. Then, the paper studies under SST+STI conditions and propose a new adaptive algorithm that achieves $\\tilde{O}(\\sqrt{\\tilde{L}KT})$ regret. This regret upper bound improves upon the previous one $\\tilde{O}(K\\sqrt{\\tilde{L}T})$.\n\nSTRENGTHS:\nThe paper is well-written and relatively easy for me to follow. The lower bound result is interesting in that it reveals that the Condorcet winner condition is not sufficient for efficient learning in the non-stationary environment. Instead, both SST and STI must hold to allow a sub-linear regret. For the upper bound, a new algorithm along with its analysis is provided to achieve an improved regret upper bound. I find the explanation and reasoning for such an algorithm framework to be quite informative. The analysis of why previous works don't have better dependence on $K$ (Section 5.1) shows the algorithm design is original.\n\nWEAKNESSES:\n1. While the impossibility result shows that SST+STI are necessary, it is not clear what is the minimax lower bound under such conditions. Is it $\\Omega(\\sqrt{LKT})$ or $\\Omega(\\sqrt{ \\tilde{L}KT})$? It is not discussed in this work what is the lower bound in terms of significant shifts. Therefore, it is not well-supported to claim the upper bound is optimal.\n\nQUESTIONS:\n1. As mentioned in the section above. What is the lower bound for the setting in this paper? In the paper, it is mentioned that one lower bound is $\\Omega(\\sqrt{LKT})$, while your upper bound is $\\Omega(\\sqrt{\\tilde{L}KT})$ with $\\tilde{L} << L$. This is contradicting without some context.\n\n2. There are a series of related works discussing ranking with/without SST or STI conditions. Could you also comment on these works:\n\n    * On Sample Complexity Upper and Lower Bounds for Exact Ranking from Noisy Comparisons (NeurIPS'19) Ren et al.\n    * Active Ranking without Strong Stochastic Transitivity (NeurIPS'22) Lou et al.\n\nLIMITATIONS:\nN/A due to its theoretical nature.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles a regret minimizing problem in a dueling bandits scenario, where the underlying preference probabilities are non-stationary but instead shifting over time. In case these preference probabilities fulfill SST (strong stochastic transitivity) and STI (stochastic triangle inequality) at any time, the authors present a novel upper bound on the dynamic regret that depends on the (unknown) number of so-called significant shifts, the number of arms and the time horizon. Moreover, they provide evidence that leaving out one of the assumptions SST and STI would result in a learning scenario where suffering linear regret is in a worst case unavoidable.\n\nSTRENGTHS:\nThis paper presents a novel algorithm for the non-stationary dueling bandits problem, which is able to learn without knowing the number of significant shifts. It comes with an upper bound on the dynamic regret. This bound and the hardness result on the learnability in case one of SST and STI are violated (Thm 3) are highly non-trivial results. Mostly, the paper and proofs are written in a convenient way.\n\nWEAKNESSES:\n- Unfortunately, this paper does not provide any experiments. Why not?\n- You say (e.g. in your conclusion and in 174ff.) that one cannot achieve a dynamic regret of order $O(\\sqrt{K\\tilde{L}T})$ outside of $SST \\cap STI$. However, you only prove that non-linear dynamic regret can not be avoided in a worst-case sense (!) under $STI \\setminus SST$ and under $SST \\setminus STI$. This (a) does not formally imply any worst-case hardness result in case none of SST and STI is fulfilled, right?, and (b) does not exclude existence of a class X' outside $SST\\cap STI$ where  $O(\\sqrt{K\\tilde{L}T})$ is achievable. \n- 162: The constant C in (3) is required to implement and execute Algorithm 2. Do you already provide it somewhere? If so, please state it here, otherwise, provide it. \n- 516: $c_1$ is not given, so $\\mathcal{E}_1$ is not well-defined.\n- Sec. 3 is called 'Hardness of significant shifts...'. However, Thm. 3 shows a negative result for a scenario with no significant shifts. How are the section title and Thm. 3 related?\n\nMinor remarks, questions and typos:\n- 135: Why do you write $\\succeq$ instead of $\\succ$?\n- 196: It seems as if METASWIFT has not been properly introduced yet.\n- 470: KL [divergence]\n- 477: \")\" missing\n- 488: observations\n- 506: Sec. 6?\n- 506: a_t^\\# is defined in Sec. 6, but not in the App. B\n- 536: Sec. 6\n- In Algo 2: Refer for the definition of $B_{s,m}$ to Algo 1\n\nQUESTIONS:\n- You have proven in Thm. 4 an upper regret bound. What could you say about a lower bound? How sharp is your bound? \n- 162: Can Thm. 3 be generalized to hold for fixed K>3?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work studies the problem of a non-stationary bandits problem. First of all, the hardness of dynamic regret minimization under significant shifts problem without SST+STI condition is discussed. Then, with such additional conditions, an algorithm is proposed with a regret of $O(\sqrt{KLT})$. An innovative arm selection strategy which maintains a set of low regret arms is used in replacement of those greedily picking the best arm so far in static environments.

STRENGTHS:
The proposed method differentiates itself with conventional arm selection strategy in stationary environments  by maintaining an acceptable level of regret of order $O(\sqrt{KT})$ and simultaneously monitoring the change of the environment.  This work may have significant use case compared to stationary algorithms given the fact the in real-life scenarios the environment is always changing.

WEAKNESSES:
Simulation study on two cases might deepen the understanding of the proposed method: Although orderwise the same, a comparison between stationary method and this work under static settings can reveal the overhead, and a comparison of the existing work such as [10] with the proposed method should verify the K factor. Given this line of research usually does not carry empirical result and is of theoretical nature. This is just a nice to have suggestion.

QUESTIONS:
See weakness.

LIMITATIONS:
The limitations are well discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of dueling bandits with distribution shifts in preferences. The authors investigate the possibility of designing an adaptive algorithm with dynamic regret that depends on the number of significant shifts in preferences. They provide an impossibility result for well-studied preference distributions and show that it is possible to design such an algorithm under certain conditions. The paper introduces novel algorithmic approaches, including the SWIFT and METASWIFT algorithms, and contributes to the understanding of regret bounds in switching dueling bandits. The authors also discuss the difficulty of efficient exploration in non-stationary dueling bandit problems and suggest the need for smarter exploration strategies to achieve optimal regret bounds. Overall, the paper presents theoretical results, algorithmic solutions, and future research directions in the field of dueling bandits.\n\nSTRENGTHS:\nThis paper introduces the novel problem of dueling bandits with distribution shifts in preferences, and proposes a unique adaptive algorithm with dynamic regret that accounts for the number of significant shifts in preferences. The authors provide a thorough theoretical analysis, rigorous proofs, and well-designed algorithmic approaches, such as the SWIFT and METASWIFT algorithms. The paper is well-written, organized, and presents clear explanations and detailed pseudocode, making it easy for readers to understand and replicate the experiments. The authors’ contributions advance the state-of-the-art in dueling bandits and provide valuable insights for researchers and practitioners. Overall, this paper demonstrates originality, quality, clarity, and significance in its contributions to the field of computer science.\n\nWEAKNESSES:\nThe paper would benefit from the inclusion of empirical evaluation to complement its theoretical analysis.  The authors should conduct extensive experiments on a wider range of datasets and problem instances to validate the performance of the proposed algorithms in practical scenarios.  Additionally, it would be valuable for the authors to provide insights into the computational complexity and scalability of the algorithms, as these are important aspects for assessing the practical relevance of their work.\n\nAnother weakness is the unclear explanation of the innovation in theoretical analysis. The authors mentioned in section 1.1 the innovation of their algorithm compared to previous work, but did not discuss how their theoretical analysis differs from existing techniques, especially in the proof analysis where some techniques from [10] and [30] were used. The authors should clearly explain the innovation in theoretical analysis and highlight the novelty and importance of their contributions in this aspect.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nThe paper does not explicitly discuss the limitations of the proposed algorithms or the potential negative consequences that may arise from their application.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}