SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, in our setting, adversarial but non-significant changes in the ordering of arms could force perpetual exploration of lowest-ranked arms. This suggests that learning an ordering should not be a subtask of our desiredueling base algorithm. Rather, the algorithm should prioritize minimizing its own regret over time. Keeping this intuition in mind, we introduce an algorithm called SWitching Interleaved FilTering (SWIFT) (see Algorithm 2 in Section 5.2) which directly tracks regret and avoids learning a fixed ordering of arms.
A new idea for switching candidate arms. A natural idea that is common to many dueling bandit algorithms (including IF) is to maintain a candidate arm  $\hat{a}$  which is always played at each round, and serves as a reference point for partially ordering other arms in contention. If the current candidate is beaten by another arm then a new candidate is chosen, and this process quickly converges to the best arm. Since the ordering of arms may change at each round, any such rule that relies on a fixed ordering is deemed to fail in our setting. Our procedure does not rely on such a fixed ordering over arms, but instead tracks the aggregate regret  $\sum_{t} \delta_{t}(a, \hat{a}_{t})$  of the changing sequence of candidate arms  $\{\hat{a}_{t}\}_{t}$  to another fixed arm  $a$ . Crucially, the candidate arm  $\hat{a}_{s}$  is always played at round  $s$  and so the history of candidate arms  $\{\hat{a}_{s}\}_{s \leq t}$  is fixed at a round  $t$ . This fact allows us to estimate the quantity  $\sum_{s=1}^{t} \delta_{s}(a, \hat{a}_{s})$  using importance-weighting at  $\sqrt{K \cdot t}$  rates via martingale concentration. An algorithmic switching criterion then switches the candidate arm  $\hat{a}_{t}$  to any arm  $a$  dominating the sequence  $\{\hat{a}_{s}\}_{s \leq t}$  over time, i.e.,  $\sum_{s=1}^{t} \delta_{s}(a, \hat{a}_{s}) \gg \sqrt{K \cdot t}$ . This simple, yet powerful, idea immediately gives us control of the regret of the candidate sequence  $\{\hat{a}_{t}\}_{t}$  which allows us to bypass the ranking-based arguments of vanilla IF and Beat-The-Mean. It also allows us to simultaneously bound the regret of a sub-optimal arm  $a$  against the sequence of candidate arms  $\sum_{s=1}^{t} \delta_{s}(\hat{a}_{s}, a)$ .
5.2 Switching Interleaved Filtering (SWIFT)
SWIFT at round  $t$  compares a candidate arm  $\hat{a}_t$  with an arm  $a_t$  (chosen uniformly at random) from an active arm set  $\mathcal{A}_t$ . Additionally, SWIFT maintains estimates  $\hat{\delta}_t(\hat{a}_t, a)$  of  $\delta_t(\hat{a}_t, a)$  which are used to (1) evict active arms  $a \in \mathcal{A}_t$  and (2) switch the candidate arm  $\hat{a}_{t+1}$  for the next round.
Estimators and Eviction/Switching Criteria. Let  $\mathcal{A}_t$  be the active arm set at round  $t$ . Let
$$
\hat {\delta} _ {t} (\hat {a} _ {t}, a) := | \mathcal {A} _ {t} | \cdot O _ {t} (\hat {a} _ {t}, a) \cdot \mathbf {1} \{(i _ {t}, j _ {t}) = (\hat {a} _ {t}, a) \} - 1 / 2, \tag {2}
$$
which is an unbiased estimator of the gap  $\delta_t(\hat{a}_t, a)$  when  $a \in \mathcal{A}_t$ . We evict an active arm  $a$  from  $\mathcal{A}_t$  at round  $t$  if for some constant  $C > 0^3$  and rounds  $s_1 < s_2 \leq t$ :
$$
\sum_ {s = s _ {1}} ^ {s _ {2}} \hat {\delta} _ {s} \left(\hat {a} _ {s}, a\right) \geq C \log (T) \sqrt {K \cdot \left(s _ {2} - s _ {1}\right) \vee K ^ {2}}, \tag {3}
$$
where  $\hat{\delta}_s(a,\hat{a}_s)\coloneqq -\hat{\delta}_s(\hat{a}_s,a)$ . Next, we switch the next candidate arm  $\hat{a}_{t + 1}\gets a$  to another arm  $a\in \mathcal{A}_t$  at round  $t$  if for some round  $s_1 < t$ :
$$
\sum_ {s = s _ {1}} ^ {t} \hat {\delta} _ {s} (a, \hat {a} _ {s}) \geq C \log (T) \sqrt {K \cdot (t - s _ {1}) \vee K ^ {2}}. \tag {4}
$$
SWIFT is formally shown in Algorithm 2, defined for generic start time  $t_{\mathrm{start}}$  and duration  $m_0$  so as to allow for recursive calls in our meta-algorithm framework.
5.3 Non-Stationary Algorithm (METASWIFT)
Algorithm 1: Meta-Elimination while Tracking Arms in SWIFT (METASWIFT)
Input: horizon  $T$    
1 Initialize: round count  $t\gets 1$    
2 Episode Initialization (setting global variables  $t_\ell ,A_{master},B_{s,m})$  .   
3  $\begin{array}{rlr}{t_{\ell}\leftarrow t.} & ; & \text{// } t_{\ell}\text{ indicates start of }\ell \text{-th episode.}\\ {\mathcal{A}_{master}\leftarrow [K]~;} & & \text{// Master active arm set.} \end{array}$    
4 For each  $m = 2,4,\ldots ,2^{\lceil \log (T)\rceil}$  and  $s = t_{\ell} + 1,\dots,T$  ..   
6 Sample and store  $B_{s,m}\sim$  Bernoulli  $\left(\frac{1}{\sqrt{m\cdot(s - t_{\ell})}}\right).$  . // Set replay schedule.   
7 Run Base-Alg  $(t_{\ell},T + 1 - t_{\ell})$    
8 if  $t <   T$  then restart from Line 2 (i.e. start a new episode).;
For the non-stationary setting with multiple (unknown) significant shifts, we run SWIFT as a base algorithm at randomly scheduled rounds and durations.
Our algorithm, dubbed METASWIFT and found in Algorithm 1, operates in episodes, starting each episode by playing a base algorithm instance of SWIFT. A running base algorithm activates its own base algorithms of varying durations (Line 8 of Algorithm 2), called replays according to a random schedule decided by the Bernoulli's  $B_{s,m}$  (see Line 6 of Algorithm 1). We refer to the (unique) base algorithm playing at round  $t$  as the active base algorithm.
Global Variables. The active arm set  $\mathcal{A}_t$  is pruned by the active base algorithm at round  $t$ , and globally shared between all running base algorithms. In addition, all other variables, i.e. the  $\ell$ -th episode start time  $t_\ell$ , round count  $t$ , schedule  $\{B_{s,m}\}_{s,m}$ , and candidate arm  $\hat{a}_t$  (and thus the quantities  $\delta_t(\hat{a}_t,a)$ ) are shared between base algorithms. Thus, while active, each Base-Alg can switch the candidate arm (4) and evict arms (3) over all intervals  $[s_1,s_2]$  elapsed since it began.
Note that only one base algorithm (the active one) can edit  $\mathcal{A}_t$  and set the candidate arm  $\hat{a}_t$  at round  $t$ , while other base algorithms can access these global variables at later rounds. By sharing these global
3The constant  $C > 0$  does not depend on  $T$ ,  $K$ , or  $\tilde{L}$ , and a suitable value can be derived from the regret analysis.
Algorithm 2: Base-Alg  $(t_{\mathrm{start}},m_0)$ : SWIFT starting at  $t_0$  and running  $m_0$  rounds
Input: starting round  $t_\mathrm{start}$  , scheduled duration  $m_0$  1 Initialize (Global) Variables:  $t\gets t_{\mathrm{start}},\mathcal{A}_t\gets [K],\hat{a}_t\gets \mathrm{Unif}\{[K]\} .$    
2 while  $t\leq T$  do   
3 Select a random arm  $a_{t}\in \mathcal{A}_{t}$  with probability  $1 / |\mathcal{A}_t|$  and play  $(\hat{a}_t,a_t)$    
4 Let  $\mathcal{A}_{\mathrm{current}}\leftarrow \mathcal{A}_t$  // Save current active arm set  $\mathcal{A}_t$  (global variable).   
5 Increment  $t\gets t + 1$    
6 if  $\exists m$  such that  $B_{t,m} > 0$  then \*/ See Algorithm 1 for definition of  $B_{s,m}*$    
7 Let  $m\coloneqq \max \{m\in \{2,4,\ldots ,2^{[\log (T)]}\} :B_{m,t} > 0\} .$  // Set replay length.   
8 Run Base-Alg(t,m).; // Replay interrupts.   
9 if  $t > t_{\mathrm{start}} + m_0$  then RETURN.;   
10 Evict bad arms:   
11  $\mathcal{A}_t\gets \mathcal{A}_{\mathrm{current}}\backslash \{a\in [K]:\exists \mathrm{rounds}[s_1,s_2]\subseteq [t_{\mathrm{start}},t)\mathrm{s.t.(3)}\mathrm{hold}\} .$    
12  $\mathcal{A}_{\mathrm{master}}\gets \mathcal{A}_{\mathrm{master}}\backslash \{a\in [K]:\exists \mathrm{rounds}[s_1,s_2]\subseteq [t_\ell ,t)\mathrm{s.t.(3)}\mathrm{hold}\} .$    
13 if (4) holds for some arm  $a\in \mathcal{A}_t$  then   
14 Switch candidate arm:  $\hat{a}_t\gets a$  // Set candidate arm  $\hat{a}_t$  (global variable).   
15 else   
16  $\hat{a}_t\gets \hat{a}_{t - 1}$    
17 Restart criterion: if  $\mathcal{A}_{\mathrm{master}} = \emptyset$  then RETURN.;   
18 RETURN.
variables, any replay can trigger a new episode: every time an arm is evicted by a replay, it is also evicted from the master arm set  $\mathcal{A}_{\mathrm{master}}$ , tracking arms' regret throughout the entire episode. A new episode is triggered when  $\mathcal{A}_{\mathrm{master}}$  becomes empty, i.e., there is no safe arm left to play.
6 Regret Analysis
6.1 Regret of METASWIFT over Significant Phases
Now, we turn to sketching the proof of Theorem 4. Full details are found in Appendix C.
Decomposing the Regret. Let  $a_{t}^{\sharp}$  denote the last safe arm at round  $t$ , or the last arm to incur significant regret in the unique phase  $[\tau_i, \tau_{i+1})$  containing round  $t$ . Then, we can decompose the dynamic regret around this safe arm using SST and STI (i.e., using Lemma 8 twice) as:
$$
\sum_ {t = 1} ^ {T} \delta_ {t} (a _ {t} ^ {*}, \hat {a} _ {t}) + \delta_ {t} (a _ {t} ^ {*}, a _ {t}) \leq 6 \sum_ {t = 1} ^ {T} \delta_ {t} (a _ {t} ^ {*}, a _ {t} ^ {\sharp}) + 3 \sum_ {t = 1} ^ {T} \delta_ {t} (a _ {t} ^ {\sharp}, \hat {a} _ {t}) + \sum_ {t = 1} ^ {T} \delta_ {t} (\hat {a} _ {t}, a _ {t}),
$$
where we recall that  $a_{t} \in \mathcal{A}_{t}$  is the other arm played (Line 3 of Algorithm 2). Next, the first sum on the above RHS is order  $\sum_{i=1}^{\tilde{L}} \sqrt{K \cdot (\tau_{i} - \tau_{i-1})}$  as the last safe arm  $a_{t}^{\sharp}$  does not incur significant regret on  $[\tau_{i}, \tau_{i+1})$ . So, it remains to bound the last two sums on the RHS above.
Episodes Align with Significant Phases. We claim that a new episode is triggered only if there is a significant shift occurs (Lemma 11). This follows from our eviction criteria (3) with Freedman's inequality for martingale concentration (Lemma 9). Then, acknowledging episodes roughly align with significant phases, we turn our attention to bounding the remaining regret in each episode.
Bounding Regret of an Episode. Let  $t_\ell$  be the start of the  $\ell$ -th episode of METASWIFT. Then, our goal is to show for all  $\ell \in [\hat{L}]$  (where  $\hat{L}$  is the random number of episodes used by the algorithm):
$$
\max  \left\{\mathbb {E} \left[ \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, \hat {a} _ {t}\right) \right], \mathbb {E} \left[ \sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(\hat {a} _ {t}, a _ {t}\right) \right] \right\} \lesssim \sum_ {i \in [ \tilde {L} + 1: [ \tau_ {i - 1}, \tau_ {i}) \cap [ t _ {\ell}, t _ {\ell + 1}) \neq \emptyset} \sqrt {K \cdot (\tau_ {i} - \tau_ {i - 1})}, \tag {5}
$$
where the RHS sum above is over the significant phases  $\left[\tau_{i-1}, \tau_i\right)$  overlapping episode  $[t_\ell, t_{\ell+1})$ . Summing over episodes  $\ell \in [\hat{L}]$  will then yield the desired total regret bound by our earlier observation that the episodes align with significant phases (see Lemma 11).
Bounding Regret of Active Arms to Candidate Arms. Bounding  $\sum_{t = t_{\ell}}^{t_{\ell +1} - 1}\delta_{t}(\hat{a}_{t},a_{t})$  follows in a similar manner as Appendix B.1 of Suk and Kpotufe [2022]. First, observe by concentration (Lemma 9) the eviction criterion (3) bounds the sums  $\sum_{t = s_1}^{s_2}\delta_t(\hat{a}_t,a)$  over intervals  $[s_1,s_2]$  where  $a$  is active. Then, accordingly, we further partition the episode rounds  $[t_\ell ,t_{\ell +1})$  into different intervals distinguishing the unique regret contributions of different active arms from varying base algorithms, on each of which we can relate the regret to our eviction criterion. Details can be found in Appendix C.3.
- Bounding Regret of Candidate Arm to Safe Arm. The first sum on the LHS of (5) will be further decomposed using the last master arm  $a_{\ell}$  which is the last arm to be evicted from the master arm set  $\mathcal{A}_{\mathrm{master}}$  in episode  $[t_{\ell}, t_{\ell + 1})$ . Carefully using SST and STI (see Lemma 13), we further decompose  $\delta_t(a_t^\sharp, \hat{a}_t)$  as:
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp}, \hat {a} _ {t}\right) \leq 2 \underbrace {\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {\sharp} , a _ {\ell}\right)} _ {\mathrm {A}} + \underbrace {\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {\ell} , \hat {a} _ {t}\right)} _ {\mathrm {B}} + 3 \underbrace {\sum_ {t = t _ {\ell}} ^ {t _ {\ell + 1} - 1} \delta_ {t} \left(a _ {t} ^ {*} , a _ {t} ^ {\sharp}\right)} _ {\mathrm {C}} \tag {6}
$$
The sum C above was already bounded earlier. So, we turn our attention to B and A.
- Bounding B. Note that the arm  $a_{\ell}$  by definition is never evicted by any base algorithm until the end of the episode  $t_{\ell + 1} - 1$ . This means that at round  $t \in [t_{\ell}, t_{\ell + 1})$ , the quantity  $\sum_{s = t_{\ell}}^{t} \hat{\delta}_s(a_{\ell}, \hat{a}_s)$  is always kept small by the candidate arm switching criterion (4). So, by concentration (Proposition 10), we have  $\sum_{s = t_{\ell}}^{t_{\ell + 1} - 1} \hat{\delta}_s(a_{\ell}, \hat{a}_s) \lesssim \sqrt{K(t_{\ell + 1} - t_{\ell})}$ .
- Bounding A The main intuition here, similar to Appendix B.2 of Suk and Kpotufe [2022], is that well-timed replays are scheduled w.h.p. to ensure fast detection of large regret of the last master arm  $a_{\ell}$ . Key in this is the notion of a bad segment of time: i.e., an interval  $[s_1,s_2]\subseteq [\tau_i,\tau_{i + 1})$  lying inside a significant phase with last safe arm  $a^{\sharp}$  where:
$$
\sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(a ^ {\sharp}, a _ {\ell}\right) \gtrsim \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)}. \tag {7}
$$
For a fixed bad segment  $[s_1, s_2]$ , the idea is that a fortuitously timed replay scheduled at round  $s_1$  and remaining active till round  $s_2$  will evict arm  $a_\ell$ .
It is not immediately obvious how to carry out this argument in the dueling bandit problem since, to detect large  $\sum_{t}\delta_{t}(a^{\sharp},a_{\ell})$ , the pair of arms  $a^{\sharp},a_{\ell}$  need to both be played which, as we discussed in Section 5.1, may not occur often enough to ensure tight estimation of the gaps.
Instead, we carefully make use of SST/STI to relate  $\delta_t(a^\sharp, a_\ell)$  to  $\delta_t(\hat{a}_t, a_\ell)$ . Note this latter quantity controls both the eviction (3) and  $\hat{a}_t$  switching (4) criteria. This allows us to convert bad intervals with large  $\sum_{t} \delta_t(a_t^\sharp, a_\ell)$  to intervals with large  $\sum_{t} \delta_t(\hat{a}_t, a_\ell)$ . Specifically, by Lemma 13, we have that (7) implies
$$
2 \sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(a ^ {\sharp}, \hat {a} _ {t}\right) + \sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(\hat {a} _ {t}, a _ {\ell}\right) + 3 \sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(a _ {t} ^ {*}, a ^ {\sharp}\right) \gtrsim \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)}. \tag {8}
$$
Then, we claim that, so long as a base algorithm  $\operatorname{Base - Alg}(s_1, m)$  is scheduled from  $s_1$  running till  $s_2$ , we will have  $\sum_{t = s_1}^{s_2} \delta_t(\hat{a}_t, a_\ell) \gtrsim \sqrt{K \cdot (s_2 - s_1)}$  which implies  $a_\ell$  will be evicted. In other words, the second sum dominates the first and third sums in (8). We repeat earlier arguments to show this:
- By the definition of the last safe arm  $a^{\sharp}, \sum_{t=s_1}^{s_2} \delta_t(a_t^*, a^{\sharp}) < \sqrt{K \cdot (s_2 - s_1)}$ .
- Meanwhile,  $\sum_{t=s_1}^{s_2} \delta_t(a^\sharp, \hat{a}_t) \lesssim \sqrt{K \cdot (s_2 - s_1)}$  by the candidate switching criterion (4) and because  $a^\sharp$  will not be evicted before round  $s_2$  lest it incurs significant regret which cannot happen by definition of  $a^\sharp$ .
Combining the above two points with (8), we have that  $\sum_{t=s_1}^{s_2} \delta_t(\hat{a}_t, a_\ell) \gtrsim \sqrt{K \cdot (s_2 - s_1)}$ , which directly aligns with our criterion (3) for evicting  $a_\ell$ . To summarize, a bad segment  $[s_1, s_2]$  in the sense of (7) is detectable using a well-timed instance of SWIFT, which happens often enough with high probability. Concretely, we argue that not too many bad segments elapse before  $a_\ell$  is evicted by a well-timed replay in the above sense and that thus the regret incurred by  $a_\ell$  is bounded by the RHS of (5). The details can be found in Appendix C.5.
We consider the problem of switching dueling bandits where the distribution over preferences can change over time. We study a notion of significant shifts in preferences and ask whether one can achieve adaptive dynamic regret of  $O(\sqrt{K\tilde{L}T})$  where  $\tilde{L}$  is the number of significant shifts. We give a negative result showing that one cannot achieve such a result outside of the SST∩STI setting, and answer this question in the affirmative under the SST∩STI setting. In the future, it would be interesting to consider other notions of shifts which are weaker than the notion of significant shift, and ask whether adaptive algorithms for the Condorcet setting can be designed with respect to these notions. Buening and Saha [2022] already give a  $O(K\sqrt{ST})$  bound for the Condorcet setting, where  $S$  is the number of changes in 'best' arm. However, their results have a suboptimal dependence on  $K$  due to reduction to "all-pairs" exploration.
We thank Samory Kpotufe for helpful discussions. We also acknowledge computing resources from Columbia University's Shared Research Computing Facility project, which is supported by NIH Research Facility Improvement Grant 1G20RR030893-01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.
Yasin Abbasi-Yadkori, Andras Gyorgy, and Nevena Lazic. A new look at dynamic regret for non-stationary stochastic bandits. arXiv preprint arXiv:2201.06532, 2022.
Arpit Agarwal, Rohan Ghuge, and Viswanath Nagarajan. Batched dueling bandits. In Proceedings of the 39th International Conference on Machine Learning, pages 89-110, 2022.
Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing Dueling Bandits to Cardinal Bandits. In Proceedings of the 31st International Conference on Machine Learning, 2014.
Robin Allesiardo, Raphaël Féraud, and Odalric-Embrym Maillard. The non-stationary stochastic multi-armed bandit problem. International Journal of Data Science and Analytics, 3(4):267-283, 2017.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The Nonstochastic Multiarmed Bandit Problem. SIAM Journal on Computing, 32(1):48-77, 2002.
Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best arm with an unknown number of distribution changes. 14th European Workshop on Reinforcement Learning (EWRL), 2018.
Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA, volume 99 of Proceedings of Machine Learning Research, pages 138-158. PMLR, 2019.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with nonstationary rewards. Advances in Neural Information Processing Systems, 27:199-207, 2014.
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pages 19-26, 2011.
Aadirupa Saha and Shubham Gupta. Optimal and efficient dynamic regret algorithms for nonstationary dueling bandits. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 19027-19049. PMLR, 2022.
Aadirupa Saha, Tomer Koren, and Yishay Mansour. Adversarial dueling bandits. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9235-9244. PMLR, 2021.
Yanan Sui, Masour Zoghi, Katja Hofmann, and Yisong Yue. Advancements in dueling bandits. In Jérôme Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 5502-5510. ijcai.org, 2018.
Joe Suk and Samory Kpotufe. Tracking most significant arm switches in bandits. In Conference on Learning Theory, pages 2160-2182. PMLR, 2022.
Tanguy Urvoy, Fabrice Clerot, Raphael Feraud, and Sami Naamane. Generic Exploration and K-armed Voting Bandits. In Proceedings of the 30th International Conference on Machine Learning, 2013.
Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 32nd International Conference on Learning Theory, 2021.
Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 649-657, 2016.
Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proceedings of the 28th International Conference on Machine Learning, 2011.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The  $k$ -armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1556, 2012a.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1556, 2012b. ISSN 0022-0000. doi: https://doi.org/10.1016/j.jcss.2011.12.028. JCSS Special Issue: Cloud Computing 2011.
Masour Zoghi, Shimon Whiteson, Remi Munos, and Maarten de Rijke. Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem. In Proceedings of the 31st International Conference on Machine Learning, 2014.
Masour Zoghi, Zohar Karnin, Shimon Whiteson, and Maarten de Rijke. Copeland Dueling Bandits. In Advances in Neural Information Processing Systems 28, 2015a.
Masour Zoghi, Shimon Whiteson, and Maarten de Rijke. MergeRUCB: A method for large-scale online ranker evaluation. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining, 2015b.
Figure 2: Plots of dynamic regret curves over time.
Algorithm	Mean Regret	Standard Deviation
S=0 Changes	METASWIFT	3048	600
ANACONDA	4863	707
IF	140	46
RANDDUEL	18688	30
S=4 Changes	METASWIFT	3346	531
ANACONDA	5331	705
IF	14142	3739
RANDDUEL	18684	23
Table 1: Table of total dynamic regrets.
Code can be found at https://github.com/joesuk/nonstationary-duel.
Synthetic Environments. We used a geometric BTL model where the arms are linearly ordered and the  $i$ -th best arm beats the  $j$ -th best arm with probability  $\mathbb{P}(i\succ j) = \frac{2^{-i}}{2^{-j} + 2^{-i}}$ . At each changepoint, the ordering of arms was randomly permuted, with a total  $T = 50,000$  rounds with  $K = 10$  arms. Regret was computed over  $N = 50$  trials of each environment with standard confidence bands shown.
Algorithms. We considered four algorithms: (1) METASWIFT (Algorithm 1, (2) the ANACONDA algorithm of Buening and Saha [2022] (3) Interleaved Filtering (which we abbreviate as IF) as specified by Yue et al. [2012b], and a baseline (3) RANDDUEL which naively plays a pair of arms selected uniformly at random every round.
Parameters. Parameters associated with each of the algorithms (e.g., the constants in displays (3) and (4), analogous quantities in ANACONDA, and IF's elimination threshold) were tuned using cross validation on randomly generated geometric BTL environments with number of changepoints varying from 0 to 1000. For fairness, all algorithms were given the chance to tune parameters on the same environments before testing.