SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arpit Agarwal
aa4931@columbia.edu
The  $K$ -armed dueling bandits problem, where the feedback is in the form of noisy pairwise preferences, has been widely studied due its applications in information retrieval, recommendation systems, etc. Motivated by concerns that user preferences/tastes can evolve over time, we consider the problem of dueling bandits with distribution shifts. Specifically, we study the recent notion of significant shifts [Suk and Kpotufe, 2022], and ask whether one can design an adaptive algorithm for the dueling problem with  $O(\sqrt{K\tilde{L}T})$  dynamic regret, where  $\tilde{L}$  is the (unknown) number of significant shifts in preferences. We show that the answer to this question depends on the properties of underlying preference distributions. Firstly, we give an impossibility result that rules out any algorithm with  $O(\sqrt{K\tilde{L}T})$  dynamic regret under the well-studied Condorcet and SST classes of preference distributions. Secondly, we show that SST∩STI is the largest amongst popular classes of preference distributions where it is possible to design such an algorithm. Overall, our results provide an almost complete resolution of the above question for the hierarchy of distribution classes.
The  $K$ -armed dueling bandits problem has been well-studied in the multi-armed bandits literature [Yue and Joachims, 2011, Yue et al., 2012b, Urvoy et al., 2013, Ailon et al., 2014, Zoghi et al., 2014, 2015a,b, Dudik et al., 2015, Jamieson et al., 2015, Komiyama et al., 2015, 2016, Ramamohan et al., 2016, Chen and Frazier, 2017, Saha and Gaillard, 2022, Agarwal et al., 2022]. In this problem, on each trial  $t \in [T]$ , the learner pulls a pair of arms and observes relative feedback between these arms indicating which arm was preferred. The feedback is typically stochastic, drawn according to a pairwise preference matrix  $\mathbf{P} \in [0,1]^{K \times K}$ , and the regret measures the 'sub-optimality' of arms with respect to a 'best' arm.
This problem has many applications, e.g. information retrieval, recommendation systems, etc, where relative feedback between arms is easy to elicit, while real-valued feedback is difficult to obtain or interpret. For example, a central task for information retrieval algorithms is to output a ranked list of documents in response to a query. The framework of online learning has been very useful for automatic parameter tuning, i.e. finding the best parameter(s), for such retrieval algorithms based on user feedback [Liu, 2009]. However, it is often difficult to get numerical feedback for an individual list of documents. Instead, one can (implicitly) compare two lists of documents by interleaving them and observing the relative number of clicks [Radlinski et al., 2008]. The availability of these pairwise comparisons allows one to tune the parameters of retrieval algorithms in real-time using the framework ofueling bandits.
However, in many such applications that rely on user generated preference feedback, there are practical concerns that the tastes/beliefs of users can change over time, resulting in a dynamically
Figure 1: The hierarchy of distribution classes. The dark region is where  $O(\sqrt{K\tilde{L}T})$  dynamic regret is not achievable, whereas the light region indicates achievability (e.g., by our Algorithm 1).
changing preference distribution. Motivated by these concerns, we consider the problem of switching dueling bandits (or non-stationary dueling bandits), where the pairwise preference matrix  $\mathbf{P}_t$  changes an unknown number of times over  $T$  rounds. The performance of the learner is evaluated using dynamic regret where sub-optimality of arms is calculated with respect to the current 'best' arm.
Saha and Gupta [2022] first studied this problem and provided an algorithm that achieves a nearly optimal (up to log terms) dynamic regret of  $\tilde{O} (\sqrt{KLT})$  where  $L$  is the total number of shifts in the preference matrix, i.e., the number of times  $\mathbf{P}_t$  differs from  $\mathbf{P}_{t + 1}$ . However, this result requires algorithm knowledge of  $L$ . Alternatively, the algorithm of Saha and Gupta [2022] can be tuned to achieve a dynamic regret rate (also nearly optimal)  $\tilde{O} (V_T^{1 / 3}K^{1 / 3}T^{2 / 3})$  in terms of the total-variation of change in preferences  $V_{T}$  over  $T$  total rounds. This is similarly limited by requiring knowledge of  $V_{T}$ .
On the other hand, recent works on the switching MAB problem show it is not only possible to design adaptive algorithms with  $\tilde{O} (\sqrt{KLT})$  dynamic regret without knowledge of the underlying environment [Auer et al., 2019], but also possible to achieve a much better bound of  $\tilde{O} (\sqrt{K\tilde{L}T})$  where  $\tilde{L}\ll L$  is the number of significant shifts [Suk and Kpotufe, 2022]. Specifically, a shift is significant when there is no 'safe' arm left to play, i.e., every arm has, on some interval  $[s_1,s_2]$ , regret order  $\Omega (\sqrt{s_2 - s_1})$ . Such a weaker measure of non-stationarity is appealing as it captures the changes in best-arm which are most severe, and allows for more optimistic regret rates over the previously known  $\sqrt{KLT}\wedge (V_TK)^{1 / 3}T^{2 / 3}$ .
Very recently, Buening and Saha [2022] considered an analogous notion of significant shifts for switching dueling bandits under the  $\mathrm{SST} \cap \mathrm{STI}^1$  assumption. They gave an algorithm that achieves a dynamic regret of  $\tilde{O}(K\sqrt{\tilde{L}T})$ , where  $\tilde{L}$  is the (unknown) number of significant shifts. However, their algorithm estimates  $\Omega(K^2)$  pairwise preferences, and hence, suffers from a sub-optimal dependence on  $K$ .
In this paper we consider the goal of designing optimal algorithms for switching dueling bandits whose regret depends on the number of significant shifts  $\tilde{L}$ . We ask the following question:
Question. Is it possible to achieve a dynamic regret of  $O(\sqrt{K\tilde{L}T})$  without knowledge of  $\tilde{L}$ ?
We show that the answer to this question depends on conditions on the preference matrices. Specifically, we consider several well-studied conditions from the dueling bandits literature, and give an almost complete resolution of the achievability of  $O(\sqrt{K\tilde{L}T})$  dynamic regret under these conditions.
We first consider the classical Condorcet winner (CW) condition where, at each time  $t \in [T]$ , there is a 'best' arm under the preference  $\mathbf{P}_t$  that stochastically beats every other arm. Such a winner arm is a benchmark in defining the aforementioned dueling dynamic regret. Our first result shows that, even under the CW condition, it is in general impossible to achieve  $O(\sqrt{K\tilde{L}T})$  dynamic regret.
Theorem 1. (Informal) There is a family of instances  $\mathcal{F}$  under Condorcet where all shifts are non-significant, i.e.  $\tilde{L} = 0$ , but no algorithm can achieve  $o(T)$  dynamic regret uniformly over  $\mathcal{F}$ .
Note that in the case when  $\tilde{L} = 0$ , one would ideally like to achieve a dynamic regret of  $O(\sqrt{KT})$ . The above theorem shows that, under the Condorcet condition when  $\tilde{L} = 0$ , not only is it impossible
$^{1}$ SST∩STI imposes a linear ordering over arms and two well-known conditions on the preference matrices: strong stochastic transitivity (SST) and stochastic triangle inequality (STI).
to achieve  $O(\sqrt{KT})$  regret, it is even impossible to achieve  $O(T^{\alpha})$  regret for any  $\alpha < 1$ . Hence, this rules out the possibility of an algorithm whose regret under this condition is sublinear in  $\tilde{L}$  and  $T$ .
The proof of the above theorem relies on a careful construction where, at each time  $t$ , the preference  $\mathbf{P}_t$  is chosen uniformly at random from two different matrices  $\mathbf{P}^+$  and  $\mathbf{P}^-$ . These matrices have different 'best' arms but there is a unique safe arm in both. However, it is impossible to identify this safe arm as all observed pairwise preferences are  $\mathrm{Ber}\left(\frac{1}{2}\right)$  over the randomness of the environment. Moreover, the theorem gives two different constructions (one ruling out SST and one STI) which together rule out all preference classes outside of SST∩STI. Our second result shows that the desired regret  $\sqrt{K\tilde{L}T}$  is in fact achievable (adaptively) under SST∩STI.
Theorem 2. (Informal) There is an algorithm that achieves a dynamic regret of  $\tilde{O} (\sqrt{K\tilde{L}T})$  under SST $\cap$ STI without requiring knowledge of  $\tilde{L}$ .
Figure 1 gives a summary of our results. Note that in stationary dueling bandits there is no separation in the regret achievable under the CW vs. SST $\cap$ STI conditions, i.e.  $O(\sqrt{KT})$  is the minimax optimal regret rate under both conditions [Saha and Gaillard, 2022]. However, our results show that in the non-stationary setting with regret in terms of significant shifts, there is a separation in adaptively achievable regret.
Key Challenge and Novelty in Regret Upper Bound: To contrast, the recent work of Buening and Saha [2022] only attains  $\tilde{O}(K\sqrt{\tilde{L}T})$  dynamic regret under SST∩STI due to inefficient exploration of arm pairs. Our more challenging goal of obtaining the optimal dependence on  $K$  introduces key difficulties in algorithmic design. In fact, even in the classical stochastic dueling bandit problem with SST∩STI, most existing results that achieve  $O(\sqrt{KT})$  regret require identifying a coarse ranking over arms to avoid suboptimal exploration of low ranked arms [Yue et al., 2012a, Yue and Joachims, 2011]. However, in the non-stationary setting, ranking the arms meaningfully is difficult as the true ordering of arms may change (insignificantly) at all rounds. Our main algorithmic innovation is to bypass the task of ranking arms and instead directly focus on minimizing the cumulative regret of played arms. This entails a new rule for selecting "candidate" arms based on cumulative regret that may be of independent interest.
Dueling bandits. The stochastic dueling bandits problem and its variants have been studied widely (see Sui et al. [2018] for a comprehensive survey). This problem was first proposed by Yue et al. [2012b], who provide an algorithm achieving instance-dependent  $O(K \log T)$  regret under the SST∩STI condition. Yue and Joachims [2011] also studied this problem under the SST∩STI condition and gave an algorithm that achieves optimal instance-dependent regret. Urvoy et al. [2013] studied this problem under the Condorcet winner condition and achieved an instance-dependent  $O(K^2 \log T)$  regret bound, which was further improved by Zoghi et al. [2014] and Komiyama et al. [2015] to  $O(K^2 + K \log T)$ . Finally, Saha and Gaillard [2022] showed that it is possible to achieve an optimal instance-dependent bound of  $O(K \log T)$  and instance-independent bound of  $O(\sqrt{KT})$  under the Condorcet condition. More general notions of winners such as Borda winner [Jamieson et al., 2015], Copeland winner [Zoghi et al., 2015a, Komiyama et al., 2016, Wu and Liu, 2016], and von Nuemann winner [Dudik et al., 2015] have also been considered. However, these works only consider the stationary setting whereas we consider the non-stationary setting.
There has also been work on adversarial dueling bandits [Saha et al., 2021, Gajane et al., 2015], however, these works only consider static regret against the 'best' arm in hindsight and whereas we consider the harder dynamic regret. Other than the two previously mentioned works [Gupta and Saha, 2022, Buening and Saha, 2022], the only other work on switching dueling bandits is Kolpaczki et al. [2022], whose procedures require knowledge of non-stationarity and only consider the weaker measure of non-stationarity  $L$  counting all changes in the preferences.
Non-stationary multi-armed bandits. Multi-armed bandits with changing rewards was first considered in the adversarial setup by Auer et al. [2002], where a version of EXP3 was shown to attain optimal dynamic regret  $\sqrt{KL}T$  when properly tuned using the number  $L$  of changes in the rewards. Later works established similar (non-adaptive) guarantees in this so-called switching bandit problem via procedures inspired by stochastic bandit algorithms [Garivier and Moulines, 2011, Kocsis
and Szepesvári, 2006]. More recent works [Auer et al., 2018, 2019, Chen et al., 2019] established the first adaptive and optimal dynamic regret guarantees, without requiring knowledge of the number of changes. An alternative parametrization of switching bandits, via a total-variation quantity, was introduced in Besbes et al. [2014] with minimax rates quantified therein and adaptive rates attained in Chen et al. [2019]. Yet another characterization, in terms of the number of best arm switches  $S$  was studied in Abbasi-Yadkori et al. [2022], establishing an adaptive regret rate of  $\sqrt{SKT}$ . Around the same time, Suk and Kpotufe [2022] introduced the aforementioned notion of significant shifts and adaptively achieved rates of the form  $\sqrt{K\tilde{L}T}$  in terms of  $\tilde{L}$  significant shifts in rewards.
We consider non-stationary dueling bandits with  $K$  arms and time-horizon  $T$ . At round  $t \in [T]$ , the pairwise preference matrix is denoted by  $\mathbf{P}_t \in [0,1]^{K \times K}$ , where the  $(i,j)$ -th entry  $P_t(i,j)$  encodes the likelihood of observing a preference for arm  $i$  in a direct comparison with arm  $j$ . The preference matrix may change arbitrarily from round to round. At round  $t$ , the learner selects a pair of actions  $(i_t,j_t) \in [K] \times [K]$  and observes the feedback  $O_t(i_t,j_t) \sim \mathrm{Ber}(P_t(i_t,j_t))$  where  $P_t(i_t,j_t)$  is the underlying preference of arm  $i_t$  over  $j_t$ . We define the pairwise gaps  $\delta_t(i,j) \coloneqq P_t(i,j) - 1/2$ .
Conditions on Preference Matrix. We consider two different conditions on preference matrices: (1) the Condorcet winner (CW) condition and (2) the strong stochastic transitivity (SST) and stochastic triangle inequality (STI), formalized below.
Definition 1. (CW condition) At each round  $t$ , there is a Condorcet winner arm, denoted by  $a_{t}^{*}$  such that  $\delta_t(a_t^*, a) \geq 0$  for all  $a \in [K] \backslash \{a_t^*\}$ . Note that  $a_{t}^{*}$  need not be unique.
Definition 2. (SST $\cap$ STI condition) At each round  $t$ , there exists a total ordering on arms, denoted by  $\succ_{t}$ , and  $\forall i \succeq_{t} j \succeq_{t} k$ :
(a)  $\delta_t(i,k)\geq \max \{\delta_t(i,j),\delta_t(j,k)\}$  (SST).
(b)  $\delta_t(i,k)\leq \delta_t(i,j) + \delta_t(i,k)$  (STI).
It's easy to see that the SST condition implies the CW condition as  $\delta_t(i,j) \geq \delta_t(i,i) = 0$  for any  $i \succ_{t} j$ . Hence, the highest ranked item under  $\succ_{t}$  in Definition 2 is the CW  $a_{t}^{*}$ . We emphasize here that the CW in Definition 1 and the total ordering on arms in Definition 2 can change at each round, even while such unknown changes in preference may not be counted as significant (see below).
Regret Notion. Our benchmark is the dynamic regret to the sequence of Condorcet winner arms:
$$
\mathrm {D R} (T) := \sum_ {t = 1} ^ {T} \frac {\delta_ {t} \left(a _ {t} ^ {*} , i _ {t}\right) + \delta_ {t} \left(a _ {t} ^ {*} , j _ {t}\right)}{2}.
$$
Here, the regret of an arm  $i$  is defined in terms of the preference gap  $\delta_t(a_t^*, i)$  between the winner arm  $a_t^*$  and  $i$ , and the regret of the pair  $(i_t, j_t)$  is the average regret of individual arms  $i_t$  and  $j_t$ . Note the this regret is well-defined under both Condorcet and SST∩STI conditions due to the existence of a unique 'best' arm  $a_t^*$ , and is non-negative due to the fact that  $\delta_t(a_t^*, i) \geq 0$  for all  $i \in [K]$ .
Measure of Non-Stationarity. We first recall the notion of Significant Condorcet Winner Switches from Buening and Saha [2022], which captures only the switches in  $a_{t}^{*}$  which are severe for regret. Throughout the paper, we'll also refer to these as significant shifts for brevity.
Definition 3 (Significant CW Switches). Define an arm  $a$  as having significant regret over  $[s_1, s_2]$  if
$$
\sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} \left(a _ {s} ^ {*}, a\right) \geq \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)}. \tag {1}
$$
We then define significant CW switches recursively as follows: let  $\tau_0 = 1$  and define the  $(i + 1)$ -th significant CW switch  $\tau_{i + 1}$  as the smallest  $t > \tau_i$  such that for each arm  $a\in [K],\exists [s_1,s_2]\subseteq [\tau_i,t]$  such that arm  $a$  has significant regret over  $[s_1,s_2]$ . We refer to the interval of rounds  $[\tau_i,\tau_{i + 1})$  as a significant phase. Let  $\tilde{L}$  be the number of significant CW switches elapsed in  $T$  rounds.
Notation. To ease notation, we'll conflate the closed, open, and half-closed intervals of real numbers  $[a,b]$ ,  $(a,b)$ , and  $[a,b)$ , with the corresponding rounds contained therein, i.e.  $[a,b] \equiv [a,b] \cap \mathbb{N}$ .
3 Hardness of Significant Shifts in the Condorcet Winner Setting
We first consider regret minimization in an environment with no significant shift in  $T$  rounds. Such an environment admits a safe arm  $a^{\sharp}$  which does not incur significant regret throughout play. Our first result shows that, under the Condorcet condition, it is not possible to distinguish the identity of  $a^{\sharp}$  from other unsafe arms, which will in turn make sublinear regret impossible.
Theorem 3. For each horizon  $T$ , there exists a finite family  $\mathcal{F}$  of switching dueling bandit environments with  $K = 3$  that satisfies the Condorcet winner condition (Definition 1) with  $\tilde{L} = 0$  significant shifts. The worst-case regret of any algorithm on an environment  $\mathcal{E}$  in this family is lower bounded as
$$
\sup_{\mathcal{E}\in \mathcal{F}}\mathbb{E}_{\mathcal{E}}\left[ \mathrm{DR}(T)\right]\geq T / 8.
$$
Proof. (sketch; details found in Appendix B) Letting  $\epsilon \ll 1 / \sqrt{T}$ , consider the preference matrices:
$$
\mathbf {P} ^ {+} := \left( \begin{array}{c c c} 1 / 2 & 1 / 2 + \epsilon & 1 \\ 1 / 2 - \epsilon & 1 / 2 & 1 / 2 + \epsilon \\ 0 & 1 / 2 - \epsilon & 1 / 2 \end{array} \right), \mathbf {P} ^ {-} := \left( \begin{array}{c c c} 1 / 2 & 1 / 2 - \epsilon & 0 \\ 1 / 2 + \epsilon & 1 / 2 & 1 / 2 - \epsilon \\ 1 & 1 / 2 + \epsilon & 1 / 2 \end{array} \right).
$$
In  $\mathbf{P}^{+}$ , arm 1 is the Condorcet winner and  $1\succ 2\succ 3$ , whereas in  $\mathbf{P}^{-}$ , 3 is the winner with  $3\succ 2\succ 1$ . Let an oblivious adversary set  $\mathbf{P}_t$  at round  $t$  to one of  $\mathbf{P}^{+}$  and  $\mathbf{P}^{-}$ , uniformly at random, inducing an environment where arm 2 remains safe for  $T$  rounds. Then, any algorithm will, over the randomness of the adversary, observe  $O_{t}(i_{t},j_{t})\sim \mathrm{Ber}(1 / 2)$  no matter the choice of arms  $(i_t,j_t)$  played, by the symmetry of  $\mathbf{P}^{+},\mathbf{P}^{-}$ . Thus, it is impossible to distinguish arms, which implies linear regret by standard Pinsker's inequality arguments. In particular, even a strategy playing arm 2 every round fails as arm 2 is unsafe in another (indistinguishable) setup with arms 1 and 2 switched in  $\mathbf{P}^{+},\mathbf{P}^{-}$ .
SST and STI Both Needed To Learn Significant Shifts. The preferences  $\mathbf{P}^{+},\mathbf{P}^{-}$  in the above proof violate STI but satisfy SST, whereas another construction using preferences  $\mathbf{P}^{+},\mathbf{P}^{-}$  which violate SST but satisfy STI also works in the proof (see Remark 2 in Appendix B). This shows that sublinear regret is impossible outside of the class SST∩STI (visualized in Figure 1).
Remark 1. Note the lower bound of Theorem 3 does not violate the established upper bounds  $\sqrt{LT}$  and  $V_{T}^{1/3}T^{2/3}$  scaling with  $L$  changes in the preference matrix or total variation  $V_{T}$  [Gupta and Saha, 2022]. Our construction in fact uses  $L = \Omega(T)$  changes in the preference matrix and  $V_{T} = \Omega(T)$  total variation. Furthermore, the regret upper bound  $\sqrt{ST}$ , in terms of  $S$  changes in Condorcet winner, of Buening and Saha [2022] is not contradicted either, for  $S = \Omega(T)$ .
4 Dynamic Regret Upper Bounds under SST/STI
Acknowledging that significant shifts are hard outside of the class SST $\cap$ STI, we now turn our attention to the achievability of  $\sqrt{K\tilde{L}T}$  regret in the SST $\cap$ STI setting. Our main result is an optimal dynamic regret upper bound attained without knowledge of the significant shift times or the number of significant shifts. Up to log terms, this is the first dynamic regret upper bound with optimal dependence on  $T$ ,  $\tilde{L}$ , and  $K$ .
Theorem 4. Suppose SST and STI hold (see Definition 2). Let  $\{\tau_{ij}\}_{i=0}^{\tilde{L}}$  denote the unknown significant shifts of Definition 3. Then, for some constant  $C_0 > 0$ , Algorithm 1 has expected dynamic regret
$$
\mathbb {E} [ \mathbf {D R} (T) ] \leq C _ {0} \log^ {3} (T) \sum_ {i = 0} ^ {\tilde {L}} \sqrt {K \cdot (\tau_ {i + 1} - \tau_ {i})},
$$
and using Jensen's inequality, this implies a regret rate of  $C_0 \log^3 (T) \sqrt{K \cdot (\tilde{L} + 1) \cdot T}$ .
The lower bound construction of Saha and Gupta [2022] in fact uses  $\Omega(L)$  significant shifts so that the  $\sqrt{L \cdot \tilde{L}}$  rate is in fact minimax optimal
In fact, this regret rate can be transformed to depend on the Condorcet winner variation introduced in Buening and Saha [2022] and the total variation quantities introduced in Gupta and Saha [2022] and inspired by the total-variation quantity from non-stationary MAB [Besbes et al., 2014]. The following corollary is shown using just the definition of the non-stationarity measures.
Corollary 5 (Regret in terms of CW Variation). Let  $V_{T} \coloneqq \sum_{t=2}^{T} \max_{a \in [K]} |P_{t}(a_{t}^{*}, a) - P_{t-1}(a_{t}^{*}, a)|$  be the unknown Condorcet winner variation. Using the same notation of Theorem 4: Algorithm 1 has expected dynamic regret
$$
\mathbb {E} [ \mathrm {D R} (T) ] \leq C _ {0} \log^ {3} (T) \left(\sqrt {K T} + \left(K V _ {T}\right) ^ {1 / 3} T ^ {2 / 3}\right).
$$
5 Algorithm
At a high level, the strategy of recent works on non-stationary multi-armed bandits [Chen et al., 2019, Wei and Luo, 2021, Suk and Kpotufe, 2022] is to first design a suitable base algorithm and then use a meta-algorithm to randomly schedule different instances of this base algorithm at variable durations across time. The key idea is that unknown time periods of significant regret can be detected fast enough with the right schedule. In order to accurately identify significant shifts, the base algorithm in question should be robust to all non-significant shifts. In the multi-armed bandit setting, a variant of the classical successive elimination algorithm [Even-Dar et al., 2006] possesses such a guarantee [Allesiardo et al., 2017], and serves as a base algorithm in Suk and Kpotufe [2022].
5.1 Difficulty of Efficient Exploration of Arms.
In the non-stationary dueling problem, a natural analogue of successive elimination is to uniformly explore the arm-pair space  $[K] \times [K]$  and eliminate arms based on observed comparisons [Urvoy et al., 2013]. The previous work [Theorem 5.1 of Buening and Saha, 2022] employs such a strategy as a base algorithm. However, such a uniform exploration approach incurs a large estimation variance of  $K^2$ , which enters into the final regret bound of  $K\sqrt{T \cdot \tilde{L}}$ . Thus, smarter exploration strategies are needed to obtain  $\sqrt{K}$  dependence.
In the stationary dueling bandit problem with SST∩STI, such efficient exploration strategies have long been known: namely, the Beat-The-Mean algorithm [Yue and Joachims, 2011] and the Interleaved Filtering (IF) algorithm [Yue et al., 2012a]. We highlight that these existing algorithms aim to learn the ordering of arms, i.e., arms are ruled out roughly in the same order as their true underlying ordering. This fact is crucial to attaining the optimal dependence in  $K$  in their regret analyses, as the higher ranked arms must be played more often against other arms to avoid the  $K^2$  cost of exploration.