SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first graphic in Figure 2 shows the regret curves in a stationary environment with  $S = 0$  changes. The second graphic shows the regret curves in a non-stationary environment with  $S = 4$  changes. Exact mean and standard deviations on final regret are given in Table 1. These do support the theoretical message that METASWIFT performs better than the existing ANACONDA algorithm in non-stationary environments due to more efficient exploration of arms (demonstrated through  $\sqrt{K}$  versus  $K$  dependence in the theoretical bounds). Moreover, we also observe that the IF algorithm which is designed for stationary environments can have almost linear regret in non-stationary environments.
B Proof of Theorem 3
Consider the following preference matrices for some  $\epsilon > 0$  (to be chosen later):
In environment  $\mathbf{P}^{+}$ , arm 1 is the Condorcet winner and we have  $1\succ 2\succ 3$ . In environment  $\mathbf{P}^{-}$ , arm 3 is the winner with  $3\succ 2\succ 1$ .
Consider a uniform mixture  $\mathcal{U}$  of the preference matrices  $\mathbf{P}^{+}$  and  $\mathbf{P}^{-}$ , Let  $\mathcal{E}$  be a (random) sequence of  $T$  environments sampled i.i.d. from  $\mathcal{U}$ , with  $\mathbf{P}_t\coloneqq (\mathcal{E})_t$  being the sampled environment at round  $t$ .
First, it is straightforward to verify in every such switching dueling bandit  $\mathcal{E}$ , arm 2 does not incur significant regret over any interval of rounds  $[s_1,s_2]\subseteq [1,T]$ , for  $\epsilon < 1 / \sqrt{T}$ . Thus, every such  $\mathcal{E}$  exhibits zero significant shifts.
Next, in what follows, we use  $\mathbb{E}_{\mathcal{E}}[\cdot]$  to denote an expectation over both the randomness of  $\mathcal{U}^{\otimes T}$  and the algorithm's feedback and decisions. If there exists a realization of  $\mathcal{E}$  such that the algorithm gets expected regret at least  $T/8$ , then we are already done. Otherwise, we have the expected regret over the random environment  $\mathcal{E}$  is bounded above by  $T/8$ . Next, define the arm-pull counts  $N(T, a) := \sum_{t=1}^{T} \mathbf{1}\{i_t = a\} + \mathbf{1}\{j_t = a\}$  for each arm  $a$ . Then, we relate these arm-pull counts to the regret:
$$
\begin{array}{l} T / 8 > \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathcal {E}} \left[ \delta_ {t} \left(i ^ {*}, i _ {t}\right) + \delta_ {t} \left(i ^ {*}, j _ {t}\right) \right] \\ \geq \frac {1}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathcal {E}} [ (\mathbf {1} \{i _ {t} = 3 \} + \mathbf {1} \{j _ {t} = 3 \}) \cdot \mathbf {1} \{(\mathcal {E}) _ {t} = \mathbf {P} ^ {+} \} \\ + \left(\mathbf {1} \left\{i _ {t} = 1 \right\} + \mathbf {1} \left\{j _ {t} = 1 \right\}\right) \cdot \mathbf {1} \left\{\left(\mathcal {E}\right) _ {t} = \mathbf {P} ^ {-} \right\} ] \\ = \frac {1}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathcal {E}} \left[ \frac {1}{2} \cdot \left(\mathbf {1} \{i _ {t} = 3 \} + \mathbf {1} \{j _ {t} = 3 \} + \mathbf {1} \{i _ {t} = 1 \} + \mathbf {1} \{j _ {t} = 1 \}\right) \right] \\ \geq \frac {1}{4} \cdot \mathbb {E} _ {\mathcal {E}} [ N (T, 3) + N (T, 1) ], \\ \end{array}
$$
where we use the tower law in the third inequality (note that  $i_t, j_t$  are independent of  $(\mathcal{E})_t$ ). Thus, in expectation over both the model noise and randomness of  $\mathcal{E}$ , arms 3 and 1 cannot be played more than  $T/2$  times without causing linear regret.
Since  $\sum_{a=1}^{3} \mathbb{E}_{\mathcal{E}}[N(T, a)] = 2T$ , we conclude that  $\mathbb{E}_{\mathcal{E}}[N(T, 2)] \geq 3T/2$ . We will next show that arm 2 is statistically indistinguishable from arm 3. To do so, we consider an analogous environment which is identical to  $\mathcal{E}$  except the identities of arms 2 and 3 are switched. Specifically, let  $\mathcal{E}'$  be a random sequence of  $T$  environments sampled i.i.d. from a uniform mixture of  $\mathbf{Q}^+$  and  $\mathbf{Q}^-$ , which are respectively  $\mathbf{P}^+$  and  $\mathbf{P}^-$  with switched entries for arms 2 and 3.
We next claim  $\mathbb{E}_{\mathcal{E}}[N(T,2)] = \mathbb{E}_{\mathcal{E}'}[N(T,2)]$ . Admitting this claim, it immediately follows that the algorithm has expected regret (over the randomness of  $\mathcal{E}'$ ) at least (using an analogous chain of inequalities as above):
$$
\mathbb {E} _ {\mathcal {E} ^ {\prime}} [ \mathrm {D R} (T) ] \geq \frac {1}{4} \cdot \mathbb {E} _ {\mathcal {E} ^ {\prime}} [ N (T, 2) ] \geq 3 T / 8.
$$
In particular, there exists a realization of  $\mathcal{E}'$  within the prior on environments on which the regret is at least  $3T/8$ .
It remains to show  $\mathbb{E}_{\mathcal{E}}[N(T,2)] = \mathbb{E}_{\mathcal{E}'}[N(T,2)]$ . This will follow from Pinsker's inequality and showing that the KL divergence between  $\mathcal{E}$  and  $\mathcal{E}'$  is zero.
We first observe that the coupling observations  $O_{t}(i,j)$  at each round  $t \in [T]$  are identically a  $\mathrm{Ber}(1/2)$  R.V. for all pairs of arms  $i,j$  in both  $\mathcal{E}$  and  $\mathcal{E}'$ , since a uniform mixture of a  $\mathrm{Ber}(1/2 + \epsilon)$  and a  $\mathrm{Ber}(1/2 - \epsilon)$  is a  $\mathrm{Ber}(1/2)$ , while so is the uniform mixture of a  $\mathrm{Ber}(1)$  and a  $\mathrm{Ber}(0)$ .
Then, since  $N(T,2) \leq 2T$ , by Pinsker's inequality [see Gupta and Saha, 2022, proof of Lemma C.1], we have:
$$
\mathbb {E} _ {\mathcal {E}} [ N (T, 2) ] - \mathbb {E} _ {\mathcal {E} ^ {\prime}} [ N (T, 2) ] \leq 2 T \sqrt {\frac {\mathrm {K L} (\mathcal {P} , \mathcal {P} ^ {\prime})}{2}},
$$
where  $\mathcal{P}$  and  $\mathcal{P}'$  are the induced distributions over the randomness  $\mathcal{U}^{\otimes T}$ , and the history of observations and decisions in  $T$  rounds by  $\mathcal{E}$  and  $\mathcal{E}'$ . Let  $\mathcal{H}_t$  be the history of randomness, observations, and decisions till round  $t$ :  $\mathcal{H}_t = \{(u_s, i_s, j_s, O_s(i_s, j_s))\}_{s \leq t}$  where  $u_s \sim \mathrm{Ber}(1/2)$  decides whether  $\mathbf{P}^{+}/\mathbf{Q}^{+}$  or  $\mathbf{P}^{-}/\mathbf{Q}^{-}$  is realized at round  $t$ . Let  $\mathcal{P}_t$  and  $\mathcal{P}'$  denote the respective marginal distributions over the round  $t$  data  $(u_t, i_t, j_t, O_t(i_t, j_t))$ . Then, repeatedly using chain rule for KL and then conditioning on the played arms  $(i_t, j_t)$  (whose identities are fixed given  $\mathcal{H}_{t-1}$ ) at round  $t$ , we get:
$$
\mathrm {K L} (\mathcal {P}, \mathcal {P} ^ {\prime}) = \sum_ {t = 1} ^ {T} \mathrm {K L} (\mathcal {P} _ {t} | \mathcal {H} _ {t - 1}, \mathcal {P} _ {t} ^ {\prime} | \mathcal {H} _ {t - 1}) = \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathcal {H} _ {t - 1}} [ \mathbb {E} _ {i _ {t}, j _ {t}} [ \mathrm {K L} (\operatorname {B e r} (1 / 2), \operatorname {B e r} (1 / 2)) ] ] = 0.
$$
Remark 2. The constructed environments  $P^{+}, P^{-}$  in the proof of Theorem 3 satisfies SST but violates STI. A similar construction which violates SST (but satisfies STI) can also be used in the proof. Let
$$
\boldsymbol {P} ^ {+} := \left( \begin{array}{c c c} 1 / 2 & 1 / 2 - \epsilon & 1 / 2 - \epsilon \\ 1 / 2 + \epsilon & 1 / 2 & 0 \\ 1 / 2 + \epsilon & 1 & 1 / 2 \end{array} \right),
$$
and let  $P^{-}$  be the same preference matrix with arms 2 and 3 switched. Note that  $3 \succ 2 \succ 1$  in  $P^{+}$  and  $2 \succ 3 \succ 1$  in  $P^{-}$ . Here, arm 1 is the "safe" arm as it always has a gap of  $\epsilon$  while arms 2 and 3 randomly alternate between being the best arm and the worst arm with a gap of  $1/2$ . Thus, both the STI and SST assumptions are required to get sublinear regret in mildly adversarial environments.
Due to these observations we have the following corollaries.
Corollary 6. For each horizon  $T$ , there exists a finite family  $\mathcal{F}$  of switching dueling bandit environments with  $K = 3$  that satisfies the SST condition with  $\tilde{L} = 0$  significant shifts. The worst-case regret of any algorithm on an environment  $\mathcal{E}$  in this family is lower bounded as
$$
\sup_{\substack{\mathcal{E}\in \mathcal{F}}}\mathbb{E}_{\mathcal{E}}\left[\mathrm{DR}(T)\right]\geq T / 8.
$$
Corollary 7. For each horizon  $T$ , there exists a finite family  $\mathcal{F}$  of switching bandit environments with  $K = 3$  that satisfies the STI condition with  $\tilde{L} = 0$  significant shifts. The worst-case regret of any algorithm on an environment  $\mathcal{E}$  in this family is lower bounded as
Throughout the proof  $c_{1}, c_{2}, \ldots$  will denote positive constants not depending on  $T$  or any distributional parameters. First, we observe the regret bound is vacuous for  $T < K$ ; so, assume  $T \geq K$ . Recall from Line 3 of Algorithm 1 that  $t_{\ell}$  is the first round of the  $\ell$ -th episode. WLOG, there are  $T$  total episodes and, by convention, we let  $t_{\ell} := T + 1$  if only  $\ell - 1$  episodes occurred by round  $T$ .
Next, we establish an elementary lemma which will help us leverage the STI and SST assumptions.
C.1 Decomposing the Regret
Lemma 8. For any three arms  $b, c$ , under SST∩STI:  $\delta_t(a_t^*, c) \leq 2 \cdot \delta_t(a_t^*, b) + \delta_t(b, c)$ .
Proof. If  $b \succeq_t c$ , this is true by STI. Otherwise,  $\delta_t(a_t^*, c) \leq \delta_t(a_t^*, b) \leq \delta_t(a_t^*, b) + \delta_t(a_t^*, b) - \delta_t(c, b)$  by SST.
Using Lemma 8 twice, we have the regret can be written as
$$
\sum_ {t = 1} ^ {T} \delta_ {t} (a _ {t} ^ {*}, \hat {a} _ {t}) + \delta_ {t} (a _ {t} ^ {*}, a _ {t}) \leq \sum_ {t = 1} ^ {T} 6 \cdot \delta_ {t} (a _ {t} ^ {*}, a _ {t} ^ {\sharp}) + 3 \cdot \delta_ {t} (a _ {t} ^ {\sharp}, \hat {a} _ {t}) + \delta_ {t} (\hat {a} _ {t}, a _ {t}).
$$
Following the discussion of Section 6, it remains to bound  $\sum_{t=1}^{T} \delta_t(a_t^\sharp, \hat{a}_t)$  and  $\sum_{t=1}^{T} \delta_t(\hat{a}_t, a_t)$  in expectation. For this, we need to relate our estimators  $\hat{\delta}_t(\hat{a}_t, a)$  to the true gaps  $\delta_t(\hat{a}_t, a)$ .
C.2 Relating Estimated Gaps to Regret
We first recall a version of Freedman's martingale concentration inequality, identical to the one used in Suk and Kpotufe [2022], Buening and Saha [2022].
Lemma 9 (Theorem 1 of Beygelzimer et al. [2011]). Let  $X_{1},\ldots ,X_{n}\in \mathbb{R}$  be a martingale difference sequence with respect to some filtration  $\{\mathcal{F}_0,\mathcal{F}_1,\dots \}$ . Assume for all  $t$  that  $X_{t}\leq R$  a.s. and that  $\sum_{i = 1}^{n}\mathbb{E}[X_i^2 |\mathcal{F}_{i - 1}]\leq V_n$  a.s. for some constant  $V_{n}$  only depending on  $n$ . Then for any  $\delta \in (0,1)$  with probability at least  $1 - \delta$ , we have:
$$
\sum_ {i = 1} ^ {n} X _ {i} \leq (e - 1) \left(\sqrt {V _ {n} \log (1 / \delta)} + R \log (1 / \delta)\right).
$$
We next apply Lemma 9 to bound the estimation error of our estimates  $\hat{\delta}_t(\hat{a}_t,a)$ , found in (2).
Proposition 10. Let  $\mathcal{E}_1$  be the event that for all rounds  $s_1 < s_2$  and all arms  $a \in [K]$ :
$$
\left| \sum_ {t = s _ {1}} ^ {s _ {2}} \hat {\delta} _ {t} \left(\hat {a} _ {t}, a\right) - \sum_ {t = s _ {1}} ^ {s _ {2}} \mathbb {E} \left[ \hat {\delta} _ {t} \left(\hat {a} _ {t}, a\right) \mid \mathcal {F} _ {t - 1} \right] \right| \leq c _ {1} \log (T) (\sqrt {K \left(s _ {2} - s _ {1}\right)} + K), \tag {9}
$$
for an appropriately large constant  $c_{1}$ , and where  $\mathcal{F} \coloneqq \{\mathcal{F}_t\}_{t=1}^T$  is the canonical filtration generated by observations and randomness of elapsed rounds. Then,  $\mathcal{E}_1$  occurs with probability at least  $1 - 1 / T^2$ .
Proof. The random variable  $\hat{\delta}_t(\hat{a}_t,a) - \mathbb{E}[\hat{\delta}_t(\hat{a}_t,a)|\mathcal{F}_{t - 1}]$  is a martingale difference bounded above by  $K$  for all rounds  $t$  and all arms  $a,a^{\prime}$ . Note here that the identity of the candidate arm  $\hat{a}_t$  is fixed conditional on the observations of the previous rounds  $\mathcal{F}_{t - 1}$ . The variance of this difference is:
$$
\begin{array}{l} \sum_ {t = s _ {1}} ^ {s _ {2}} \mathbb {E} [ \hat {\delta} _ {t} ^ {2} (\hat {a} _ {t}, a) \mid \mathcal {F} _ {t - 1} ] \leq \sum_ {t = s _ {1}} ^ {s _ {2}} | \mathcal {A} _ {t} | ^ {2} \mathbb {E} [ \mathbf {1} \{j _ {t} = a \} | \mathcal {F} _ {t - 1} ] \\ \leq \sum_ {t = s _ {1}} ^ {s _ {2}} \left| \mathcal {A} _ {t} \right| ^ {2} \cdot \frac {1}{\left| \mathcal {A} _ {t} \right|} \\ \leq K \cdot \left(s _ {2} - s _ {1} + 1\right). \\ \leq 2 K \cdot \left(s _ {2} - s _ {1}\right) \\ \end{array}
$$
Then, the result follows from Lemma 9 and taking union bounds over arms  $a$  and rounds  $s_1, s_2$ .
Since the contribution to the expected regret is small outside of the high-probability good event  $\mathcal{E}_1$ , going forward we will assume as necessary that (9) holds for all arms  $a \in [K]$  and rounds  $s_1, s_2$ . The next result asserts that episodes roughly correspond to significant shifts in the sense that a restart (Line 8 of Algorithm 1) occurs only if a significant shift has been detected.
Lemma 11. On event  $\mathcal{E}_1$ , for each episode  $[t_{\ell}, t_{\ell + 1})$  with  $t_{\ell + 1} \leq T$  (i.e., an episode which concludes with a restart), there exists a significant shift  $\tau_i \in [t_{\ell}, t_{\ell + 1})$ .
$$
\mathbb {E} [ \hat {\delta} _ {t} (\hat {a} _ {t}, a) | \mathcal {F} _ {t - 1} ] = \left\{ \begin{array}{l l} \delta_ {t} (\hat {a} _ {t}, a) & a \in \mathcal {A} _ {t} \\ - 1 / 2 & a \notin \mathcal {A} _ {t} \end{array} \right..
$$
Thus, by concentration (Proposition 10) and the eviction criteria (3) with large enough constant  $C > 0$ , we have that an arm  $a$  being evicted over interval  $[s_1, s_2]$  implies  $\sum_{t=s_1}^{s_2} \delta_t(\hat{a}_t, a) > \sqrt{K \cdot (s_2 - s_1)}$ . By the SST condition, this means that
$$
\sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(a _ {t} ^ {*}, a\right) \geq \sum_ {t = s _ {1}} ^ {s _ {2}} \delta_ {t} \left(\hat {a} _ {t}, a\right) > \sqrt {K \cdot \left(s _ {2} - s _ {1}\right)}.
$$
This means, over the course of episode  $[t_{\ell}, t_{\ell + 1})$ , every arm  $a \in [K]$  incurs significant regret meaning a significant shift must take place between rounds  $t_{\ell}$  and  $t_{\ell + 1} - 1$ .
Following the outline of Section 6, we now turn our attention to bounding the regrets  $\delta_t(a_t^\sharp, \hat{a}_t)$  and  $\delta_t(\hat{a}_t, a_t)$  over a single episode  $[t_\ell, t_{\ell+1})$ .
C.3 Bounding  $\mathbb{E}[\sum_{t = t_\ell}^{t_{\ell +1} - 1}\delta_t(\hat{a}_t,a_t)]$  : Regret of Active Arms to Candidate Arm
We first decompose the total sum of regrets  $\mathbb{E}[\sum_{t=1}^{T} \delta_t(\hat{a}_t, a_t)]$  based on which arm  $a_t$  chooses within the active set  $\mathcal{A}_t$ . Using tower law, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \delta_ {t} \left(\hat {a} _ {t}, a _ {t}\right) \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} \left[ \delta_ {t} \left(\hat {a} _ {t}, a _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} \left(\hat {a} _ {t} , a\right)}{\left| \mathcal {A} _ {t} \right|} \right].
$$
Splitting the above RHS back along episodes, we obtain the sum  $\mathbb{E}[\sum_{t = t_{\ell}}^{t_{\ell +1} - 1}\sum_{a\in \mathcal{A}_t}\delta_t(\hat{a}_t,a) / |\mathcal{A}_t|]$ . Next, we condition on the good event  $\mathcal{E}_1$  on which the concentration bounds of Proposition 10 hold. Additionally, we divide up the rounds  $t$  into those before arm  $a$  is evicted from  $\mathcal{A}_{\mathrm{master}}$  and those after. Suppose arm  $a$  is evicted from  $\mathcal{A}_{\mathrm{master}}$  at round  $t_\ell^a\in [t_\ell ,t_{\ell +1})$ . In particular, this means arm  $a\in \mathcal{A}_t$  for all  $t\in [t_{\ell},t_{\ell}^{a})$ . Thus, it suffices to bound:
$$
\mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} \right\} \cdot \left(\sum_ {a = 1} ^ {K} \sum_ {t = t _ {\ell}} ^ {t _ {\ell} ^ {a} - 1} \frac {\delta_ {t} \left(\hat {a} _ {t} , a\right)}{\left| \mathcal {A} _ {t} \right|} + \sum_ {a = 1} ^ {K} \sum_ {t = t _ {\ell} ^ {a}} ^ {t _ {\ell + 1} - 1} \frac {\delta_ {t} \left(\hat {a} _ {t} , a\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \mathbf {1} \left\{a \in \mathcal {A} _ {t} \right\}\right) \right]. \tag {10}
$$
Suppose WLOG that  $t_{\ell}^{1} \leq t_{\ell}^{2} \leq \dots \leq t_{\ell}^{K}$ . Then, for each round  $t < t_{\ell}^{a}$  all arms  $a' \geq a$  are retained in  $\mathcal{A}_{\mathrm{master}}$  and thus retained in the candidate arm set  $\mathcal{A}_t$ . Thus,  $|\mathcal{A}_t| \geq K + 1 - a$  for all  $t \leq t_{\ell}^{a}$ .
Then, the first double sum in (10) can be bounded by combining our eviction criterion (3) with our concentration bounds Proposition 10. Since arm  $a$  is not evicted from  $\mathcal{A}_t$  till round  $t_\ell^a$ , on event  $\mathcal{E}_1$  we have for some  $c_2 > 0$ :
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell} ^ {a} - 1} \delta_ {t} (\hat {a} _ {t}, a) = \sum_ {t = t _ {\ell}} ^ {t _ {\ell} ^ {a} - 1} \mathbb {E} [ \hat {\delta} _ {t} (\hat {a} _ {t}, a) \mid \mathcal {F} _ {t - 1} ] \leq c _ {2} \log (T) \sqrt {K (t _ {\ell} ^ {a} - t _ {\ell}) \vee K ^ {2}}
$$
Then, using the fact that  $|\mathcal{A}_t| \geq K + 1 - a$  for all  $t \in [t_\ell, t_\ell^a)$ , we have:
$$
\sum_ {t = t _ {\ell}} ^ {t _ {\ell} ^ {a} - 1} \frac {\delta_ {t} (\hat {a} _ {t} , a)}{| \mathcal {A} _ {t} |} \leq \frac {c _ {2} \log (T) \sqrt {K (t _ {\ell} ^ {a} - t _ {\ell}) \vee K ^ {2}}}{K + 1 - a}.
$$
Then, summing the above R.H.S. over all arms  $a$ , we have on event  $\mathcal{E}_1$ :
$$
\sum_ {a = 1} ^ {K} \sum_ {t = t _ {\ell}} ^ {t _ {\ell} ^ {a} - 1} \frac {\delta_ {t} (\hat {a} _ {t} , a)}{| \mathcal {A} _ {t} |} \leq c _ {2} \log (K) \log (T) \sqrt {K (t _ {\ell + 1} - t _ {\ell}) \vee K ^ {2}}.
$$
Next, we handle the second double sum in (10). We first observe that if arm  $a$  is played after round  $t_{\ell}^{a}$ , then it must due to a scheduled replay. The difficulty here is that replays may interrupt each other and so care must be taken in managing the relative regret contribution  $\sum_{t}\delta_{t}(\hat{a}_{t},a)$  (which may be negative if  $a\prec \hat{a}_t$ ) of different overlapping replays.
Fixing an arm  $a$ , our strategy is to partition the rounds when  $a$  is played by a replay after round  $t_{\ell}^{a}$  according to which replay is active and not accounted for by another replay. This involves carefully designating a subclass of replays whose durations while playing  $a$  span all the rounds where  $a$  is played after  $t_{\ell}^{a}$ . Then, we cover the times when  $a$  is played by a collection of intervals corresponding to the schedules of this subclass of replays, on each of which we can employ the eviction criterion (3) and concentration like before.
For this purpose, we define the following terminology (which is all w.r.t. a fixed arm  $a$ ):
Definition 4.
(i) For each scheduled and activated Base-Alg(s, m), let the round  $M(s, m)$  be the minimum of two quantities: (a) the last round in  $[s, s + m]$  when arm a is retained by Base-Alg(s, m) and all of its children, and (b) the last round that Base-Alg(s, m) is active and not permanently interrupted. Call the interval  $[s, M(s, m)]$  the active interval of Base-Alg(s, m).
Figure 3: Shown are replay scheduled durations (in gray) with dots marking when arm  $a$  is reintroduced to  $\mathcal{A}_t$ . Black segments indicate the period  $[s, M(s, m)]$  for proper and subproper replays. Note that the rounds where  $a \in \mathcal{A}_t$  in the left unlabeled replay's duration are accounted for by the larger proper replay.
(ii) Call a replay Base-Alg(s,m) proper if there is no other scheduled replay Base-Alg(s',m') such that  $[s,s + m]\subset (s^{\prime},s^{\prime} + m^{\prime})$  where Base-Alg(s',m') will become active again after round  $s + m$ . In other words, a proper replay is not scheduled inside the scheduled range of rounds of another replay. Let PROPER(tl,t+1) be the set of proper replays scheduled to start before round t+1.
(iii) Call a scheduled replay Base-Alg(s,m) subproper if it is non-proper and if each of its ancestor replays (i.e., previously scheduled replays whose durations have not concluded) Base-Alg(s',m') satisfies  $M(s',m') < s$ . In other words, a subproper replay either permanently interrupts its parent or does not, but is scheduled after its parent (and all its ancestors) stops playing arm a. Let SUBPROPER(t<sub>1</sub>, t<sub>2</sub>) be the set of all subproper replays scheduled before round t<sub>1</sub>.
Equipped with this language, we now show some basic claims which essentially reduce analyzing the complicated hierarchy of replays to analyzing the active intervals of replays in  $\mathrm{PROPER}(t_{\ell}, t_{\ell+1}) \cup \mathrm{SUBPROPER}(t_{\ell}, t_{\ell+1})$ .
$$
\left\{\left[ s, M (s, m) \right]: \text {B a s e - A l g} (s, m) \in \text {P R O P E R} \left(t _ {\ell}, t _ {\ell + 1}\right) \cup \text {S u b P R O P E R} \left(t _ {\ell}, t _ {\ell + 1}\right) \right\},
$$
Proof. Clearly, the classes of replays  $\mathrm{PROPER}(t_{\ell}, t_{\ell + 1})$  and  $\mathrm{SUBPROPER}(t_{\ell}, t_{\ell + 1})$  are disjoint. Next, we show the respective active intervals  $[s, M(s, m)]$  and  $[s', M(s', m')']$  of any two Base-Alg  $(s, m)$  and Base-Alg  $(s', m') \in \mathrm{PROPER}(t_{\ell}, t_{\ell + 1}) \cup \mathrm{SUBPROPER}(t_{\ell}, t_{\ell + 1})$  are disjoint.
1. Proper replay vs. subproper replay: a subproper replay can only be scheduled after the round  $M(s, m)$  of the most recent proper replay Base-Alg(s, m) (which is necessarily an ancestor). Thus, the active intervals of proper replays and subproper replays are disjoint.
2. Two distinct proper replays: two such replays can only permanently interrupt each other, and since  $M(s,m)$  always occurs before the permanent interruption of Base-Alg(s,m), we have the active intervals of two such replays are disjoint.
3. Two distinct subproper replays: consider two non-proper replays Base-  $\mathbf{Alg}(s,m)$  ,Base-  $\mathbf{Alg}(s^{\prime},m^{\prime})\in$  SUBPROPER  $(t_{\ell},t_{\ell +1})$  with  $s^\prime >s$  . The only way their active intervals intersect is if Base-  $\mathbf{Alg}(s,m)$  is an ancestor of Base-  $\mathbf{Alg}(s^{\prime},m^{\prime})$  Then, if Base-  $\mathbf{Alg}(s',m')$  is subproper, we must have  $s^\prime >M(s,m)$  , which means that  $[s^{\prime},M(s^{\prime},m^{\prime})]$  and  $[s,M(s,m)]$  are disjoint.
Next, we claim that the active intervals  $[s,M(s,m)]$  for Base-  $\mathbf{Alg}(s,m)\in \mathrm{PROPER}(t_{\ell},t_{\ell +1})\cup$  SUBPROPER(tl,t+1) contain all the rounds where a is played after being evicted from Amaster. To show this, we first observe that for each round  $t$  when a replay is active, there is a unique proper
replay associated to  $t$ , namely the proper replay scheduled most recently. Next, note that any round  $t > t_{\ell}^{a}$  where arm  $a \in \mathcal{A}_t$  must belong to the active interval  $[s, M(s, m)]$  of the unique proper replay Base-Alg  $(s, m)$  associated to round  $t$ , or else satisfies  $t > M(s, m)$  in which case a unique subproper replay Base-Alg  $(s', m') \in \mathrm{SUBPROPER}(t_{\ell}, t_{\ell+1})$  was active and not yet permanently interrupted by round  $t$ . Thus, it must be the case that  $t \in [s', M(s', m')]$ .
At the same time, every round  $t \in [s, M(s, m)]$  for a proper or subproper Base-Alg  $(s, m)$  is clearly a round where  $a \in \mathcal{A}_t$  and no such round is accounted for twice by Proposition 12. Thus,
$$
\{t\in (t_{\ell}^{a},t_{\ell +1}):a\in \mathcal{A}_{t}\} = \bigsqcup_{\substack{\text{Base - Alg} (s,m)\in \text{PROPER} (t_{\ell},t_{\ell +1})\cup \text{SUBPROPER} (t_{\ell},t_{\ell +1})}}[s,M(s,m)].
$$
Then, we can rewrite the second double sum in (10) as:
$$
\sum_ {a = 1} ^ {K} \sum_ {\text {B a s e - A l g} (s, m) \in \text {P R O P E R} (t _ {\ell}, t _ {\ell + 1}) \cup \text {S U B P R O P E R} (t _ {\ell}, t _ {\ell + 1})} \mathbf {1} \{B _ {s, m} = 1 \} \sum_ {t = s \vee t _ {\ell} ^ {a}} ^ {M (s, m)} \frac {\delta_ {t} (\hat {a} _ {t} , a)}{| \mathcal {A} _ {t} |}.
$$
Recall in the above that the Bernoulli  $B_{s,m}$  (see Line 6 of Algorithm 1) decides whether  $\operatorname{Base - Alg}(s,m)$  is scheduled.