SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {B \in \mathcal {T} _ {r}} K \log (T) \leq K \log (T) \cdot r ^ {- d} \leq c _ {1 1} \log (T) K ^ {\frac {2}{2 + d}} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {d}{2 + d}} \leq c _ {1 1} \log (T) K ^ {\frac {1}{2 + d}} \cdot (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}}.
$$
Additionally, we have by Lemma 10:
$$
\sqrt {(\tau_ {i} ^ {a} - \tau_ {i}) \cdot \mu (B)} \leq \sqrt {(\tau_ {i + 1} - \tau_ {i}) \cdot \mu (B)} \leq c _ {7} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1}{2 + d}} K ^ {\frac {d / 2}{2 + d}} \leq c _ {8} K ^ {\frac {1}{2 + d}} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}}.
$$
Then, plugging the above into (16) and summing over bins  $B$  at level  $r$ , we have the regret in episode  $\left[\tau_{i},\tau_{i + 1}\right)$  is with probability at least  $1 - 1 / T^2$  w.r.t. the distribution of  $\mathbf{X}_T$ :
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = \tau_ {i}} ^ {\tau_ {i + 1} - 1} \delta_ {t} (\pi_ {t}) \middle | \mathbf {X} _ {T} \right] = \mathbb {E} \left[ \sum_ {B \in \mathcal {T} _ {r}} \sum_ {t = \tau_ {i}} ^ {\tau_ {i + 1} - 1} \sum_ {a \in \mathcal {G} _ {t}} \frac {\delta_ {t} (a) \cdot \mathbf {1} \{X _ {t} \in B \}}{| \mathcal {G} _ {t} |} \middle | \mathbf {X} _ {T} \right] \\ = \mathbb {E} \left[ \sum_ {B \in \mathcal {T} _ {r}} \sum_ {a \in [ K ]} \sum_ {t = \tau_ {i}} ^ {\tau_ {i} ^ {a}} \frac {\delta_ {t} (a) \cdot \mathbf {1} \{X _ {t} \in B \}}{| \mathcal {G} _ {t} |} \Bigg | \mathbf {X} _ {T} \right] \\ \leq c _ {1 2} \log (K) \sum_ {B \in \mathcal {T} _ {r}} \log^ {1 / 2} (T) r ^ {d} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} + K \log (T) \\ \leq c _ {1 3} \log (K) \log (T) \cdot \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}, \\ \end{array}
$$
where we use the strong density assumption to bound  $\sum_{B\in \mathcal{T}_r}r^d\leq \sum_{B\in \mathcal{T}_r}c_d^{-1}\cdot \mu (B)\leq c_d^{-1}$  in the last inequality. Summing the regret over all experienced significant phases  $[\tau_i,\tau_{i + 1})$  gives the desired result.
D Proof of CMETA Regret Upper Bound (Theorem 3)
Recall from Line 3 of Algorithm 1 that  $t_\ell$  is the first round of the  $\ell$ -th episode. WLOG, there are  $T$  total episodes and, by convention, we let  $t_\ell \doteq T + 1$  if only  $\ell - 1$  episodes occurred by round  $T$ .
We first quickly handle the simple case of  $T < K$ . In this case, the regret bound of Theorem 3 is vacuous since by the sub-additivity of  $x \mapsto x^{\frac{1 + d}{2 + d}}$ :
$$
\sum_ {i = 0} ^ {\tilde {L}} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} \geq \left(\tau_ {\tilde {L} + 1} - \tau_ {0}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} \geq T ^ {\frac {1 + d}{2 + d}} \cdot T ^ {\frac {1}{2 + d}} = T.
$$
Thus, it remains to show Theorem 3 for  $T \geq K$ .
We first transform the expected regret into a more suitable form, which will allow us to analyze regret in a similar fashion to the proof of the oracle regret bound (Appendix C).
D.1 Decomposing the Regret
We first transform the regret into a more convenient form. Let  $\mathcal{F} \doteq \{\mathcal{F}_t\}_{t=1}^T$  be the filtration with  $\mathcal{F}_t$  generated by  $\{\pi_s, Y_s^{\pi_s}\}_{s=1}^t$  conditional on a fixed  $\mathbf{X}_T$ . Then,
$$
\begin{array}{l} \mathbb {E} \left[ R _ {T} \left(\pi , \mathbf {X} _ {T}\right) \mid \mathbf {X} _ {T} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} \left[ \delta_ {t} \left(\pi_ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \mid \mathbf {X} _ {T} \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} \left(\pi_ {t}\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \left| \mathbf {X} _ {T} \right. \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} (a)}{| \mathcal {A} _ {t} |} \Bigg | \mathbf {X} _ {T} \right]. \\ \end{array}
$$
Now, it suffices to bound the above R.H.S. on the good event  $\mathcal{E}_1 \cap \mathcal{E}_2$  where the bounds of Lemmas 8 and 9 hold. Going forward in the rest of the analysis, we will assume said bounds hold wherever convenient.
Next, as alluded to in defining the oracle procedure (Definition 7), until the end of a significant phase  $\left[\tau_{i},\tau_{i + 1}\right)$ , there is a safe arm in each bin  $B$  at level  $r_{\tau_{i + 1} - \tau_i}$  which is experienced.
Definition 9 (local last safe arm in each phase  $a_{t}^{\sharp}$ ). For a round  $t \in [\tau_i, \tau_{i+1})$ , let  $B$  be the bin at level  $r_{\tau_{i+1}-\tau_i}$  which contains  $X_t$  and let  $t_i(B)$  be the last round in  $[\tau_i, \tau_{i+1})$  such that  $X_{t_i(B)} \in B$ . Then, by Definition 6, there is a (local) last safe arm  $a_t^{\sharp}$  which does not yet incur significant regret in bin  $B$  in the following sense: for all  $[s_1, s_2] \subseteq [\tau_i, t_i(B)]$  letting  $r = r_{s_2 - s_1}$  and  $B' \in \mathcal{T}_r$  such that  $B' \supseteq B$  we have:
$$
\sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} \left(a _ {t} ^ {\sharp}\right) \cdot \mathbf {1} \left\{X _ {s} \in B ^ {\prime} \right\} <   \sqrt {K \cdot n _ {B ^ {\prime}} ([ s _ {1} , s _ {2} ])} + r \cdot n _ {B ^ {\prime}} ([ s _ {1}, s _ {2} ]).
$$
Remark 6. The local last safe arms  $\{a_t^\sharp\}_t$  only depend on the distribution of  $X_T$  and not on the realized rewards  $Y_T$ . In particular, the sequence  $\{a_t^\sharp\}_t$  is fixed conditional on  $X_T$ .
We first decompose the regret at round  $t$  as (a) the regret of the local safe arm  $a_{t}^{\sharp}$  and (b) the regret of arm  $a$  to  $a_{t}^{\sharp}$ . In other words, it suffices to bound:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} (a)}{| \mathcal {A} _ {t} |} \mathbf {1} \{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \} \middle | \mathbf {X} _ {T} \right] = \sum_ {t = 1} ^ {T} \delta_ {t} (a _ {t} ^ {\sharp}) \mathbf {1} \{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \} + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} (a _ {t} ^ {\sharp} , a)}{| \mathcal {A} _ {t} |} \mathbf {1} \{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \} \middle | \mathbf {X} _ {T} \right].
$$
Note that the expectation on the first sum disappears since  $a_{t}^{\sharp}$  is only a function of  $\mathbf{X}_T$  and the mean reward functions  $\{f_t^a (\cdot)\}_{t,a}$ .
D.2 Bounding the Regret of the Local Last Safe Arm
Bounding  $\sum_{t=1}^{T} \delta_t(a_t^\sharp) \cdot \mathbf{1}\{\mathcal{E}_1 \cap \mathcal{E}_2\}$  will be similar to the proof of Proposition 2. We show that the oracle procedure could have essentially just played arm  $a_t^\sharp$  every round.
Fix a phase  $\left[\tau_{i},\tau_{i + 1}\right)$  and let  $r = r_{\tau_{i + 1} - \tau_i}$ . Fix a bin  $B\in \mathcal{T}_r$  and let  $a_{i}(B)$  be the local last safe arm  $a_{t}^{\sharp}$  of the last round  $t\in [\tau_i,\tau_{i + 1})$  such that  $X_{t}\in B$ . Then,  $a_{t}^{\sharp} = a_{i}(B)$  for every round  $t\in [\tau_i,\tau_{i + 1})$  such that  $X_{t}\in B$ . Then, we have by Definition 6 that for bin  $B^{\prime}\supseteq B$  at level  $r_{t - \tau_i}$ :
$$
\sum_ {s = \tau_ {i}} ^ {t} \delta_ {s} \left(a _ {i} (B)\right) \cdot \mathbf {1} \left\{X _ {s} \in B ^ {\prime} \right\} \leq \sqrt {K \cdot n _ {B ^ {\prime}} \left(\left[ \tau_ {i} , t \right]\right)} + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} \left(\left[ \tau_ {i}, t \right]\right).
$$
Then, by Lemma 9, we have:
$$
\begin{array}{l} \sum_ {s = \tau_ {i}} ^ {t} \delta_ {s} (a _ {i} (B)) \cdot \mathbf {1} \{X _ {s} \in B \} \leq c _ {4} \left(\log^ {1 / 2} (T) \cdot r ^ {d} \cdot (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} \right. \\ \left. + K \log (T) + \sqrt {\log (T) \mu (B) \cdot \left(t - \tau_ {i} + 1\right)}\right). \tag {17} \\ \end{array}
$$
Then, summing the above over bins in the same fashion as the proof of Proposition 2 gives:
$$
\sum_ {t = \tau_ {i}} ^ {\tau_ {i + 1} - 1} \delta_ {t} (a _ {t} ^ {\sharp}) = \sum_ {B \in \mathcal {T} _ {r}} \sum_ {s = \tau_ {i}} ^ {\tau_ {i + 1} - 1} \delta_ {s} (a _ {i} (B)) \cdot \mathbf {1} \{X _ {s} \in B \} \leq c _ {1 4} \log (T) \cdot (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
Finally, summing over phases  $\left[\tau_{i},\tau_{i + 1}\right)$  we have  $\sum_{t = 1}^{T}\delta_{t}(a_{t}^{\sharp})$  is of the right order.
D.3 Relating Episodes to Significant Phases
We next show that w.h.p. a restart occurs (i.e., a new episode begins) only if a significant shift has occurred sometime within the episode. Recall from Definition 6 that  $\tau_{1},\tau_{2},\ldots ,\tau_{\tilde{L}}$  are the times of the significant shifts and that  $t_1,\dots ,t_T$  are the episode start times.
Lemma 11 (Restart Implies Significant Shift). On event  $\mathcal{E}_1$ , for each episode  $[t_{\ell}, t_{\ell+1})$  with  $t_{\ell+1} \leq T$  (i.e., an episode which concludes with a restart), there exists a significant shift  $\tau_i \in [t_{\ell}, t_{\ell+1}]$ .
Proof. Fix an episode  $[t_{\ell}, t_{\ell + 1})$ . Then, by Line 11 of Algorithm 1, there is a bin  $B$  such that every arm  $a \in [K]$  was evicted from  $B$  at some round in the episode, i.e. (5) is true for each arm  $a$  on some interval  $[s_1, s_2] \subseteq [t_{\ell}, t_{\ell + 1})$ . It suffices to show that this implies a significat shift has occurred between rounds  $t_{\ell}$  and  $t_{\ell + 1}$ .
Suppose (5) first triggers the eviction of arm  $a$  at time  $t$  in  $B' \supseteq B$  over interval  $[s_1, s_2]$  where  $r(B') = r_{s_2 - s_1}$ . By concentration (10) and our eviction criteria (5), we have that there is an arm  $a' \neq a$  such that (using the notation of Proposition 7) for large enough  $C_0 > 0$  and some  $c_{14} > 0$ :
$$
\begin{array}{l} \sum_ {s = s _ {1}} ^ {s _ {2}} \mathbb {E} \left[ \hat {\delta} _ {s} ^ {B} \left(a ^ {\prime}, a\right) \mid \mathcal {F} _ {s - 1} \right] \geq c _ {1 4} \left(\sqrt {K \log (T) \cdot n _ {B ^ {\prime}} ([ s _ {1} , s _ {2} ])} + (K \log (T)) ^ {2} \right. \\ + r \left(B ^ {\prime}\right) \cdot n _ {B ^ {\prime}} \left(\left[ s _ {1}, s _ {2} \right]\right)). \tag {18} \\ \end{array}
$$
Next, if arm  $a$  is evicted from  $\mathcal{A}(B')$  at round  $t$ , then we have by the definition of  $\hat{\delta}_s^{B'}(a', a)$  (4):
$$
\mathbb {E} [ \hat {\delta} _ {s} ^ {B ^ {\prime}} (a ^ {\prime}, a) \mid \mathcal {F} _ {s - 1} ] = \left\{ \begin{array}{l l} \delta_ {s} (a ^ {\prime}, a) \cdot \mathbf {1} \{X _ {s} \in B ^ {\prime} \} & a, a ^ {\prime} \in \mathcal {A} _ {s} \\ - f _ {s} ^ {a} (X _ {s}) \cdot \mathbf {1} \{X _ {s} \in B \} & a \in \mathcal {A} _ {s}, a ^ {\prime} \notin \mathcal {A} _ {s} \\ 0 & a \not \in \mathcal {A} _ {s} \end{array} \right..
$$
In any case, the above L.H.S. conditional expectation is bounded above by  $\delta_s(a) \cdot \mathbf{1}\{X_s \in B'\}$ . Thus, (18) implies arm  $a$  incurs significant regret ( $\star$ ) in  $B'$  on  $[s_1, s_2]$ :
$$
\sum_ {s = s _ {1}} ^ {s _ {2}} \delta_ {s} (a) \cdot \mathbf {1} \{X _ {s} \in B ^ {\prime} \} \geq \sqrt {K \cdot n _ {B ^ {\prime}} ([ s _ {2} , s _ {2} ])} + r (B ^ {\prime}) \cdot n _ {B ^ {\prime}} ([ s _ {1}, s _ {2} ]).
$$
Then, since every arm  $a$  is evicted in bin  $B$  by round  $t$ , a significant shift must have occurred in episode  $[t_{\ell}, t_{\ell + 1}]$ .
D.4 Regret of CMETA to the Last Safe Arm
It remains to bound  $\mathbb{E}[\sum_{t=1}^{T}\sum_{a\in\mathcal{A}_t}\delta_t(a_t^\sharp,a)/|\mathcal{A}_t||\mathbf{X}_t]$ . We further decompose this sum over  $t$  into episodes and then the blocks (see Definition 8) where a particular choice of level is used within the episode. The following notation will be useful.
Definition 10. Let PHASES  $(\ell, r) \doteq \{i \in [\tilde{L}]: [\tau_i, \tau_{i+1}) \cap [s_\ell(r), e_\ell(r)] \neq \emptyset\}$  be the phases which intersect block  $[s_\ell(r), e_\ell(r))$ , let  $T(i, r, \ell) \doteq |[\tau_i, \tau_{i+1}) \cap [s_\ell(r), e_\ell(r)]|$  be the effective length of the phase as observed in block  $[s_\ell(r), e_\ell(r)]$ .
Similarly, define PHASES  $(\ell) \doteq \{i \in [\tilde{L}]: [\tau_i, \tau_{i+1}) \cap [t_\ell, t_{\ell+1}) \neq \emptyset\}$  as the phases which intersect episode  $[t_\ell, t_{\ell+1})$ .
It will in fact suffice to show w.h.p. w.r.t. the distribution of  $\mathbf{X}_T$ , for each episode  $[t_\ell, t_{\ell+1})$ , each block  $[s_\ell(r), e_\ell(r)]$  in  $[t_\ell, t_{\ell+1})$ , and each bin  $B \in \mathcal{T}_r$ :
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} (a _ {t} ^ {\sharp} , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \} \cdot \mathbf {1} \{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \} \Bigg | \mathbf {X} _ {T} \right] \\ \leq c _ {1 5} \log (K) \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \right\} \left(\log (T) + \log^ {2} (T) \sum_ {i \in \text {P H A S E S} (\ell , r)} r (B) ^ {d} \cdot T (i, r, \ell) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}\right) \mid \mathbf {X} _ {T} \right] \tag {19} \\ \end{array}
$$
D.5 Summing the Per-(Bin, Block, Episode) Regret over Bins, Blocks, and Episodes.
Admitting (19), we show that the total dynamic regret over  $T$  rounds is of the desired order.
Recall from earlier that there are WLOG  $T$  total episodes with the convention that  $t_{\ell} \doteq T + 1$  if only  $\ell$  episodes occur by round  $T$ . Then, summing our per-bin regret bound (19) over all the bins  $B \in \mathcal{T}_r$
at level  $r$  gives (using strong density to bound  $\sum_{B \in r} r^d \leq \frac{C_d}{c_d}$ ):
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {B \in \mathcal {T} _ {r}} \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} \left(a _ {t} ^ {\sharp} , a\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \mathbf {1} \{X _ {t} \in B \} \cdot \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \right\} \middle | \mathbf {X} _ {T} \right] \\ \leq c _ {1 6} \log (K) \mathbb {E} \left[ \mathbf {1} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \right\} \left(\sum_ {B \in \mathcal {T} _ {r}} \log (T) + \log^ {2} (T) \sum_ {i \in \text {P H A S E S} (\ell , r)} T (i, r, \ell) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}\right) \mid \mathbf {X} _ {T} \right]. \tag {20} \\ \end{array}
$$
Next, summing over the different levels  $r$  (of which there are at most  $\log(T)$  used in any episode), we obtain by Jensen's inequality on the concave function  $z \mapsto z^{\frac{1 + d}{2 + d}}$ :
$$
\begin{array}{l} \sum_ {r \in \mathcal {R}} \sum_ {i \in \text {P H A S E S} (\ell , r)} T (i, r, \ell) ^ {\frac {1 + d}{2 + d}} = \sum_ {i \in \text {P H A S E S} (\ell)} \sum_ {r \in \mathcal {R}: i \in \text {P H A S E S} (\ell , r)} T (i, r, \ell) ^ {\frac {1 + d}{2 + d}} \\ \leq \sum_ {i \in \text {P H A S E S} (\ell)} \left(\log (T) \sum_ {r \in \mathcal {R}: i \in \text {P H A S E S} (\ell , r)} T (i, r, \ell)\right) ^ {\frac {1 + d}{2 + d}}. \\ \end{array}
$$
Now, we have
$$
\sum_ {r \in \mathcal {R}: i \in \mathrm {P H A S E S} (\ell , r)} T (i, r, \ell) = \sum_ {r \in \mathcal {R}: i \in \mathrm {P H A S E S} (\ell , r)} | [ \tau_ {i}, \tau_ {i + 1}) \cap [ s _ {\ell} (r), e _ {\ell} (r) ] | = \tau_ {i + 1} - \tau_ {i} + 1.
$$
We also have (via Fact 1 about level  $r_{t_{\ell +1} - t_{\ell}}$  which is the smallest level used in episode  $[t_{\ell},t_{\ell +1})$ ).
$$
\begin{array}{l} \sum_ {r \in \mathcal {R}} \sum_ {B \in \mathcal {T} _ {r}} \log (T) \leq \sum_ {r \in \mathcal {R}} r ^ {- d} \cdot \log (T) \\ \leq c _ {1 7} \log^ {2} (T) \left(\frac {t _ {\ell + 1} - t _ {\ell}}{K}\right) ^ {\frac {d}{2 + d}} \\ \leq c_{18}\log^{2}(T)\sum_{i\in \text{PHASES} (\ell)}(\tau_{i + 1} - \tau_{i})^{\frac{1 + d}{2 + d}}\cdot K^{\frac{1}{2 + d}}. \\ \end{array}
$$
Thus, combining the above inequalities with (20), we obtain overall bound:
$$
c _ {1 8} \log (K) \log^ {3} (T) \mathbb {E} \left[ \mathbf {1} \{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \} \sum_ {i \in \mathrm {P H A S E S} (\ell)} \left(\tau_ {i + 1} - \tau_ {i}\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} \right].
$$
Recall now that  $\mathcal{E}_1$  is the good event over which the concentration bounds of Proposition 7 hold. Then, using the fact that, on event  $\mathcal{E}_1$ , each phase  $[\tau_i, \tau_{i+1})$  intersects at most two episodes (Lemma 11), summing the above R.H.S over episodes  $\ell \in [T]$  gives us (since at most  $\log(T)$  blocks per episode) order
$$
2 \log (K) \log^ {3} (T) \sum_ {i = 1} ^ {\tilde {L}} (\tau_ {i + 1} - \tau_ {i}) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
It then remains to show the per-(bin, block, episode) regret bound (19).
D.6 Bounding the Per-(Bin, Block, Episode) Regret to the Last Safe Arm
To show (19), we first fix a block  $[s_{\ell}(r), e_{\ell}(r)]$  and a bin  $B \in \mathcal{T}_r$ . We then further decompose  $\delta_t(a_t^\sharp, a)$  in two parts:
(a) The regret of  $a$  to the local last master arm, denoted by  $a_{r}(B)$ , to be evicted from  $\mathcal{A}_{\mathrm{master}}(B)$  in block  $[s_{\ell}(r), e_{\ell}(r)]$  (ties are broken arbitrarily).
(b) The regret of the local last master arm  $a_{r}(B)$  to the last safe arm  $a_{t}^{\sharp}$ .
In other words, the L.H.S. of (19) is decomposed as:
$$
\underbrace {\mathbb {E} \left[ \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \sum_ {a \in \mathcal {A} _ {t}} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \} \Bigg | \mathbf {X} _ {T} \right]} _ {(a)} + \underbrace {\mathbb {E} \left[ \sum_ {t = s _ {\ell} (r)} ^ {e _ {\ell} (r)} \delta_ {t} (a _ {t} ^ {\sharp} , a _ {r} (B)) \cdot \mathbf {1} \{X _ {t} \in B \} \Bigg | \mathbf {X} _ {T} \right]} _ {(b)}.
$$
We will show both (a) and (b) are of order (19).
- Bounding the Regret of Other Arms to the Local Last Master Arm  $a_{r}(B)$ . We start by partitioning the rounds  $t$  such that  $X_{t} \in B$  and  $a \in \mathcal{A}_t$  in (a) according to before or after they are evicted from  $\mathcal{A}_{\mathrm{master}}(B)$ . Suppose arm  $a$  is evicted from  $\mathcal{A}_{\mathrm{master}}(B)$  at round  $t_r^a \in [s_\ell (r),e_\ell (r)]$  (formally, we let  $t_r^a \doteq e_\ell (r)$  if  $a$  is not evicted in block  $[s_{\ell}(r),e_{\ell}(r)]$ ). Then, it suffices to bound:
$$
\mathbb {E} \left[ \sum_ {a = 1} ^ {K} \sum_ {t = s _ {\ell} (r)} ^ {t _ {r} ^ {a} - 1} \frac {\delta_ {t} \left(a _ {r} (B) , a\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \mathbf {1} \{X _ {t} \in B \} + \sum_ {a = 1} ^ {K} \sum_ {t = t _ {r} ^ {a}} ^ {e _ {\ell} (r)} \frac {\delta_ {t} \left(a _ {r} (B) , a\right)}{\left| \mathcal {A} _ {t} \right|} \cdot \mathbf {1} \{a \in \mathcal {A} _ {t} \} \cdot \mathbf {1} \{X _ {t} \in B \} \Bigg | \mathbf {X} _ {T} \right]. \tag {21}
$$
Suppose WLOG that  $t_r^1 \leq t_r^2 \leq \dots \leq t_r^K$ . Then, for each round  $t < t_r^a$  all arms  $a' \geq a$  are retained in  $\mathcal{A}_{\mathrm{master}}(B)$  and thus retained in the candidate arm set  $\mathcal{A}_t$  for all rounds  $t$  where  $X_t \in B$ . Importantly, at each round  $t$  a level of at least  $r$  is used since a child Base-Alg can only use a higher level than the master Base-Alg. Thus,  $|\mathcal{A}_t| \geq K + 1 - a$  for all  $t \leq t_r^a$ .
Next, we bound the first double sum in (21), i.e. the regret of playing  $a$  to  $a_{r}(B)$  from  $s_{\ell}(r)$  to  $t_{r}^{a} - 1$ . Applying our concentration bounds (Proposition 7), since arm  $a$  is not evicted from  $\mathcal{A}(B)$  till round  $t_{r}^{a}$ , on event  $\mathcal{E}_{1}$  we have for some  $c_{19} > 0$  and any other arm  $a' \in \mathcal{A}(B)$  through round  $t_{r}^{a} - 1$  (i.e.,  $a' \in \mathcal{A}_{t}$  for all  $t \in [t_{\ell}, t_{r}^{a})$  such that  $X_{t} \in B$  since we always use level at least  $r$  at such a round  $t$ ): for bin  $B' \supseteq B$  at level  $r_{t_{r}^{a} - 1 - s_{\ell}(r)}$ : on event  $\mathcal{E}_{1}$  (note that we necessarily always have  $\mathcal{A}(B') \supseteq \mathcal{A}(B)$  for  $B' \supseteq B$  by Line 14 of Algorithm 2):
$$
\sum_ {t = s _ {\ell} (r)} ^ {t _ {r} ^ {a} - 1} \mathbb {E} [ \hat {\delta} _ {s} ^ {B ^ {\prime}} (a ^ {\prime}, a) \mid \mathcal {F} _ {t - 1} ] \leq c _ {1 9} \sqrt {K \log (T) \cdot (n _ {B ^ {\prime}} ([ s _ {\ell} (r) , t _ {r} ^ {a})) \vee K \log (T))} + r (B ^ {\prime}) \cdot n _ {B ^ {\prime}} ([ s _ {\ell} (r), t _ {r} ^ {a})).
$$
Next, since  $a, a' \in \mathcal{A}_t$  for each  $t \in [s_\ell(r), t_r^a - 1)$  such that  $X_t \in B$ , we have:
$$
\forall t \in \left[ s _ {\ell} (r), t _ {r} ^ {a}\right), X _ {t} \in B: \mathbb {E} \left[ \hat {\delta} _ {t} ^ {B} \left(a ^ {\prime}, a\right) \mid \mathcal {F} _ {t - 1} \right] = \delta_ {t} \left(a ^ {\prime}, a\right).
$$
Thus, we conclude by (5):
$$
\sum_ {t = s _ {\ell} (r)} ^ {t _ {r} ^ {a} - 1} \delta_ {t} (a ^ {\prime}, a) \cdot \mathbf {1} \{X _ {t} \in B \} \leq c _ {1 9} \sqrt {K \log (T) \cdot \left(n _ {B ^ {\prime}} \left([ s _ {\ell} (r) , t _ {r} ^ {a}\right)\right) \vee K \log (T))} + r (B ^ {\prime}) \cdot n _ {B ^ {\prime}} \left([ s _ {\ell} (r), t _ {r} ^ {a})\right).
$$
Thus, by Lemma 9, and since  $B' \supseteq B$ , we conclude for any such  $a'$  on event  $\mathcal{E}_1$ :  $\sum_{t=s_\ell(r)}^{t_r^a-1} \frac{\delta_t(a', a)}{|\mathcal{A}_t|}$ .  $\mathbf{1}\{X_t \in B\}$  is at most
$$
\frac {c _ {4} \left(\log^ {1 / 2} (T) r ^ {d} \cdot K ^ {\frac {1}{2 + d}} \cdot \left(t _ {r} ^ {a} - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} + K \log (T) + \sqrt {\log (T) \left(t _ {r} ^ {a} - s _ {\ell} (r)\right) \cdot \mu (B)}\right)}{K + 1 - a}, \tag {22}
$$
where we use the fact that  $|\mathcal{A}_t| \geq K + 1 - a$  for all  $t \in [s_\ell(r), t_r^a)$ . Since this last bound holds uniformly for all  $a' \in \mathcal{A}(B)$  through round  $t_r^a - 1$ , it must hold for  $a' = a_r(B)$ , the local last master arm.
Then, summing over all arms  $a$ , we have on event  $\mathcal{E}_1$ :
$$
\begin{array}{l} \sum_ {a = 1} ^ {K} \sum_ {t = s _ {\ell} (r)} ^ {t _ {r} - 1} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \} \leq c _ {4} \log (K) \left(\log^ {1 / 2} (T) \cdot r ^ {d} \cdot (e _ {\ell} (r) - s _ {\ell} (r)) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} + \right. \\ \left. K \log (T) + \sqrt {\log (T) \left(t _ {r} ^ {a} - s _ {\ell} (r)\right) \cdot \mu (B)}\right). \tag {23} \\ \end{array}
$$
Next note that by Lemma 10:
$$
\begin{array}{l} \sqrt {(t _ {r} ^ {a} - s _ {\ell} (r)) \cdot \mu (B)} \leq \sqrt {(e _ {\ell} (r) - s _ {\ell} (r)) \cdot \mu (B)} \\ \leq c _ {7} K ^ {\frac {d / 2}{2 + d}} \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1}{2 + d}} \\ \leq c _ {7} K ^ {\frac {1}{2 + d}} \cdot r ^ {d} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}}. \\ \end{array}
$$
Additionally, since  $K \leq e_{\ell}(r) - s_{\ell}(r)$  (Fact 4), we have:
$$
K \log (T) \leq \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1}{2 + d}} K ^ {\frac {1 + d}{2 + d}} \propto r ^ {d} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
Thus, combining the above two displays with (23) gives us
$$
\sum_ {a = 1} ^ {K} \sum_ {t = s _ {\ell} (r)} ^ {t _ {r} ^ {a} - 1} \frac {\delta_ {t} (a _ {r} (B) , a)}{| \mathcal {A} _ {t} |} \cdot \mathbf {1} \{X _ {t} \in B \} \leq c _ {2 0} \log (K) \log (T) \cdot r ^ {d} \cdot \left(e _ {\ell} (r) - s _ {\ell} (r)\right) ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}}.
$$
We next show the second double sum in (21) has an upper bound similar to the above. For this, we first observe that if arm  $a$  is played in bin  $B$  after round  $t_r^a$ , then it must be due to an active replay. The difficulty here is that replays may interrupt each other and so care must be taken in managing the contribution of  $\sum_t \delta_t(a_r(B), a)$  (which may be negative) by different overlapping replays.